fix(transforms): log the compression failures that silently become "no savings" (#4092)

## Description

Five places in the compression path caught every exception and returned
the input unchanged, with no log line. A compression bug therefore
looked exactly like content that did not compress. Each now logs a
WARNING that names the exception type and counts, and a DEBUG line that
names the exception types and code locations (`describe_exception`). No
level logs the exception text or a traceback, because compressor and
store errors can quote tool output or credentials. Return values do not
change.

Refs #4136

Depends on #4138 (`headroom/log_safety.py`); its files show in this diff
until it merges.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `compression_batches.compress_batch_with_router`: warn when
`router.compress` raises (the batch still passes through).
- `cross_turn_dedup.dedup_blocks`: warn on failure. The `"error": True`
stat it returned was never read.
- `lossless_compaction.compact_lossless`: warn when a fold raises, with
the kind and length. It still never raises.
- `kompress_compressor.store_kompress_in_ccr`: warn on a CCR store
failure. The output then has no retrieval marker.
- `smart_crusher` CCR mirror: a store write failure is now WARNING, not
DEBUG.
- Each WARNING carries `type(exc).__name__` plus counts or the hash, and
never the exception text. The detail is a separate DEBUG line built with
`describe_exception(e)`, which gives types and code locations and never
the message or traceback (unless the operator sets
`HEADROOM_DEBUG_DUMP=full`). It only runs when DEBUG is enabled.
- New `tests/test_transforms_failure_logging.py`: one parametrized test
over all five paths. It checks pass-through, one WARNING and one DEBUG
detail, and formats every captured record with `logging.Formatter` to
assert a content canary and a credential canary (raised in the exception
and its cause) are absent. All five fail on main, and all five fail on
the earlier `exc_info=True` version of this PR. A second parametrized
test checks `describe_exception` is not called when DEBUG is off. A
third runs the real `compact_lossless` fallback with DEBUG on and an
`OSError` whose errno is out of the C int range (`2**31`, and
`10**10000` built at runtime), with the integer digit limit on and off.
It checks the original content comes back and the DEBUG line stays
bounded.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_transforms_failure_logging.py   # with main's product code
5 failed
$ pytest tests/test_transforms_failure_logging.py tests/test_lossless_mode.py tests/test_cross_turn_dedup.py tests/test_compression_batches.py tests/test_kompress_marker_gate.py tests/test_garbled_compression_fixes.py tests/test_ccr.py tests/test_lossless_excluded_compaction.py tests/test_lossless_first_dispatch.py tests/test_bash_search_lossless_fold.py tests/test_lossless_timestamp_guard.py tests/test_remote_kompress_dropin.py tests/test_ccr_marker_policy.py tests/test_responses_cross_turn_dedup.py tests/test_proxy_handlers_batch.py
258 passed in 24.04s
$ ruff check / ruff format (0.16.8): All checks passed;  mypy (5 modules): Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) forced failure: `compact_lossless` with
`collapse_runs` patched to raise an error whose text contains a fake
token, logging at INFO.
- Observed result: The proxy serves both requests (200) with savings.
The forced failure returns the content unchanged and logs one WARNING
naming RuntimeError, without the token text. Output below.
- Not tested: A live proxy request that hits one of the five failure
paths: they need an injected bug, which the direct check and the unit
tests supply.

```text
branch: cleanup/transforms-log-silent-failures @ f2557723
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-64662.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ HOME=$T/home HEADROOM_WORKSPACE_DIR=$T/ws CODEX_HOME=$T/codex python -c "<compact_lossless with collapse_runs patched to raise RuntimeError('token sk-live-123 in tool output')>"
headroom.transforms.lossless_compaction - WARNING - Lossless compaction (log) failed (RuntimeError) on 205 chars; leaving content as-is
unchanged: True
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only log output. Five failure paths
now log at WARNING; one of them was DEBUG.
- Kill switch / disable path: Not applicable; the change only adds log
lines.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

The image tile-optimizer item from the audit was dropped. A
non-decodable image makes PIL raise `UnidentifiedImageError` there, so a
warning would fire on ordinary bad input. There is no once-guard. Each
warning fires only when a compressor raises or the CCR store fails, the
same idiom as the existing CCR write-through warnings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-16 14:30:13 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 85520c2941
commit 580cc83a9a
7 changed files with 351 additions and 6 deletions
+14 -1
View File
@@ -3,9 +3,11 @@
from __future__ import annotations
import hashlib
import logging
import re
from dataclasses import dataclass
from ..log_safety import describe_exception, safe_id
from .compression_units import (
_CCR_MARKER_RE,
_LOSSY_UNMARKED_STRATEGIES,
@@ -19,6 +21,8 @@ from .compression_units import (
from .content_router import RouterCompressionResult
from .tag_protector import protect_tags, restore_tags
logger = logging.getLogger(__name__)
DEFAULT_MAX_BATCH_BYTES = 2048
DEFAULT_MAX_BATCH_UNITS = 16
@@ -254,7 +258,16 @@ def compress_batch_with_router(
question=batch.entries[0].routed.unit.question,
bias=batch.entries[0].routed.unit.bias,
)
except Exception:
except Exception as e:
# The exception text can quote tool output, so no level logs it;
# describe_exception keeps only types and code locations.
logger.warning(
"Batch compression failed (%s); passing %d units through uncompressed",
safe_id(e),
len(batch.entries),
)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Batch compression failure detail: %s", describe_exception(e))
return _passthrough_batch_results(
batch,
tokenizer=tokenizer,
+15 -1
View File
@@ -28,11 +28,16 @@ Pure stdlib, deterministic, never raises (returns input unchanged on any error).
from __future__ import annotations
import logging
import re
from dataclasses import dataclass
__all__ = ["DedupBlock", "dedup_blocks", "is_prefix_monotonic"]
from ..log_safety import describe_exception, safe_id
logger = logging.getLogger(__name__)
# A run must be at least this many lines AND this many chars to be worth a
# pointer. Small dups are left alone (fragmenting context is not worth it) —
# and a larger floor keeps the pointer comfortably shorter than the span it
@@ -284,7 +289,16 @@ def dedup_blocks(
out_blocks.append(DedupBlock(text="\n".join(out), turn=blk.turn, protected=False))
return out_blocks, stats
except Exception: # never break the proxy
except Exception as e: # never break the proxy
# The exception text can quote tool output, so no level logs it;
# describe_exception keeps only types and code locations.
logger.warning(
"Cross-turn dedup failed (%s) on %d blocks; leaving them as-is",
safe_id(e),
len(blocks),
)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Cross-turn dedup failure detail: %s", describe_exception(e))
return blocks, {"spans_folded": 0, "lines_removed": 0, "chars_removed": 0, "error": True}
+9 -1
View File
@@ -29,6 +29,7 @@ from functools import lru_cache
from typing import Any, Literal
from ..config import TransformResult
from ..log_safety import describe_exception, safe_id
from ..offline import OFFLINE_ENV, OfflineEgressBlocked
from ..onnx_runtime import (
ONNX_CPU_ARENA_ENV,
@@ -1702,7 +1703,14 @@ def store_kompress_in_ccr(original: str, compressed: str, original_tokens: int)
strategy="kompress",
)
return cache_key
except Exception:
except Exception as e:
# Store backends can echo the payload in errors, so no level logs it;
# describe_exception keeps only types and code locations.
logger.warning(
"Kompress CCR store write failed (%s); output is not retrievable", safe_id(e)
)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Kompress CCR store failure detail: %s", describe_exception(e))
return None
+16 -1
View File
@@ -14,6 +14,7 @@ unchanged. Nothing here raises.
from __future__ import annotations
import logging
import os
import re
@@ -32,6 +33,10 @@ __all__ = [
"compact_lossless",
]
from ..log_safety import describe_exception, safe_id
logger = logging.getLogger(__name__)
# ANSI CSI SGR (color/style) escape sequences: ESC [ ... m. Color is
# non-semantic, so stripping it is a safe (one-way) lossless-of-meaning op.
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")
@@ -533,6 +538,16 @@ def compact_lossless(content: str, kind: str) -> str:
if expand_runs(unfold_repeated_blocks(candidate)) != content:
return content
return candidate if _smaller(candidate, content) else content
except Exception:
except Exception as e:
# The exception text can quote the content, so no level logs it;
# describe_exception keeps only types and code locations.
logger.warning(
"Lossless compaction (%s) failed (%s) on %d chars; leaving content as-is",
kind,
safe_id(e),
len(content),
)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("Lossless compaction failure detail: %s", describe_exception(e))
return content
return content
+7 -2
View File
@@ -53,6 +53,7 @@ from typing import Any
from ..ccr.tool_injection import CCR_TOOL_NAME
from ..config import CCRConfig, TransformResult, is_tool_excluded, unwrap_tool_call_name
from ..log_safety import describe_exception, safe_id
from ..tokenizer import Tokenizer
from ..utils import compute_short_hash, create_tool_digest_marker, deep_copy_messages
from .base import Transform
@@ -1197,8 +1198,12 @@ class SmartCrusher(Transform):
"CCR mirror: invalid hash %r from rendered marker",
ccr_hash,
)
except Exception as e: # pragma: no cover - defensive
logger.debug("CCR mirror: store.store() raised (%s)", e)
except Exception as e:
# Store backends can echo the payload in errors, so no level logs it;
# describe_exception keeps only types and code locations.
logger.warning("CCR mirror: store write failed (%s) for hash %s", safe_id(e), ccr_hash)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("CCR mirror store failure detail: %s", describe_exception(e))
def _extract_context_from_messages(self, messages: list[dict[str, Any]]) -> str:
"""Build a query string from the last 5 user messages + recent
+34
View File
@@ -200,6 +200,40 @@ def test_compact_lossless_never_raises_on_empty() -> None:
assert compact_lossless("", kind) == ""
def test_compact_lossless_logs_a_fold_that_raises(monkeypatch) -> None:
import logging
import headroom.transforms.lossless_compaction as lc
records: list[logging.LogRecord] = []
class _Capture(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
records.append(record)
handler = _Capture(level=logging.WARNING)
lc.logger.addHandler(handler)
try:
log = "worker 7 retrying connection to upstream\n" * 5
# A fold that works logs nothing.
assert compact_lossless(log, "log") != log
assert records == []
def _boom(_content: str) -> str:
raise RuntimeError("fold bug")
monkeypatch.setattr(lc, "collapse_runs", _boom)
assert compact_lossless(log, "log") == log
finally:
lc.logger.removeHandler(handler)
assert len(records) == 1
message = records[0].getMessage()
assert "Lossless compaction (log) failed (<RuntimeError>)" in message
# The exception text can quote content, so it stays out of the WARNING.
assert "fold bug" not in message
# --------------------------------------------------------------------------
# End-to-end: ContentRouter(lossless=True) invariant
# --------------------------------------------------------------------------
+256
View File
@@ -0,0 +1,256 @@
"""Compression failure paths log no exception text at any level.
Each path catches the failure and passes content through unchanged. The
WARNING names the exception type and counts, and the DEBUG detail line gives
only exception types and code locations, because the exception text (and its
cause chain) can quote tool output, stored payloads or credentials.
"""
from __future__ import annotations
import logging
import sys
from collections.abc import Callable, Iterator
import pytest
import headroom.cache.compression_store as compression_store
from headroom.transforms import (
compression_batches,
cross_turn_dedup,
kompress_compressor,
lossless_compaction,
smart_crusher,
)
from headroom.transforms.compression_batches import (
CompressionBatchEntry,
build_compression_batches,
compress_batch_with_router,
)
from headroom.transforms.compression_units import CompressionUnit, RoutedCompressionUnit
CONTENT = "customer-tool-output-canary-7f3a"
CREDENTIAL = "sk-test-credential-canary-91bd"
class _Capture(logging.Handler):
def __init__(self) -> None:
super().__init__(level=logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
@pytest.fixture
def capture() -> Iterator[Callable[[logging.Logger], _Capture]]:
attached: list[tuple[logging.Logger, _Capture, int]] = []
def attach(logger: logging.Logger) -> _Capture:
handler = _Capture()
attached.append((logger, handler, logger.level))
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
return handler
yield attach
for logger, handler, level in attached:
logger.removeHandler(handler)
logger.setLevel(level)
def _boom(*_args: object, **_kwargs: object) -> object:
try:
raise ValueError(f"backend rejected {CONTENT} for key {CREDENTIAL}")
except ValueError as cause:
raise RuntimeError(f"failed while handling {CONTENT} with {CREDENTIAL}") from cause
def _run_batch(monkeypatch: pytest.MonkeyPatch) -> None:
def entry(index: int) -> CompressionBatchEntry:
unit = CompressionUnit(
text="x" * 150,
provider="openai",
endpoint="responses",
role="tool",
item_type="local_shell_call_output",
cache_zone="live",
mutable=True,
min_bytes=512,
)
return CompressionBatchEntry(
entry_id=f"u{index}",
routed=RoutedCompressionUnit(unit=unit, slot=(index, ("output", None))),
)
class _Router:
compress = staticmethod(_boom)
class _Counter:
def count_text(self, text: str) -> int:
return len(text)
batches, _ = build_compression_batches([entry(i) for i in range(4)], min_batch_bytes=512)
results = compress_batch_with_router(batches[0], router=_Router(), tokenizer=_Counter())
assert [result.compressed for _, result in results] == ["x" * 150] * 4
def _run_dedup(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(cross_turn_dedup, "_longest_match", _boom)
blocks = [
cross_turn_dedup.DedupBlock(text="line a\nline b\nline c\nline d", turn=i, protected=False)
for i in range(2)
]
out, stats = cross_turn_dedup.dedup_blocks(blocks)
assert out == blocks
assert stats["error"] is True
def _run_lossless(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(lossless_compaction, "collapse_runs", _boom)
log = "worker 7 retrying connection to upstream\n" * 5
assert lossless_compaction.compact_lossless(log, "log") == log
def _run_kompress_store(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(compression_store, "get_compression_store", _boom)
assert kompress_compressor.store_kompress_in_ccr("original words", "compressed", 2) is None
def _run_crusher_mirror(monkeypatch: pytest.MonkeyPatch) -> None:
class _Store:
store = staticmethod(_boom)
class _Rust:
def ccr_get(self, _hash: str) -> str:
return "[1, 2, 3]"
monkeypatch.setattr(compression_store, "get_compression_store", lambda: _Store())
crusher = smart_crusher.SmartCrusher()
monkeypatch.setattr(crusher, "_rust", _Rust())
crusher._mirror_single_hash_to_python_store("a" * 24, "smart_crusher", "", None)
_PATHS = pytest.mark.parametrize(
("logger", "run", "expected"),
[
(compression_batches.logger, _run_batch, "Batch compression failed (<RuntimeError>)"),
(cross_turn_dedup.logger, _run_dedup, "Cross-turn dedup failed (<RuntimeError>)"),
(lossless_compaction.logger, _run_lossless, "Lossless compaction (log) failed"),
(
kompress_compressor.logger,
_run_kompress_store,
"Kompress CCR store write failed (<RuntimeError>)",
),
(smart_crusher.logger, _run_crusher_mirror, "CCR mirror: store write failed"),
],
ids=["batch", "dedup", "lossless", "kompress-ccr", "crusher-mirror"],
)
@_PATHS
def test_failure_logs_no_exception_text_at_any_level(
capture: Callable[[logging.Logger], _Capture],
monkeypatch: pytest.MonkeyPatch,
logger: logging.Logger,
run: Callable[[pytest.MonkeyPatch], None],
expected: str,
) -> None:
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
handler = capture(logger)
run(monkeypatch)
formatter = logging.Formatter()
for record in handler.records:
rendered = formatter.format(record)
assert CONTENT not in rendered
assert CREDENTIAL not in rendered
warnings = [r for r in handler.records if r.levelno == logging.WARNING]
assert len(warnings) == 1
message = warnings[0].getMessage()
assert expected in message
assert "RuntimeError" in message
details = [r for r in handler.records if r.levelno == logging.DEBUG]
assert len(details) == 1
assert "RuntimeError" in details[0].getMessage()
assert "caused by ValueError" in details[0].getMessage()
@_PATHS
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
def test_malformed_exception_names_cannot_leak_from_fallbacks(
capture, monkeypatch, logger, run, expected, level
) -> None:
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
def raise_malformed(*args, **kwargs):
raise error_type("untrusted backend content")
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
monkeypatch.setitem(globals(), "_boom", raise_malformed)
handler = capture(logger)
monkeypatch.setattr(logger, "level", level)
run(monkeypatch)
assert len([record for record in handler.records if record.levelno == logging.WARNING]) == 1
for record in handler.records:
rendered = logging.Formatter().format(record)
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
assert "FORGED_LOG_ENTRY" not in rendered
assert "\n" not in rendered
@_PATHS
def test_failure_skips_the_detail_when_debug_is_off(
monkeypatch: pytest.MonkeyPatch,
logger: logging.Logger,
run: Callable[[pytest.MonkeyPatch], None],
expected: str,
) -> None:
def _not_called(_exc: BaseException) -> str:
raise AssertionError("describe_exception ran with DEBUG disabled")
monkeypatch.setattr(sys.modules[logger.name], "describe_exception", _not_called)
monkeypatch.setattr(logger, "level", logging.WARNING)
run(monkeypatch)
@pytest.mark.parametrize("unlimited_digits", [False, True], ids=["digit-limit", "no-digit-limit"])
@pytest.mark.parametrize(
"errno_value",
[2**31, 10 ** int("10000")],
ids=["strerror-overflow", "huge-errno"],
)
def test_lossless_fallback_survives_out_of_range_errno(
capture: Callable[[logging.Logger], _Capture],
monkeypatch: pytest.MonkeyPatch,
errno_value: int,
unlimited_digits: bool,
) -> None:
set_digits = getattr(sys, "set_int_max_str_digits", None)
if unlimited_digits:
if set_digits is None:
pytest.skip("this interpreter has no integer digit limit to disable")
previous = sys.get_int_max_str_digits()
set_digits(0)
def _raise_os_error(_content: str) -> str:
raise OSError(errno_value, f"backend failed on {CONTENT}")
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
monkeypatch.setattr(lossless_compaction, "collapse_runs", _raise_os_error)
handler = capture(lossless_compaction.logger)
log = "worker 7 retrying connection to upstream\n" * 5
try:
assert lossless_compaction.compact_lossless(log, "log") == log
finally:
if unlimited_digits and set_digits is not None:
set_digits(previous)
details = [r for r in handler.records if r.levelno == logging.DEBUG]
assert len(details) == 1
rendered = logging.Formatter().format(details[0])
assert "OSError [Errno out of range]" in rendered
assert CONTENT not in rendered
assert len(rendered) < 1000