Files
headroom/tests/test_transforms_failure_logging.py
580cc83a9a fix(transforms): log the compression failures that silently become "no savings" (#4092)
## Description

Five places in the compression path caught every exception and returned
the input unchanged, with no log line. A compression bug therefore
looked exactly like content that did not compress. Each now logs a
WARNING that names the exception type and counts, and a DEBUG line that
names the exception types and code locations (`describe_exception`). No
level logs the exception text or a traceback, because compressor and
store errors can quote tool output or credentials. Return values do not
change.

Refs #4136

Depends on #4138 (`headroom/log_safety.py`); its files show in this diff
until it merges.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `compression_batches.compress_batch_with_router`: warn when
`router.compress` raises (the batch still passes through).
- `cross_turn_dedup.dedup_blocks`: warn on failure. The `"error": True`
stat it returned was never read.
- `lossless_compaction.compact_lossless`: warn when a fold raises, with
the kind and length. It still never raises.
- `kompress_compressor.store_kompress_in_ccr`: warn on a CCR store
failure. The output then has no retrieval marker.
- `smart_crusher` CCR mirror: a store write failure is now WARNING, not
DEBUG.
- Each WARNING carries `type(exc).__name__` plus counts or the hash, and
never the exception text. The detail is a separate DEBUG line built with
`describe_exception(e)`, which gives types and code locations and never
the message or traceback (unless the operator sets
`HEADROOM_DEBUG_DUMP=full`). It only runs when DEBUG is enabled.
- New `tests/test_transforms_failure_logging.py`: one parametrized test
over all five paths. It checks pass-through, one WARNING and one DEBUG
detail, and formats every captured record with `logging.Formatter` to
assert a content canary and a credential canary (raised in the exception
and its cause) are absent. All five fail on main, and all five fail on
the earlier `exc_info=True` version of this PR. A second parametrized
test checks `describe_exception` is not called when DEBUG is off. A
third runs the real `compact_lossless` fallback with DEBUG on and an
`OSError` whose errno is out of the C int range (`2**31`, and
`10**10000` built at runtime), with the integer digit limit on and off.
It checks the original content comes back and the DEBUG line stays
bounded.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_transforms_failure_logging.py   # with main's product code
5 failed
$ pytest tests/test_transforms_failure_logging.py tests/test_lossless_mode.py tests/test_cross_turn_dedup.py tests/test_compression_batches.py tests/test_kompress_marker_gate.py tests/test_garbled_compression_fixes.py tests/test_ccr.py tests/test_lossless_excluded_compaction.py tests/test_lossless_first_dispatch.py tests/test_bash_search_lossless_fold.py tests/test_lossless_timestamp_guard.py tests/test_remote_kompress_dropin.py tests/test_ccr_marker_policy.py tests/test_responses_cross_turn_dedup.py tests/test_proxy_handlers_batch.py
258 passed in 24.04s
$ ruff check / ruff format (0.16.8): All checks passed;  mypy (5 modules): Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) forced failure: `compact_lossless` with
`collapse_runs` patched to raise an error whose text contains a fake
token, logging at INFO.
- Observed result: The proxy serves both requests (200) with savings.
The forced failure returns the content unchanged and logs one WARNING
naming RuntimeError, without the token text. Output below.
- Not tested: A live proxy request that hits one of the five failure
paths: they need an injected bug, which the direct check and the unit
tests supply.

```text
branch: cleanup/transforms-log-silent-failures @ f2557723
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-64662.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ HOME=$T/home HEADROOM_WORKSPACE_DIR=$T/ws CODEX_HOME=$T/codex python -c "<compact_lossless with collapse_runs patched to raise RuntimeError('token sk-live-123 in tool output')>"
headroom.transforms.lossless_compaction - WARNING - Lossless compaction (log) failed (RuntimeError) on 205 chars; leaving content as-is
unchanged: True
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only log output. Five failure paths
now log at WARNING; one of them was DEBUG.
- Kill switch / disable path: Not applicable; the change only adds log
lines.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

The image tile-optimizer item from the audit was dropped. A
non-decodable image makes PIL raise `UnidentifiedImageError` there, so a
warning would fire on ordinary bad input. There is no once-guard. Each
warning fires only when a compressor raises or the CCR store fails, the
same idiom as the existing CCR write-through warnings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:30:13 -07:00

257 lines
8.9 KiB
Python

"""Compression failure paths log no exception text at any level.
Each path catches the failure and passes content through unchanged. The
WARNING names the exception type and counts, and the DEBUG detail line gives
only exception types and code locations, because the exception text (and its
cause chain) can quote tool output, stored payloads or credentials.
"""
from __future__ import annotations
import logging
import sys
from collections.abc import Callable, Iterator
import pytest
import headroom.cache.compression_store as compression_store
from headroom.transforms import (
compression_batches,
cross_turn_dedup,
kompress_compressor,
lossless_compaction,
smart_crusher,
)
from headroom.transforms.compression_batches import (
CompressionBatchEntry,
build_compression_batches,
compress_batch_with_router,
)
from headroom.transforms.compression_units import CompressionUnit, RoutedCompressionUnit
CONTENT = "customer-tool-output-canary-7f3a"
CREDENTIAL = "sk-test-credential-canary-91bd"
class _Capture(logging.Handler):
def __init__(self) -> None:
super().__init__(level=logging.DEBUG)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self.records.append(record)
@pytest.fixture
def capture() -> Iterator[Callable[[logging.Logger], _Capture]]:
attached: list[tuple[logging.Logger, _Capture, int]] = []
def attach(logger: logging.Logger) -> _Capture:
handler = _Capture()
attached.append((logger, handler, logger.level))
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
return handler
yield attach
for logger, handler, level in attached:
logger.removeHandler(handler)
logger.setLevel(level)
def _boom(*_args: object, **_kwargs: object) -> object:
try:
raise ValueError(f"backend rejected {CONTENT} for key {CREDENTIAL}")
except ValueError as cause:
raise RuntimeError(f"failed while handling {CONTENT} with {CREDENTIAL}") from cause
def _run_batch(monkeypatch: pytest.MonkeyPatch) -> None:
def entry(index: int) -> CompressionBatchEntry:
unit = CompressionUnit(
text="x" * 150,
provider="openai",
endpoint="responses",
role="tool",
item_type="local_shell_call_output",
cache_zone="live",
mutable=True,
min_bytes=512,
)
return CompressionBatchEntry(
entry_id=f"u{index}",
routed=RoutedCompressionUnit(unit=unit, slot=(index, ("output", None))),
)
class _Router:
compress = staticmethod(_boom)
class _Counter:
def count_text(self, text: str) -> int:
return len(text)
batches, _ = build_compression_batches([entry(i) for i in range(4)], min_batch_bytes=512)
results = compress_batch_with_router(batches[0], router=_Router(), tokenizer=_Counter())
assert [result.compressed for _, result in results] == ["x" * 150] * 4
def _run_dedup(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(cross_turn_dedup, "_longest_match", _boom)
blocks = [
cross_turn_dedup.DedupBlock(text="line a\nline b\nline c\nline d", turn=i, protected=False)
for i in range(2)
]
out, stats = cross_turn_dedup.dedup_blocks(blocks)
assert out == blocks
assert stats["error"] is True
def _run_lossless(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(lossless_compaction, "collapse_runs", _boom)
log = "worker 7 retrying connection to upstream\n" * 5
assert lossless_compaction.compact_lossless(log, "log") == log
def _run_kompress_store(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(compression_store, "get_compression_store", _boom)
assert kompress_compressor.store_kompress_in_ccr("original words", "compressed", 2) is None
def _run_crusher_mirror(monkeypatch: pytest.MonkeyPatch) -> None:
class _Store:
store = staticmethod(_boom)
class _Rust:
def ccr_get(self, _hash: str) -> str:
return "[1, 2, 3]"
monkeypatch.setattr(compression_store, "get_compression_store", lambda: _Store())
crusher = smart_crusher.SmartCrusher()
monkeypatch.setattr(crusher, "_rust", _Rust())
crusher._mirror_single_hash_to_python_store("a" * 24, "smart_crusher", "", None)
_PATHS = pytest.mark.parametrize(
("logger", "run", "expected"),
[
(compression_batches.logger, _run_batch, "Batch compression failed (<RuntimeError>)"),
(cross_turn_dedup.logger, _run_dedup, "Cross-turn dedup failed (<RuntimeError>)"),
(lossless_compaction.logger, _run_lossless, "Lossless compaction (log) failed"),
(
kompress_compressor.logger,
_run_kompress_store,
"Kompress CCR store write failed (<RuntimeError>)",
),
(smart_crusher.logger, _run_crusher_mirror, "CCR mirror: store write failed"),
],
ids=["batch", "dedup", "lossless", "kompress-ccr", "crusher-mirror"],
)
@_PATHS
def test_failure_logs_no_exception_text_at_any_level(
capture: Callable[[logging.Logger], _Capture],
monkeypatch: pytest.MonkeyPatch,
logger: logging.Logger,
run: Callable[[pytest.MonkeyPatch], None],
expected: str,
) -> None:
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
handler = capture(logger)
run(monkeypatch)
formatter = logging.Formatter()
for record in handler.records:
rendered = formatter.format(record)
assert CONTENT not in rendered
assert CREDENTIAL not in rendered
warnings = [r for r in handler.records if r.levelno == logging.WARNING]
assert len(warnings) == 1
message = warnings[0].getMessage()
assert expected in message
assert "RuntimeError" in message
details = [r for r in handler.records if r.levelno == logging.DEBUG]
assert len(details) == 1
assert "RuntimeError" in details[0].getMessage()
assert "caused by ValueError" in details[0].getMessage()
@_PATHS
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
def test_malformed_exception_names_cannot_leak_from_fallbacks(
capture, monkeypatch, logger, run, expected, level
) -> None:
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
def raise_malformed(*args, **kwargs):
raise error_type("untrusted backend content")
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
monkeypatch.setitem(globals(), "_boom", raise_malformed)
handler = capture(logger)
monkeypatch.setattr(logger, "level", level)
run(monkeypatch)
assert len([record for record in handler.records if record.levelno == logging.WARNING]) == 1
for record in handler.records:
rendered = logging.Formatter().format(record)
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
assert "FORGED_LOG_ENTRY" not in rendered
assert "\n" not in rendered
@_PATHS
def test_failure_skips_the_detail_when_debug_is_off(
monkeypatch: pytest.MonkeyPatch,
logger: logging.Logger,
run: Callable[[pytest.MonkeyPatch], None],
expected: str,
) -> None:
def _not_called(_exc: BaseException) -> str:
raise AssertionError("describe_exception ran with DEBUG disabled")
monkeypatch.setattr(sys.modules[logger.name], "describe_exception", _not_called)
monkeypatch.setattr(logger, "level", logging.WARNING)
run(monkeypatch)
@pytest.mark.parametrize("unlimited_digits", [False, True], ids=["digit-limit", "no-digit-limit"])
@pytest.mark.parametrize(
"errno_value",
[2**31, 10 ** int("10000")],
ids=["strerror-overflow", "huge-errno"],
)
def test_lossless_fallback_survives_out_of_range_errno(
capture: Callable[[logging.Logger], _Capture],
monkeypatch: pytest.MonkeyPatch,
errno_value: int,
unlimited_digits: bool,
) -> None:
set_digits = getattr(sys, "set_int_max_str_digits", None)
if unlimited_digits:
if set_digits is None:
pytest.skip("this interpreter has no integer digit limit to disable")
previous = sys.get_int_max_str_digits()
set_digits(0)
def _raise_os_error(_content: str) -> str:
raise OSError(errno_value, f"backend failed on {CONTENT}")
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
monkeypatch.setattr(lossless_compaction, "collapse_runs", _raise_os_error)
handler = capture(lossless_compaction.logger)
log = "worker 7 retrying connection to upstream\n" * 5
try:
assert lossless_compaction.compact_lossless(log, "log") == log
finally:
if unlimited_digits and set_digits is not None:
set_digits(previous)
details = [r for r in handler.records if r.levelno == logging.DEBUG]
assert len(details) == 1
rendered = logging.Formatter().format(details[0])
assert "OSError [Errno out of range]" in rendered
assert CONTENT not in rendered
assert len(rendered) < 1000