mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
Five places in the compression path caught every exception and returned
the input unchanged, with no log line. A compression bug therefore
looked exactly like content that did not compress. Each now logs a
WARNING that names the exception type and counts, and a DEBUG line that
names the exception types and code locations (`describe_exception`). No
level logs the exception text or a traceback, because compressor and
store errors can quote tool output or credentials. Return values do not
change.
Refs #4136
Depends on #4138 (`headroom/log_safety.py`); its files show in this diff
until it merges.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `compression_batches.compress_batch_with_router`: warn when
`router.compress` raises (the batch still passes through).
- `cross_turn_dedup.dedup_blocks`: warn on failure. The `"error": True`
stat it returned was never read.
- `lossless_compaction.compact_lossless`: warn when a fold raises, with
the kind and length. It still never raises.
- `kompress_compressor.store_kompress_in_ccr`: warn on a CCR store
failure. The output then has no retrieval marker.
- `smart_crusher` CCR mirror: a store write failure is now WARNING, not
DEBUG.
- Each WARNING carries `type(exc).__name__` plus counts or the hash, and
never the exception text. The detail is a separate DEBUG line built with
`describe_exception(e)`, which gives types and code locations and never
the message or traceback (unless the operator sets
`HEADROOM_DEBUG_DUMP=full`). It only runs when DEBUG is enabled.
- New `tests/test_transforms_failure_logging.py`: one parametrized test
over all five paths. It checks pass-through, one WARNING and one DEBUG
detail, and formats every captured record with `logging.Formatter` to
assert a content canary and a credential canary (raised in the exception
and its cause) are absent. All five fail on main, and all five fail on
the earlier `exc_info=True` version of this PR. A second parametrized
test checks `describe_exception` is not called when DEBUG is off. A
third runs the real `compact_lossless` fallback with DEBUG on and an
`OSError` whose errno is out of the C int range (`2**31`, and
`10**10000` built at runtime), with the integer digit limit on and off.
It checks the original content comes back and the DEBUG line stays
bounded.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms_failure_logging.py # with main's product code
5 failed
$ pytest tests/test_transforms_failure_logging.py tests/test_lossless_mode.py tests/test_cross_turn_dedup.py tests/test_compression_batches.py tests/test_kompress_marker_gate.py tests/test_garbled_compression_fixes.py tests/test_ccr.py tests/test_lossless_excluded_compaction.py tests/test_lossless_first_dispatch.py tests/test_bash_search_lossless_fold.py tests/test_lossless_timestamp_guard.py tests/test_remote_kompress_dropin.py tests/test_ccr_marker_policy.py tests/test_responses_cross_turn_dedup.py tests/test_proxy_handlers_batch.py
258 passed in 24.04s
$ ruff check / ruff format (0.16.8): All checks passed; mypy (5 modules): Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) forced failure: `compact_lossless` with
`collapse_runs` patched to raise an error whose text contains a fake
token, logging at INFO.
- Observed result: The proxy serves both requests (200) with savings.
The forced failure returns the content unchanged and logs one WARNING
naming RuntimeError, without the token text. Output below.
- Not tested: A live proxy request that hits one of the five failure
paths: they need an injected bug, which the direct check and the unit
tests supply.
```text
branch: cleanup/transforms-log-silent-failures @ f2557723
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-64662.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```
```text
$ HOME=$T/home HEADROOM_WORKSPACE_DIR=$T/ws CODEX_HOME=$T/codex python -c "<compact_lossless with collapse_runs patched to raise RuntimeError('token sk-live-123 in tool output')>"
headroom.transforms.lossless_compaction - WARNING - Lossless compaction (log) failed (RuntimeError) on 205 chars; leaving content as-is
unchanged: True
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only log output. Five failure paths
now log at WARNING; one of them was DEBUG.
- Kill switch / disable path: Not applicable; the change only adds log
lines.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
Not applicable.
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.
The image tile-optimizer item from the audit was dropped. A
non-decodable image makes PIL raise `UnidentifiedImageError` there, so a
warning would fire on ordinary bad input. There is no once-guard. Each
warning fires only when a compressor raises or the CCR store fails, the
same idiom as the existing CCR write-through warnings.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
257 lines
8.9 KiB
Python
257 lines
8.9 KiB
Python
"""Compression failure paths log no exception text at any level.
|
|
|
|
Each path catches the failure and passes content through unchanged. The
|
|
WARNING names the exception type and counts, and the DEBUG detail line gives
|
|
only exception types and code locations, because the exception text (and its
|
|
cause chain) can quote tool output, stored payloads or credentials.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import sys
|
|
from collections.abc import Callable, Iterator
|
|
|
|
import pytest
|
|
|
|
import headroom.cache.compression_store as compression_store
|
|
from headroom.transforms import (
|
|
compression_batches,
|
|
cross_turn_dedup,
|
|
kompress_compressor,
|
|
lossless_compaction,
|
|
smart_crusher,
|
|
)
|
|
from headroom.transforms.compression_batches import (
|
|
CompressionBatchEntry,
|
|
build_compression_batches,
|
|
compress_batch_with_router,
|
|
)
|
|
from headroom.transforms.compression_units import CompressionUnit, RoutedCompressionUnit
|
|
|
|
CONTENT = "customer-tool-output-canary-7f3a"
|
|
CREDENTIAL = "sk-test-credential-canary-91bd"
|
|
|
|
|
|
class _Capture(logging.Handler):
|
|
def __init__(self) -> None:
|
|
super().__init__(level=logging.DEBUG)
|
|
self.records: list[logging.LogRecord] = []
|
|
|
|
def emit(self, record: logging.LogRecord) -> None:
|
|
self.records.append(record)
|
|
|
|
|
|
@pytest.fixture
|
|
def capture() -> Iterator[Callable[[logging.Logger], _Capture]]:
|
|
attached: list[tuple[logging.Logger, _Capture, int]] = []
|
|
|
|
def attach(logger: logging.Logger) -> _Capture:
|
|
handler = _Capture()
|
|
attached.append((logger, handler, logger.level))
|
|
logger.addHandler(handler)
|
|
logger.setLevel(logging.DEBUG)
|
|
return handler
|
|
|
|
yield attach
|
|
for logger, handler, level in attached:
|
|
logger.removeHandler(handler)
|
|
logger.setLevel(level)
|
|
|
|
|
|
def _boom(*_args: object, **_kwargs: object) -> object:
|
|
try:
|
|
raise ValueError(f"backend rejected {CONTENT} for key {CREDENTIAL}")
|
|
except ValueError as cause:
|
|
raise RuntimeError(f"failed while handling {CONTENT} with {CREDENTIAL}") from cause
|
|
|
|
|
|
def _run_batch(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
def entry(index: int) -> CompressionBatchEntry:
|
|
unit = CompressionUnit(
|
|
text="x" * 150,
|
|
provider="openai",
|
|
endpoint="responses",
|
|
role="tool",
|
|
item_type="local_shell_call_output",
|
|
cache_zone="live",
|
|
mutable=True,
|
|
min_bytes=512,
|
|
)
|
|
return CompressionBatchEntry(
|
|
entry_id=f"u{index}",
|
|
routed=RoutedCompressionUnit(unit=unit, slot=(index, ("output", None))),
|
|
)
|
|
|
|
class _Router:
|
|
compress = staticmethod(_boom)
|
|
|
|
class _Counter:
|
|
def count_text(self, text: str) -> int:
|
|
return len(text)
|
|
|
|
batches, _ = build_compression_batches([entry(i) for i in range(4)], min_batch_bytes=512)
|
|
results = compress_batch_with_router(batches[0], router=_Router(), tokenizer=_Counter())
|
|
assert [result.compressed for _, result in results] == ["x" * 150] * 4
|
|
|
|
|
|
def _run_dedup(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setattr(cross_turn_dedup, "_longest_match", _boom)
|
|
blocks = [
|
|
cross_turn_dedup.DedupBlock(text="line a\nline b\nline c\nline d", turn=i, protected=False)
|
|
for i in range(2)
|
|
]
|
|
out, stats = cross_turn_dedup.dedup_blocks(blocks)
|
|
assert out == blocks
|
|
assert stats["error"] is True
|
|
|
|
|
|
def _run_lossless(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setattr(lossless_compaction, "collapse_runs", _boom)
|
|
log = "worker 7 retrying connection to upstream\n" * 5
|
|
assert lossless_compaction.compact_lossless(log, "log") == log
|
|
|
|
|
|
def _run_kompress_store(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setattr(compression_store, "get_compression_store", _boom)
|
|
assert kompress_compressor.store_kompress_in_ccr("original words", "compressed", 2) is None
|
|
|
|
|
|
def _run_crusher_mirror(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
class _Store:
|
|
store = staticmethod(_boom)
|
|
|
|
class _Rust:
|
|
def ccr_get(self, _hash: str) -> str:
|
|
return "[1, 2, 3]"
|
|
|
|
monkeypatch.setattr(compression_store, "get_compression_store", lambda: _Store())
|
|
crusher = smart_crusher.SmartCrusher()
|
|
monkeypatch.setattr(crusher, "_rust", _Rust())
|
|
crusher._mirror_single_hash_to_python_store("a" * 24, "smart_crusher", "", None)
|
|
|
|
|
|
_PATHS = pytest.mark.parametrize(
|
|
("logger", "run", "expected"),
|
|
[
|
|
(compression_batches.logger, _run_batch, "Batch compression failed (<RuntimeError>)"),
|
|
(cross_turn_dedup.logger, _run_dedup, "Cross-turn dedup failed (<RuntimeError>)"),
|
|
(lossless_compaction.logger, _run_lossless, "Lossless compaction (log) failed"),
|
|
(
|
|
kompress_compressor.logger,
|
|
_run_kompress_store,
|
|
"Kompress CCR store write failed (<RuntimeError>)",
|
|
),
|
|
(smart_crusher.logger, _run_crusher_mirror, "CCR mirror: store write failed"),
|
|
],
|
|
ids=["batch", "dedup", "lossless", "kompress-ccr", "crusher-mirror"],
|
|
)
|
|
|
|
|
|
@_PATHS
|
|
def test_failure_logs_no_exception_text_at_any_level(
|
|
capture: Callable[[logging.Logger], _Capture],
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
logger: logging.Logger,
|
|
run: Callable[[pytest.MonkeyPatch], None],
|
|
expected: str,
|
|
) -> None:
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
handler = capture(logger)
|
|
run(monkeypatch)
|
|
|
|
formatter = logging.Formatter()
|
|
for record in handler.records:
|
|
rendered = formatter.format(record)
|
|
assert CONTENT not in rendered
|
|
assert CREDENTIAL not in rendered
|
|
|
|
warnings = [r for r in handler.records if r.levelno == logging.WARNING]
|
|
assert len(warnings) == 1
|
|
message = warnings[0].getMessage()
|
|
assert expected in message
|
|
assert "RuntimeError" in message
|
|
|
|
details = [r for r in handler.records if r.levelno == logging.DEBUG]
|
|
assert len(details) == 1
|
|
assert "RuntimeError" in details[0].getMessage()
|
|
assert "caused by ValueError" in details[0].getMessage()
|
|
|
|
|
|
@_PATHS
|
|
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
|
def test_malformed_exception_names_cannot_leak_from_fallbacks(
|
|
capture, monkeypatch, logger, run, expected, level
|
|
) -> None:
|
|
error_type = type("PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY", (RuntimeError,), {})
|
|
|
|
def raise_malformed(*args, **kwargs):
|
|
raise error_type("untrusted backend content")
|
|
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
monkeypatch.setitem(globals(), "_boom", raise_malformed)
|
|
handler = capture(logger)
|
|
monkeypatch.setattr(logger, "level", level)
|
|
run(monkeypatch)
|
|
assert len([record for record in handler.records if record.levelno == logging.WARNING]) == 1
|
|
for record in handler.records:
|
|
rendered = logging.Formatter().format(record)
|
|
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
|
assert "FORGED_LOG_ENTRY" not in rendered
|
|
assert "\n" not in rendered
|
|
|
|
|
|
@_PATHS
|
|
def test_failure_skips_the_detail_when_debug_is_off(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
logger: logging.Logger,
|
|
run: Callable[[pytest.MonkeyPatch], None],
|
|
expected: str,
|
|
) -> None:
|
|
def _not_called(_exc: BaseException) -> str:
|
|
raise AssertionError("describe_exception ran with DEBUG disabled")
|
|
|
|
monkeypatch.setattr(sys.modules[logger.name], "describe_exception", _not_called)
|
|
monkeypatch.setattr(logger, "level", logging.WARNING)
|
|
run(monkeypatch)
|
|
|
|
|
|
@pytest.mark.parametrize("unlimited_digits", [False, True], ids=["digit-limit", "no-digit-limit"])
|
|
@pytest.mark.parametrize(
|
|
"errno_value",
|
|
[2**31, 10 ** int("10000")],
|
|
ids=["strerror-overflow", "huge-errno"],
|
|
)
|
|
def test_lossless_fallback_survives_out_of_range_errno(
|
|
capture: Callable[[logging.Logger], _Capture],
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
errno_value: int,
|
|
unlimited_digits: bool,
|
|
) -> None:
|
|
set_digits = getattr(sys, "set_int_max_str_digits", None)
|
|
if unlimited_digits:
|
|
if set_digits is None:
|
|
pytest.skip("this interpreter has no integer digit limit to disable")
|
|
previous = sys.get_int_max_str_digits()
|
|
set_digits(0)
|
|
|
|
def _raise_os_error(_content: str) -> str:
|
|
raise OSError(errno_value, f"backend failed on {CONTENT}")
|
|
|
|
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
|
monkeypatch.setattr(lossless_compaction, "collapse_runs", _raise_os_error)
|
|
handler = capture(lossless_compaction.logger)
|
|
log = "worker 7 retrying connection to upstream\n" * 5
|
|
try:
|
|
assert lossless_compaction.compact_lossless(log, "log") == log
|
|
finally:
|
|
if unlimited_digits and set_digits is not None:
|
|
set_digits(previous)
|
|
|
|
details = [r for r in handler.records if r.levelno == logging.DEBUG]
|
|
assert len(details) == 1
|
|
rendered = logging.Formatter().format(details[0])
|
|
assert "OSError [Errno out of range]" in rendered
|
|
assert CONTENT not in rendered
|
|
assert len(rendered) < 1000
|