Files
headroom/tests/test_transforms_search_compressor.py
ea947ca421 refactor(transforms): share one CCR persist helper for the Rust-backed compressors (#4094)
## Description

DiffCompressor, LogCompressor and SearchCompressor each carried an
identical private `_persist_to_python_ccr` method. It writes the
Rust-emitted CCR entry through to the production CompressionStore with
`explicit_hash` (#816). This PR replaces the three copies with one
function, `headroom.transforms.base.persist_rust_ccr_entry`. Its
warnings name the compressor that failed and keep the store's exception
text out of every log level.

Refs #4136

Depends on #4138 (`headroom/log_safety.py`); its files show in this diff
until it merges.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Add `persist_rust_ccr_entry(original, compressed, cache_key, *,
source)` to `transforms/base.py`, using a lazy import of the compression
store as before.
- The three `compress()` methods call it with `source="diff"`, `"log"`
or `"search"`.
- Remove the three private copies, and the loggers in `diff_compressor`
and `search_compressor` that are now unused.
- The write-failure WARNING names the source and exception type. The
detail goes to DEBUG as `describe_exception(e)` (types and code
locations, no message or traceback), because store backends can echo the
payload. The import-failure warning uses it too. `explicit_hash` and
fail-open are unchanged.
- Tests:
- the three #816 bridge tests become one parametrized test over the
helper;
- they no longer skip without the Rust extension, because the helper is
pure Python;
- a new test formats every captured DEBUG and WARNING record and asserts
the original-content and credential canaries are absent. It fails on the
earlier `exc_info=True` version of this PR.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_search_compressor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py tests/test_transforms_log_compressor.py tests/test_log_compressor.py tests/test_search_compressor.py tests/test_transforms_content_router.py
280 passed in 15.69s
$ python -c "import sys; sys.modules['headroom._core'] = None; import pytest; pytest.main([... 'tests/test_ccr_rust_marker_hash_bridge.py'])"
3 passed
$ ruff check / format: All checks passed;  mypy base.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
(head 1bd479d6) with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) a real `DiffCompressor.compress()` on a 60-file diff,
then a retrieval of its CCR entry under the marker key; (3) the real
`persist_rust_ccr_entry` with logging at DEBUG, against a rejecting
backend whose exception echoes the source (with a credential in its
cause) and a backend raising `OSError` with errno `10**10000`,
`HEADROOM_DEBUG_DUMP` unset.
- Observed result: The proxy serves both requests (200) with savings.
The diff compressor emits a cache_key, and the entry can be retrieved
under exactly that key. On a store failure the helper returns `None`
(fail-open); the WARNING names source, class and cache key, and the
DEBUG line names only types and code locations. Neither canary appears
in any rendered line (grep count 0). Output below.
- Not tested: A model-issued retrieval through a live proxy; the direct
check reads the same store.

```text
branch: cleanup/transforms-shared-ccr-persist @ 1bd479d6
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-57958.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python diff_direct.py   # DiffCompressor(min_lines_for_ccr=10).compress(60-file diff); store.retrieve(cache_key)
cache_key: 76142b4e8a8086abb3709b9f
retrievable under marker key: True
```

```text
$ python persist_proof.py   # real persist_rust_ccr_entry, DEBUG on, two failing backends
WARNING headroom.transforms.base CCR store write failed (diff, RuntimeError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only
DEBUG headroom.transforms.base CCR store write failure detail: RuntimeError at persist_proof.py:12 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry; caused by ValueError at persist_proof.py:10 in store
returned: None
WARNING headroom.transforms.base CCR store write failed (diff, OSError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only
DEBUG headroom.transforms.base CCR store write failure detail: OSError [Errno out of range] at persist_proof.py:15 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry
returned: None
$ grep -c "CANARY\|sk-test" persist_proof.out
0
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No, except that the warning text now
includes the compressor name and comes from the
`headroom.transforms.base` logger.
- Kill switch / disable path: Not applicable; the stored data and keys
are unchanged.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

`DiffCompressor.compress_with_stats` is unchanged here. It is public, so
it is deprecated in its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:30:21 -07:00

122 lines
4.8 KiB
Python

from __future__ import annotations
from types import SimpleNamespace
import pytest
from headroom.transforms.base import persist_rust_ccr_entry
from headroom.transforms.search_compressor import (
SearchCompressionResult,
SearchCompressor,
SearchCompressorConfig,
)
def test_compress_scores_caps_and_summarizes_search_results() -> None:
compressor = SearchCompressor(
SearchCompressorConfig(
max_matches_per_file=3,
max_total_matches=4,
max_files=2,
context_keywords=["auth"],
enable_ccr=False,
)
)
content = "\n".join(
[
"src/auth.py:10:ERROR auth failed",
"src/auth.py-11-warning auth retry",
"src/auth.py:12:plain auth line",
"src/auth.py:13:another auth line",
"src/db.py:2:warning token expired",
"not a match",
]
)
result = compressor.compress(content, context="find auth error")
# "not a match" is not a search row; both files are parsed.
assert result.original_match_count == 5
assert result.files_affected == 2
# The error line ranks first and is kept; auth.py is capped at 3 matches.
assert "src/auth.py:10:ERROR auth failed" in result.compressed
assert "src/db.py:2:warning token expired" in result.compressed
assert result.summaries == {"src/auth.py": "[... and 1 more matches in src/auth.py]"}
assert "[... and 1 more matches in src/auth.py]" in result.compressed
def test_search_compressor_compress_paths_and_ccr() -> None:
"""Phase 3e.2: `compress()` is now a single Rust call, so this test
exercises end-to-end behavior instead of monkeypatching internal
helpers (which the old orchestration relied on). The CCR plumbing
is verified via `cache_key` presence + the marker string format
Rust emits."""
compressor = SearchCompressor(
SearchCompressorConfig(enable_ccr=True, min_matches_for_ccr=2, context_keywords=["auth"])
)
no_match = compressor.compress("plain text only")
assert no_match.original_match_count == 0
assert no_match.compressed == "plain text only"
# Build a large input so compute_optimal_k's min_k=5 floor doesn't
# absorb everything and compression actually fires (must drop the
# ratio below `min_compression_ratio_for_ccr=0.8`).
lines = [f"src/auth.py:{i}:auth event {i}" for i in range(1, 51)]
lines += [f"src/db.py:{i}:db query {i}" for i in range(1, 31)]
content = "\n".join(lines)
result = compressor.compress(content, context="auth", bias=0.5) # low bias = drop more
assert result.original_match_count == 80
assert result.files_affected == 2
assert result.compressed_match_count < result.original_match_count
assert result.cache_key is not None
assert result.compressed.endswith(f". Retrieve more: hash={result.cache_key}]")
# Summaries appear for any file whose matches were dropped.
assert isinstance(result.summaries, dict)
assert len(result.summaries) >= 1
def test_persist_rust_ccr_entry(monkeypatch: pytest.MonkeyPatch) -> None:
"""CCR write-through delegates to the production `CompressionStore`.
Failures are logged (not silently swallowed) — this pins both paths."""
seen: dict[str, tuple[str, str, str | None]] = {}
monkeypatch.setitem(
__import__("sys").modules,
"headroom.cache.compression_store",
SimpleNamespace(
get_compression_store=lambda: SimpleNamespace(
store=lambda original, compressed, original_item_count=0, explicit_hash=None: (
seen.setdefault("call", (original, compressed, explicit_hash)) or "stored-key"
)
)
),
)
persist_rust_ccr_entry("orig", "comp", "abc123", source="search")
# explicit_hash carries the Rust marker key so retrieval of the
# marker hash finds the entry (issue #816).
assert seen["call"] == ("orig", "comp", "abc123")
# Loud failure: the store raises, but persist swallows + logs (no
# exception propagates to the compress callsite).
def broken_store() -> SimpleNamespace:
raise RuntimeError("boom")
monkeypatch.setitem(
__import__("sys").modules,
"headroom.cache.compression_store",
SimpleNamespace(get_compression_store=broken_store),
)
persist_rust_ccr_entry("orig", "comp", "abc123", source="search") # must not raise
def test_search_compression_result_properties() -> None:
"""Result-property contract preserved across the port."""
result = SearchCompressionResult(
compressed="tiny",
original="this is a much longer original string",
original_match_count=10,
compressed_match_count=4,
files_affected=2,
compression_ratio=0.3,
)
assert result.tokens_saved_estimate > 0
assert result.matches_omitted == 6