mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description DiffCompressor, LogCompressor and SearchCompressor each carried an identical private `_persist_to_python_ccr` method. It writes the Rust-emitted CCR entry through to the production CompressionStore with `explicit_hash` (#816). This PR replaces the three copies with one function, `headroom.transforms.base.persist_rust_ccr_entry`. Its warnings name the compressor that failed and keep the store's exception text out of every log level. Refs #4136 Depends on #4138 (`headroom/log_safety.py`); its files show in this diff until it merges. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Add `persist_rust_ccr_entry(original, compressed, cache_key, *, source)` to `transforms/base.py`, using a lazy import of the compression store as before. - The three `compress()` methods call it with `source="diff"`, `"log"` or `"search"`. - Remove the three private copies, and the loggers in `diff_compressor` and `search_compressor` that are now unused. - The write-failure WARNING names the source and exception type. The detail goes to DEBUG as `describe_exception(e)` (types and code locations, no message or traceback), because store backends can echo the payload. The import-failure warning uses it too. `explicit_hash` and fail-open are unchanged. - Tests: - the three #816 bridge tests become one parametrized test over the helper; - they no longer skip without the Rust extension, because the helper is pure Python; - a new test formats every captured DEBUG and WARNING record and asserts the original-content and credential canaries are absent. It fails on the earlier `exc_info=True` version of this PR. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_search_compressor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py tests/test_transforms_log_compressor.py tests/test_log_compressor.py tests/test_search_compressor.py tests/test_transforms_content_router.py 280 passed in 15.69s $ python -c "import sys; sys.modules['headroom._core'] = None; import pytest; pytest.main([... 'tests/test_ccr_rust_marker_hash_bridge.py'])" 3 passed $ ruff check / format: All checks passed; mypy base.py: Success ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree (head1bd479d6) with the prebuilt `headroom._core`. Temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real home. - Exact command / steps: (1) shared smoke harness: a real proxy from this checkout against a local fake upstream, with one Anthropic and one OpenAI request carrying a compressible tool result (`smoke.sh <worktree>`); (2) a real `DiffCompressor.compress()` on a 60-file diff, then a retrieval of its CCR entry under the marker key; (3) the real `persist_rust_ccr_entry` with logging at DEBUG, against a rejecting backend whose exception echoes the source (with a credential in its cause) and a backend raising `OSError` with errno `10**10000`, `HEADROOM_DEBUG_DUMP` unset. - Observed result: The proxy serves both requests (200) with savings. The diff compressor emits a cache_key, and the entry can be retrieved under exactly that key. On a store failure the helper returns `None` (fail-open); the WARNING names source, class and cache key, and the DEBUG line names only types and code locations. Neither canary appears in any rendered line (grep count 0). Output below. - Not tested: A model-issued retrieval through a live proxy; the direct check reads the same store. ```text branch: cleanup/transforms-shared-ccr-persist @1bd479d6health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-57958.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: $SMOKE) ``` ```text $ python diff_direct.py # DiffCompressor(min_lines_for_ccr=10).compress(60-file diff); store.retrieve(cache_key) cache_key: 76142b4e8a8086abb3709b9f retrievable under marker key: True ``` ```text $ python persist_proof.py # real persist_rust_ccr_entry, DEBUG on, two failing backends WARNING headroom.transforms.base CCR store write failed (diff, RuntimeError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only DEBUG headroom.transforms.base CCR store write failure detail: RuntimeError at persist_proof.py:12 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry; caused by ValueError at persist_proof.py:10 in store returned: None WARNING headroom.transforms.base CCR store write failed (diff, OSError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only DEBUG headroom.transforms.base CCR store write failure detail: OSError [Errno out of range] at persist_proof.py:15 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry returned: None $ grep -c "CANARY\|sk-test" persist_proof.out 0 ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Not applicable; no rollout-managed behaviour is touched. - Stable/default behavior changed: No, except that the warning text now includes the compressor name and comes from the `headroom.transforms.base` logger. - Kill switch / disable path: Not applicable; the stored data and keys are unchanged. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this single commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Documentation changes are not in this PR. They are recorded in the cleanup docs ledger for the single follow-up docs PR. `DiffCompressor.compress_with_stats` is unchanged here. It is public, so it is deprecated in its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
122 lines
4.8 KiB
Python
122 lines
4.8 KiB
Python
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from headroom.transforms.base import persist_rust_ccr_entry
|
|
from headroom.transforms.search_compressor import (
|
|
SearchCompressionResult,
|
|
SearchCompressor,
|
|
SearchCompressorConfig,
|
|
)
|
|
|
|
|
|
def test_compress_scores_caps_and_summarizes_search_results() -> None:
|
|
compressor = SearchCompressor(
|
|
SearchCompressorConfig(
|
|
max_matches_per_file=3,
|
|
max_total_matches=4,
|
|
max_files=2,
|
|
context_keywords=["auth"],
|
|
enable_ccr=False,
|
|
)
|
|
)
|
|
content = "\n".join(
|
|
[
|
|
"src/auth.py:10:ERROR auth failed",
|
|
"src/auth.py-11-warning auth retry",
|
|
"src/auth.py:12:plain auth line",
|
|
"src/auth.py:13:another auth line",
|
|
"src/db.py:2:warning token expired",
|
|
"not a match",
|
|
]
|
|
)
|
|
result = compressor.compress(content, context="find auth error")
|
|
|
|
# "not a match" is not a search row; both files are parsed.
|
|
assert result.original_match_count == 5
|
|
assert result.files_affected == 2
|
|
# The error line ranks first and is kept; auth.py is capped at 3 matches.
|
|
assert "src/auth.py:10:ERROR auth failed" in result.compressed
|
|
assert "src/db.py:2:warning token expired" in result.compressed
|
|
assert result.summaries == {"src/auth.py": "[... and 1 more matches in src/auth.py]"}
|
|
assert "[... and 1 more matches in src/auth.py]" in result.compressed
|
|
|
|
|
|
def test_search_compressor_compress_paths_and_ccr() -> None:
|
|
"""Phase 3e.2: `compress()` is now a single Rust call, so this test
|
|
exercises end-to-end behavior instead of monkeypatching internal
|
|
helpers (which the old orchestration relied on). The CCR plumbing
|
|
is verified via `cache_key` presence + the marker string format
|
|
Rust emits."""
|
|
compressor = SearchCompressor(
|
|
SearchCompressorConfig(enable_ccr=True, min_matches_for_ccr=2, context_keywords=["auth"])
|
|
)
|
|
no_match = compressor.compress("plain text only")
|
|
assert no_match.original_match_count == 0
|
|
assert no_match.compressed == "plain text only"
|
|
|
|
# Build a large input so compute_optimal_k's min_k=5 floor doesn't
|
|
# absorb everything and compression actually fires (must drop the
|
|
# ratio below `min_compression_ratio_for_ccr=0.8`).
|
|
lines = [f"src/auth.py:{i}:auth event {i}" for i in range(1, 51)]
|
|
lines += [f"src/db.py:{i}:db query {i}" for i in range(1, 31)]
|
|
content = "\n".join(lines)
|
|
result = compressor.compress(content, context="auth", bias=0.5) # low bias = drop more
|
|
assert result.original_match_count == 80
|
|
assert result.files_affected == 2
|
|
assert result.compressed_match_count < result.original_match_count
|
|
assert result.cache_key is not None
|
|
assert result.compressed.endswith(f". Retrieve more: hash={result.cache_key}]")
|
|
# Summaries appear for any file whose matches were dropped.
|
|
assert isinstance(result.summaries, dict)
|
|
assert len(result.summaries) >= 1
|
|
|
|
|
|
def test_persist_rust_ccr_entry(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""CCR write-through delegates to the production `CompressionStore`.
|
|
Failures are logged (not silently swallowed) — this pins both paths."""
|
|
seen: dict[str, tuple[str, str, str | None]] = {}
|
|
monkeypatch.setitem(
|
|
__import__("sys").modules,
|
|
"headroom.cache.compression_store",
|
|
SimpleNamespace(
|
|
get_compression_store=lambda: SimpleNamespace(
|
|
store=lambda original, compressed, original_item_count=0, explicit_hash=None: (
|
|
seen.setdefault("call", (original, compressed, explicit_hash)) or "stored-key"
|
|
)
|
|
)
|
|
),
|
|
)
|
|
persist_rust_ccr_entry("orig", "comp", "abc123", source="search")
|
|
# explicit_hash carries the Rust marker key so retrieval of the
|
|
# marker hash finds the entry (issue #816).
|
|
assert seen["call"] == ("orig", "comp", "abc123")
|
|
|
|
# Loud failure: the store raises, but persist swallows + logs (no
|
|
# exception propagates to the compress callsite).
|
|
def broken_store() -> SimpleNamespace:
|
|
raise RuntimeError("boom")
|
|
|
|
monkeypatch.setitem(
|
|
__import__("sys").modules,
|
|
"headroom.cache.compression_store",
|
|
SimpleNamespace(get_compression_store=broken_store),
|
|
)
|
|
persist_rust_ccr_entry("orig", "comp", "abc123", source="search") # must not raise
|
|
|
|
|
|
def test_search_compression_result_properties() -> None:
|
|
"""Result-property contract preserved across the port."""
|
|
result = SearchCompressionResult(
|
|
compressed="tiny",
|
|
original="this is a much longer original string",
|
|
original_match_count=10,
|
|
compressed_match_count=4,
|
|
files_affected=2,
|
|
compression_ratio=0.3,
|
|
)
|
|
assert result.tokens_saved_estimate > 0
|
|
assert result.matches_omitted == 6
|