mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
refactor(transforms): share one CCR persist helper for the Rust-backed compressors (#4094)
## Description DiffCompressor, LogCompressor and SearchCompressor each carried an identical private `_persist_to_python_ccr` method. It writes the Rust-emitted CCR entry through to the production CompressionStore with `explicit_hash` (#816). This PR replaces the three copies with one function, `headroom.transforms.base.persist_rust_ccr_entry`. Its warnings name the compressor that failed and keep the store's exception text out of every log level. Refs #4136 Depends on #4138 (`headroom/log_safety.py`); its files show in this diff until it merges. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Add `persist_rust_ccr_entry(original, compressed, cache_key, *, source)` to `transforms/base.py`, using a lazy import of the compression store as before. - The three `compress()` methods call it with `source="diff"`, `"log"` or `"search"`. - Remove the three private copies, and the loggers in `diff_compressor` and `search_compressor` that are now unused. - The write-failure WARNING names the source and exception type. The detail goes to DEBUG as `describe_exception(e)` (types and code locations, no message or traceback), because store backends can echo the payload. The import-failure warning uses it too. `explicit_hash` and fail-open are unchanged. - Tests: - the three #816 bridge tests become one parametrized test over the helper; - they no longer skip without the Rust extension, because the helper is pure Python; - a new test formats every captured DEBUG and WARNING record and asserts the original-content and credential canaries are absent. It fails on the earlier `exc_info=True` version of this PR. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_search_compressor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py tests/test_transforms_log_compressor.py tests/test_log_compressor.py tests/test_search_compressor.py tests/test_transforms_content_router.py 280 passed in 15.69s $ python -c "import sys; sys.modules['headroom._core'] = None; import pytest; pytest.main([... 'tests/test_ccr_rust_marker_hash_bridge.py'])" 3 passed $ ruff check / format: All checks passed; mypy base.py: Success ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree (head1bd479d6) with the prebuilt `headroom._core`. Temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real home. - Exact command / steps: (1) shared smoke harness: a real proxy from this checkout against a local fake upstream, with one Anthropic and one OpenAI request carrying a compressible tool result (`smoke.sh <worktree>`); (2) a real `DiffCompressor.compress()` on a 60-file diff, then a retrieval of its CCR entry under the marker key; (3) the real `persist_rust_ccr_entry` with logging at DEBUG, against a rejecting backend whose exception echoes the source (with a credential in its cause) and a backend raising `OSError` with errno `10**10000`, `HEADROOM_DEBUG_DUMP` unset. - Observed result: The proxy serves both requests (200) with savings. The diff compressor emits a cache_key, and the entry can be retrieved under exactly that key. On a store failure the helper returns `None` (fail-open); the WARNING names source, class and cache key, and the DEBUG line names only types and code locations. Neither canary appears in any rendered line (grep count 0). Output below. - Not tested: A model-issued retrieval through a live proxy; the direct check reads the same store. ```text branch: cleanup/transforms-shared-ccr-persist @1bd479d6health: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-57958.log): 2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes. 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base 4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. (smoke dir: $SMOKE) ``` ```text $ python diff_direct.py # DiffCompressor(min_lines_for_ccr=10).compress(60-file diff); store.retrieve(cache_key) cache_key: 76142b4e8a8086abb3709b9f retrievable under marker key: True ``` ```text $ python persist_proof.py # real persist_rust_ccr_entry, DEBUG on, two failing backends WARNING headroom.transforms.base CCR store write failed (diff, RuntimeError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only DEBUG headroom.transforms.base CCR store write failure detail: RuntimeError at persist_proof.py:12 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry; caused by ValueError at persist_proof.py:10 in store returned: None WARNING headroom.transforms.base CCR store write failed (diff, OSError); cache_key aaaaaaaaaaaaaaaaaaaaaaaa remains in-marker only DEBUG headroom.transforms.base CCR store write failure detail: OSError [Errno out of range] at persist_proof.py:15 in store <- headroom/transforms/base.py:40 in persist_rust_ccr_entry returned: None $ grep -c "CANARY\|sk-test" persist_proof.out 0 ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Not applicable; no rollout-managed behaviour is touched. - Stable/default behavior changed: No, except that the warning text now includes the compressor name and comes from the `headroom.transforms.base` logger. - Kill switch / disable path: Not applicable; the stored data and keys are unchanged. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this single commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. Documentation changes are not in this PR. They are recorded in the cleanup docs ledger for the single follow-up docs PR. `DiffCompressor.compress_with_stats` is unchanged here. It is public, so it is deprecated in its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
580cc83a9a
commit
ea947ca421
@@ -2,12 +2,55 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
from ..config import TransformResult
|
||||
from ..log_safety import describe_exception, safe_id
|
||||
from ..tokenizer import Tokenizer
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def persist_rust_ccr_entry(original: str, compressed: str, cache_key: str, *, source: str) -> None:
|
||||
"""Store a Rust-emitted CCR entry in the production ``CompressionStore``.
|
||||
|
||||
The Rust search/diff/log compressors emit a retrieval marker but keep the
|
||||
original only in their in-memory test store. This writes it through to the
|
||||
long-lived Python store so the marker resolves. Failures are logged at
|
||||
warning level with ``source`` (the compressor name): a store hiccup must not
|
||||
break the response, just degrade retrieval.
|
||||
"""
|
||||
try:
|
||||
from ..cache.compression_store import get_compression_store
|
||||
except ImportError as e:
|
||||
logger.warning(
|
||||
"CCR store import failed (%s); cache_key %s won't persist: %s",
|
||||
source,
|
||||
cache_key,
|
||||
safe_id(e),
|
||||
)
|
||||
logger.debug("CCR store import failure detail: %s", describe_exception(e))
|
||||
return
|
||||
try:
|
||||
store: Any = get_compression_store()
|
||||
# The Rust-emitted marker embeds MD5(original)[:24], but store() has
|
||||
# defaulted to SHA-256(original)[:24] since PR #395. Pass the marker's
|
||||
# key explicitly so retrieving the marker hash finds the entry (#816).
|
||||
store.store(original, compressed, explicit_hash=cache_key)
|
||||
except Exception as e:
|
||||
# Store backends can echo the payload in errors. The WARNING names only
|
||||
# the exception type; the DEBUG record has types and code locations, or
|
||||
# the full traceback (payload included) once HEADROOM_DEBUG_DUMP=full.
|
||||
logger.warning(
|
||||
"CCR store write failed (%s, %s); cache_key %s remains in-marker only",
|
||||
source,
|
||||
safe_id(e),
|
||||
cache_key,
|
||||
)
|
||||
logger.debug("CCR store write failure detail: %s", describe_exception(e))
|
||||
|
||||
|
||||
class Transform(ABC):
|
||||
"""Abstract base class for message transforms."""
|
||||
|
||||
@@ -20,12 +20,11 @@ fallback. Build it locally with `scripts/build_rust_extension.sh`
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import warnings
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from .base import persist_rust_ccr_entry
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -114,7 +113,7 @@ class DiffCompressor:
|
||||
# call. Without this, every diff CCR marker emitted in
|
||||
# production is dangling — the regression fixed in the
|
||||
# audit-cleanup PR.
|
||||
self._persist_to_python_ccr(content, r.compressed, cache_key)
|
||||
persist_rust_ccr_entry(content, r.compressed, cache_key, source="diff")
|
||||
return DiffCompressionResult(
|
||||
compressed=r.compressed,
|
||||
original_line_count=r.original_line_count,
|
||||
@@ -127,31 +126,6 @@ class DiffCompressor:
|
||||
cache_key=cache_key,
|
||||
)
|
||||
|
||||
def _persist_to_python_ccr(self, original: str, compressed: str, cache_key: str) -> None:
|
||||
"""Promote a Rust-emitted cache_key into the production Python
|
||||
CompressionStore. Failures are logged at warning level — a
|
||||
store hiccup must not break the response, just degrade
|
||||
retrieval. Mirrors the same helper on log_compressor.py and
|
||||
search_compressor.py."""
|
||||
try:
|
||||
from ..cache.compression_store import get_compression_store
|
||||
except ImportError as e:
|
||||
logger.warning("CCR store import failed; cache_key %s won't persist: %s", cache_key, e)
|
||||
return
|
||||
try:
|
||||
store: Any = get_compression_store()
|
||||
# The Rust-emitted marker embeds MD5(original)[:24], but
|
||||
# store() has defaulted to SHA-256(original)[:24] since
|
||||
# PR #395. Pass the marker's key explicitly so retrieving
|
||||
# the marker hash actually finds the entry (issue #816).
|
||||
store.store(original, compressed, explicit_hash=cache_key)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"CCR store write failed; cache_key %s remains in-marker only: %s",
|
||||
cache_key,
|
||||
e,
|
||||
)
|
||||
|
||||
def compress_with_stats(
|
||||
self, content: str, context: str = ""
|
||||
) -> tuple[DiffCompressionResult, Any]:
|
||||
|
||||
@@ -37,12 +37,11 @@ exists only for unit testing.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any, cast
|
||||
from typing import cast
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from .base import persist_rust_ccr_entry
|
||||
|
||||
|
||||
class LogFormat(Enum):
|
||||
@@ -201,7 +200,7 @@ class LogCompressor:
|
||||
rust_result = self._rust.compress(content, bias)
|
||||
cache_key: str | None = rust_result.cache_key
|
||||
if cache_key is not None:
|
||||
self._persist_to_python_ccr(content, rust_result.compressed, cache_key)
|
||||
persist_rust_ccr_entry(content, rust_result.compressed, cache_key, source="log")
|
||||
|
||||
stats_dict = {k: int(v) for k, v in cast("dict[str, int]", rust_result.stats).items()}
|
||||
return LogCompressionResult(
|
||||
@@ -215,28 +214,6 @@ class LogCompressor:
|
||||
stats=stats_dict,
|
||||
)
|
||||
|
||||
def _persist_to_python_ccr(self, original: str, compressed: str, cache_key: str) -> None:
|
||||
"""Promote a Rust-emitted cache_key into the production Python
|
||||
CompressionStore. Failures are logged at warning level."""
|
||||
try:
|
||||
from ..cache.compression_store import get_compression_store
|
||||
except ImportError as e:
|
||||
logger.warning("CCR store import failed; cache_key %s won't persist: %s", cache_key, e)
|
||||
return
|
||||
try:
|
||||
store: Any = get_compression_store()
|
||||
# The Rust-emitted marker embeds MD5(original)[:24], but
|
||||
# store() has defaulted to SHA-256(original)[:24] since
|
||||
# PR #395. Pass the marker's key explicitly so retrieving
|
||||
# the marker hash actually finds the entry (issue #816).
|
||||
store.store(original, compressed, explicit_hash=cache_key)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"CCR store write failed; cache_key %s remains in-marker only: %s",
|
||||
cache_key,
|
||||
e,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"LogCompressor",
|
||||
|
||||
@@ -39,12 +39,10 @@ exactly what the previous Python implementation provided.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, cast
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from typing import cast
|
||||
|
||||
from .base import persist_rust_ccr_entry
|
||||
|
||||
# ─── Public dataclasses (preserve existing import surface) ──────────────────
|
||||
|
||||
@@ -178,7 +176,7 @@ class SearchCompressor:
|
||||
# store. The Rust crate already wrote to its in-memory test
|
||||
# store; promote that to the long-lived Python store so the
|
||||
# marker remains retrievable beyond the request lifecycle.
|
||||
self._persist_to_python_ccr(content, rust_result.compressed, cache_key)
|
||||
persist_rust_ccr_entry(content, rust_result.compressed, cache_key, source="search")
|
||||
|
||||
summaries = dict(cast("dict[str, str]", rust_result.summaries))
|
||||
return SearchCompressionResult(
|
||||
@@ -192,36 +190,6 @@ class SearchCompressor:
|
||||
summaries=summaries,
|
||||
)
|
||||
|
||||
# ─── Internal CCR persistence ───────────────────────────────────────
|
||||
|
||||
def _persist_to_python_ccr(self, original: str, compressed: str, cache_key: str) -> None:
|
||||
"""Promote the Rust-emitted cache_key into the production Python
|
||||
`CompressionStore`. Failures are surfaced via logging instead of
|
||||
being silently swallowed (see no-silent-fallbacks rule).
|
||||
|
||||
Note: the Rust path computes the hash and embeds it in the
|
||||
emitted marker text — the Rust hash IS the canonical one
|
||||
(MD5(original)[:24]). The store must be keyed by that exact
|
||||
hash or the marker dangles.
|
||||
"""
|
||||
try:
|
||||
from ..cache.compression_store import get_compression_store
|
||||
except ImportError as e:
|
||||
logger.warning("CCR store import failed; cache_key %s won't persist: %s", cache_key, e)
|
||||
return
|
||||
|
||||
try:
|
||||
store: Any = get_compression_store()
|
||||
# The Rust-emitted marker embeds MD5(original)[:24], but
|
||||
# store() has defaulted to SHA-256(original)[:24] since
|
||||
# PR #395. Pass the marker's key explicitly so retrieving
|
||||
# the marker hash actually finds the entry (issue #816).
|
||||
store.store(original, compressed, explicit_hash=cache_key)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"CCR store write failed; cache_key %s remains in-marker only: %s", cache_key, e
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"SearchCompressor",
|
||||
|
||||
@@ -4,18 +4,20 @@ The Rust side embeds ``MD5(original)[:24]`` in the emitted
|
||||
``Retrieve more: hash=...`` marker, but since PR #395
|
||||
``CompressionStore.store()`` defaults to ``SHA-256(original)[:24]``.
|
||||
PR #395 fixed the SmartCrusher path by passing ``explicit_hash``
|
||||
(see ``test_ccr_row_drop_store_bridge.py``); the three
|
||||
``_persist_to_python_ccr`` shims on the Rust-accelerated transforms
|
||||
were never migrated, so every marker they emitted dangled —
|
||||
retrieval returned "Entry not found or expired" inside any TTL.
|
||||
(see ``test_ccr_row_drop_store_bridge.py``); the CCR write-through
|
||||
used by the Rust-accelerated transforms was never migrated, so every
|
||||
marker they emitted dangled — retrieval returned "Entry not found or
|
||||
expired" inside any TTL.
|
||||
|
||||
These tests pin the cross-language contract at the shim layer: the
|
||||
store entry must be keyed by the exact hash the marker embeds.
|
||||
These tests pin the cross-language contract at the shared
|
||||
``persist_rust_ccr_entry`` helper: the store entry must be keyed by the
|
||||
exact hash the marker embeds.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -23,11 +25,7 @@ from headroom.cache.compression_store import (
|
||||
get_compression_store,
|
||||
reset_compression_store,
|
||||
)
|
||||
from headroom.transforms.diff_compressor import DiffCompressor
|
||||
from headroom.transforms.log_compressor import LogCompressor
|
||||
from headroom.transforms.search_compressor import SearchCompressor
|
||||
|
||||
pytest.importorskip("headroom._core", reason="Rust extension required")
|
||||
from headroom.transforms.base import persist_rust_ccr_entry
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -37,6 +35,30 @@ def _fresh_store():
|
||||
reset_compression_store()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG])
|
||||
def test_persist_warning_rejects_malformed_exception_names(monkeypatch, caplog, level) -> None:
|
||||
from headroom.cache import compression_store
|
||||
from headroom.transforms import base
|
||||
|
||||
canary = "PRIVATE_CREDENTIAL_CANARY\nFORGED_LOG_ENTRY"
|
||||
error_type = type(canary, (RuntimeError,), {})
|
||||
|
||||
class BrokenStore:
|
||||
def store(self, original, compressed, **kwargs):
|
||||
raise error_type("untrusted backend text")
|
||||
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
monkeypatch.setattr(compression_store, "get_compression_store", lambda: BrokenStore())
|
||||
with caplog.at_level(level, logger=base.logger.name):
|
||||
assert persist_rust_ccr_entry("original", "compressed", "a" * 24, source="diff") is None
|
||||
assert len([record for record in caplog.records if record.levelno == logging.WARNING]) == 1
|
||||
for record in caplog.records:
|
||||
rendered = logging.Formatter().format(record)
|
||||
assert "PRIVATE_CREDENTIAL_CANARY" not in rendered
|
||||
assert "FORGED_LOG_ENTRY" not in rendered
|
||||
assert "\n" not in rendered
|
||||
|
||||
|
||||
def _rust_marker_key(original: str) -> str:
|
||||
"""The hash the Rust side embeds in emitted markers."""
|
||||
return hashlib.md5(original.encode()).hexdigest()[:24]
|
||||
@@ -61,25 +83,105 @@ def _assert_round_trip(original: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_search_compressor_shim_stores_under_marker_key() -> None:
|
||||
original = "src/app.py:12: def handle_request(payload):\n" * 40
|
||||
SearchCompressor()._persist_to_python_ccr(
|
||||
original, "compressed search output", _rust_marker_key(original)
|
||||
@pytest.mark.parametrize(
|
||||
("source", "original"),
|
||||
[
|
||||
("search", "src/app.py:12: def handle_request(payload):\n" * 40),
|
||||
("diff", "+added line of code\n-removed line of code\n" * 40),
|
||||
("log", "2026-06-11T09:00:00Z INFO worker heartbeat ok seq=1\n" * 40),
|
||||
],
|
||||
)
|
||||
def test_persist_stores_under_marker_key(source: str, original: str) -> None:
|
||||
persist_rust_ccr_entry(
|
||||
original, f"compressed {source} output", _rust_marker_key(original), source=source
|
||||
)
|
||||
_assert_round_trip(original)
|
||||
|
||||
|
||||
def test_diff_compressor_shim_stores_under_marker_key() -> None:
|
||||
original = "+added line of code\n-removed line of code\n" * 40
|
||||
DiffCompressor()._persist_to_python_ccr(
|
||||
original, "compressed diff output", _rust_marker_key(original)
|
||||
)
|
||||
_assert_round_trip(original)
|
||||
def test_persist_failure_logs_no_exception_text_at_any_level(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
import logging
|
||||
|
||||
import headroom.cache.compression_store as compression_store
|
||||
import headroom.transforms.base as base
|
||||
|
||||
content = "def private_source_canary_4c1e(): pass"
|
||||
credential = "sk-test-credential-canary-91bd"
|
||||
|
||||
class _Store:
|
||||
def store(self, original: str, *_args: object, **_kwargs: object) -> str:
|
||||
try:
|
||||
raise ValueError(f"backend rejected key {credential}")
|
||||
except ValueError as cause:
|
||||
raise RuntimeError(f"cannot store {original}") from cause
|
||||
|
||||
monkeypatch.delenv("HEADROOM_DEBUG_DUMP", raising=False)
|
||||
monkeypatch.setattr(compression_store, "get_compression_store", lambda: _Store())
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Capture(level=logging.DEBUG)
|
||||
base.logger.addHandler(handler)
|
||||
monkeypatch.setattr(base.logger, "level", logging.DEBUG)
|
||||
try:
|
||||
persist_rust_ccr_entry(content, "comp", "a" * 24, source="diff")
|
||||
finally:
|
||||
base.logger.removeHandler(handler)
|
||||
|
||||
formatter = logging.Formatter()
|
||||
for record in records:
|
||||
rendered = formatter.format(record)
|
||||
assert content not in rendered
|
||||
assert credential not in rendered
|
||||
|
||||
warnings = [r for r in records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert "(diff, <RuntimeError>)" in warnings[0].getMessage()
|
||||
assert "a" * 24 in warnings[0].getMessage()
|
||||
details = [r for r in records if r.levelno == logging.DEBUG]
|
||||
assert len(details) == 1
|
||||
assert "RuntimeError" in details[0].getMessage()
|
||||
assert "caused by ValueError" in details[0].getMessage()
|
||||
|
||||
|
||||
def test_log_compressor_shim_stores_under_marker_key() -> None:
|
||||
original = "2026-06-11T09:00:00Z INFO worker heartbeat ok seq=1\n" * 40
|
||||
LogCompressor()._persist_to_python_ccr(
|
||||
original, "compressed log output", _rust_marker_key(original)
|
||||
)
|
||||
_assert_round_trip(original)
|
||||
def test_persist_warning_stays_one_line_when_full_dump_is_on(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""HEADROOM_DEBUG_DUMP=full puts the traceback in DEBUG only, never the WARNING."""
|
||||
import headroom.cache.compression_store as compression_store
|
||||
import headroom.transforms.base as base
|
||||
|
||||
class _Store:
|
||||
def store(self, *_args: object, **_kwargs: object) -> str:
|
||||
raise RuntimeError("cannot store original")
|
||||
|
||||
monkeypatch.setenv("HEADROOM_DEBUG_DUMP", "full")
|
||||
monkeypatch.setattr(compression_store, "get_compression_store", lambda: _Store())
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Capture(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
handler = _Capture(level=logging.DEBUG)
|
||||
base.logger.addHandler(handler)
|
||||
monkeypatch.setattr(base.logger, "level", logging.DEBUG)
|
||||
try:
|
||||
persist_rust_ccr_entry("original", "comp", "a" * 24, source="log")
|
||||
finally:
|
||||
base.logger.removeHandler(handler)
|
||||
|
||||
warnings = [r.getMessage() for r in records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
assert "(log, <RuntimeError>)" in warnings[0]
|
||||
assert "\n" not in warnings[0]
|
||||
assert "Traceback" not in warnings[0]
|
||||
assert "cannot store original" not in warnings[0]
|
||||
details = [r.getMessage() for r in records if r.levelno == logging.DEBUG]
|
||||
assert len(details) == 1
|
||||
assert "Traceback" in details[0]
|
||||
assert "cannot store original" in details[0]
|
||||
|
||||
@@ -4,6 +4,7 @@ from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.transforms.base import persist_rust_ccr_entry
|
||||
from headroom.transforms.search_compressor import (
|
||||
SearchCompressionResult,
|
||||
SearchCompressor,
|
||||
@@ -73,12 +74,9 @@ def test_search_compressor_compress_paths_and_ccr() -> None:
|
||||
assert len(result.summaries) >= 1
|
||||
|
||||
|
||||
def test_search_compressor_persist_to_python_ccr(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 3e.2: CCR persistence is now in `_persist_to_python_ccr`,
|
||||
which delegates to the production `CompressionStore`. Failures are
|
||||
logged (not silently swallowed) — this pins both paths."""
|
||||
compressor = SearchCompressor()
|
||||
|
||||
def test_persist_rust_ccr_entry(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""CCR write-through delegates to the production `CompressionStore`.
|
||||
Failures are logged (not silently swallowed) — this pins both paths."""
|
||||
seen: dict[str, tuple[str, str, str | None]] = {}
|
||||
monkeypatch.setitem(
|
||||
__import__("sys").modules,
|
||||
@@ -91,7 +89,7 @@ def test_search_compressor_persist_to_python_ccr(monkeypatch: pytest.MonkeyPatch
|
||||
)
|
||||
),
|
||||
)
|
||||
compressor._persist_to_python_ccr("orig", "comp", "abc123")
|
||||
persist_rust_ccr_entry("orig", "comp", "abc123", source="search")
|
||||
# explicit_hash carries the Rust marker key so retrieval of the
|
||||
# marker hash finds the entry (issue #816).
|
||||
assert seen["call"] == ("orig", "comp", "abc123")
|
||||
@@ -106,7 +104,7 @@ def test_search_compressor_persist_to_python_ccr(monkeypatch: pytest.MonkeyPatch
|
||||
"headroom.cache.compression_store",
|
||||
SimpleNamespace(get_compression_store=broken_store),
|
||||
)
|
||||
compressor._persist_to_python_ccr("orig", "comp", "abc123") # must not raise
|
||||
persist_rust_ccr_entry("orig", "comp", "abc123", source="search") # must not raise
|
||||
|
||||
|
||||
def test_search_compression_result_properties() -> None:
|
||||
|
||||
Reference in New Issue
Block a user