mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description When the code compressor declined a block of source code, the router passed the block to Kompress. Kompress is a prose model. It drops low-salience words, and in code those words are keywords such as `for`, `in`, `if` and `import`. The result no longer parses. #4214 shows a fenced Python script that comes back with `for` and `in` removed. This PR stops that in the Python router and in the Rust live-zone dispatcher. When the code compressor declines, the code is forwarded unchanged. Closes #4214 Credit: the Python change is oxura's work from https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018 (277ca1f). Their commit is the first commit here, rebased onto `main`, and oxura is its author. The test, docs and Rust commits were added on top. oxura asked us to open this PR from our fork ([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)), and said they have not run this branch, so the model and native results below are ours, not theirs. ## Behaviour change to review With `enable_code_aware=False` (the default), source code now stays unchanged. Before this PR, it was sent to Kompress and lost keywords. With `enable_code_aware=True`, code that the code compressor declines also stays unchanged. Before, Kompress ran on it. A code block the code compressor does compress is unaffected. Prose, logs, JSON and tables are unaffected. Side effect: a code block that Kompress used to shrink (and break) now keeps its full size. This is the intended trade, but it means fewer tokens saved on code when `enable_code_aware=False`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update - [ ] New feature - [ ] Breaking change (the default-config behaviour change above is intended, and it only stops damage to code) ## Changes Made Five commits: - `596764c3` `fix(router): keep source code out of prose fallbacks`. Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in `headroom/transforms/content_router.py`. This is 277ca1f rebased onto `main`, and it differs from 277ca1f in two ways only: the two test-file conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because `AGENTS.md` says not to add content under `wiki/`. - `7705a4e6` `test(router): repin the code-to-Kompress tests and add the #4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by: oxura` trailer. Five tests asserted the old fallback. They are rewritten to assert the new behaviour (code unchanged, Kompress never sees it) instead of being deleted. Adds a regression test for the script from #4214, with `enable_code_aware` both off and on. Updates `docs/content/docs/code-compression.mdx` and `docs/content/docs/how-compression-works.mdx`, which said that detected code goes to Kompress. - `6631668a` `fix(live-zone): leave source code unchanged when the code compressor declines`. Author: Tao Hansen. Same fix in `crates/headroom-core/src/transforms/live_zone.rs`: a declined `SourceCode` block is now a NoOp, not a Kompress input. Tests: `tests/live_zone_source_code_decline.rs` (single test, needs no model), and new cases in `tests/live_zone_dispatch.rs`. - `d8776283` `fix(router): turn off the code compressor's internal Kompress fallback`. Author: Tao Hansen. Follows review: the router's code compressor was still built with `fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown languages, Perl and AST failures could build a `KompressCompressor` inside `CodeAwareCompressor`. The router now always sets `fallback_to_kompress=False`. The #4719 test that pinned `True` (`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as `test_enabled_kompress_still_turns_off_code_aware_fallback`, and the unknown-language, Perl (by language hint, since `detect_language()` reports Perl-dominant code as unknown) and AST-failure paths are tested with spies on `is_kompress_available` and `KompressCompressor`, with `enable_kompress` both off and on. `docs/content/docs/code-compression.mdx` notes that the router always turns the option off. - `c1f8464e` `fix(router): keep passthrough code from external compressors and stale caches`. Author: Tao Hansen. Follows review: `_try_external_compressor` skips `PASSTHROUGH` blocks (code with `prefer_code_aware_for_code=False`, whole or as a mixed section), which would otherwise be offered to a prose-only external compressor as `text/plain`. The disk cache namespace now also hashes `_ROUTING_POLICY_REVISION`, so a same-version restart does not replay code that Kompress damaged before this change. Tests: two external-dispatch cases with a `text/plain` external, and one namespace test. All three fail on the previous head. I did not touch `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`): see below, the failures are environment failures that also fail on `main` - [x] Linting passes (`ruff check .`, `ruff format --check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py`) - [x] New tests added for new behaviour - [x] Manual testing performed (real model replay, below) ### Test Output macOS, Python 3.14. Head is now `c1f8464e`. ```text After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean. After the rebase onto mainc05fbdddand the review fix (d8776283): router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure) ruff check / ruff format --check headroom tests: clean cargo test -p headroom-core: 1151 passed, 0 failed Earlier run on9df60c7c(before the rebase): router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed: test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget, a timing test the branch does not touch; it passes 3 of 3 times when rerun alone ruff check .: All checks passed! ruff format --check headroom tests: 1714 files already formatted mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file cargo test -p headroom-core: 22 test binaries, 1151 passed, 0 failed cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean pytest tests (full): 31 failed, 19208 passed, 513 skipped ``` Rust bug-fix tests, fail before and pass after (the router-test commit's source with the new Rust tests, then the live-zone commit; measured before the rebase as `9df60c7c`, and the Rust code is unchanged since): ```text before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on7705a4e6) source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed) after: all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok ``` The dispatch fail-before was measured on the earlier, code-identical branch (only documentation files differ). The dispatch test only fails before the fix when the Kompress model is loaded (`HF_HOME` with the model, `ORT_DYLIB_PATH` set). The single-test binary fails before the fix on any host. Python regression: with the new tests run against `origin/main`'s router (no fix), 9 test cases fail (the #4214 fence test in both configs, the repinned tests, and the fenced-code keyword test). With the fix they pass. ### Failures in the full `pytest tests` run The full run on the branch had 31 failures, none in router or compression code. I compared them with pristine `origin/main` (200c838c): - 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path is too long under this checkout directory (`OSError`). They fail the same way in isolation on `main`. - 17 other failures are in `test_proxy_health.py`, `test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`, `test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`, `test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A full run on `main` fails 29 tests, and the set differs from run to run (the two sets overlap by 17 of 31). I re-ran the 14 failures that were not in the `main` list in isolation on the branch: 12 pass, the other 2 are the UDS tests above. - Known flaky test: `tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads` failed once in a loaded run and passes on rerun. File repeated 3 times on `596764c3` and on `origin/main`: 16 passed each time. It is timing based and not caused by this change. It passed in the 558-test router run above. ## Real Behavior Proof - Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`, real `kompress-v2-base` int8 ONNX model from an offline copy of the public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the venv's onnxruntime. - Fence: a 1324-character valid Python script. The issue quotes only four of its lines, so the rest is a reconstruction in the same style, not the reporter's bytes. - Exact command / steps: `ContentRouter.compress` on the fence followed by about 40 lines of prose, for four configs (default; `enable_code_aware=True`; `prefer_code_aware_for_code=False`; `enable_code_aware=True` with lossless-then-lossy). Also `headroom.compress` on a full Anthropic message list with the fence in a `tool_result`. - Observed result: Before (v0.44.0): in the default and code-aware configs the fence comes back as 1144 characters and does not parse (`import json, collections` becomes `json, collections`; `[s for s in ...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs and in `headroom.compress`, the fence is byte-identical, 1324 characters, and parses. Prose control: the same router on prose still compresses. A prose-only block went from 5709 to 5357 characters (strategy TEXT). A prose / fence / prose block went from 12760 to 12056 characters (strategy MIXED), and the fence in it is byte-identical and parses. - Not tested: the reporter's exact 1324-character fence (only four lines are public, so the tests use the reconstructed fence described above); the Rust live-zone path end to end through the proxy with a real request (covered by `cargo test` only); platforms other than macOS; the full proxy on a live deployment. ## Other places where code can still reach a prose model I checked every path from code to a prose or ML compressor, in Python and Rust. | Path | State | | --- | --- | | Python `CODE_AWARE` branch, code compressor declines or is disabled | Fixed here: the block stays unchanged | | Python post-strategy fallback after "no savings" (`content_router.py`, `fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so unchanged | | Python lossless-then-lossy (opt-in) after the lossless fold | `CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged | | Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in) | Prose-like output, not code. Unchanged | | Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged | | Python mixed content with fenced code (`_compress_mixed`) | Each fence goes through the `CODE_AWARE` branch, so it is fixed by this change. Covered by the new fence tests | | Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before this PR | | Python `_experimental_compress_read` (`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment that runs Kompress on file reads, which can be code. Off by default. Left as is | | Python size gate and deadline fallbacks in `_try_ml_compressor` | Run only after a caller chose Kompress, so code no longer reaches them | | Python `CodeAwareCompressor` internal fallback (`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST failure) | Fixed in commit 4: the router always builds it with `fallback_to_kompress=False`. Covered by Kompress-spy tests | | Python external compressor dispatch for `PASSTHROUGH` code | Fixed in commit 5: never offered to an external compressor | | Disk result cache made before this change | Fixed in commit 5: a new namespace through `_ROUTING_POLICY_REVISION` | | Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) | | Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It is the single remaining call to `kompress_or_noop` | | Rust: prose text with an embedded fenced code block | **Not reproduced.** I built synthetic tool results of prose around the fenced Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher classified each one as `source_code`, so commit 3 leaves it unchanged, and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has no fence-aware splitting, but I found no input where that sends code to Kompress, so this PR files no follow-up. | | Python `config_compressor.py` docstring says YAML/TOML/INI "fall through to the lossy prose path" | Documentation only: a stale comment, not a code path. Config content has its own `CONFIG` strategy. Can be refreshed in a later change | ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, in the default config code is no longer sent to Kompress (see "Behaviour change to review"). Tokens saved on code drop in return for code that stays valid. - Kill switch / disable path: none added. Revert the PR to restore the old fallback. - Unsafe override required: no. - Qualification impact: none expected. Compression of code by the code compressor is unchanged. - Rollback path: revert the three commits. No data, schema or config migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (`ruff`, `cargo fmt`, `clippy` clean) - [x] I have made corresponding changes to the documentation (`docs/content/docs/code-compression.mdx`, `docs/content/docs/how-compression-works.mdx`) - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` - [ ] New and existing unit tests pass locally: the only failures are environment failures that also fail on `main`, listed above ## Additional Notes - Rust has no fence-aware splitting like the Python router has. A synthetic prose-plus-fence repro did not reach Kompress (see the table above), so no follow-up issue is filed. - One stale comment remains in `content_router.py` (`_try_ml_compressor`, the Phase 0 size-gate note lists `CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not changed here to keep the review commits as they are. Say if you want it fixed in this PR. - Reviewers: the Python change is easiest to review commit by commit. Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and the Rust fix. --------- Co-authored-by: oxura <grozafroza44@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
443 lines
17 KiB
Python
443 lines
17 KiB
Python
"""Tests for routing a selected EXTERNAL compressor through the content router.
|
||
|
||
Scope 3 of the pluggable-compressor work: an opt-in ``headroom.compressor``
|
||
entry point, when SELECTED, actually compresses matching real traffic through
|
||
:meth:`ContentRouter._apply_strategy_to_content`, with fail-open fallback to the
|
||
built-in dispatch. BACKWARD COMPATIBILITY is the hard requirement — with nothing
|
||
selected the branch is inert and the default request path is byte-identical.
|
||
|
||
Covers:
|
||
(a) a selected external compressor compresses a matching block end-to-end;
|
||
(b) its ``recoverable`` (hash -> original) map is retrievable from the CCR store;
|
||
(c) fail-open: an external that raises / malforms / expands falls back to the
|
||
built-in path (never breaks the request);
|
||
(d) NOT selected, or a non-matching content type, leaves the built-in path
|
||
untouched (the external is never even invoked);
|
||
plus the proxy seam that threads external names to the router.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import hashlib
|
||
|
||
import pytest
|
||
|
||
from headroom.cache.compression_store import (
|
||
get_compression_store,
|
||
reset_compression_store,
|
||
)
|
||
from headroom.proxy.server import _external_compressor_selection
|
||
from headroom.transforms.compressor_registry import (
|
||
CompressInput,
|
||
CompressorDescriptor,
|
||
CompressOutput,
|
||
)
|
||
from headroom.transforms.content_router import (
|
||
ContentRouter,
|
||
ContentRouterConfig,
|
||
)
|
||
|
||
# A JSON array reliably routes to SMART_CRUSHER (content type application/json)
|
||
# and is not touched by the STAGE-0 lossless fold, so it reaches the external
|
||
# dispatch branch. Big enough that the reference compressor's output shrinks it.
|
||
_JSON_ARRAY = (
|
||
"["
|
||
+ ",".join(
|
||
f'{{"id":{i},"name":"item-{i}","status":"active","value":{i * 7},'
|
||
f'"note":"a fairly long descriptive field number {i} to add bulk"}}'
|
||
for i in range(40)
|
||
)
|
||
+ "]"
|
||
)
|
||
|
||
|
||
@pytest.fixture
|
||
def _memory_ccr(monkeypatch):
|
||
"""Isolated in-memory CCR store + offline content detection per test.
|
||
|
||
``HEADROOM_DETECT_BACKEND=python`` forces the pure-Python regex detector so
|
||
``compress()`` never touches the native Magika/ONNX detector (which needs a
|
||
model download and blocks in this offline environment). The external-dispatch
|
||
branch under test is independent of the detector backend.
|
||
"""
|
||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "python")
|
||
reset_compression_store()
|
||
yield
|
||
reset_compression_store()
|
||
|
||
|
||
class _WordTokenizer:
|
||
"""Word-count tokenizer stub — no model, deterministic, offline-safe."""
|
||
|
||
def count_text(self, text: object) -> int:
|
||
return len(str(text).split())
|
||
|
||
def count_messages(self, messages: list[dict]) -> int:
|
||
return sum(self.count_text(m.get("content", "")) for m in messages)
|
||
|
||
|
||
def _json_array(tag: str) -> str:
|
||
"""A distinct, spaced JSON array (>50 word-tokens, >500 chars) that the
|
||
pure-Python detector routes to SMART_CRUSHER (application/json)."""
|
||
return (
|
||
"["
|
||
+ ",".join(
|
||
f'{{"id":{i},"tag":"{tag}","note":"long descriptive field number {i} '
|
||
f'to add real bulk here"}}'
|
||
for i in range(40)
|
||
)
|
||
+ "]"
|
||
)
|
||
|
||
|
||
def _tool_msg(call_id: str, content: str) -> dict:
|
||
# tool_call_id with no matching assistant tool_calls -> not excluded -> the
|
||
# non-frozen one reaches compression (matches Bash/shell output).
|
||
return {"role": "tool", "tool_call_id": call_id, "content": content}
|
||
|
||
|
||
class _RecordingExternal:
|
||
"""Reference in-process external ``Compressor`` for the router tests.
|
||
|
||
Deterministically shrinks its input to a short marker string and, unless
|
||
``recoverable=False``, returns a ``{hash: original}`` recovery map keyed by
|
||
the same hash it embeds in the output — mirroring how SmartCrusher's markers
|
||
point back into the CCR store.
|
||
"""
|
||
|
||
def __init__(
|
||
self,
|
||
name: str = "ext_json",
|
||
content_types: tuple[str, ...] = ("application/json",),
|
||
*,
|
||
lossless: bool = False,
|
||
recoverable: bool = True,
|
||
raises: bool = False,
|
||
expand: bool = False,
|
||
malformed: bool = False,
|
||
empty: bool = False,
|
||
bad_hash: bool = False,
|
||
) -> None:
|
||
self._name = name
|
||
self._content_types = list(content_types)
|
||
self._lossless = lossless
|
||
self._recoverable = recoverable
|
||
self._raises = raises
|
||
self._expand = expand
|
||
self._malformed = malformed
|
||
self._empty = empty
|
||
self._bad_hash = bad_hash
|
||
self.calls: list[CompressInput] = []
|
||
|
||
@property
|
||
def descriptor(self) -> CompressorDescriptor:
|
||
return CompressorDescriptor(
|
||
name=self._name,
|
||
content_types=self._content_types,
|
||
lossless=self._lossless,
|
||
cost_tier="fast",
|
||
recoverable=self._recoverable,
|
||
)
|
||
|
||
def compress(self, inp: CompressInput): # noqa: ANN201 - matches protocol
|
||
self.calls.append(inp)
|
||
if self._raises:
|
||
raise RuntimeError("external boom")
|
||
if self._malformed:
|
||
return {"not": "a CompressOutput"}
|
||
original = inp.content
|
||
digest = hashlib.sha256(original.encode()).hexdigest()[:24]
|
||
key = "zznothex" if self._bad_hash else digest
|
||
if self._empty:
|
||
content = ""
|
||
elif self._expand:
|
||
content = original + (" PADDING" * 200)
|
||
else:
|
||
content = f"[external-compressed <<ccr:{digest}>>]"
|
||
recoverable = {key: original} if self._recoverable else {}
|
||
return CompressOutput(
|
||
content=content,
|
||
tokens_before=len(original.split()),
|
||
tokens_after=len(content.split()),
|
||
lossless=self._lossless,
|
||
markers=[f"external:{self._name}"],
|
||
recoverable=recoverable,
|
||
warnings=["reference-warning"],
|
||
)
|
||
|
||
|
||
def _cfg(**kwargs) -> ContentRouterConfig:
|
||
"""Router config with Kompress OFF so the built-in fallback path never
|
||
loads the ModernBERT ML model (keeps these unit tests fast and offline).
|
||
The external-dispatch branch under test is independent of this flag."""
|
||
kwargs.setdefault("enable_kompress", False)
|
||
return ContentRouterConfig(**kwargs)
|
||
|
||
|
||
def _router_with_external(comp: _RecordingExternal, selection):
|
||
"""Build a router with ``comp`` registered and ``selection`` active."""
|
||
router = ContentRouter(_cfg(active_external_compressors=selection))
|
||
router.compressor_registry.register(comp, replace=True)
|
||
# Re-resolve now that the external compressor is registered (the router
|
||
# resolves the selection once at construction, before this injection).
|
||
router._active_external_compressors = router._resolve_active_external_compressors()
|
||
return router
|
||
|
||
|
||
# ─────────────────────────── (a) end-to-end ──────────────────────────────────
|
||
|
||
|
||
def test_selected_external_compresses_matching_block_end_to_end(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls, "external compressor should have been invoked"
|
||
assert "external:ext_json" in result.strategy_chain
|
||
assert "external-compressed" in result.compressed
|
||
assert len(result.compressed) < len(_JSON_ARRAY)
|
||
# The CompressInput carried the block + its detected MIME content type.
|
||
assert comp.calls[0].content == _JSON_ARRAY
|
||
assert comp.calls[0].content_type == "application/json"
|
||
|
||
|
||
@pytest.mark.parametrize("mime", ["text/csv", "*/*"])
|
||
def test_selected_external_preserves_untracked_git_status_bytes(_memory_ccr, mime):
|
||
status = (
|
||
"On branch feature/status\n\nUntracked files:\n"
|
||
' (use "git add <file>..." to include in what will be committed)\n'
|
||
+ "".join(f"\tuntracked path {i:03d} α.txt\n" for i in range(120))
|
||
)
|
||
compressor = _RecordingExternal(name="ext_status", content_types=(mime,))
|
||
router = _router_with_external(compressor, ["ext_status"])
|
||
|
||
direct = router.compress(status)
|
||
messages = [
|
||
{"role": "user", "content": "List every untracked path."},
|
||
_tool_msg("status", status),
|
||
{"role": "user", "content": "Keep the exact path names."},
|
||
]
|
||
tool_result = router.apply(messages, _WordTokenizer(), model_limit=100_000)
|
||
|
||
assert direct.compressed == status
|
||
assert tool_result.messages[1]["content"] == status
|
||
|
||
|
||
def test_native_misclassification_keeps_git_status_format(_memory_ccr, monkeypatch):
|
||
from types import SimpleNamespace
|
||
|
||
import headroom._core
|
||
import headroom._ort
|
||
from headroom.transforms.content_detector import ContentType
|
||
from headroom.transforms.content_router import _detect_content
|
||
|
||
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
|
||
monkeypatch.setattr("headroom.transforms.content_router._detect_native_unhealthy", False)
|
||
monkeypatch.setattr(headroom._ort, "rust_ort_runtime_compatible", lambda: True)
|
||
monkeypatch.setattr(
|
||
headroom._core,
|
||
"detect_content_type",
|
||
lambda _content: SimpleNamespace(content_type="html", confidence=1.0),
|
||
)
|
||
status = "On branch feature/status\n\nUntracked files:\n\tnew path α.txt\n"
|
||
|
||
detected = _detect_content(status)
|
||
|
||
assert detected.content_type is ContentType.TABULAR
|
||
assert detected.metadata["format"] == "git_status"
|
||
|
||
|
||
# ─────────────────────────── (b) recoverable map ─────────────────────────────
|
||
|
||
|
||
def test_recoverable_map_is_retrievable(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
router.compress(_JSON_ARRAY)
|
||
|
||
digest = hashlib.sha256(_JSON_ARRAY.encode()).hexdigest()[:24]
|
||
entry = get_compression_store().retrieve(digest)
|
||
assert entry is not None, "recoverable entry should be in the CCR store"
|
||
assert entry.original_content == _JSON_ARRAY
|
||
assert entry.compression_strategy == "external:ext_json"
|
||
|
||
|
||
def test_non_hex_recoverable_hash_falls_back_to_builtin(_memory_ccr):
|
||
# A malformed (non-hex) recovery hash must not break the request; the store
|
||
# cannot keep that entry, so the external output (whose marker would point
|
||
# nowhere) is discarded and the built-in path runs.
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), bad_hash=True)
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert "external:ext_json" not in result.strategy_chain
|
||
assert "external-compressed" not in result.compressed
|
||
assert get_compression_store().retrieve("zznothex") is None
|
||
|
||
|
||
# ─────────────────────────── (c) fail-open ───────────────────────────────────
|
||
|
||
|
||
def test_external_raise_falls_back_to_builtin(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), raises=True)
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls, "external should have been attempted"
|
||
# Fell back: no external marker in the chain, request not broken.
|
||
assert "external:ext_json" not in result.strategy_chain
|
||
assert result.compressed and result.compressed.strip()
|
||
|
||
|
||
def test_external_malformed_output_falls_back_to_builtin(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), malformed=True)
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls
|
||
assert "external:ext_json" not in result.strategy_chain
|
||
|
||
|
||
def test_external_expansion_falls_back_to_builtin(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), expand=True)
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls
|
||
assert "external:ext_json" not in result.strategy_chain
|
||
# Never expands: the returned block is no larger than the input.
|
||
assert len(result.compressed) <= len(_JSON_ARRAY)
|
||
|
||
|
||
def test_external_empty_output_falls_back_to_builtin(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), empty=True)
|
||
router = _router_with_external(comp, ["ext_json"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls
|
||
assert "external:ext_json" not in result.strategy_chain
|
||
assert result.compressed.strip(), "non-empty input must never blank out"
|
||
|
||
|
||
# ─────────────── (d) not selected / non-matching → built-in ──────────────────
|
||
|
||
|
||
def test_not_selected_leaves_builtin_path_unchanged(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
|
||
# Registered but NOT selected.
|
||
router = _router_with_external(comp, None)
|
||
|
||
baseline = ContentRouter(_cfg()).compress(_JSON_ARRAY)
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls == [], "unselected external must never be invoked"
|
||
assert "external:ext_json" not in result.strategy_chain
|
||
# Byte-identical to a plain router with no external registered at all.
|
||
assert result.compressed == baseline.compressed
|
||
assert result.strategy_chain == baseline.strategy_chain
|
||
|
||
|
||
def test_non_matching_content_type_leaves_builtin_path_unchanged(_memory_ccr):
|
||
# Selected, but declares a content type the JSON block never has.
|
||
comp = _RecordingExternal(name="ext_diff", content_types=("text/x-diff",))
|
||
router = _router_with_external(comp, ["ext_diff"])
|
||
|
||
baseline = ContentRouter(_cfg()).compress(_JSON_ARRAY)
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls == [], "non-matching external must never be invoked"
|
||
assert "external:ext_diff" not in result.strategy_chain
|
||
assert result.compressed == baseline.compressed
|
||
|
||
|
||
def test_default_config_has_no_external_selection():
|
||
assert ContentRouterConfig().active_external_compressors is None
|
||
router = ContentRouter(ContentRouterConfig())
|
||
assert router._active_external_compressors == []
|
||
|
||
|
||
# ─────────────────────────── proxy seam ──────────────────────────────────────
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"selection,expected",
|
||
[
|
||
(None, None),
|
||
(set(), None),
|
||
({"", " "}, None),
|
||
({"smart_crusher"}, None), # built-in only → no external
|
||
({"smart_crusher", "kompress"}, None),
|
||
({"my_ext"}, ["my_ext"]),
|
||
({"kompress", "my_ext"}, ["my_ext"]),
|
||
({"b_ext", "a_ext"}, ["a_ext", "b_ext"]), # sorted
|
||
({"*"}, ["*"]),
|
||
({"*", "my_ext"}, ["*"]), # wildcard wins
|
||
],
|
||
)
|
||
def test_external_compressor_selection_helper(selection, expected):
|
||
assert _external_compressor_selection(selection) == expected
|
||
|
||
|
||
def test_wildcard_selection_activates_registered_external(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
|
||
router = _router_with_external(comp, ["*"])
|
||
|
||
result = router.compress(_JSON_ARRAY)
|
||
|
||
assert comp.calls
|
||
assert "external:ext_json" in result.strategy_chain
|
||
|
||
|
||
# ───────────── explicit passthrough code never reaches an external ─────────────
|
||
|
||
_PY_FENCE = "\n".join(
|
||
["import json", ""]
|
||
+ [
|
||
f"def handler_{i}(event):\n if event.get('k{i}'):\n return json.dumps(event)\n return None\n"
|
||
for i in range(30)
|
||
]
|
||
)
|
||
_PROSE = " ".join(
|
||
["The nightly job failed after the deploy and the operator wants a short summary."] * 40
|
||
)
|
||
|
||
|
||
def test_passthrough_code_section_is_not_offered_to_a_text_external(_memory_ccr):
|
||
# With prefer_code_aware_for_code=False the router keeps source code
|
||
# unmangled (PASSTHROUGH). A selected prose-only external compressor must
|
||
# not receive that code: mixed content splits the fence into its own section.
|
||
comp = _RecordingExternal(name="ext_text", content_types=("text/plain",))
|
||
router = ContentRouter(
|
||
_cfg(active_external_compressors=["ext_text"], prefer_code_aware_for_code=False)
|
||
)
|
||
router.compressor_registry.register(comp, replace=True)
|
||
router._active_external_compressors = router._resolve_active_external_compressors()
|
||
content = f"{_PROSE}\n\n```python\n{_PY_FENCE}\n```\n\n{_PROSE}"
|
||
|
||
result = router.compress(content)
|
||
|
||
assert all("def handler_0" not in call.content for call in comp.calls)
|
||
assert f"```python\n{_PY_FENCE}\n```" in result.compressed
|
||
|
||
|
||
def test_passthrough_whole_code_block_is_not_offered_to_a_text_external(_memory_ccr):
|
||
comp = _RecordingExternal(name="ext_text", content_types=("text/plain",))
|
||
router = ContentRouter(
|
||
_cfg(active_external_compressors=["ext_text"], prefer_code_aware_for_code=False)
|
||
)
|
||
router.compressor_registry.register(comp, replace=True)
|
||
router._active_external_compressors = router._resolve_active_external_compressors()
|
||
|
||
result = router.compress(_PY_FENCE)
|
||
|
||
assert not comp.calls
|
||
assert result.compressed == _PY_FENCE
|