Files
headroom/tests/test_router_external_dispatch.py
3aafdf2dcc fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description

When the code compressor declined a block of source code, the router
passed the block to Kompress. Kompress is a prose model. It drops
low-salience words, and in code those words are keywords such as `for`,
`in`, `if` and `import`. The result no longer parses. #4214 shows a
fenced Python script that comes back with `for` and `in` removed.

This PR stops that in the Python router and in the Rust live-zone
dispatcher. When the code compressor declines, the code is forwarded
unchanged.

Closes #4214

Credit: the Python change is oxura's work from
https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018
(277ca1f). Their commit is the first commit here, rebased onto `main`,
and oxura is its author. The test, docs and Rust commits were added on
top. oxura asked us to open this PR from our fork
([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)),
and said they have not run this branch, so the model and native results
below are ours, not theirs.

## Behaviour change to review

With `enable_code_aware=False` (the default), source code now stays
unchanged. Before this PR, it was sent to Kompress and lost keywords.
With `enable_code_aware=True`, code that the code compressor declines
also stays unchanged. Before, Kompress ran on it.
A code block the code compressor does compress is unaffected. Prose,
logs, JSON and tables are unaffected.

Side effect: a code block that Kompress used to shrink (and break) now
keeps its full size. This is the intended trade, but it means fewer
tokens saved on code when `enable_code_aware=False`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
- [ ] New feature
- [ ] Breaking change (the default-config behaviour change above is
intended, and it only stops damage to code)

## Changes Made

Five commits:

- `596764c3` `fix(router): keep source code out of prose fallbacks`.
Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in
`headroom/transforms/content_router.py`. This is 277ca1f rebased onto
`main`, and it differs from 277ca1f in two ways only: the two test-file
conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because
`AGENTS.md` says not to add content under `wiki/`.
- `7705a4e6` `test(router): repin the code-to-Kompress tests and add the
#4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by:
oxura` trailer. Five tests asserted the old fallback. They are rewritten
to assert the new behaviour (code unchanged, Kompress never sees it)
instead of being deleted. Adds a regression test for the script from
#4214, with `enable_code_aware` both off and on. Updates
`docs/content/docs/code-compression.mdx` and
`docs/content/docs/how-compression-works.mdx`, which said that detected
code goes to Kompress.
- `6631668a` `fix(live-zone): leave source code unchanged when the code
compressor declines`. Author: Tao Hansen. Same fix in
`crates/headroom-core/src/transforms/live_zone.rs`: a declined
`SourceCode` block is now a NoOp, not a Kompress input. Tests:
`tests/live_zone_source_code_decline.rs` (single test, needs no model),
and new cases in `tests/live_zone_dispatch.rs`.
- `d8776283` `fix(router): turn off the code compressor's internal
Kompress fallback`. Author: Tao Hansen. Follows review: the router's
code compressor was still built with
`fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown
languages, Perl and AST failures could build a `KompressCompressor`
inside `CodeAwareCompressor`. The router now always sets
`fallback_to_kompress=False`. The #4719 test that pinned `True`
(`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as
`test_enabled_kompress_still_turns_off_code_aware_fallback`, and the
unknown-language, Perl (by language hint, since `detect_language()`
reports Perl-dominant code as unknown) and AST-failure paths are tested
with spies on `is_kompress_available` and `KompressCompressor`, with
`enable_kompress` both off and on.
`docs/content/docs/code-compression.mdx` notes that the router always
turns the option off.
- `c1f8464e` `fix(router): keep passthrough code from external
compressors and stale caches`. Author: Tao Hansen. Follows review:
`_try_external_compressor` skips `PASSTHROUGH` blocks (code with
`prefer_code_aware_for_code=False`, whole or as a mixed section), which
would otherwise be offered to a prose-only external compressor as
`text/plain`. The disk cache namespace now also hashes
`_ROUTING_POLICY_REVISION`, so a same-version restart does not replay
code that Kompress damaged before this change. Tests: two
external-dispatch cases with a `text/plain` external, and one namespace
test. All three fail on the previous head.

I did not touch `CHANGELOG.md`.

## Testing

- [x] Unit tests pass (`pytest`): see below, the failures are
environment failures that also fail on `main`
- [x] Linting passes (`ruff check .`, `ruff format --check .`)
- [x] Type checking passes (`mypy
headroom/transforms/content_router.py`)
- [x] New tests added for new behaviour
- [x] Manual testing performed (real model replay, below)

### Test Output

macOS, Python 3.14. Head is now `c1f8464e`.

```text
After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean.
After the rebase onto main c05fbddd and the review fix (d8776283):
  router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped
  tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail
    (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure)
  ruff check / ruff format --check headroom tests: clean
  cargo test -p headroom-core: 1151 passed, 0 failed
Earlier run on 9df60c7c (before the rebase):
router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed:
  test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget,
  a timing test the branch does not touch; it passes 3 of 3 times when rerun alone
ruff check .:           All checks passed!
ruff format --check headroom tests: 1714 files already formatted
mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
cargo test -p headroom-core:   22 test binaries, 1151 passed, 0 failed
cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean
pytest tests (full):     31 failed, 19208 passed, 513 skipped
```

Rust bug-fix tests, fail before and pass after (the router-test commit's
source with the new Rust tests, then the live-zone commit; measured
before the rebase as `9df60c7c`, and the Rust code is unchanged since):

```text
before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on 7705a4e6)
        source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed)
after:  all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok
```

The dispatch fail-before was measured on the earlier, code-identical
branch (only documentation files differ). The dispatch test only fails
before the fix when the Kompress model is loaded (`HF_HOME` with the
model, `ORT_DYLIB_PATH` set). The single-test binary fails before the
fix on any host.

Python regression: with the new tests run against `origin/main`'s router
(no fix), 9 test cases fail (the #4214 fence test in both configs, the
repinned tests, and the fenced-code keyword test). With the fix they
pass.

### Failures in the full `pytest tests` run

The full run on the branch had 31 failures, none in router or
compression code. I compared them with pristine `origin/main`
(200c838c):

- 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path
is too long under this checkout directory (`OSError`). They fail the
same way in isolation on `main`.
- 17 other failures are in `test_proxy_health.py`,
`test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`,
`test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`,
`test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A
full run on `main` fails 29 tests, and the set differs from run to run
(the two sets overlap by 17 of 31). I re-ran the 14 failures that were
not in the `main` list in isolation on the branch: 12 pass, the other 2
are the UDS tests above.
- Known flaky test:
`tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`
failed once in a loaded run and passes on rerun. File repeated 3 times
on `596764c3` and on `origin/main`: 16 passed each time. It is timing
based and not caused by this change. It passed in the 558-test router
run above.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`,
real `kompress-v2-base` int8 ONNX model from an offline copy of the
public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the
venv's onnxruntime.
- Fence: a 1324-character valid Python script. The issue quotes only
four of its lines, so the rest is a reconstruction in the same style,
not the reporter's bytes.
- Exact command / steps: `ContentRouter.compress` on the fence followed
by about 40 lines of prose, for four configs (default;
`enable_code_aware=True`; `prefer_code_aware_for_code=False`;
`enable_code_aware=True` with lossless-then-lossy). Also
`headroom.compress` on a full Anthropic message list with the fence in a
`tool_result`.
- Observed result: Before (v0.44.0): in the default and code-aware
configs the fence comes back as 1144 characters and does not parse
(`import json, collections` becomes `json, collections`; `[s for s in
...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs
and in `headroom.compress`, the fence is byte-identical, 1324
characters, and parses. Prose control: the same router on prose still
compresses. A prose-only block went from 5709 to 5357 characters
(strategy TEXT). A prose / fence / prose block went from 12760 to 12056
characters (strategy MIXED), and the fence in it is byte-identical and
parses.
- Not tested: the reporter's exact 1324-character fence (only four lines
are public, so the tests use the reconstructed fence described above);
the Rust live-zone path end to end through the proxy with a real request
(covered by `cargo test` only); platforms other than macOS; the full
proxy on a live deployment.

## Other places where code can still reach a prose model

I checked every path from code to a prose or ML compressor, in Python
and Rust.

| Path | State |
| --- | --- |
| Python `CODE_AWARE` branch, code compressor declines or is disabled |
Fixed here: the block stays unchanged |
| Python post-strategy fallback after "no savings" (`content_router.py`,
`fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only
applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so
unchanged |
| Python lossless-then-lossy (opt-in) after the lossless fold |
`CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged |
| Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in)
| Prose-like output, not code. Unchanged |
| Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged |
| Python mixed content with fenced code (`_compress_mixed`) | Each fence
goes through the `CODE_AWARE` branch, so it is fixed by this change.
Covered by the new fence tests |
| Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before
this PR |
| Python `_experimental_compress_read`
(`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment
that runs Kompress on file reads, which can be code. Off by default.
Left as is |
| Python size gate and deadline fallbacks in `_try_ml_compressor` | Run
only after a caller chose Kompress, so code no longer reaches them |
| Python `CodeAwareCompressor` internal fallback
(`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST
failure) | Fixed in commit 4: the router always builds it with
`fallback_to_kompress=False`. Covered by Kompress-spy tests |
| Python external compressor dispatch for `PASSTHROUGH` code | Fixed in
commit 5: never offered to an external compressor |
| Disk result cache made before this change | Fixed in commit 5: a new
namespace through `_ROUTING_POLICY_REVISION` |
| Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) |
| Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It
is the single remaining call to `kompress_or_noop` |
| Rust: prose text with an embedded fenced code block | **Not
reproduced.** I built synthetic tool results of prose around the fenced
Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher
classified each one as `source_code`, so commit 3 leaves it unchanged,
and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has
no fence-aware splitting, but I found no input where that sends code to
Kompress, so this PR files no follow-up. |
| Python `config_compressor.py` docstring says YAML/TOML/INI "fall
through to the lossy prose path" | Documentation only: a stale comment,
not a code path. Config content has its own `CONFIG` strategy. Can be
refreshed in a later change |

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, in the default config code is no
longer sent to Kompress (see "Behaviour change to review"). Tokens saved
on code drop in return for code that stays valid.
- Kill switch / disable path: none added. Revert the PR to restore the
old fallback.
- Unsafe override required: no.
- Qualification impact: none expected. Compression of code by the code
compressor is unchanged.
- Rollback path: revert the three commits. No data, schema or config
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines (`ruff`, `cargo
fmt`, `clippy` clean)
- [x] I have made corresponding changes to the documentation
(`docs/content/docs/code-compression.mdx`,
`docs/content/docs/how-compression-works.mdx`)
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
- [ ] New and existing unit tests pass locally: the only failures are
environment failures that also fail on `main`, listed above

## Additional Notes

- Rust has no fence-aware splitting like the Python router has. A
synthetic prose-plus-fence repro did not reach Kompress (see the table
above), so no follow-up issue is filed.
- One stale comment remains in `content_router.py`
(`_try_ml_compressor`, the Phase 0 size-gate note lists
`CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not
changed here to keep the review commits as they are. Say if you want it
fixed in this PR.
- Reviewers: the Python change is easiest to review commit by commit.
Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and
the Rust fix.

---------

Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:25:06 -06:00

443 lines
17 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Tests for routing a selected EXTERNAL compressor through the content router.
Scope 3 of the pluggable-compressor work: an opt-in ``headroom.compressor``
entry point, when SELECTED, actually compresses matching real traffic through
:meth:`ContentRouter._apply_strategy_to_content`, with fail-open fallback to the
built-in dispatch. BACKWARD COMPATIBILITY is the hard requirement — with nothing
selected the branch is inert and the default request path is byte-identical.
Covers:
(a) a selected external compressor compresses a matching block end-to-end;
(b) its ``recoverable`` (hash -> original) map is retrievable from the CCR store;
(c) fail-open: an external that raises / malforms / expands falls back to the
built-in path (never breaks the request);
(d) NOT selected, or a non-matching content type, leaves the built-in path
untouched (the external is never even invoked);
plus the proxy seam that threads external names to the router.
"""
from __future__ import annotations
import hashlib
import pytest
from headroom.cache.compression_store import (
get_compression_store,
reset_compression_store,
)
from headroom.proxy.server import _external_compressor_selection
from headroom.transforms.compressor_registry import (
CompressInput,
CompressorDescriptor,
CompressOutput,
)
from headroom.transforms.content_router import (
ContentRouter,
ContentRouterConfig,
)
# A JSON array reliably routes to SMART_CRUSHER (content type application/json)
# and is not touched by the STAGE-0 lossless fold, so it reaches the external
# dispatch branch. Big enough that the reference compressor's output shrinks it.
_JSON_ARRAY = (
"["
+ ",".join(
f'{{"id":{i},"name":"item-{i}","status":"active","value":{i * 7},'
f'"note":"a fairly long descriptive field number {i} to add bulk"}}'
for i in range(40)
)
+ "]"
)
@pytest.fixture
def _memory_ccr(monkeypatch):
"""Isolated in-memory CCR store + offline content detection per test.
``HEADROOM_DETECT_BACKEND=python`` forces the pure-Python regex detector so
``compress()`` never touches the native Magika/ONNX detector (which needs a
model download and blocks in this offline environment). The external-dispatch
branch under test is independent of the detector backend.
"""
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "python")
reset_compression_store()
yield
reset_compression_store()
class _WordTokenizer:
"""Word-count tokenizer stub — no model, deterministic, offline-safe."""
def count_text(self, text: object) -> int:
return len(str(text).split())
def count_messages(self, messages: list[dict]) -> int:
return sum(self.count_text(m.get("content", "")) for m in messages)
def _json_array(tag: str) -> str:
"""A distinct, spaced JSON array (>50 word-tokens, >500 chars) that the
pure-Python detector routes to SMART_CRUSHER (application/json)."""
return (
"["
+ ",".join(
f'{{"id":{i},"tag":"{tag}","note":"long descriptive field number {i} '
f'to add real bulk here"}}'
for i in range(40)
)
+ "]"
)
def _tool_msg(call_id: str, content: str) -> dict:
# tool_call_id with no matching assistant tool_calls -> not excluded -> the
# non-frozen one reaches compression (matches Bash/shell output).
return {"role": "tool", "tool_call_id": call_id, "content": content}
class _RecordingExternal:
"""Reference in-process external ``Compressor`` for the router tests.
Deterministically shrinks its input to a short marker string and, unless
``recoverable=False``, returns a ``{hash: original}`` recovery map keyed by
the same hash it embeds in the output — mirroring how SmartCrusher's markers
point back into the CCR store.
"""
def __init__(
self,
name: str = "ext_json",
content_types: tuple[str, ...] = ("application/json",),
*,
lossless: bool = False,
recoverable: bool = True,
raises: bool = False,
expand: bool = False,
malformed: bool = False,
empty: bool = False,
bad_hash: bool = False,
) -> None:
self._name = name
self._content_types = list(content_types)
self._lossless = lossless
self._recoverable = recoverable
self._raises = raises
self._expand = expand
self._malformed = malformed
self._empty = empty
self._bad_hash = bad_hash
self.calls: list[CompressInput] = []
@property
def descriptor(self) -> CompressorDescriptor:
return CompressorDescriptor(
name=self._name,
content_types=self._content_types,
lossless=self._lossless,
cost_tier="fast",
recoverable=self._recoverable,
)
def compress(self, inp: CompressInput): # noqa: ANN201 - matches protocol
self.calls.append(inp)
if self._raises:
raise RuntimeError("external boom")
if self._malformed:
return {"not": "a CompressOutput"}
original = inp.content
digest = hashlib.sha256(original.encode()).hexdigest()[:24]
key = "zznothex" if self._bad_hash else digest
if self._empty:
content = ""
elif self._expand:
content = original + (" PADDING" * 200)
else:
content = f"[external-compressed <<ccr:{digest}>>]"
recoverable = {key: original} if self._recoverable else {}
return CompressOutput(
content=content,
tokens_before=len(original.split()),
tokens_after=len(content.split()),
lossless=self._lossless,
markers=[f"external:{self._name}"],
recoverable=recoverable,
warnings=["reference-warning"],
)
def _cfg(**kwargs) -> ContentRouterConfig:
"""Router config with Kompress OFF so the built-in fallback path never
loads the ModernBERT ML model (keeps these unit tests fast and offline).
The external-dispatch branch under test is independent of this flag."""
kwargs.setdefault("enable_kompress", False)
return ContentRouterConfig(**kwargs)
def _router_with_external(comp: _RecordingExternal, selection):
"""Build a router with ``comp`` registered and ``selection`` active."""
router = ContentRouter(_cfg(active_external_compressors=selection))
router.compressor_registry.register(comp, replace=True)
# Re-resolve now that the external compressor is registered (the router
# resolves the selection once at construction, before this injection).
router._active_external_compressors = router._resolve_active_external_compressors()
return router
# ─────────────────────────── (a) end-to-end ──────────────────────────────────
def test_selected_external_compresses_matching_block_end_to_end(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
router = _router_with_external(comp, ["ext_json"])
result = router.compress(_JSON_ARRAY)
assert comp.calls, "external compressor should have been invoked"
assert "external:ext_json" in result.strategy_chain
assert "external-compressed" in result.compressed
assert len(result.compressed) < len(_JSON_ARRAY)
# The CompressInput carried the block + its detected MIME content type.
assert comp.calls[0].content == _JSON_ARRAY
assert comp.calls[0].content_type == "application/json"
@pytest.mark.parametrize("mime", ["text/csv", "*/*"])
def test_selected_external_preserves_untracked_git_status_bytes(_memory_ccr, mime):
status = (
"On branch feature/status\n\nUntracked files:\n"
' (use "git add <file>..." to include in what will be committed)\n'
+ "".join(f"\tuntracked path {i:03d} α.txt\n" for i in range(120))
)
compressor = _RecordingExternal(name="ext_status", content_types=(mime,))
router = _router_with_external(compressor, ["ext_status"])
direct = router.compress(status)
messages = [
{"role": "user", "content": "List every untracked path."},
_tool_msg("status", status),
{"role": "user", "content": "Keep the exact path names."},
]
tool_result = router.apply(messages, _WordTokenizer(), model_limit=100_000)
assert direct.compressed == status
assert tool_result.messages[1]["content"] == status
def test_native_misclassification_keeps_git_status_format(_memory_ccr, monkeypatch):
from types import SimpleNamespace
import headroom._core
import headroom._ort
from headroom.transforms.content_detector import ContentType
from headroom.transforms.content_router import _detect_content
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
monkeypatch.setattr("headroom.transforms.content_router._detect_native_unhealthy", False)
monkeypatch.setattr(headroom._ort, "rust_ort_runtime_compatible", lambda: True)
monkeypatch.setattr(
headroom._core,
"detect_content_type",
lambda _content: SimpleNamespace(content_type="html", confidence=1.0),
)
status = "On branch feature/status\n\nUntracked files:\n\tnew path α.txt\n"
detected = _detect_content(status)
assert detected.content_type is ContentType.TABULAR
assert detected.metadata["format"] == "git_status"
# ─────────────────────────── (b) recoverable map ─────────────────────────────
def test_recoverable_map_is_retrievable(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
router = _router_with_external(comp, ["ext_json"])
router.compress(_JSON_ARRAY)
digest = hashlib.sha256(_JSON_ARRAY.encode()).hexdigest()[:24]
entry = get_compression_store().retrieve(digest)
assert entry is not None, "recoverable entry should be in the CCR store"
assert entry.original_content == _JSON_ARRAY
assert entry.compression_strategy == "external:ext_json"
def test_non_hex_recoverable_hash_falls_back_to_builtin(_memory_ccr):
# A malformed (non-hex) recovery hash must not break the request; the store
# cannot keep that entry, so the external output (whose marker would point
# nowhere) is discarded and the built-in path runs.
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), bad_hash=True)
router = _router_with_external(comp, ["ext_json"])
result = router.compress(_JSON_ARRAY)
assert "external:ext_json" not in result.strategy_chain
assert "external-compressed" not in result.compressed
assert get_compression_store().retrieve("zznothex") is None
# ─────────────────────────── (c) fail-open ───────────────────────────────────
def test_external_raise_falls_back_to_builtin(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), raises=True)
router = _router_with_external(comp, ["ext_json"])
result = router.compress(_JSON_ARRAY)
assert comp.calls, "external should have been attempted"
# Fell back: no external marker in the chain, request not broken.
assert "external:ext_json" not in result.strategy_chain
assert result.compressed and result.compressed.strip()
def test_external_malformed_output_falls_back_to_builtin(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), malformed=True)
router = _router_with_external(comp, ["ext_json"])
result = router.compress(_JSON_ARRAY)
assert comp.calls
assert "external:ext_json" not in result.strategy_chain
def test_external_expansion_falls_back_to_builtin(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), expand=True)
router = _router_with_external(comp, ["ext_json"])
result = router.compress(_JSON_ARRAY)
assert comp.calls
assert "external:ext_json" not in result.strategy_chain
# Never expands: the returned block is no larger than the input.
assert len(result.compressed) <= len(_JSON_ARRAY)
def test_external_empty_output_falls_back_to_builtin(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",), empty=True)
router = _router_with_external(comp, ["ext_json"])
result = router.compress(_JSON_ARRAY)
assert comp.calls
assert "external:ext_json" not in result.strategy_chain
assert result.compressed.strip(), "non-empty input must never blank out"
# ─────────────── (d) not selected / non-matching → built-in ──────────────────
def test_not_selected_leaves_builtin_path_unchanged(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
# Registered but NOT selected.
router = _router_with_external(comp, None)
baseline = ContentRouter(_cfg()).compress(_JSON_ARRAY)
result = router.compress(_JSON_ARRAY)
assert comp.calls == [], "unselected external must never be invoked"
assert "external:ext_json" not in result.strategy_chain
# Byte-identical to a plain router with no external registered at all.
assert result.compressed == baseline.compressed
assert result.strategy_chain == baseline.strategy_chain
def test_non_matching_content_type_leaves_builtin_path_unchanged(_memory_ccr):
# Selected, but declares a content type the JSON block never has.
comp = _RecordingExternal(name="ext_diff", content_types=("text/x-diff",))
router = _router_with_external(comp, ["ext_diff"])
baseline = ContentRouter(_cfg()).compress(_JSON_ARRAY)
result = router.compress(_JSON_ARRAY)
assert comp.calls == [], "non-matching external must never be invoked"
assert "external:ext_diff" not in result.strategy_chain
assert result.compressed == baseline.compressed
def test_default_config_has_no_external_selection():
assert ContentRouterConfig().active_external_compressors is None
router = ContentRouter(ContentRouterConfig())
assert router._active_external_compressors == []
# ─────────────────────────── proxy seam ──────────────────────────────────────
@pytest.mark.parametrize(
"selection,expected",
[
(None, None),
(set(), None),
({"", " "}, None),
({"smart_crusher"}, None), # built-in only → no external
({"smart_crusher", "kompress"}, None),
({"my_ext"}, ["my_ext"]),
({"kompress", "my_ext"}, ["my_ext"]),
({"b_ext", "a_ext"}, ["a_ext", "b_ext"]), # sorted
({"*"}, ["*"]),
({"*", "my_ext"}, ["*"]), # wildcard wins
],
)
def test_external_compressor_selection_helper(selection, expected):
assert _external_compressor_selection(selection) == expected
def test_wildcard_selection_activates_registered_external(_memory_ccr):
comp = _RecordingExternal(name="ext_json", content_types=("application/json",))
router = _router_with_external(comp, ["*"])
result = router.compress(_JSON_ARRAY)
assert comp.calls
assert "external:ext_json" in result.strategy_chain
# ───────────── explicit passthrough code never reaches an external ─────────────
_PY_FENCE = "\n".join(
["import json", ""]
+ [
f"def handler_{i}(event):\n if event.get('k{i}'):\n return json.dumps(event)\n return None\n"
for i in range(30)
]
)
_PROSE = " ".join(
["The nightly job failed after the deploy and the operator wants a short summary."] * 40
)
def test_passthrough_code_section_is_not_offered_to_a_text_external(_memory_ccr):
# With prefer_code_aware_for_code=False the router keeps source code
# unmangled (PASSTHROUGH). A selected prose-only external compressor must
# not receive that code: mixed content splits the fence into its own section.
comp = _RecordingExternal(name="ext_text", content_types=("text/plain",))
router = ContentRouter(
_cfg(active_external_compressors=["ext_text"], prefer_code_aware_for_code=False)
)
router.compressor_registry.register(comp, replace=True)
router._active_external_compressors = router._resolve_active_external_compressors()
content = f"{_PROSE}\n\n```python\n{_PY_FENCE}\n```\n\n{_PROSE}"
result = router.compress(content)
assert all("def handler_0" not in call.content for call in comp.calls)
assert f"```python\n{_PY_FENCE}\n```" in result.compressed
def test_passthrough_whole_code_block_is_not_offered_to_a_text_external(_memory_ccr):
comp = _RecordingExternal(name="ext_text", content_types=("text/plain",))
router = ContentRouter(
_cfg(active_external_compressors=["ext_text"], prefer_code_aware_for_code=False)
)
router.compressor_registry.register(comp, replace=True)
router._active_external_compressors = router._resolve_active_external_compressors()
result = router.compress(_PY_FENCE)
assert not comp.calls
assert result.compressed == _PY_FENCE