Files
headroom/tests/test_code_aware_kompress_fallback.py
3aafdf2dcc fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description

When the code compressor declined a block of source code, the router
passed the block to Kompress. Kompress is a prose model. It drops
low-salience words, and in code those words are keywords such as `for`,
`in`, `if` and `import`. The result no longer parses. #4214 shows a
fenced Python script that comes back with `for` and `in` removed.

This PR stops that in the Python router and in the Rust live-zone
dispatcher. When the code compressor declines, the code is forwarded
unchanged.

Closes #4214

Credit: the Python change is oxura's work from
https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018
(277ca1f). Their commit is the first commit here, rebased onto `main`,
and oxura is its author. The test, docs and Rust commits were added on
top. oxura asked us to open this PR from our fork
([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)),
and said they have not run this branch, so the model and native results
below are ours, not theirs.

## Behaviour change to review

With `enable_code_aware=False` (the default), source code now stays
unchanged. Before this PR, it was sent to Kompress and lost keywords.
With `enable_code_aware=True`, code that the code compressor declines
also stays unchanged. Before, Kompress ran on it.
A code block the code compressor does compress is unaffected. Prose,
logs, JSON and tables are unaffected.

Side effect: a code block that Kompress used to shrink (and break) now
keeps its full size. This is the intended trade, but it means fewer
tokens saved on code when `enable_code_aware=False`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
- [ ] New feature
- [ ] Breaking change (the default-config behaviour change above is
intended, and it only stops damage to code)

## Changes Made

Five commits:

- `596764c3` `fix(router): keep source code out of prose fallbacks`.
Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in
`headroom/transforms/content_router.py`. This is 277ca1f rebased onto
`main`, and it differs from 277ca1f in two ways only: the two test-file
conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because
`AGENTS.md` says not to add content under `wiki/`.
- `7705a4e6` `test(router): repin the code-to-Kompress tests and add the
#4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by:
oxura` trailer. Five tests asserted the old fallback. They are rewritten
to assert the new behaviour (code unchanged, Kompress never sees it)
instead of being deleted. Adds a regression test for the script from
#4214, with `enable_code_aware` both off and on. Updates
`docs/content/docs/code-compression.mdx` and
`docs/content/docs/how-compression-works.mdx`, which said that detected
code goes to Kompress.
- `6631668a` `fix(live-zone): leave source code unchanged when the code
compressor declines`. Author: Tao Hansen. Same fix in
`crates/headroom-core/src/transforms/live_zone.rs`: a declined
`SourceCode` block is now a NoOp, not a Kompress input. Tests:
`tests/live_zone_source_code_decline.rs` (single test, needs no model),
and new cases in `tests/live_zone_dispatch.rs`.
- `d8776283` `fix(router): turn off the code compressor's internal
Kompress fallback`. Author: Tao Hansen. Follows review: the router's
code compressor was still built with
`fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown
languages, Perl and AST failures could build a `KompressCompressor`
inside `CodeAwareCompressor`. The router now always sets
`fallback_to_kompress=False`. The #4719 test that pinned `True`
(`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as
`test_enabled_kompress_still_turns_off_code_aware_fallback`, and the
unknown-language, Perl (by language hint, since `detect_language()`
reports Perl-dominant code as unknown) and AST-failure paths are tested
with spies on `is_kompress_available` and `KompressCompressor`, with
`enable_kompress` both off and on.
`docs/content/docs/code-compression.mdx` notes that the router always
turns the option off.
- `c1f8464e` `fix(router): keep passthrough code from external
compressors and stale caches`. Author: Tao Hansen. Follows review:
`_try_external_compressor` skips `PASSTHROUGH` blocks (code with
`prefer_code_aware_for_code=False`, whole or as a mixed section), which
would otherwise be offered to a prose-only external compressor as
`text/plain`. The disk cache namespace now also hashes
`_ROUTING_POLICY_REVISION`, so a same-version restart does not replay
code that Kompress damaged before this change. Tests: two
external-dispatch cases with a `text/plain` external, and one namespace
test. All three fail on the previous head.

I did not touch `CHANGELOG.md`.

## Testing

- [x] Unit tests pass (`pytest`): see below, the failures are
environment failures that also fail on `main`
- [x] Linting passes (`ruff check .`, `ruff format --check .`)
- [x] Type checking passes (`mypy
headroom/transforms/content_router.py`)
- [x] New tests added for new behaviour
- [x] Manual testing performed (real model replay, below)

### Test Output

macOS, Python 3.14. Head is now `c1f8464e`.

```text
After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean.
After the rebase onto main c05fbddd and the review fix (d8776283):
  router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped
  tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail
    (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure)
  ruff check / ruff format --check headroom tests: clean
  cargo test -p headroom-core: 1151 passed, 0 failed
Earlier run on 9df60c7c (before the rebase):
router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed:
  test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget,
  a timing test the branch does not touch; it passes 3 of 3 times when rerun alone
ruff check .:           All checks passed!
ruff format --check headroom tests: 1714 files already formatted
mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
cargo test -p headroom-core:   22 test binaries, 1151 passed, 0 failed
cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean
pytest tests (full):     31 failed, 19208 passed, 513 skipped
```

Rust bug-fix tests, fail before and pass after (the router-test commit's
source with the new Rust tests, then the live-zone commit; measured
before the rebase as `9df60c7c`, and the Rust code is unchanged since):

```text
before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on 7705a4e6)
        source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed)
after:  all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok
```

The dispatch fail-before was measured on the earlier, code-identical
branch (only documentation files differ). The dispatch test only fails
before the fix when the Kompress model is loaded (`HF_HOME` with the
model, `ORT_DYLIB_PATH` set). The single-test binary fails before the
fix on any host.

Python regression: with the new tests run against `origin/main`'s router
(no fix), 9 test cases fail (the #4214 fence test in both configs, the
repinned tests, and the fenced-code keyword test). With the fix they
pass.

### Failures in the full `pytest tests` run

The full run on the branch had 31 failures, none in router or
compression code. I compared them with pristine `origin/main`
(200c838c):

- 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path
is too long under this checkout directory (`OSError`). They fail the
same way in isolation on `main`.
- 17 other failures are in `test_proxy_health.py`,
`test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`,
`test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`,
`test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A
full run on `main` fails 29 tests, and the set differs from run to run
(the two sets overlap by 17 of 31). I re-ran the 14 failures that were
not in the `main` list in isolation on the branch: 12 pass, the other 2
are the UDS tests above.
- Known flaky test:
`tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`
failed once in a loaded run and passes on rerun. File repeated 3 times
on `596764c3` and on `origin/main`: 16 passed each time. It is timing
based and not caused by this change. It passed in the 558-test router
run above.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`,
real `kompress-v2-base` int8 ONNX model from an offline copy of the
public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the
venv's onnxruntime.
- Fence: a 1324-character valid Python script. The issue quotes only
four of its lines, so the rest is a reconstruction in the same style,
not the reporter's bytes.
- Exact command / steps: `ContentRouter.compress` on the fence followed
by about 40 lines of prose, for four configs (default;
`enable_code_aware=True`; `prefer_code_aware_for_code=False`;
`enable_code_aware=True` with lossless-then-lossy). Also
`headroom.compress` on a full Anthropic message list with the fence in a
`tool_result`.
- Observed result: Before (v0.44.0): in the default and code-aware
configs the fence comes back as 1144 characters and does not parse
(`import json, collections` becomes `json, collections`; `[s for s in
...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs
and in `headroom.compress`, the fence is byte-identical, 1324
characters, and parses. Prose control: the same router on prose still
compresses. A prose-only block went from 5709 to 5357 characters
(strategy TEXT). A prose / fence / prose block went from 12760 to 12056
characters (strategy MIXED), and the fence in it is byte-identical and
parses.
- Not tested: the reporter's exact 1324-character fence (only four lines
are public, so the tests use the reconstructed fence described above);
the Rust live-zone path end to end through the proxy with a real request
(covered by `cargo test` only); platforms other than macOS; the full
proxy on a live deployment.

## Other places where code can still reach a prose model

I checked every path from code to a prose or ML compressor, in Python
and Rust.

| Path | State |
| --- | --- |
| Python `CODE_AWARE` branch, code compressor declines or is disabled |
Fixed here: the block stays unchanged |
| Python post-strategy fallback after "no savings" (`content_router.py`,
`fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only
applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so
unchanged |
| Python lossless-then-lossy (opt-in) after the lossless fold |
`CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged |
| Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in)
| Prose-like output, not code. Unchanged |
| Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged |
| Python mixed content with fenced code (`_compress_mixed`) | Each fence
goes through the `CODE_AWARE` branch, so it is fixed by this change.
Covered by the new fence tests |
| Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before
this PR |
| Python `_experimental_compress_read`
(`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment
that runs Kompress on file reads, which can be code. Off by default.
Left as is |
| Python size gate and deadline fallbacks in `_try_ml_compressor` | Run
only after a caller chose Kompress, so code no longer reaches them |
| Python `CodeAwareCompressor` internal fallback
(`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST
failure) | Fixed in commit 4: the router always builds it with
`fallback_to_kompress=False`. Covered by Kompress-spy tests |
| Python external compressor dispatch for `PASSTHROUGH` code | Fixed in
commit 5: never offered to an external compressor |
| Disk result cache made before this change | Fixed in commit 5: a new
namespace through `_ROUTING_POLICY_REVISION` |
| Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) |
| Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It
is the single remaining call to `kompress_or_noop` |
| Rust: prose text with an embedded fenced code block | **Not
reproduced.** I built synthetic tool results of prose around the fenced
Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher
classified each one as `source_code`, so commit 3 leaves it unchanged,
and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has
no fence-aware splitting, but I found no input where that sends code to
Kompress, so this PR files no follow-up. |
| Python `config_compressor.py` docstring says YAML/TOML/INI "fall
through to the lossy prose path" | Documentation only: a stale comment,
not a code path. Config content has its own `CONFIG` strategy. Can be
refreshed in a later change |

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, in the default config code is no
longer sent to Kompress (see "Behaviour change to review"). Tokens saved
on code drop in return for code that stays valid.
- Kill switch / disable path: none added. Revert the PR to restore the
old fallback.
- Unsafe override required: no.
- Qualification impact: none expected. Compression of code by the code
compressor is unchanged.
- Rollback path: revert the three commits. No data, schema or config
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines (`ruff`, `cargo
fmt`, `clippy` clean)
- [x] I have made corresponding changes to the documentation
(`docs/content/docs/code-compression.mdx`,
`docs/content/docs/how-compression-works.mdx`)
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
- [ ] New and existing unit tests pass locally: the only failures are
environment failures that also fail on `main`, listed above

## Additional Notes

- Rust has no fence-aware splitting like the Python router has. A
synthetic prose-plus-fence repro did not reach Kompress (see the table
above), so no follow-up issue is filed.
- One stale comment remains in `content_router.py`
(`_try_ml_compressor`, the Phase 0 size-gate note lists
`CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not
changed here to keep the review commits as they are. Say if you want it
fixed in this PR.
- Reviewers: the Python change is easiest to review commit by commit.
Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and
the Rust fix.

---------

Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:25:06 -06:00

166 lines
5.9 KiB
Python

"""The code_aware Kompress fallback honours ``enable_kompress=False`` (#4719).
``CodeAwareCompressor`` falls back to Kompress for code it cannot parse (unknown
language, Perl, no tree-sitter, AST failure). That fallback instantiates its own
``KompressCompressor``, so ``ContentRouter._get_kompress()`` returning ``None``
was not enough: the router must also switch the code compressor's fallback off.
"""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
import headroom.transforms.code_compressor as cc
import headroom.transforms.kompress_compressor as kc
from headroom.transforms.code_compressor import CodeLanguage, detect_language
from headroom.transforms.compressor_registry import CompressInput
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
# Pseudo-assembly that detect_language() cannot identify (the issue's repro).
UNKNOWN_CODE = "\n".join(
f"step {i}: load register r{i % 8} <- mem[{i * 4}]; jump-if-zero r{i % 8} label_{i}"
for i in range(80)
)
@pytest.fixture(autouse=True)
def _tree_sitter_present(monkeypatch: pytest.MonkeyPatch) -> None:
# The router only builds the code compressor when tree-sitter parses; the
# unknown-language fallback never reaches the parser, so stub the probe to
# exercise the router path in environments without the ``code`` extra.
monkeypatch.setattr(cc, "_check_tree_sitter_available", lambda: True)
@pytest.fixture()
def kompress_spies(monkeypatch: pytest.MonkeyPatch) -> tuple[MagicMock, MagicMock]:
available = MagicMock(return_value=True)
compressor = MagicMock()
compressor.return_value.compress.return_value = SimpleNamespace(
compressed="kompressed",
original_tokens=100,
compressed_tokens=1,
compression_ratio=0.01,
)
monkeypatch.setattr(kc, "is_kompress_available", available)
monkeypatch.setattr(kc, "KompressCompressor", compressor)
return available, compressor
def test_disabled_kompress_turns_off_code_aware_fallback() -> None:
router = ContentRouter(ContentRouterConfig(enable_code_aware=True, enable_kompress=False))
compressor = router._get_code_compressor()
assert compressor is not None
assert compressor.config.fallback_to_kompress is False
def test_enabled_kompress_still_turns_off_code_aware_fallback() -> None:
# Kompress may be on for prose, but it must never compress code (#4214):
# the router's code compressor leaves blocks it cannot handle unchanged.
router = ContentRouter(ContentRouterConfig(enable_code_aware=True))
compressor = router._get_code_compressor()
assert compressor is not None
assert compressor.config.fallback_to_kompress is False
PERL_CODE = "\n".join(
["#!/usr/bin/perl", "use strict;", "use warnings;"]
+ [
f'my $v{i} = $ARGV[{i % 3}] // {i}; print "$v{i}\\n" if $v{i} =~ /^\\d+$/;'
for i in range(60)
]
)
PYTHON_CODE = "\n".join(
["import json", ""]
+ [
f"def handler_{i}(event):\n if event.get('k{i}'):\n return json.dumps(event)\n return None\n"
for i in range(40)
]
)
def _code_aware_entry(enable_kompress: bool):
router = ContentRouter(
ContentRouterConfig(enable_code_aware=True, enable_kompress=enable_kompress)
)
entry = router.compressor_registry.get("code_aware")
assert entry is not None
return router, entry
@pytest.mark.parametrize("enable_kompress", [False, True])
def test_unknown_language_code_passes_through_without_kompress(
kompress_spies: tuple[MagicMock, MagicMock], enable_kompress: bool
) -> None:
assert detect_language(UNKNOWN_CODE)[0] == CodeLanguage.UNKNOWN
_, entry = _code_aware_entry(enable_kompress)
out = entry.compress(CompressInput(content=UNKNOWN_CODE, content_type="text/x-code"))
available, compressor = kompress_spies
available.assert_not_called()
compressor.assert_not_called()
assert out.content == UNKNOWN_CODE
@pytest.mark.parametrize("enable_kompress", [False, True])
def test_perl_code_passes_through_without_kompress(
kompress_spies: tuple[MagicMock, MagicMock], enable_kompress: bool
) -> None:
# detect_language() reports Perl-dominant code as UNKNOWN on purpose, so the
# Perl branch is reached through an explicit language hint.
_, entry = _code_aware_entry(enable_kompress)
out = entry.compress(
CompressInput(content=PERL_CODE, content_type="text/x-code", config={"language": "perl"})
)
available, compressor = kompress_spies
available.assert_not_called()
compressor.assert_not_called()
assert out.content == PERL_CODE
@pytest.mark.parametrize("enable_kompress", [False, True])
def test_ast_failure_passes_through_without_kompress(
kompress_spies: tuple[MagicMock, MagicMock],
monkeypatch: pytest.MonkeyPatch,
enable_kompress: bool,
) -> None:
router, entry = _code_aware_entry(enable_kompress)
code_compressor = router._get_code_compressor()
assert code_compressor is not None
def _boom(*_args, **_kwargs):
raise RuntimeError("simulated AST failure")
monkeypatch.setattr(code_compressor, "_compress_with_ast", _boom)
out = entry.compress(CompressInput(content=PYTHON_CODE, content_type="text/x-code"))
available, compressor = kompress_spies
available.assert_not_called()
compressor.assert_not_called()
assert out.content == PYTHON_CODE
def test_disk_cache_namespace_includes_the_routing_policy_revision(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# Output cached before #4214 (code sent to Kompress) must not be replayed
# after the fix by a restart with the same version and config.
import headroom.transforms.content_router as cr
config = ContentRouterConfig()
current = cr._config_namespace(config)
monkeypatch.setattr(cr, "_ROUTING_POLICY_REVISION", "earlier-policy")
assert cr._config_namespace(config) != current