mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
## Description When the code compressor declined a block of source code, the router passed the block to Kompress. Kompress is a prose model. It drops low-salience words, and in code those words are keywords such as `for`, `in`, `if` and `import`. The result no longer parses. #4214 shows a fenced Python script that comes back with `for` and `in` removed. This PR stops that in the Python router and in the Rust live-zone dispatcher. When the code compressor declines, the code is forwarded unchanged. Closes #4214 Credit: the Python change is oxura's work from https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018 (277ca1f). Their commit is the first commit here, rebased onto `main`, and oxura is its author. The test, docs and Rust commits were added on top. oxura asked us to open this PR from our fork ([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)), and said they have not run this branch, so the model and native results below are ours, not theirs. ## Behaviour change to review With `enable_code_aware=False` (the default), source code now stays unchanged. Before this PR, it was sent to Kompress and lost keywords. With `enable_code_aware=True`, code that the code compressor declines also stays unchanged. Before, Kompress ran on it. A code block the code compressor does compress is unaffected. Prose, logs, JSON and tables are unaffected. Side effect: a code block that Kompress used to shrink (and break) now keeps its full size. This is the intended trade, but it means fewer tokens saved on code when `enable_code_aware=False`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update - [ ] New feature - [ ] Breaking change (the default-config behaviour change above is intended, and it only stops damage to code) ## Changes Made Five commits: - `596764c3` `fix(router): keep source code out of prose fallbacks`. Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in `headroom/transforms/content_router.py`. This is 277ca1f rebased onto `main`, and it differs from 277ca1f in two ways only: the two test-file conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because `AGENTS.md` says not to add content under `wiki/`. - `7705a4e6` `test(router): repin the code-to-Kompress tests and add the #4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by: oxura` trailer. Five tests asserted the old fallback. They are rewritten to assert the new behaviour (code unchanged, Kompress never sees it) instead of being deleted. Adds a regression test for the script from #4214, with `enable_code_aware` both off and on. Updates `docs/content/docs/code-compression.mdx` and `docs/content/docs/how-compression-works.mdx`, which said that detected code goes to Kompress. - `6631668a` `fix(live-zone): leave source code unchanged when the code compressor declines`. Author: Tao Hansen. Same fix in `crates/headroom-core/src/transforms/live_zone.rs`: a declined `SourceCode` block is now a NoOp, not a Kompress input. Tests: `tests/live_zone_source_code_decline.rs` (single test, needs no model), and new cases in `tests/live_zone_dispatch.rs`. - `d8776283` `fix(router): turn off the code compressor's internal Kompress fallback`. Author: Tao Hansen. Follows review: the router's code compressor was still built with `fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown languages, Perl and AST failures could build a `KompressCompressor` inside `CodeAwareCompressor`. The router now always sets `fallback_to_kompress=False`. The #4719 test that pinned `True` (`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as `test_enabled_kompress_still_turns_off_code_aware_fallback`, and the unknown-language, Perl (by language hint, since `detect_language()` reports Perl-dominant code as unknown) and AST-failure paths are tested with spies on `is_kompress_available` and `KompressCompressor`, with `enable_kompress` both off and on. `docs/content/docs/code-compression.mdx` notes that the router always turns the option off. - `c1f8464e` `fix(router): keep passthrough code from external compressors and stale caches`. Author: Tao Hansen. Follows review: `_try_external_compressor` skips `PASSTHROUGH` blocks (code with `prefer_code_aware_for_code=False`, whole or as a mixed section), which would otherwise be offered to a prose-only external compressor as `text/plain`. The disk cache namespace now also hashes `_ROUTING_POLICY_REVISION`, so a same-version restart does not replay code that Kompress damaged before this change. Tests: two external-dispatch cases with a `text/plain` external, and one namespace test. All three fail on the previous head. I did not touch `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`): see below, the failures are environment failures that also fail on `main` - [x] Linting passes (`ruff check .`, `ruff format --check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py`) - [x] New tests added for new behaviour - [x] Manual testing performed (real model replay, below) ### Test Output macOS, Python 3.14. Head is now `c1f8464e`. ```text After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean. After the rebase onto mainc05fbdddand the review fix (d8776283): router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure) ruff check / ruff format --check headroom tests: clean cargo test -p headroom-core: 1151 passed, 0 failed Earlier run on9df60c7c(before the rebase): router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed: test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget, a timing test the branch does not touch; it passes 3 of 3 times when rerun alone ruff check .: All checks passed! ruff format --check headroom tests: 1714 files already formatted mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file cargo test -p headroom-core: 22 test binaries, 1151 passed, 0 failed cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean pytest tests (full): 31 failed, 19208 passed, 513 skipped ``` Rust bug-fix tests, fail before and pass after (the router-test commit's source with the new Rust tests, then the live-zone commit; measured before the rebase as `9df60c7c`, and the Rust code is unchanged since): ```text before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on7705a4e6) source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed) after: all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok ``` The dispatch fail-before was measured on the earlier, code-identical branch (only documentation files differ). The dispatch test only fails before the fix when the Kompress model is loaded (`HF_HOME` with the model, `ORT_DYLIB_PATH` set). The single-test binary fails before the fix on any host. Python regression: with the new tests run against `origin/main`'s router (no fix), 9 test cases fail (the #4214 fence test in both configs, the repinned tests, and the fenced-code keyword test). With the fix they pass. ### Failures in the full `pytest tests` run The full run on the branch had 31 failures, none in router or compression code. I compared them with pristine `origin/main` (200c838c): - 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path is too long under this checkout directory (`OSError`). They fail the same way in isolation on `main`. - 17 other failures are in `test_proxy_health.py`, `test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`, `test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`, `test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A full run on `main` fails 29 tests, and the set differs from run to run (the two sets overlap by 17 of 31). I re-ran the 14 failures that were not in the `main` list in isolation on the branch: 12 pass, the other 2 are the UDS tests above. - Known flaky test: `tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads` failed once in a loaded run and passes on rerun. File repeated 3 times on `596764c3` and on `origin/main`: 16 passed each time. It is timing based and not caused by this change. It passed in the 558-test router run above. ## Real Behavior Proof - Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`, real `kompress-v2-base` int8 ONNX model from an offline copy of the public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the venv's onnxruntime. - Fence: a 1324-character valid Python script. The issue quotes only four of its lines, so the rest is a reconstruction in the same style, not the reporter's bytes. - Exact command / steps: `ContentRouter.compress` on the fence followed by about 40 lines of prose, for four configs (default; `enable_code_aware=True`; `prefer_code_aware_for_code=False`; `enable_code_aware=True` with lossless-then-lossy). Also `headroom.compress` on a full Anthropic message list with the fence in a `tool_result`. - Observed result: Before (v0.44.0): in the default and code-aware configs the fence comes back as 1144 characters and does not parse (`import json, collections` becomes `json, collections`; `[s for s in ...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs and in `headroom.compress`, the fence is byte-identical, 1324 characters, and parses. Prose control: the same router on prose still compresses. A prose-only block went from 5709 to 5357 characters (strategy TEXT). A prose / fence / prose block went from 12760 to 12056 characters (strategy MIXED), and the fence in it is byte-identical and parses. - Not tested: the reporter's exact 1324-character fence (only four lines are public, so the tests use the reconstructed fence described above); the Rust live-zone path end to end through the proxy with a real request (covered by `cargo test` only); platforms other than macOS; the full proxy on a live deployment. ## Other places where code can still reach a prose model I checked every path from code to a prose or ML compressor, in Python and Rust. | Path | State | | --- | --- | | Python `CODE_AWARE` branch, code compressor declines or is disabled | Fixed here: the block stays unchanged | | Python post-strategy fallback after "no savings" (`content_router.py`, `fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so unchanged | | Python lossless-then-lossy (opt-in) after the lossless fold | `CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged | | Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in) | Prose-like output, not code. Unchanged | | Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged | | Python mixed content with fenced code (`_compress_mixed`) | Each fence goes through the `CODE_AWARE` branch, so it is fixed by this change. Covered by the new fence tests | | Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before this PR | | Python `_experimental_compress_read` (`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment that runs Kompress on file reads, which can be code. Off by default. Left as is | | Python size gate and deadline fallbacks in `_try_ml_compressor` | Run only after a caller chose Kompress, so code no longer reaches them | | Python `CodeAwareCompressor` internal fallback (`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST failure) | Fixed in commit 4: the router always builds it with `fallback_to_kompress=False`. Covered by Kompress-spy tests | | Python external compressor dispatch for `PASSTHROUGH` code | Fixed in commit 5: never offered to an external compressor | | Disk result cache made before this change | Fixed in commit 5: a new namespace through `_ROUTING_POLICY_REVISION` | | Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) | | Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It is the single remaining call to `kompress_or_noop` | | Rust: prose text with an embedded fenced code block | **Not reproduced.** I built synthetic tool results of prose around the fenced Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher classified each one as `source_code`, so commit 3 leaves it unchanged, and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has no fence-aware splitting, but I found no input where that sends code to Kompress, so this PR files no follow-up. | | Python `config_compressor.py` docstring says YAML/TOML/INI "fall through to the lossy prose path" | Documentation only: a stale comment, not a code path. Config content has its own `CONFIG` strategy. Can be refreshed in a later change | ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, in the default config code is no longer sent to Kompress (see "Behaviour change to review"). Tokens saved on code drop in return for code that stays valid. - Kill switch / disable path: none added. Revert the PR to restore the old fallback. - Unsafe override required: no. - Qualification impact: none expected. Compression of code by the code compressor is unchanged. - Rollback path: revert the three commits. No data, schema or config migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (`ruff`, `cargo fmt`, `clippy` clean) - [x] I have made corresponding changes to the documentation (`docs/content/docs/code-compression.mdx`, `docs/content/docs/how-compression-works.mdx`) - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` - [ ] New and existing unit tests pass locally: the only failures are environment failures that also fail on `main`, listed above ## Additional Notes - Rust has no fence-aware splitting like the Python router has. A synthetic prose-plus-fence repro did not reach Kompress (see the table above), so no follow-up issue is filed. - One stale comment remains in `content_router.py` (`_try_ml_compressor`, the Phase 0 size-gate note lists `CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not changed here to keep the review commits as they are. Say if you want it fixed in this PR. - Reviewers: the Python change is easiest to review commit by commit. Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and the Rust fix. --------- Co-authored-by: oxura <grozafroza44@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
166 lines
5.9 KiB
Python
166 lines
5.9 KiB
Python
"""The code_aware Kompress fallback honours ``enable_kompress=False`` (#4719).
|
|
|
|
``CodeAwareCompressor`` falls back to Kompress for code it cannot parse (unknown
|
|
language, Perl, no tree-sitter, AST failure). That fallback instantiates its own
|
|
``KompressCompressor``, so ``ContentRouter._get_kompress()`` returning ``None``
|
|
was not enough: the router must also switch the code compressor's fallback off.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
import headroom.transforms.code_compressor as cc
|
|
import headroom.transforms.kompress_compressor as kc
|
|
from headroom.transforms.code_compressor import CodeLanguage, detect_language
|
|
from headroom.transforms.compressor_registry import CompressInput
|
|
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
|
|
|
# Pseudo-assembly that detect_language() cannot identify (the issue's repro).
|
|
UNKNOWN_CODE = "\n".join(
|
|
f"step {i}: load register r{i % 8} <- mem[{i * 4}]; jump-if-zero r{i % 8} label_{i}"
|
|
for i in range(80)
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _tree_sitter_present(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# The router only builds the code compressor when tree-sitter parses; the
|
|
# unknown-language fallback never reaches the parser, so stub the probe to
|
|
# exercise the router path in environments without the ``code`` extra.
|
|
monkeypatch.setattr(cc, "_check_tree_sitter_available", lambda: True)
|
|
|
|
|
|
@pytest.fixture()
|
|
def kompress_spies(monkeypatch: pytest.MonkeyPatch) -> tuple[MagicMock, MagicMock]:
|
|
available = MagicMock(return_value=True)
|
|
compressor = MagicMock()
|
|
compressor.return_value.compress.return_value = SimpleNamespace(
|
|
compressed="kompressed",
|
|
original_tokens=100,
|
|
compressed_tokens=1,
|
|
compression_ratio=0.01,
|
|
)
|
|
monkeypatch.setattr(kc, "is_kompress_available", available)
|
|
monkeypatch.setattr(kc, "KompressCompressor", compressor)
|
|
return available, compressor
|
|
|
|
|
|
def test_disabled_kompress_turns_off_code_aware_fallback() -> None:
|
|
router = ContentRouter(ContentRouterConfig(enable_code_aware=True, enable_kompress=False))
|
|
|
|
compressor = router._get_code_compressor()
|
|
|
|
assert compressor is not None
|
|
assert compressor.config.fallback_to_kompress is False
|
|
|
|
|
|
def test_enabled_kompress_still_turns_off_code_aware_fallback() -> None:
|
|
# Kompress may be on for prose, but it must never compress code (#4214):
|
|
# the router's code compressor leaves blocks it cannot handle unchanged.
|
|
router = ContentRouter(ContentRouterConfig(enable_code_aware=True))
|
|
|
|
compressor = router._get_code_compressor()
|
|
|
|
assert compressor is not None
|
|
assert compressor.config.fallback_to_kompress is False
|
|
|
|
|
|
PERL_CODE = "\n".join(
|
|
["#!/usr/bin/perl", "use strict;", "use warnings;"]
|
|
+ [
|
|
f'my $v{i} = $ARGV[{i % 3}] // {i}; print "$v{i}\\n" if $v{i} =~ /^\\d+$/;'
|
|
for i in range(60)
|
|
]
|
|
)
|
|
|
|
PYTHON_CODE = "\n".join(
|
|
["import json", ""]
|
|
+ [
|
|
f"def handler_{i}(event):\n if event.get('k{i}'):\n return json.dumps(event)\n return None\n"
|
|
for i in range(40)
|
|
]
|
|
)
|
|
|
|
|
|
def _code_aware_entry(enable_kompress: bool):
|
|
router = ContentRouter(
|
|
ContentRouterConfig(enable_code_aware=True, enable_kompress=enable_kompress)
|
|
)
|
|
entry = router.compressor_registry.get("code_aware")
|
|
assert entry is not None
|
|
return router, entry
|
|
|
|
|
|
@pytest.mark.parametrize("enable_kompress", [False, True])
|
|
def test_unknown_language_code_passes_through_without_kompress(
|
|
kompress_spies: tuple[MagicMock, MagicMock], enable_kompress: bool
|
|
) -> None:
|
|
assert detect_language(UNKNOWN_CODE)[0] == CodeLanguage.UNKNOWN
|
|
_, entry = _code_aware_entry(enable_kompress)
|
|
|
|
out = entry.compress(CompressInput(content=UNKNOWN_CODE, content_type="text/x-code"))
|
|
|
|
available, compressor = kompress_spies
|
|
available.assert_not_called()
|
|
compressor.assert_not_called()
|
|
assert out.content == UNKNOWN_CODE
|
|
|
|
|
|
@pytest.mark.parametrize("enable_kompress", [False, True])
|
|
def test_perl_code_passes_through_without_kompress(
|
|
kompress_spies: tuple[MagicMock, MagicMock], enable_kompress: bool
|
|
) -> None:
|
|
# detect_language() reports Perl-dominant code as UNKNOWN on purpose, so the
|
|
# Perl branch is reached through an explicit language hint.
|
|
_, entry = _code_aware_entry(enable_kompress)
|
|
|
|
out = entry.compress(
|
|
CompressInput(content=PERL_CODE, content_type="text/x-code", config={"language": "perl"})
|
|
)
|
|
|
|
available, compressor = kompress_spies
|
|
available.assert_not_called()
|
|
compressor.assert_not_called()
|
|
assert out.content == PERL_CODE
|
|
|
|
|
|
@pytest.mark.parametrize("enable_kompress", [False, True])
|
|
def test_ast_failure_passes_through_without_kompress(
|
|
kompress_spies: tuple[MagicMock, MagicMock],
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
enable_kompress: bool,
|
|
) -> None:
|
|
router, entry = _code_aware_entry(enable_kompress)
|
|
code_compressor = router._get_code_compressor()
|
|
assert code_compressor is not None
|
|
|
|
def _boom(*_args, **_kwargs):
|
|
raise RuntimeError("simulated AST failure")
|
|
|
|
monkeypatch.setattr(code_compressor, "_compress_with_ast", _boom)
|
|
|
|
out = entry.compress(CompressInput(content=PYTHON_CODE, content_type="text/x-code"))
|
|
|
|
available, compressor = kompress_spies
|
|
available.assert_not_called()
|
|
compressor.assert_not_called()
|
|
assert out.content == PYTHON_CODE
|
|
|
|
|
|
def test_disk_cache_namespace_includes_the_routing_policy_revision(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
# Output cached before #4214 (code sent to Kompress) must not be replayed
|
|
# after the fix by a restart with the same version and config.
|
|
import headroom.transforms.content_router as cr
|
|
|
|
config = ContentRouterConfig()
|
|
current = cr._config_namespace(config)
|
|
monkeypatch.setattr(cr, "_ROUTING_POLICY_REVISION", "earlier-policy")
|
|
|
|
assert cr._config_namespace(config) != current
|