mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
143
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
303f521812 |
fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description
A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.
This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.
Closes #4874
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored
$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored
# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
`fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib
compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds
`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
it the string renders as the JSON string literal (`"""[]"""`), which
`expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
included for CI, where the wheel is built normally.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
f4e3ad4b11 |
fix(content-router): keep list-form tool_result text blocks on separate lines (#4739)
## Description
A `tool_result` whose `content` is a list of several `text` blocks gets
flattened with `"".join(...)` in `ContentRouter` before the
read-protection check and before compression. MCP tools that return more
than one `TextContent` and litellm-style clients send this shape. When a
block does not end with a newline, the last line of one block and the
first line of the next one become a single line. The compressed result
is written back as one block, so the model sees a line the tool never
produced.
This PR joins the blocks with a newline unless the block already ends
with one, at both flatten sites.
Fixes #4729
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new helper
`_join_tool_result_text_blocks()` that joins list-form `text` blocks
with `\n`, skipping the separator when a block already ends with a
newline. Empty blocks add no extra line.
- Used at both places that flattened with `""`: the read-protection
check and the compression path. The wire shape does not change (the
existing `[{"type": "text", ...}]` re-wrap is kept), and results that
are not compressed keep their original blocks.
- `tests/test_transforms/test_content_router.py`: one `router.apply`
test with a two-block log `tool_result` that asserts that the output was
compressed and that every log line in it is a line the tool actually
returned, plus four parametrized cases for the helper.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Exact source: head `5305d8acb5f505e635d1287cae30ed7f8d9eb595`, base
`c05fbdddc873078dd97cb9ede841e75ef7ed0bef`. The PR has three commits:
`f273fbb9` (the fix, written on `1da8930a`) and two merges of `main`
(`2cba6c5b`, `5305d8ac`), both without conflicts. `git diff
c05fbddd...5305d8ac` still touches only the two files listed above.
Upstream CI on `5305d8ac`, all `completed` / `success`: CI
[43338122484](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122484),
Merge Conflicts
[43338122336](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122336),
Changelog Guard
[43338122370](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122370),
rust
[43338122318](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122318),
Evaluation Suite
[43338122592](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122592),
Security
[43338122460](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122460).
Local runs below were built from `git archive 5305d8ac` in a
`python:3.13-slim-bookworm` image with `uv sync --frozen --extra proxy
--extra dev`. ruff ran on the two touched files and mypy on
`content_router.py`. pytest ran on the content router test modules. For
the reverse check, `content_router.py` was replaced with the copy from
`c05fbddd` and the tests kept.
A wider run over the 122 test files that touch the content router gave 9
failed / 2669 passed on `5305d8ac` and 9 failed / 2664 passed on
`c05fbddd`. The 9 failures are the same set on both (memory_bridge,
offline_egress docs, tabular; they come from the test image, not this
change), and the extra 5 passes are the new tests.
### Test Output
```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new tests, 5 consecutive runs, all identical:
======================= 5 passed, 66 deselected in 5.01s =======================
# test_content_router.py + test_content_router_ccr_retrieve_exemption.py + test_content_router_shell_small_results.py:
============================= 125 passed in 5.61s ==============================
# with content_router.py restored to
|
||
|
|
3aafdf2dcc |
fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description When the code compressor declined a block of source code, the router passed the block to Kompress. Kompress is a prose model. It drops low-salience words, and in code those words are keywords such as `for`, `in`, `if` and `import`. The result no longer parses. #4214 shows a fenced Python script that comes back with `for` and `in` removed. This PR stops that in the Python router and in the Rust live-zone dispatcher. When the code compressor declines, the code is forwarded unchanged. Closes #4214 Credit: the Python change is oxura's work from https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018 (277ca1f). Their commit is the first commit here, rebased onto `main`, and oxura is its author. The test, docs and Rust commits were added on top. oxura asked us to open this PR from our fork ([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)), and said they have not run this branch, so the model and native results below are ours, not theirs. ## Behaviour change to review With `enable_code_aware=False` (the default), source code now stays unchanged. Before this PR, it was sent to Kompress and lost keywords. With `enable_code_aware=True`, code that the code compressor declines also stays unchanged. Before, Kompress ran on it. A code block the code compressor does compress is unaffected. Prose, logs, JSON and tables are unaffected. Side effect: a code block that Kompress used to shrink (and break) now keeps its full size. This is the intended trade, but it means fewer tokens saved on code when `enable_code_aware=False`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update - [ ] New feature - [ ] Breaking change (the default-config behaviour change above is intended, and it only stops damage to code) ## Changes Made Five commits: - `596764c3` `fix(router): keep source code out of prose fallbacks`. Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in `headroom/transforms/content_router.py`. This is 277ca1f rebased onto `main`, and it differs from 277ca1f in two ways only: the two test-file conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because `AGENTS.md` says not to add content under `wiki/`. - `7705a4e6` `test(router): repin the code-to-Kompress tests and add the #4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by: oxura` trailer. Five tests asserted the old fallback. They are rewritten to assert the new behaviour (code unchanged, Kompress never sees it) instead of being deleted. Adds a regression test for the script from #4214, with `enable_code_aware` both off and on. Updates `docs/content/docs/code-compression.mdx` and `docs/content/docs/how-compression-works.mdx`, which said that detected code goes to Kompress. - `6631668a` `fix(live-zone): leave source code unchanged when the code compressor declines`. Author: Tao Hansen. Same fix in `crates/headroom-core/src/transforms/live_zone.rs`: a declined `SourceCode` block is now a NoOp, not a Kompress input. Tests: `tests/live_zone_source_code_decline.rs` (single test, needs no model), and new cases in `tests/live_zone_dispatch.rs`. - `d8776283` `fix(router): turn off the code compressor's internal Kompress fallback`. Author: Tao Hansen. Follows review: the router's code compressor was still built with `fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown languages, Perl and AST failures could build a `KompressCompressor` inside `CodeAwareCompressor`. The router now always sets `fallback_to_kompress=False`. The #4719 test that pinned `True` (`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as `test_enabled_kompress_still_turns_off_code_aware_fallback`, and the unknown-language, Perl (by language hint, since `detect_language()` reports Perl-dominant code as unknown) and AST-failure paths are tested with spies on `is_kompress_available` and `KompressCompressor`, with `enable_kompress` both off and on. `docs/content/docs/code-compression.mdx` notes that the router always turns the option off. - `c1f8464e` `fix(router): keep passthrough code from external compressors and stale caches`. Author: Tao Hansen. Follows review: `_try_external_compressor` skips `PASSTHROUGH` blocks (code with `prefer_code_aware_for_code=False`, whole or as a mixed section), which would otherwise be offered to a prose-only external compressor as `text/plain`. The disk cache namespace now also hashes `_ROUTING_POLICY_REVISION`, so a same-version restart does not replay code that Kompress damaged before this change. Tests: two external-dispatch cases with a `text/plain` external, and one namespace test. All three fail on the previous head. I did not touch `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`): see below, the failures are environment failures that also fail on `main` - [x] Linting passes (`ruff check .`, `ruff format --check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py`) - [x] New tests added for new behaviour - [x] Manual testing performed (real model replay, below) ### Test Output macOS, Python 3.14. Head is now `c1f8464e`. ```text After the second review fix ( |
||
|
|
c21b0813bd |
fix(compress): stop adding markers the caller did not ask for (#4678)
`/v1/compress` returns a body that someone else's client forwards
unchanged, so anything Headroom adds there reaches the provider. Two
things got added that the caller never asked for.
## Stale-Read retrieval hashes in marker-free modes
Read lifecycle replaces a Read that a later edit made stale with a short
notice. That notice always ended in `Retrieve original: hash=…` and
wrote the original to the CCR store, even in the default and
`lossy_inline` modes. The docs call those modes marker-free, and their
callers have no retrieval tool, so the hash pointed nowhere.
- Marker-free modes keep the notice ("re-read the file for current
content") but drop the hash and skip the store write.
- Lossless routers (proxy `--lossless`,
`ContentRouterConfig(lossless=True)`) skip read lifecycle entirely,
because the notice drops the file's bytes.
- Read maturation now recognises the hashless notice. Otherwise it would
replace the notice with a marker whose hash points at the notice itself.
## Cache breakpoints the client didn't send
On the session path, `normalize_message_cache_control` fell back to
placing `{"type": "ephemeral"}` on the last block message when the
client marked nothing. A caller that chose not to cache got billed cache
writes. `/v1/compress` now passes `place_unmarked=False`: if the client
marked no message, markers are stripped and none are placed. Proxy
handlers keep today's behaviour, and client markers in a misaligned list
still go through the old path.
## Tests
Each new test fails on main.
- `test_marker_free_modes_mark_a_stale_read_without_a_retrieval_hash`,
for the default mode and `lossy_inline`
- `test_lossless_router_keeps_a_stale_read`
- `test_session_turns_add_no_marker_the_caller_did_not_send`
- `test_respects_lifecycle_markers`, extended to the hashless notice
The related suites (gateway, transforms, read maturation, prefix
tracker, cache_control, compress endpoint and session) pass: 1371 passed
and 9 skipped before the review fixes, then 222 of 222 on the touched
files after them. Lint is clean on CI's ruff 0.17.0, and mypy is clean
on the touched modules.
## Not in this PR
- Read maturation (`handlers/anthropic.py`) is not gated on `--lossless`
or `--no-ccr` either. It's the same kind of bug on the proxy path, so it
belongs in a separate fix.
- The "lossless densify only" pipeline in `compress.py` sets
`ccr_enabled=False` without `lossless=True`, so read lifecycle still
runs there.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4678"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
## Real Behavior Proof
Maintainer qualification on Windows, Python 3.12.13, using an actual
Uvicorn server bound to an owned loopback socket and HTTPX over real
HTTP. The normal `/v1/compress` endpoint and compression pipeline ran
with offline mode and Kompress disabled; no handler, compression stage,
HTTP response or store was mocked. Workspace, configuration, savings and
TOIN paths were isolated. All 243 native inputs and the reused artifact
SHA-256
`5be63a4f31d3e38e7427d28380ecb977616e3f6a3dd6a1c59af5ff6361a05978` were
verified; this was not a fresh native build.
- Exact command / steps: the committed probe is `python -m pytest -q -s
tests/test_compress_runtime_http.py`. The qualifying run used the
maintainer loopback-only launcher with an explicit verified native path
and SHA-256, followed by the four focused
session/API/maturation/lifecycle suites. All 138 tests passed. The probe
starts the actual server, GETs `/health`, and sends nine POSTs to
`/v1/compress`: a Read followed by Edit in default, lossy-inline and
lossless modes, then three session turns without caller cache markers
and three turns with a caller marker. It stops and joins the server in
`finally`.
- Observed result: health and all nine compression requests returned
HTTP 200. Default and lossy-inline emitted stale notices with no
retrieval hash or CCR hashes, saving 3,576 tokens per request. Lossless
preserved the original Read bytes. All three unmarked session turns
returned zero cache markers; all three marked controls retained exactly
one. Session compression saved 7,350 / 15,901 / 25,958 tokens. Server
shutdown joined successfully.
- Before-fix control: the same real-server requests with all six
affected production modules replaced by current main
`200c838c7393d20ae56a0c338f832ed27133c398` failed five assertions:
default and lossy-inline included retrieval hashes, and each unmarked
session turn added a cache marker. Lossless and the three marked
controls passed on main and are not claimed as reproduced regressions.
Candidate source was restored byte-for-byte.
- Not tested: provider forwarding or billing, a live model, deployment
behavior, or the separate proxy read-maturation scope already listed
above. External startup connection attempts were blocked by the
qualifying launcher; this is not a zero-attempt network claim. No author
readiness or self-review declaration is added by this maintainer
evidence.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
8cdce172ea |
fix(router): ignore shell operators inside quoted arguments in read detection (#4646)
## Description Item 3 of #4478, the quoted-pattern case that #4521 left open. Read detection scanned the raw command text for shell syntax without regard to quoting: `_segment_is_read` treated any `>` as a redirect (`content_router.py:893` on `e4a60ed7`) and split the pipeline at the first `|`, and `_command_segments` split on `;`, `&&` and `||`. A quiet sed whose script holds one of these characters, such as `sed -n '/if x > 0/,/return/p' a.py` or the GNU alternation `sed -n '/def \|class /p' a.py`, was therefore not detected as a file read and lost read protection. The three scans now run on a copy of the command in which quoted, backslash-escaped and commented characters are masked, and the match positions are mapped back onto the original text. Two shell rules keep word boundaries intact (review round 2): a backslash-newline is a line continuation and is masked as two spaces, so a redirect or `tee` on the next line still counts as a write; and an unquoted `#` at the start of a word masks the rest of its line without changing quote state, so a quote mark inside a comment cannot hide the next line's separators. Unquoted operators behave as before. Refs #4478 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: new `_mask_quoted(command)`, which returns a same-length string with characters inside `'...'` / `"..."`, backslash-escaped characters and comment text replaced by `_` (the quote characters are kept; a backslash-newline becomes two spaces; a comment's newline stays visible). `_segment_is_read` runs its redirect/`tee` check and its pipeline split on the masked string; `_command_segments` finds `;` / `&&` / `||` on the masked string and slices the original. - `tests/test_transforms/test_read_command_detection.py`: `test_quoted_shell_operators_are_arguments`, 25 cases. Quoted `>`, `>>`, `|`, `;`, `&&` and `tee` inside a sed script or a file name are reads; a real redirect, pipe-to-`tee` or separator next to a quoted script still counts; `bash -lc "cat foo.py > copy.py"` is still not a read (the inner command is checked on its own, as before). Round 2 adds: continued `>`, `>>` and `| tee` are writes; a comment with an unmatched `"` or `'` followed by `ls; cat foo.py` / `true; cat foo.py` on the next line is a read; `cat foo.py # > not a redirect` is a read; a `#` inside a word is not a comment. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) (run on `headroom/transforms/content_router.py` only, see below) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # fail before (source at |
||
|
|
f333edf20f |
fix(router): keep tagged blocks out of the lossy size-gate fallback (#4543)
## Description
When a text block is over the Kompress size gate
(`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000) and contains protected
custom tags, `ContentRouter._try_ml_compressor` passed the
`{{HEADROOM_TAG_N}}` placeholder text to the TextCrusher / LogCompressor
fallback. Two things went wrong:
1. Neither fallback takes a `ccr_original` argument, so CCR stored the
placeholder text. `headroom_retrieve` returned `{{HEADROOM_TAG_0}}`,
`{{HEADROOM_TAG_1}}`, ... instead of the tagged content.
2. When the fallback dropped the lines holding the placeholders,
`restore_tags` discarded each lost wrap (by design,
`tag_protector_placeholder_lost`). The tagged content was then gone from
the output sent upstream too.
This is the same defect class as #1209 (Kompress stored the placeholder
intermediate) and #2694 (SmartCrusher, `ccr_source`), on the size-gate
branch those fixes did not reach.
The fix: when tags were protected, the size-gate branch returns the
original `content` unchanged instead of running the lossy fallback. That
is what the request-budget branch directly above it already does for the
same reason (the bytes must stay stable and intact). I chose this over
threading a `ccr_original` through the fallbacks because
`TextCrusher.compress` and `LogCompressor.compress` have no such
parameter, and the output would still lose the tagged lines through
`restore_tags`. Passing through is the smallest change and loses
nothing; the cost is that a gated block with custom tags is no longer
shrunk by the fallback.
Fixes #4542
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: in the Kompress size-gate
branch of `_try_ml_compressor`, return `content` unchanged when
`protected` is non-empty. The now-unreachable `restore_tags` call at the
end of that branch is removed (the fallback only runs when nothing is
protected).
- `tests/test_transforms/test_kompress_size_gate.py`: new regression
test `test_gate_keeps_tagged_blocks_out_of_lossy_fallback`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Fail before (fix reverted, new test kept):
```text
FAILED tests/test_transforms/test_kompress_size_gate.py::test_gate_keeps_tagged_blocks_out_of_lossy_fallback
========================= 1 failed, 5 passed in 2.19s ==========================
```
Pass after:
```text
tests/test_transforms/test_kompress_size_gate.py ...... [100%]
============================== 6 passed in 1.70s ===============================
```
Related suites (`tests/test_transforms` plus
`tests/test_ccr_tag_placeholder_regression.py`, excluding
`test_text_crusher_cjk_eval.py`, which imports `benchmarks/` that my
test copy does not include):
```text
FAILED tests/test_transforms/test_kompress_compressor.py::TestLazyImports::test_unloadable_native_dep_degrades_instead_of_raising
1 failed, 520 passed, 78 skipped in 24.66s
```
That one failure (`assert attempts["torch"] == 1`) is identical with my
change reverted (`1 failed, 30 passed, 11 skipped` on
`test_kompress_compressor.py` alone), so it is not caused by this PR.
```text
$ ruff check headroom/transforms/content_router.py tests/test_transforms/test_kompress_size_gate.py
All checks passed!
$ ruff format --check <same files>
2 files already formatted
```
## Real Behavior Proof
- Environment: Ubuntu (WSL2), Python 3.12.15, `main` at `a2d4fe72` with
the `_core` extension from the 0.42.0 wheel (no commits under
`tag_protector.rs`, `log_compressor.rs` or `crates/headroom-py` since
0.42.0), `HEADROOM_OFFLINE=1`, in-memory CCR.
- Exact command / steps: ran the script from #4542 (a short header, then
N lines of `<item>row i value</item> note text here for item number i`,
through `ContentRouter(config=ContentRouterConfig()).compress(text)`)
for N=3000 and N=4000, before and after the change, and retrieved the
CCR entry behind any `hash=` marker.
- Observed result: before the change, N=4000 became `[4006 lines
omitted]\n[4006 lines compressed to 0. Retrieve more:
hash=dda7b8a79b740b17a0a5461b]`, and the retrieved original was 229,839
bytes, not equal to the input, with 0 `<item>` and 4,000
`{{HEADROOM_TAG`. After the change, N=4000 comes back byte-identical
(253,839 chars, 4,000 `<item>`, 0 placeholders, no CCR marker). N=3000
is unchanged in both runs (189,839 chars, 3,000 `<item>`).
- Not tested: the Rust-side `tag_protector` and a freshly built `_core`
(I used the prebuilt 0.42.0 wheel's); a live proxy run; mypy; the case
where the gated block has no protected tags (unchanged code path,
covered by the existing gate tests); other platforms.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. This is a bug fix inside the
existing size-gate branch.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly. A block over the size
gate that contains protected custom tags is now returned unchanged
instead of being run through TextCrusher/LogCompressor with the tags
lost. Blocks with no protected tags take the same path as before.
- Kill switch / disable path: `HEADROOM_KOMPRESS_MAX_TOKENS=0` disables
the gate entirely (existing behavior).
- Unsafe override required: No
- Qualification impact: None expected. Token savings drop only for gated
blocks that carry custom tags, which previously lost the tagged content.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (one
unrelated failure, see Test Output)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Docs unchanged: no public API or documented behavior changes. An
alternative would be to teach TextCrusher/LogCompressor a `ccr_original`
and also keep the tagged lines; that touches two compressors (and the
Rust side) for a block shape that is rare, so I kept the change to the
router. Happy to do it that way if you prefer.
Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4543"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `38f4e6e6c37ae4b7b46bf97be50ad5fe320c2094`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.
Mechanically merged upstream/main
|
||
|
|
db30d82dff |
fix(router): log the ONNX fallbacks at info instead of warning (#4535)
<!-- maintainer-current-main --> ## Maintainer latest-main checkpoint Published `fe40ae66e3d0dd9e7df00be81716c42370139004` incorporates upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 23 ORT/dylib/logging/Gemini tests pass; Ruff/format/diff pass. Main sync is clean; expected-fallback source changes remain only the two INFO log levels. Shared Gemini request-fixture repair is test-only. No new code finding. The inherited main shard-2 failure was confirmed in actual hosted logs and locally (four Gemini failures). This head carries the maintainer fixture repair: real ASGI Request/JSON bytes replace an unused body-reader mock. All six test-function ASTs/assertions are preserved; all 10 Gemini cases pass. Contributor production logic was not rewritten by the sync/fixture follow-up. Earlier runtime captures retain their original commit/platform scope. Fresh exact-head hosted qualification is requested; nothing merged by this review. <!-- /maintainer-current-main --> ## Description A plain `pip install headroom-ai` has no ONNX Runtime (it comes with `[proxy]`), and the pure-Python detector is the supported path there. The first `compress()` call still logged "Native content detection requires ONNX Runtime 1.24+; using pure-Python detection for this process" as a WARNING. Every library user sees it on first import; both coding agents in our sandbox runs saw it, and one spent a step investigating it before continuing. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/_ort.py`: the sibling "onnxruntime x.y exposes an older C API than Rust detection requires" message drops to INFO as well. It fires on Intel macOS, where the `proxy` extra pins `onnxruntime <1.24` by design (`docs/installation.mdx`: Rust ML detection stays disabled there), so a supported `[proxy]` install warned on first `compress()` too. Its test now asserts the level. - `headroom/transforms/content_router.py`: `logger.warning` to `logger.info` for the ORT-compat fallback, with a comment on why. The existing circuit breaker (`_detect_native_unhealthy`) already makes it fire once per process; unchanged. - `tests/test_transforms/test_ort_dylib.py::test_content_router_bypasses_native_detector_for_incompatible_ort`: now captures at INFO, asserts the record's level is INFO and that a second call logs nothing. ## Testing - [x] Unit tests: the test fails before the fix (`assert 30 == 20`) and passes after; `tests/test_transforms/test_ort_dylib.py` 10 passed. - [x] `ruff check` / `ruff format --check` ### Test Output ```text pytest tests/test_transforms/test_ort_dylib.py: 10 passed (fails before the fix: assert 30 == 20) source build, default logging: 20031 -> 95 FATAL kept: True (no fallback line on stderr) source build, INFO enabled: INFO headroom.transforms.content_router: Native content detection requires ONNX Runtime 1.24+; using pure-Python detection for this process. ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.12, clean venv with the published 0.42.0 wheel (no onnxruntime). - Exact command / steps: `HEADROOM_OFFLINE=1 python -c "from headroom import compress; ..."` on a 3,000-line log. - Observed result: the warning line on stderr on first import, then `20031 -> 95`. - After-fix, this branch built from source (`uv pip install -e .` with maturin, fresh venv with tiktoken, pydantic, filelock, opentelemetry-api; no onnxruntime): ```text headroom from /Users/tejas/headroom-wt-onnx-info/headroom/__init__.py 20031 -> 95 FATAL kept: True ``` No fallback line on stderr at default logging. With `logging.basicConfig(level=INFO)` the same run prints `INFO headroom.transforms.content_router: Native content detection requires ONNX Runtime 1.24+; using pure-Python detection for this process.` once. - Not tested: the native (ORT present) path, which this change does not touch. ## Runtime Rollout Safety - Rollout-managed feature(s): none; two log records change level from WARNING to INFO, no behaviour change - Minimum rollout channel: stable (ships with the next release) - Stable/default behavior changed: only log level; detection and compression paths are untouched - Kill switch / disable path: not needed; `HEADROOM_LOG_LEVEL`-style logger config still surfaces the line at INFO - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Maintainer post-merge revisit: main |
||
|
|
7c4af752ad |
fix(transforms): stop the dense-line elider cutting Chinese, Japanese and Thai prose (#4295)
## Description `is_dense_line()` (`headroom/transforms/dense_line_elider.py`, #3685) decides a line is minified or encoded output if it is at least 300 characters long and less than 6% of it is spaces. Chinese, Japanese and Thai don't put spaces between words, so an ordinary paragraph in those languages passes the same test: a 455-character Chinese paragraph has zero ASCII spaces. Once the dense lines in a block add up to 2,000 characters, which is about five paragraphs of a fetched Chinese news or docs page, `elide_dense_lines()` keeps the first 160 and last 80 characters of **each paragraph** and replaces the middle with `...[N chars of dense machine-generated content elided]...`. This runs on the router's text fallback, so `headroom.compress()` and the proxy both cut readable prose mid-sentence and tell the model it was "machine-generated content". The original is CCR-retrievable, but the model has no reason to suspect it's missing article text. Closes: N/A (no matching issue found) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/dense_line_elider.py`: new `MAX_NON_ASCII_RATIO = 0.1`. `is_dense_line()` returns `False` when more than 10% of a line's characters are non-ASCII. 10% rather than 20% (changed after review) so that CJK prose inside scraped HTML, where long ASCII attributes or URLs make up most of the line, is kept too. Minified JS/CSS, base64, JWT/RSC payloads and hex blobs are almost entirely ASCII, so they are still elided. The check is `not line.isascii() and n - len(line.encode("ascii", "ignore")) > n * 0.1`, which runs in C: about 0.003 ms on a 1.1 MB ASCII line and 2 ms on a 1.1 MB line with some CJK in it. `is_dense_line()` already spends hundreds of ms on such lines in its JSON scan. - `tests/test_transforms/test_dense_line_elider.py`: - `test_space_free_scripts_are_prose_not_dense` (zh / ja / th): six long paragraphs pass through `elide_dense_lines()` byte-identical. - `test_cjk_prose_wrapped_in_long_ascii_markup_is_kept`: six `<p data-article-url="…long URL…">` lines with 84 Chinese characters each (10–20% non-ASCII) pass through unchanged; - `test_dense_lines_still_elided_next_to_cjk_prose`: minified JS and base64 next to a Chinese paragraph are still elided (2 lines), and the paragraph is kept. ## Testing - [x] Unit tests pass (`pytest`): `tests/test_transforms/test_dense_line_elider.py`, `tests/test_json_protection_boundaries.py`, `tests/test_content_router*.py`, `tests/test_transforms_content_router.py`, `tests/test_transforms/test_content_router.py`, not the full suite - [x] Linting passes (`ruff check .`): on the changed files - [ ] Type checking passes (`mypy headroom`): not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New tests on `main` (fix reverted, tests kept): ```text $ python -m pytest tests/test_transforms/test_dense_line_elider.py -q FAILED tests/test_transforms/test_dense_line_elider.py::test_space_free_scripts_are_prose_not_dense[ja] FAILED tests/test_transforms/test_dense_line_elider.py::test_space_free_scripts_are_prose_not_dense[th] FAILED tests/test_transforms/test_dense_line_elider.py::test_space_free_scripts_are_prose_not_dense[zh] FAILED tests/test_transforms/test_dense_line_elider.py::test_cjk_prose_wrapped_in_long_ascii_markup_is_kept FAILED tests/test_transforms/test_dense_line_elider.py::test_dense_lines_still_elided_next_to_cjk_prose ========================= 5 failed, 9 passed in 6.65s ========================== ``` With the fix: ```text $ python -m pytest tests/test_transforms/test_dense_line_elider.py tests/test_json_protection_boundaries.py \ tests/test_content_router*.py tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py -q ======================= 254 passed, 2 skipped in 18.97s ======================== $ ruff check headroom/transforms/dense_line_elider.py tests/test_transforms/test_dense_line_elider.py All checks passed! $ ruff format --check headroom/transforms/dense_line_elider.py tests/test_transforms/test_dense_line_elider.py 2 files already formatted ``` (On the first revision, one run of that command had a failure I didn't capture. Two reruns passed. The machine had a load average of about 50 at the time, and the router's deadline tests are timing-sensitive.) ## Real Behavior Proof - Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this branch (native `headroom._core` from the 0.41.0 wheel), `HEADROOM_OFFLINE=1`, public `headroom.compress()` API - Exact command / steps: called `headroom.compress(messages, model="gpt-4o", model_limit=128000, protect_recent=0)` on an OpenAI-format conversation whose `fetch_page` tool result is a Chinese article: a title plus six 455-character paragraphs (2,751 chars). I counted elision markers in the compressed tool result and printed an excerpt, on `main` and on this branch. - Observed result: on `main` all 6 paragraphs are cut (`router:text:0.59`, 2,751 → 1,918 chars), each with a `...[215 chars of dense machine-generated content elided]...` marker in mid-sentence. On this branch the article goes through unchanged (0 of 6 elided). ```text == main chars 2751 -> 1918 | transforms: [..., 'router:text:0.59'] paragraphs elided: 6 of 6 excerpt: 能相似的方式做出反应的智能机器,该领域的研究包括机器人、语 ...[215 chars of dense machine-generated content elided]... 一个分支,它企图了解 == this branch chars 2751 -> 2751 | transforms: ['router:protected:user_message', 'router:protected:user_message'] paragraphs elided: 0 of 6 ``` - Not tested: the live proxy. Minified bundles whose string literals push non-ASCII above 10% (a synthetic i18n bundle at 14% is now kept) are left alone rather than elided. That's the false negative the architecture principles prefer, and ASCII-dominated bundles, base64 and RSC are still elided. Other unspaced scripts (Lao, Khmer, Burmese) are covered by the same rule, but I didn't test them. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: long lines that are more than 10% non-ASCII are no longer elided. ASCII-dominated dense lines (minified JS/CSS, base64, RSC) are elided exactly as before. - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes This is not the same as the Kompress/ML text path. That path only runs once the elider has declined a block, and I haven't checked how it treats CJK. Maintainer requalification (Windows/Python 3.12, current-main integration, existing native extension): the public `headroom.compress()` API was exercised with six paragraphs each of the Chinese, Japanese, and Thai samples as a `fetch_page` tool result, `model="gpt-4o", model_limit=128000, protect_recent=0`, offline. For each language, current main produced six dense-elision markers and changed the text; this branch produced zero markers and byte-identical text. The broader router suite reported 306 passes and seven failures; all seven exact failure IDs reproduced on actual unchanged main (embedded-JSON/mixed-content assertions), so they are not attributed to this patch. Changed-file lint, formatting, and whitespace checks passed. No live provider/proxy or fresh native build is claimed; hosted sync qualification remains pending. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
54f48fb1c2 |
fix(code): keep semicolon-separated statements valid (#4630)
## Description
Python's grammar exposes the `;` between two statements on one physical
line as a root-level anonymous token. The leftover top-level sweep
captured it as a fragment, so assembly put a bare `;` on its own line:
```python
v=json.load(open(B+'v.json'))
;
m=json.load(open(B+'m.json'))
```
The result no longer parses, `compress()` discards it and returns the
original, and the proxy logs `Code compression produced invalid syntax
for python (...) returning original` for every such tool result. Two
live fenced blocks (220 and 117 tokens) hit this in the same proxy
session.
The line-based statement collection has a second symptom: when two
statements share one physical line (`x=1; y=2`), both AST nodes map to
the same row. Function bodies appended that row twice; class bodies
appended the full line twice, producing syntactically valid but wrong
output (`x=1; y=2` executed twice, compression ratio inflated to 1.429).
A third symptom surfaced once the output became valid: Python runs
top-level statements in order, but imports are always emitted first.
`import sys; sys.path.insert(0, 'vendor'); import plugin` now compressed
to valid code with `import plugin` hoisted above the path update, so the
plugin failed to load with `ModuleNotFoundError`.
Demoting such imports fixed ordering inside the top-level bucket, but
definitions are assembled before that bucket. `import sys; marker = 1;
import decimal` followed by `class Money(decimal.Decimal)` still moved
the class ahead of `import decimal` and raised `NameError` at definition
time.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Skip unnamed `;`/`,` children in the top-level leftover sweep so a
standalone separator can never become a code fragment.
- Merge overlapping row ranges in `_compress_function_ast` before the
budget loop. Taking the union (instead of dropping the later range)
preserves continuation lines of a multi-line statement that starts after
`;`.
- Track the last emitted row in `_compress_class_ast` and emit shared
rows once, trimming only rows already emitted so continuation lines
survive.
- For Python only, keep a root-level import behind executable top-level
code: the top-level sweep demotes such imports to the top-level code
stream (source order preserved) instead of hoisting them above the code
they depend on. Other languages are untouched because their imports/uses
are order-independent or must lead the file.
- For Python only, decline reassembly when executable top-level code
precedes any declaration. Definitions are emitted before the top-level
code bucket, so a class base, decorator, or default expression after
that code would be evaluated before the code or import it needs. Safe
reassembly cannot be guaranteed, so the original snippet is returned
unchanged. Snippets with no declarations keep compressing with demoted
imports in source order.
- Added `TestSemicolonSeparatedStatements` with seven regression tests:
top-level fallback, class-body duplication, class-body continuation,
function-body shared-line/continuation behavior, import-order
preservation that executes the compressed output, and executable
class-base and decorator/default ordering regressions.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches untyped helpers
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
109 passed in 5.22s
$ .venv/bin/ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ .venv/bin/ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
$ # same new tests against unpatched origin/main
|
||
|
|
7c150ec6b0 |
fix(code): keep Rust attributes attached to their items (#4624)
## Description
Rust outer attributes (`#[derive(...)]`, `#[inline]`, `#[cfg(...)]`) are
grammar siblings of the item they decorate, exactly like doc comments.
`_get_leading_comment_text` only collected comment node types, so
attributes fell through to the leftover top-level sweep and drifted away
from their item. On a mixed file the compressed output looked like this:
```rust
#[derive(Debug, Clone, PartialEq)]
pub struct Item {
pub id: u64,
pub name: String,
}
pub fn log_item(item: &Item, sink: &mut Vec<u64>) {
sink.push(item.id);
// [3 lines omitted]
}
/// Logs one item.
#[inline]
#[cfg(feature = "extra")]
```
The attribute block ends up after the last item, and `rustc` rejects the
file with `error: expected item after attributes`. Tree-sitter accepts
it, so the compressor's syntax check does not catch the corruption.
Rust line comments also include their trailing newline in the node text
(`/// doc\n`), so simply attaching them to the same preamble would
introduce blank lines.
Rust also allows blank lines between an outer attribute and the item it
decorates:
```rust
#[inline]
pub fn record(item: &Item) {}
```
The attribute is still attached to the function. Only ordinary comments
require row adjacency; attributes must survive the gap as well.
Comments can also sit between the attribute and its item:
```rust
#[inline]
// Keep with the function below.
pub fn record(item: &Item) {}
```
A blank line before that comment ended the preamble scan before the
attribute was reached. Detached comments are now skipped while scanning,
so the attribute behind one is still collected; the comments themselves
keep the ordinary adjacency policy.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `_ATTACHED_NODE_TYPES = {"attribute_item"}` and made the
leading-sibling collector treat those nodes like comments.
- Renamed the collected list to `preamble`, since it now holds comments
and attributes, and updated the docstring.
- Strip the trailing newline from each collected sibling before joining
(Rust line comments carry one in their node text).
- Keep the blank-line adjacency rule for ordinary comments only;
attributes always attach to the following item, since Rust permits
whitespace between an outer attribute and its item.
- Skip detached comments while scanning the preamble so an outer
attribute behind one is still collected, without capturing those
comments.
- Added `TestRustAttributeAttachment` with four regression tests,
including blank-line-separated attributes and attributes behind an
intervening comment.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches an untyped helper
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_transforms/test_code_compressor.py -q
96 passed in 7.41s
$ uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ uv run ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
```
The new blank-line and intervening-comment tests fail on the unpatched
helper and pass with this change.
## Real Behavior Proof
- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from `uv sync
--extra dev --extra code`, `rustc 1.99.0`.
- Exact command / steps: compressed the samples below with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1,
max_body_lines=1, enable_ccr=False, semantic_analysis=False))`, wrote
each output to disk, then ran `rustc --edition 2021 --cfg
'feature="extra"' --crate-type lib`.
- Observed result: before the fix the compressed output ended with the
orphaned `/// Logs one item.` / `#[inline]` / `#[cfg(...)]` block and
`rustc` exited with `error: expected item after attributes`; with the
first fix ratio `0.808`, the preamble stayed attached, and `rustc` exit
0. For the blank-line sample, before the follow-up the attribute was
detached and `rustc` failed; after the follow-up the output emits
`#[inline]\npub fn record(...)`, ratio `0.795`, and `rustc` exit 0. For
the comment-between sample, before this change the attribute and comment
were emitted after the function and `rustc` failed; after the change the
output emits `#[inline]\npub fn record(...)`, ratio `0.746`, and `rustc`
exit 0. The detached comment keeps the ordinary policy and remains in
the leftover sweep.
- Not tested: attributes on `impl`/`trait`/`use` items (same sibling
mechanism, no dedicated fixture), other grammars (only Rust emits
`attribute_item`), and end-to-end proxy traffic.
Compressed output after the fix (compiled successfully):
```rust
#[derive(Debug, Clone, PartialEq)]
pub struct Item {
pub id: u64,
pub name: String,
}
/// Logs one item.
#[inline]
#[cfg(feature = "extra")]
pub fn log_item(item: &Item, sink: &mut Vec<u64>) {
sink.push(item.id);
// [3 lines omitted]
}
```
Blank-line attribute sample after the follow-up (compiled successfully):
```rust
pub struct Item { pub id: u64 }
#[inline]
pub fn record(item: &Item, sink: &mut Vec<u64>) {
sink.push(item.id);
// [2 lines omitted]
}
```
Comment-between sample after the follow-up (compiled successfully):
```rust
pub struct Item { pub id: u64 }
#[inline]
pub fn record(item: &Item, sink: &mut Vec<u64>) {
sink.push(item.id);
// [3 lines omitted]
}
// Keep with the function below.
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: Rust attributes are now emitted with
their item instead of being separated or orphaned
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The leftover path that caused this is shared by every language, but only
the Rust grammar emits `attribute_item` siblings today, so the behavior
change is scoped to Rust. The Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`) has no
equivalent preamble collection; parity there is out of scope for this
PR.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
0d0cda22b4 |
fix(code): keep multi-line class headers' opening brace (#4623)
## Description
The code-aware compressor rebuilds a class/impl header from the lines
before the member container, then re-adds the container's start line
only when that line begins with `{`. A multi-line signature can put the
opening brace on a continuation line shared with the signature tail:
```rust
impl<T> Wrapper<T>
where T: Value + Debug + Clone {
```
```java
public class Calc
extends Base
implements Runnable {
```
In that case the rebuilt header loses its `{`, the rebuilt source no
longer parses, the syntax check rejects the output, and the compressor
returns the input unchanged — so these containers are never compressed.
On a live proxy this shows up as repeated runtime warnings while
compressing Rust code:
```text
Code compression produced invalid syntax for rust (614 tokens), returning original
Code compression produced invalid syntax for rust (366 tokens), returning original
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_compress_class_ast` keeps the member container's start line whenever
the container node begins with the opening brace, instead of requiring
the line to start with `{`.
- Colon-signature languages (Python) are unchanged: their block node
starts at the first body statement, which can itself begin with `{`, so
they stay on the colon path.
- Added `TestMultiLineClassHeaders` with a Rust `impl ... where ... {`
and a Java `extends`/`implements` header.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
adds a branch to existing untyped helper logic
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_transforms/test_code_compressor.py -q
94 passed, 2 warnings in 24.25s
$ uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ uv run ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
```
Both new tests fail on `main` (the compressor logs `invalid syntax ...
returning original` and `compression_ratio == 1.0`) and pass with this
change.
## Real Behavior Proof
- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from `uv sync
--extra dev --extra code`, base commit `7e0094d`; `rustc 1.99.0`, `javac
17.0.21`.
- Exact command / steps: compressed the samples with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1,
max_body_lines=2, enable_ccr=False, semantic_analysis=False))`, wrote
the compressed output to disk, then compiled it with `rustc --edition
2021 --crate-type lib <out>.rs` and `javac <out>/Calc.java`.
- Observed result: before the fix both samples logged `Code compression
produced invalid syntax ... returning original` and `compression_ratio`
was `1.0`; after the fix Rust ratio `0.943` with `where T: Value + Debug
+ Clone {` retained and `rustc` exit 0, Java ratio `0.803` with
`implements Runnable {` retained and `javac` exit 0.
- Not tested: C++, JavaScript/TypeScript and C# multi-line class headers
(same code path, but no dedicated fixtures were added), and end-to-end
proxy traffic.
Compressed Rust output (compiled successfully):
```rust
impl<T> Wrapper<T>
where T: Value + Debug + Clone {
pub fn record(&self, sink: &mut Vec<u64>) {
for item in &self.items {
sink.push(item.id());
}
// [2 lines omitted]
}
}
```
Compressed Java header:
```java
public class Calc
extends Base
implements Runnable {
public void compute(int x) {
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: compression now succeeds for
containers that previously fell back to the original
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The same header logic exists in the Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`), which
currently keeps only `node_lines[..sig_end]` and has the same gap. This
PR intentionally stays scoped to the Python implementation; the Rust
port can follow separately if maintainers want parity.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
1b9d296c19 |
fix(proxy): preserve Cline headroom__headroom_retrieve results from recompression (#4586)
## Description Cline names MCP tools `<server>__<tool>`, so with the default `headroom` server the retrieval tool reaches the proxy as `headroom__headroom_retrieve`. `_tool_name_aliases()` in `headroom/config.py` maps OpenCode's single-underscore `headroom_headroom_retrieve` and the `mcp__` / `mcp_` wrappers to `headroom_retrieve`, but not the double-underscore form. `is_tool_excluded()` therefore returns `False` for it, so both the ContentRouter `ccr_retrieve` guard and the OpenAI handler's `headroom_retrieve_call_ids` miss the call. The retrieved content then goes through normal compression again, which produces the retrieval loop described in the issue. This follows #3782: the double-underscore spelling now gets the same exact-name alias as the single-underscore one. I kept it an exact match on the default server name, not a generic `*__headroom_retrieve` suffix rule, so the change doesn't widen matching for other exclusion entries. The content-based guard suggested in the issue (detecting a `hash` + `original_content` payload whatever the tool name is) is a larger behavior change, and this PR leaves it out. Closes #4585 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/config.py`: `_tool_name_aliases()` now maps `headroom__headroom_retrieve` (case-insensitive) to `headroom_retrieve`, alongside the existing OpenCode spelling. - `tests/test_tool_name_aliases.py`: adds the Cline spelling to the parametrized exclusion test. - `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`: new test that sends an OpenAI-shaped `headroom__headroom_retrieve` call and result through `ContentRouter.apply()` and asserts the result passes through unchanged with `router:excluded:ccr_retrieve`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Both new test cases fail on unmodified `main` (`2 failed, 16 passed`) and pass with the fix. Lint, format and mypy were run on the changed files only. ### Test Output ```text $ .venv/bin/python -m pytest tests/test_tool_name_aliases.py tests/test_transforms/test_content_router_ccr_retrieve_exemption.py tests/test_ccr_tool_injection_cache_stability.py tests/test_hermes_tool_call_unwrap.py tests/test_openai_responses_compression_units.py -q -p no:cacheprovider -o addopts= 125 passed, 1 warning in 6.16s # same two files on unmodified main (headroom/config.py reverted): FAILED tests/test_tool_name_aliases.py::test_retrieve_tool_aliases_are_excluded[headroom__headroom_retrieve] FAILED tests/test_transforms/test_content_router_ccr_retrieve_exemption.py::TestContentRouterCcrRetrieveExemption::test_cline_server_tool_mcp_name_not_recompressed 2 failed, 16 passed in 3.38s $ .venv/bin/ruff check headroom/config.py tests/test_tool_name_aliases.py tests/test_transforms/test_content_router_ccr_retrieve_exemption.py All checks passed! $ .venv/bin/ruff format --check headroom/config.py tests/test_tool_name_aliases.py tests/test_transforms/test_content_router_ccr_retrieve_exemption.py 3 files already formatted $ .venv/bin/mypy headroom/config.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Ubuntu 26.04 x86_64, Python 3.12, offline, no provider - Exact command / steps: `.venv/bin/python proof.py`, a standalone script outside pytest. It prints `is_tool_excluded(name, DEFAULT_EXCLUDE_TOOLS)` for the three retrieve spellings, then runs `ContentRouter(ContentRouterConfig(min_section_tokens=10)).apply()` on an OpenAI-shaped assistant `tool_calls` entry named `headroom__headroom_retrieve` followed by a 3,680-char JSON tool result, and prints the output size and the transforms that were applied. - Observed result: on unmodified `main`, the Cline name is not excluded and the retrieve result is crushed: ```text is_tool_excluded('headroom_retrieve') = True is_tool_excluded('headroom_headroom_retrieve') = True is_tool_excluded('headroom__headroom_retrieve') = False input chars=3680 output chars=1861 unchanged=False new_ccr_marker=False transforms: ['router:smart_crusher:0.40'] ``` With this branch, the result passes through byte-for-byte: ```text is_tool_excluded('headroom_retrieve') = True is_tool_excluded('headroom_headroom_retrieve') = True is_tool_excluded('headroom__headroom_retrieve') = True input chars=3680 output chars=3680 unchanged=True new_ccr_marker=False transforms: ['router:excluded:ccr_retrieve'] ``` - Not tested: a live Cline session and real provider traffic through the proxy, and Cline's hashed-name fallback for tool names over 64 characters (not reachable with the default `headroom` server name, which gives a 27-character tool name). ## Runtime Rollout Safety - Rollout-managed feature(s): none; this is the default tool-exclusion code path, with no flag - Minimum rollout channel: stable; the only effect is that one more tool-name spelling is excluded from compression - Stable/default behavior changed: yes, but only for tools named exactly `headroom__headroom_retrieve` (case-insensitive), whose results are now kept verbatim instead of compressed - Kill switch / disable path: none added; the change is a one-line alias in `_tool_name_aliases()` - Unsafe override required: no - Qualification impact: none; other tool names keep their current exclusion behavior - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4586" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
ac8423965d |
fix(code): preserve syntax in CCR retrieval comments (#4583)
## Description `CodeAwareCompressor.compress()` validates the AST-compressed source before appending its CCR retrieval marker. The marker always started with `#`, so JavaScript, TypeScript, Go and other C-family outputs became invalid after validation while still reporting `syntax_valid=True`. Use the detected language's existing `LangConfig.comment_prefix` for that marker. Python keeps `#`; C-family languages use `//`. Addresses **item 4** of #4478. This deliberately does not close the umbrella issue or change its other three items. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Reuse the existing per-language comment configuration when appending CCR metadata. - Exercise real compression with CCR enabled for Python, JavaScript, TypeScript and Go. Parse the final output and verify that the recovery hash retrieves the exact original source. - No dependency, public API, rollout flag or changelog changes. ## Testing - [x] New regression tests fail before the fix and pass after it. - [x] Six relevant compressor suites pass: **149 passed**. - [x] `ruff check .` passes; both changed files pass `ruff format --check`. - [x] Project pre-commit checks (including isolated mypy) and commitlint pass. - [x] Manual compression + Node syntax check + real in-memory CCR retrieval performed. - The full development environment's `mypy headroom --ignore-missing-imports` reports two existing errors in `memory/adapters/embedders.py:389` and `integrations/langchain/chat_model.py:380`. A baseline run with the unmodified compressor via `mypy --shadow-file` reports exactly the same errors. These files are not changed here. ### Test Output ```text Before: pytest tests/test_transforms/test_code_compressor.py -k ccr_marker_preserves -q 3 failed, 1 passed, 92 deselected (JavaScript, TypeScript and Go fail the final-source syntax assertion.) After: same command 4 passed, 92 deselected pytest tests/test_transforms/test_code_compressor.py \ tests/test_code_compressor_language_alias.py \ tests/test_code_compressor_syntax_breaker.py \ tests/test_code_compressor_bash.py \ tests/test_code_compressor_thread_safety.py \ tests/test_transforms/test_code_compressor_cjk.py -q 149 passed in 12.19s ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.12.14, Node 24.16.0, tree-sitter-language-pack 0.13.0. Python source based on `1fe8549f`; development/code dependencies installed from `uv.lock`, with the official headroom-ai 0.42.0 wheel also installed. No provider or model API call. - Exact command / steps: run the script below from the checkout with `HEADROOM_OFFLINE=1`, then run `node --check /tmp/headroom-ccr-check.mjs`. Run once on the base revision and once with this patch. ```python from pathlib import Path from headroom.cache.compression_store import CompressionStore, set_request_compression_store from headroom.transforms.code_compressor import CodeAwareCompressor, CodeCompressorConfig from tests.test_transforms.test_code_compressor import generate_javascript_code store = CompressionStore() set_request_compression_store(store) code = generate_javascript_code() result = CodeAwareCompressor( CodeCompressorConfig(min_tokens_for_compression=10) ).compress(code, language="javascript") Path("/tmp/headroom-ccr-check.mjs").write_text( result.compressed, encoding="utf-8", newline="\n" ) print(result.syntax_valid, result.compressed.splitlines()[-1]) assert result.cache_key assert store.retrieve(result.cache_key).original_content == code ``` - Observed result: Before the fix, Headroom reports `True`, but Node exits 1 with `SyntaxError: Invalid or unexpected token` at `# [200 tokens compressed. Retrieve more: hash=... Expires in 5m.]`. - After: marker starts `// [200 tokens compressed...`; Node exits 0; CCR retrieval equals the input byte for byte. - Not tested: live provider/proxy traffic, Windows/Linux, full repository test suite, freshly built Rust extension or the full Rust workspace checks. `make ci-precheck` stops at `cargo fmt` because this machine has no `cargo`; the Python change does not modify Rust code. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this fixes the existing code-aware compressor, which is not gated by the runtime rollout registry. - Minimum rollout channel: Stable; no beta/canary/dev channel or unsafe override is required. - Stable/default behavior changed: only the CCR comment delimiter on languages whose configured delimiter is not `#`. - Kill switch / disable path: Set `CodeCompressorConfig(enable_ccr=False)` to disable the CCR marker and original-content storage. - Unsafe override required: no. - Qualification impact: final source now remains parseable for the covered languages; CCR payload/hash behavior is preserved. - Rollback path: Revert commit `69afcfa7e3acac52b99e638af2331538b3dadea6`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] The change follows project style and uses the existing language configuration. - [x] Regression tests cover final output syntax and original-content recovery. - [x] Relevant existing tests pass locally. - [x] `CHANGELOG.md` is unchanged; the Conventional Commit title supplies the release-note entry. - [x] No user-facing documentation change is needed for this syntax-preservation fix. ## Additional Notes Prepared with AI assistance (Codex), with executed local regression tests and a Node syntax-check reproduction. The scope is restricted to #4478 item 4. An independent Codex review found no blocking defects, checked language coverage and marker consumers, and independently ran the main compressor test file (96 passed), changed-file Ruff and `git diff --check`. This is automated review, not maintainer approval. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4583" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
93279f3208 |
fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description Closes #3697. An unchanged SmartCrusher result now keeps the original bytes and reports no structural compression. Formatting-only savings come from the existing Rust JsonMinifier, not from pretending the JSON value changed. The minifier preserves duplicate keys, order, escapes and number literals. A mixed result cannot use one section's CCR marker to hide unmarked loss in another. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace savings through JsonMinifier in Python and the eligible native live zone. - Keep all records and JSON literals; do not store originals or inject CCR markers for lossless minification. - Preserve mixed routing and reject uncovered lossy sections in both tool-result formats. Explicit no-CCR behavior is unchanged. - Replace incidental forwarding tests with native minification and complete-record regressions; document the distinction. ## Testing - [ ] Entire Python suite — not claimed. - [x] Four consumer/native modules: `python -m pytest -q tests/test_recursive_json.py tests/test_transforms_content_router.py tests/test_transforms/test_smart_crusher_rust_parity.py tests/test_transforms/test_smart_crusher_lossless_default.py` - [x] Changed-file Ruff checks; normal commit hooks, including mypy and commitlint. - [x] Manual public-API runs, with and without the HTML extra. ### Test Output Consumer/native modules: ```text collected 136 items ============================= 136 passed in 36.43s ============================= ``` Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native build, Python checks, commitlint): ```text ======================== 180 passed, 1 warning in 8.35s ======================== ✅ ci-precheck PASSED — safe to push. To https://github.com/oxura/headroom.git 64688217..0002af3e HEAD -> fix/3697-unchanged-json-passthrough ``` The first push stopped on an inherited commit footer because `origin/main` was stale. Fetching the actual upstream ref fixed the lint range; the normal push then passed. No source merge, lint-rule change or hook bypass. Hosted checks on `0002af3e` now pass: all four Python shards, Windows permissions, smoke tests, native wrappers, parity and Docker e2e. [Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks). ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual extension built from this checkout. No provider billing calls. - Exact command / steps: the earlier localhost `/v1/compress` and TransformPipeline runs exercised the original fix. For this follow-up, call `ContentRouter.compress` on a data-only HTML page containing 60 complete records, with `smart_crusher_lossless_only=True`, `smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run it once without trafilatura, then with the locked 2.0.0 extra. The new public-API test contains the full input/config. Separately call `route_embedded_json` on a compact array followed by a padded array, with Unicode and CRLF surroundings. - Observed result: both HTML modes keep all 60 records and the surrounding markup, with no unrecoverable section. The unchanged first JSON span does not stop the second span's compaction; surrounding text stays exact. Final Codecov is 96%; the two remaining lines are the deferred HTML fallback return. The real no-HTML-extra API run executed both, and local coverage also exercises the unchanged-span branch. - Not tested: live provider acceptance, invoices or production cache hit rates. No native eligibility or admission gates changed. ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing JSON routing. - Minimum rollout channel: unchanged. - Stable/default behavior changed: eligible formatting savings return as json_minifier; standalone SmartCrusher remains byte-exact on no-ops. - Kill switch / disable path: existing routing configuration and HEADROOM_LIVE_ZONE_DISABLE_ARMS. - Unsafe override required: no. - Qualification impact: no-op, formatting, structural and CCR paths exercised separately. - Rollback path: revert the minifier integration; no persistence migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes Addresses the [minifier review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285). The maintainer’s main merge is retained. The latest follow-up only changes consumer tests. AI-assisted; no subagents. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
75df01d627 |
fix(codec): reject densified blocks with mismatched row counts (#4271)
## Description
`expand_compacted()` now rejects blocks whose `[N]` header disagrees
with the number of CSV records. Previously, `[3]{id:int}\n1\n2\n`
returned two records as though the original array had been
reconstructed, leaving callers unable to distinguish truncation from
successful expansion.
The decoder returns its existing `None` failure result on mismatch.
Valid multiline cells, single-column missing values, empty arrays and
the tolerated trailing blank line retain their behavior.
Compatibility limit: existing trailing-empty-row normalization can
discard an extra empty CSV record. This patch does not distinguish that
record from a trailing blank-line artifact; it validates the count after
that normalization.
## Type of Change
- [x] Bug fix
## Changes Made
- Validate the parsed CSV record count after dictionary legends and
existing trailing-newline normalization.
## Testing
- [x] Focused unit tests pass (full suite limitation below)
- [x] Linting passes
- [x] Type checking passes on macOS (Windows limitation below)
- [x] New regression tests added
- [x] Manual testing performed
- Eight new regression cases failed before the fix and pass after it
(missing/extra rows, empty bodies, dictionary-encoded values, and
JSON-string wrappers).
- Codec and live-agent densification suites: **41 passed on Windows and
41 passed on macOS arm64**.
- Repository-wide `ruff check .`: passed (Ruff 0.17.0).
- Changed-file `ruff format --check`: passed.
- `mypy --follow-imports=silent --ignore-missing-imports
headroom/transforms/compaction_codec.py`: passed.
- Full `pytest -x` attempt stopped at collection: optional `fastapi` is
not installed in this environment. The full suite is not claimed green.
- Repository pre-commit hooks passed except full-project mypy on
Windows: seven existing POSIX-only `os` attribute errors in unchanged
`fileperms.py` and `proxy/interceptors/astgrep.py`. Scoped codec type
checking passed.
- Full-project mypy 1.20.2 on macOS: **no issues found in 572 source
files** (`uvx --from mypy==1.20.2 mypy --ignore-missing-imports
headroom`).
### Test Output
```text
python -m pytest tests/test_transforms/test_compaction_codec.py tests/test_densify_agent_mode.py -q
Windows / Python 3.13.12: 41 passed in 2.74s
macOS arm64 / Python 3.13.15: 41 passed in 2.99s
```
## Real Behavior Proof
- Environment: Windows, Python 3.13.12, current Python source with the
published Headroom 0.41.0 native wheel. No provider API calls, ML model
downloads or mocks. `HEADROOM_TELEMETRY=off`, `HEADROOM_BEACON=0`,
`HEADROOM_OFFLINE=1`; `gpt-4o` used only for token counting.
- Exact command / steps: Run the Python snippet below from the checkout
after installing the native extension.
```python
import json
from headroom import densify, expand_compacted
records = [{"id": i, "name": f"item_{i}", "status": "active"} for i in range(100)]
result = densify(
[{"role": "tool", "tool_call_id": "demo", "content": json.dumps(records)}],
model="gpt-4o", protect_recent=0,
)
block = result.messages[0]["content"]
if block.startswith('"'):
block = json.loads(block)
assert expand_compacted(block) == records
truncated = block.rstrip("\n").rsplit("\n", 1)[0] + "\n"
print(f"Full block: {len(records)} records recovered exactly")
print(f"After removing the last data row: {expand_compacted(truncated)!r}")
assert expand_compacted(truncated) is None
```
- Observed result: All 100 records recovered exactly; deleting the final
row returns `None`.
```text
Full block: 100 records recovered exactly
After removing the last data row: None
```
- Not tested: full optional-dependency suite, native-source rebuild,
live provider/proxy traffic. This change does not validate every
possible malformed CSV cell; its scope is the declared row count.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: not applicable.
- Stable/default behavior changed: malformed row-count blocks return
`None`.
- Kill switch / disable path: no new flag; revert this commit.
- Unsafe override required: no.
- Qualification impact: no dependency or wire-format change.
- Rollback path: revert this commit to restore prior decoder behavior.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (decoder
docstring)
- [x] My changes generate no new warnings in the checks run
- [x] I have added tests that prove my fix is effective
- [x] New and existing focused unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Screenshots (if applicable)
Not applicable; decoder behavior is shown above.
## Additional Notes
The pre-push hook's documented bypass was used because Windows has no
`make`, and the Mac validation environment has no Rust toolchain. The
full native `ci-precheck` is unverified. Windows pre-commit mypy was
skipped after the same pinned full-project check passed on macOS; all
other pre-commit checks passed.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4271"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
|
||
|
|
312ffec8ea |
fix(router): preserve read protection for quiet sed options (#4521)
## Description With `HEADROOM_PROTECT_READS=1`, `sed -n '1,60p' file.py` achieves byte-exact read protection, but equivalent `-ne`, `-En`, `--quiet`, and `--silent` forms currently do not. The classifier previously checked for a literal `-n` token across the raw command string, which also caused it to mistake `-n` inside an expression argument as a quiet flag. This PR properly parses sed options before classifying whether a command is a read. It handles grouped flags and long options, respects script arguments and `--` delimiters, and excludes in-place edits (`-i`). GNU and BSD `-l` forms remain recognized. Addresses the quiet-option parsing portion of item 3 in #4478 (quoted redirect parsing and other items in that issue remain separate). Note that `sed -n1,80p` is invalid GNU sed syntax and remains unhandled. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Recognize quiet sed options using standard-library shell tokenization and robust option parsing. - Keep script arguments separate from options, and exclude in-place edits from protected reads. - Add command-classification unit tests and end-to-end `ContentRouter.apply()` regression coverage for all five quiet forms. ## Testing - [x] Unit tests pass (`pytest`): focused and related suites below - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`): native Linux pre-commit configuration - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Focused tests against unchanged main's router: 16 failed, 44 passed # This patch: 60 passed # Read detection, Responses read protection, read audit, maturation, lifecycle: 147 passed # make ci-precheck-rust (Rust 1.95.0): # cargo fmt --all -- --check; cargo clippy --workspace -- -D warnings; # cargo test --workspace 1659 passed, 3 ignored # make ci-precheck-python (including native extension rebuild): 177 passed, 4 skipped # ruff check . All checks passed! # ruff format --check . 1834 files already formatted ``` The native extension rebuild and import verification pass cleanly. The repository's unmodified pre-commit configuration passes on native Linux (WSL Ubuntu, Python 3.14.4), including mypy, Ruff, formatting, conflict checks, version checks, and commit-message hooks. On Windows, the default mypy hook reports seven POSIX-only `os` attribute errors in unchanged files (`fileperms.py` and `proxy/interceptors/astgrep.py`), matching unmodified main (#4520). Similarly, broader Windows test failures (18 failed, 5367 passed, 190 skipped, 2 errors) reproduce identically on main and stem from known platform differences (symlink privileges, SIGHUP handling, and Windows environment-variable size limits). ## Real Behavior Proof - Environment: Windows x86_64, Python 3.13.9, GNU sed 4.9, current-main Python sources with native `headroom._core` extension. ML and code compression are disabled to directly isolate the read-protection route. - Exact command / steps: Run the script below with main's router versus this patch. It executes real GNU sed against a temporary Python source file and passes each tool result through `ContentRouter.apply()`. - Observed result: The standard `-n` control is protected on both versions. On main, the other four forms miss `router:read_protected`. With this patch, all five forms correctly trigger read protection and preserve the 7,998-character output byte-for-byte. - Not tested: Focuses on command parsing and routing behavior. BSD option syntax is validated via parser unit tests; live proxy traffic and model calls were not required. Save as `repro_sed_quiet_reads.py`, then run `python repro_sed_quiet_reads.py`: ```python """Run real GNU sed output through ContentRouter, without a model or provider.""" import json import os from pathlib import Path import subprocess import tempfile os.environ["HEADROOM_OFFLINE"] = "1" os.environ["HEADROOM_TELEMETRY"] = "off" os.environ["HEADROOM_PROTECT_READS"] = "1" from headroom.transforms.content_router import ContentRouter, ContentRouterConfig from headroom.tokenizers.estimator import EstimatingTokenCounter sed = Path("C:/Program Files/Git/usr/bin/sed.exe") source = "\n\n".join( f"def compute_{i}(value):\n" ' """Compute a checked result for a caller."""\n' " result = value * 2\n" " if result < 0:\n" ' raise ValueError("negative result")\n' " return result\n" for i in range(45) ) with tempfile.TemporaryDirectory() as tmp: path = Path(tmp) / "sample.py" path.write_text(source, encoding="utf-8", newline="\n") for flags in [["-n"], ["-ne"], ["--quiet"], ["--silent"], ["-En"]]: output = subprocess.check_output( [str(sed), *flags, "1,400p", str(path)], text=True, encoding="utf-8" ) command = f"sed {' '.join(flags)} '1,400p' {path.as_posix()}" router = ContentRouter(ContentRouterConfig( enable_code_aware=False, enable_kompress=False, protect_analysis_context=False, protect_recent_reads_fraction=0.0, )) messages = [ {"role": "user", "content": "Read the module."}, {"role": "assistant", "content": None, "tool_calls": [{ "id": "read-1", "type": "function", "function": { "name": "Bash", "arguments": json.dumps({"command": command}) } }]}, {"role": "tool", "tool_call_id": "read-1", "content": output}, ] result = router.apply(messages, EstimatingTokenCounter(), protect_recent=0) actual = result.messages[2]["content"] print(json.dumps({ "flags": flags, "input_chars": len(output), "output_chars": len(actual), "byte_exact": output == actual, "transforms": result.transforms_applied, }), flush=True) ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: sed read classification; byte-exact routing changes when `HEADROOM_PROTECT_READS` is enabled (default remains unchanged). - Kill switch / disable path: `HEADROOM_PROTECT_READS=0` disables read protection. - Unsafe override required: no. - Qualification impact: equivalent quiet reads now retain their original output; no savings or task-quality measurement is claimed. - Rollback path: revert this patch; no persisted-state migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing focused unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes I implemented and verified this with Codex assistance for drafting and local tests. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
31f1304026 |
fix(privacy): turn off ONNX Runtime's Microsoft telemetry by default (#4602)
## Description Since 1.31.0, ONNX Runtime's Linux and macOS builds include Microsoft's 1DS telemetry client (`onnxruntime/core/platform/posix/telemetry_1ds.cc`, absent through 1.30.0). It is **on by default**. `OneDsTelemetry::Initialize` turns it off only when a CI variable (`GITHUB_ACTIONS`, `TF_BUILD`, ...), `ORT_RUNNING_UNIT_TESTS` or `ORT_DISABLE_TELEMETRY` is set. Otherwise ORT: - writes a device id under `~/Library/Application Support/Microsoft/DeveloperTools/.onnxruntime/` on macOS, and sends a SHA-256 of it with each event; - queues events in an `onnxruntime.db` next to it and uploads them to `https://mobile.events.data.microsoft.com/OneCollector/1.0`; - sends a `SessionCreation` event for each session, with `modelFileName`, `modelGraphName`, `modelProducerName`, `modelGraphHash`, `modelWeightHash`, execution provider ids and versions, and hardware vendor and device types. Headroom requires `onnxruntime>=1.24.0` on Python 3.11+ and currently resolves 1.31.1. The proxy loads ORT twice: the Python package (Kompress, image router) and the dylib the Rust core loads through `ORT_DYLIB_PATH` (Magika, embeddings). Each copy initializes the uploader. In my runs the proxy opened a TLS connection to Microsoft about 11 s after start, while the models loaded, and sent about 4 KB. `HEADROOM_OFFLINE` doesn't touch it, and `docs/content/docs/security-model.mdx` doesn't list this destination. CI doesn't see it because GitHub Actions sets `GITHUB_ACTIONS`, which ORT treats as CI. The same client is in the manylinux wheels (checked in `onnxruntime-1.31.1-cp311-cp311-manylinux_2_28_aarch64.whl`), so Docker deployments are affected too. ORT reads `ORT_DISABLE_TELEMETRY` once per copy, when that copy initializes, and then never creates the uploader or the device id. So `import headroom` now sets it with `os.environ.setdefault("ORT_DISABLE_TELEMETRY", "1")`, in the same place as the existing `ensure_ort_dylib_pinned()` hook, before anything in the package can load ORT. An explicit caller value wins, including `0` for anyone who wants ORT's telemetry on. Design choice: this is off always, not only under `HEADROOM_OFFLINE` (as `pin_crewai_telemetry_off` does for CrewAI). The events describe the user's models and hardware and go to a third party the user never chose. Headroom's own beacon is opt-out and documented, but this upload was neither documented nor tied to any Headroom setting. If maintainers prefer offline-only, `_OfflineEnvPin("ORT_DISABLE_TELEMETRY", "1")` in `headroom/offline.py` is a small change to this PR. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/_ort.py`: `disable_ort_telemetry_by_default()` sets `ORT_DISABLE_TELEMETRY=1` unless the variable is already set. - `headroom/__init__.py`: calls it at package import, next to `ensure_ort_dylib_pinned()`. - `crates/headroom-core/src/transforms/magika_detector.rs`: `init_ort_from_path` commits the environment with `.with_telemetry(ort_telemetry_opted_in(ORT_DISABLE_TELEMETRY))`. The `ort` crate's default is `true`, and it calls `EnableTelemetryEvents`. With this change, telemetry is off when the variable is unset or set to a value ORT treats as disabled, and on for any other explicit value, so `ORT_DISABLE_TELEMETRY=0` re-enables it for Rust-loaded models too. This only stops later events, after the uploader has started, so it is a backup for Rust-only callers; the environment variable is the real fix. A unit test covers each case. - `tests/test_transforms/test_ort_dylib.py`: the default is set; a caller's value (`0`) is kept; a subprocess `import headroom` leaves `ORT_DISABLE_TELEMETRY=1` with `onnxruntime` not yet imported. - `docs/content/docs/security-model.mdx`: a row for `mobile.events.data.microsoft.com` in the network egress table. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_transforms/test_ort_dylib.py -k importing # without the headroom/__init__.py change E AssertionError: assert {'ort_loaded'...'value': None} == {'ort_loaded'... 'value': '1'} 1 failed, 12 deselected $ pytest -q tests/test_transforms/test_ort_dylib.py tests/test_package_init_lazy.py tests/test_offline_egress_chokepoint.py \ tests/test_transforms/test_detect_fallback_1123.py tests/test_router_external_dispatch.py 223 passed, 2 skipped in 57.30s $ ruff check / ruff format --check on the changed .py files (ruff 0.16.8) All checks passed! $ mypy headroom/_ort.py headroom/memory/adapters/embedders.py:389: error: Unsupported operand types for - ("None" and "int") # same single error on main $ cargo clippy --all-features --tests -D warnings / cargo fmt --check -p headroom-core clean $ cargo test -p headroom-core --all-features --lib ort_telemetry 1 passed ``` ## Real Behavior Proof - Environment: macOS 27 (arm64), Python 3.11.17, onnxruntime 1.31.1 (CoreML, Azure and CPU providers), no CI variables set; `origin/main` at `fc1ba891` and this branch, each in its own worktree with the same `headroom-ai[all]` environment. - Exact command / steps: (1) a bare script that creates one CPU `InferenceSession` for `chopratejas/technique-router-onnx` and idles 60 s, with `ORT_DISABLE_TELEMETRY` unset, then set to `1`. (2) `PYTHONPATH=<worktree> python -m headroom.cli proxy --port 8799 --no-telemetry` with `HEADROOM_UPDATE_CHECK=off` and `ORT_DISABLE_TELEMETRY` unset, running until Kompress reports healthy. Both record every TCP flow of the process with `nettop -m tcp -p <pid> -L 0 -s 1`, and remote owners come from `whois`. - Observed result: (1) unset: one TLS flow to a Microsoft Corporation address (4.150.223.107:443, about 9.5 KB in and 4 KB out), with `onnxruntime.db` touched; set to `1`: no TCP flows, and the database is untouched. (2) `main`: 2 of 2 runs open a flow to Microsoft (74.151.238.96, 40.74.98.197) about 11 s after start, with about 9.5 KB in and 4 KB out; the collector rotates across Azure addresses. This branch: no flow to Microsoft in 6 runs; only the Anthropic upstream, plus litellm's GitHub pricing fetch in some runs. - Not tested: Linux and Windows at runtime. The Linux wheel contains the same client (`OneDsTelemetry` and the collector URL appear in `libonnxruntime.so.1.31.1`), but I only measured macOS. Windows uses ORT's separate ETW provider, which doesn't read this variable. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, process environment default - Stable/default behavior changed: yes; ONNX Runtime no longer uploads usage telemetry to Microsoft from Headroom processes unless `ORT_DISABLE_TELEMETRY` is set to a false value - Kill switch / disable path: `ORT_DISABLE_TELEMETRY=0` restores ORT's own default - Unsafe override required: no - Qualification impact: none; inference paths, providers and models are unchanged - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes A host that imports `onnxruntime` before `headroom` has already initialized ORT, so it has to set the variable itself. This is the same limitation `pin_litellm_cost_map_local` documents. In one early run, the fixed proxy did open the Microsoft flow. I couldn't reproduce it in six later runs, and its log reports a different computed `__version__` (`0.43.0-dev` against `0.42.1-dev` for the same tree), so the import state of that run is uncertain. Setting the variable before the process starts was clean in every run, for both the bare script and the proxy. --------- Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com> |
||
|
|
3c1fcbdfde |
fix(compaction): reject negative dictionary indices and non-array legends in expand_compacted (#4470)
## Description
`expand_compacted()` promises `None` when a densified block cannot be
losslessly reversed, but value-factored (`__dict:K`) cells were decoded
with raw Python indexing:
- a wire index of `-1` silently decoded to the **last** legend value
(data corruption, no error);
- a non-array legend (`@tag=42`, `@tag="ab"`, `@tag={"a":1}`,
`@tag=null`) raised `TypeError`/`KeyError` or was indexed character by
character.
Fixes #4280
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/compaction_codec.py`: reject a legend whose JSON
payload is not a list; reject negative dictionary indices before
indexing. Both paths return `None`, as the docstring already promises.
- `tests/test_transforms/test_compaction_codec.py`: regression tests for
the negative index, four non-array legend shapes, and a guard that valid
first/last indices, a quoted literal `"-1"`, `null`, and missing cells
in a dictionary column still decode.
## Testing
- [x] Unit tests pass locally
- [x] New tests added for the fix
```
$ python -m pytest tests/test_transforms/test_compaction_codec.py tests/test_densify_agent_mode.py -q
37 passed in 1.58s
# with the codec fix reverted, new tests:
5 failed, 1 passed, 17 deselected
$ ruff check / ruff format --check (touched files)
All checks passed! / 2 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, local build of headroom with
native `headroom._core`, branch rebased on upstream/main
- Exact command / steps: `expand_compacted('[1]{tag:string}
__dict:1\n@tag=["first","last"]\n-1\n')` and
`expand_compacted('[1]{tag:string} __dict:1\n@tag=42\n0\n')`, plus the
codec and densify-agent-mode test files above
- Observed result: before: `[{"tag": "last"}]` and `TypeError`; after:
`None` for both; full codec + agent densification suites pass (37 tests)
- Not tested: declared row-count mismatch validation (separate issue
#4271)
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, decoder correctness fix
- Stable/default behavior changed: only for malformed blocks, which now
return None instead of wrong data or an exception
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4470"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
4c3799fd81 |
fix(read-lifecycle): keep Reads fresh after a failed Edit or Write (#4461)
## Description `ReadLifecycleManager` counted every `Edit`/`Write` tool_use as a change to the file, so an Edit whose tool_result had `is_error: true` still turned earlier Reads of that file STALE and replaced their content with a marker, although the file never changed. This collects the tool_use ids whose tool_result is an error and leaves them out of the edit operations. Fixes #4460 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `ReadLifecycleManager._failed_tool_use_ids` (Anthropic `tool_result` blocks with `is_error` set) and skip those ids when building the edit operations in `_build_file_operation_index`. - Add two tests to `tests/test_transforms/test_read_lifecycle.py`: a failed Edit leaves the Read fresh; a later successful Edit still makes it stale. ## Testing Rebased onto upstream/main `3fef15d3` (2026-10-24): the new tests pass (25 passed); with only the source change reverted, 1 fails. The real-behavior run below was done on `e995eb9f`. Review round 1 ( |
||
|
|
cddd2fd60e |
fix(transforms): tolerate malformed assistant tool calls in ContentRouter.apply (#4459)
## Description A tool call with `"function": null` (or a non-object), `arguments` that decode to a list/string/number, or a `file_path` that is not a string raised out of `ReadLifecycleManager`, `ReadMaturationManager` and `ContentRouter._build_tool_name_map`. `TransformPipeline.apply` re-raised and counted a circuit-breaker failure, so three such requests in a row switched compression off for the cooldown (60 s default) for every request on that pipeline. The three scanners now skip the malformed call, as the surrounding `isinstance(tc, dict)` checks already do for a non-dict `tool_calls` entry. Fixes #4458 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/read_lifecycle.py` (`_build_tool_metadata`): skip a call whose `function` is not a dict; only read `file_path` / `offset` / `limit` when the decoded `arguments` is a dict; treat a non-string `file_path` as absent (it is later used as a dict key). Same non-string `file_path` guard in the Anthropic `tool_use` branch. - `headroom/transforms/read_maturation.py` (`_scan_activity`): same `function` and decoded-`arguments` guards; non-dict `input` and non-string `file_path` treated as absent in both branches. - `headroom/transforms/content_router.py` (`_build_tool_name_map`): skip a call whose `function` is not a dict. - `tests/test_transforms/test_malformed_tool_calls.py`: 9 malformed shapes (`function` null / string, `arguments` decoding to list / string / number, `file_path` list in OpenAI arguments and in `tool_use`, `path` dict, `input` list) run through each of the three entry points (27 cases). ## Testing Rebased onto upstream/main `3fef15d3` (2026-10-24): the new tests pass (27 passed); with only the source change reverted, 25 fail. The real-behavior run below was done on `e995eb9f`. Review round 1 ( |
||
|
|
e11afdbdf8 |
fix(smart-crusher): match audit-safe protected_patterns against unescaped text (#4294)
## Description Audit-safe mode (`SmartCrusherConfig(audit_safe=True, protected_patterns=[...])`, #1705) guarantees that rows matching a protected regex survive compression. The regexes run against `SmartCrusher._canon(row)`, which is `json.dumps(row, sort_keys=True, default=str)`. `json.dumps` defaults to `ensure_ascii=True`, so every non-ASCII character is escaped before the pattern sees it: ```python _canon({"city": "Kraków"}) # '{"city": "Krak\\u00f3w"}' re.search("Kraków", ...) # None ``` A pattern that contains any non-ASCII character can never match: accented names, CJK error text, currency symbols such as `€`. `_scan_protected_rows` returns `[]`, nothing is spliced back or verified, and the row is sampled out like any other. Nothing warns, because from the crusher's point of view there was nothing to protect. That's the exact failure audit-safe mode exists to prevent. Closes: N/A (no matching issue found) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: `_canon()` serializes with `ensure_ascii=False`, so patterns are matched against the real characters. `_canon` is also the identity key in `_splice_missing_protected` / verification, and both sides of that comparison go through it, so identity stays consistent. Rows without non-ASCII text serialize exactly as before. - `tests/test_transforms/test_smart_crusher_audit_safe.py`: `test_protected_patterns_match_non_ascii_text`, parametrized over `Kraków`, `错误` and `Zürich`, builds a 600-row array where row 437's `city` is the pattern. It checks that `_scan_protected_rows` finds that row and that it survives a real `_smart_crush_content` lossy sample (600 → ~16 rows). ## Testing - [x] Unit tests pass (`pytest`): `tests/test_transforms/`, `tests/test_garbled_compression_fixes.py`, `tests/test_smart_crusher.py`, `tests/test_smart_crusher_toin_attachment.py`, not the full suite - [x] Linting passes (`ruff check .`): on the changed files - [ ] Type checking passes (`mypy headroom`): not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New test on `main` (fix reverted, test kept) fails at the scan: ```text $ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q -k non_ascii E AssertionError: assert [] == [{'id': 437, ...', 'ms': 185}] E Right contains one more item: {'id': 437, 'city': 'Kraków', 'ms': 185} ``` With the scan assertion removed, so the crush outcome on `main` is visible, all three rows are sampled out: ```text E AssertionError: assert {'id': 437, 'city': 'Kraków', 'ms': 185} in [{'id': 0, 'city': 'Berlin', 'ms': 0}, {'id': 1, ...}, ...] E AssertionError: assert {'id': 437, 'city': '错误', 'ms': 185} in [...] E AssertionError: assert {'id': 437, 'city': 'Zürich', 'ms': 185} in [...] ======================= 3 failed, 11 deselected in 9.52s ======================= ``` With the fix: ```text $ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q ============================= 14 passed in 28.95s ============================== $ python -m pytest tests/test_transforms tests/test_garbled_compression_fixes.py tests/test_smart_crusher.py tests/test_smart_crusher_toin_attachment.py -q ================== 519 passed, 78 skipped in 67.42s (0:01:07) ================== $ ruff check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py All checks passed! $ ruff format --check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this branch (native `headroom._core` from the 0.41.0 wheel), `HEADROOM_OFFLINE=1` - Exact command / steps: ran the full `SmartCrusher.apply(messages, tokenizer)` transform (the path `test_apply_preserves_protected_rows_in_tool_message` uses for the proxy) with `SmartCrusherConfig(audit_safe=True, protected_patterns=["Kraków"], min_tokens_to_crush=10)` on an OpenAI tool message holding a 600-row JSON array (UTF-8, `ensure_ascii=False`, as tool output arrives) in which one row's `city` is `Kraków`. I parsed the compressed tool content and checked for that row, on `main` and on this branch. - Observed result: on `main` the protected row is dropped: 16 of 600 rows kept, `Kraków` row kept: False. On this branch it is kept: 17 of 600 rows, the 16 sampled plus the spliced protected row. ```text == main rows kept: 16 of 600 | Kraków row kept: False == this branch rows kept: 17 of 600 | Kraków row kept: True ``` - Not tested: the proxy end to end with an audit-safe config; patterns that deliberately match the escaped form (a regex like `Krak\\u00f3w` with an escaped backslash). I don't know of anyone relying on that, and such a pattern stops matching after this change. A regex escape like `Krak\u00f3w` means the character `ó` to `re`, so it never matched on `main` and now does. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for audit-safe users whose `protected_patterns` or protected rows contain non-ASCII text: those rows are now protected as documented. ASCII-only rows serialize identically, so existing audit-safe behaviour is unchanged. - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Docs unchecked: the audit-safe docs don't mention ASCII. This makes non-ASCII patterns behave the way the docs already imply. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
9762a14a40 |
fix(smart-crusher): preserve wrapped retrieval results in apply (#4149)
## Current-head security-gate integration Head `17ac399ac38930a265d8e5108c703110cd3e2ebb` incorporates upstream main `49751ec4` (#4178). The upstream change is Cargo.lock only; contributor logic and behavioral regressions are unchanged. This section supersedes older current-head/check-status references below; earlier runtime evidence is retained as historical implementation qualification. Normal commit/pre-push hooks passed without bypass: Rust formatting, warnings-as-errors Clippy, complete workspace tests, native editable rebuild/import verification, **181 critical Python tests**, and branch-range commitlint. Fresh local `cargo audit --file Cargo.lock` also passed (546 dependencies, no findings). Hosted checks for this new head must complete independently; local results are not a claim that all hosted checks are green. ## Description Closes #4147. Related: #3563, #3837 and the already merged router fix #3915. Close the remaining direct Python `SmartCrusher.apply()` parity gap: resolve originating wrapper calls with the existing shared `unwrap_tool_call_name` parser before applying the CCR retrieval exemption. This is not a duplicate proxy/router fix, a blanket wrapper exclusion, or a new content-based recovery heuristic. ## Type of Change - [x] Bug fix (non-breaking correction) - [x] Documentation update ## Changes Made - Resolve OpenAI originating function arguments and Anthropic tool-use input to their effective tool names in the existing call-ID index. - Preserve null-function handling, missing-ID/name behavior and the shared parser's conservative ambiguity/malformed-input rules. - Add four real-native regression cases: OpenAI/Anthropic × Hermes `tool_call`/`functions.exec`, each checking byte-exact retrieval output and a genuinely compressed ordinary wrapper-output control. - Remove old tautological compression assertions and retain named-retrieval consumer coverage. - Document direct `apply()` recovery identity, the need for originating call history, and the context-free `crush()` limit. Native compression, thresholds, CCR storage and the parser itself are unchanged. ## Testing - [x] Real native regression and shared wrapper safety suites pass - [x] Ruff passes on the two changed Python files - [x] Scoped mypy passes on the changed source file - [x] Actual rebuilt native API and real tokenizer exercised before and after ### Test Output ```text Before: 4 wrapped-recovery regressions failed at byte preservation. After: 65 passed, 1 warning in 126.20s Ruff: All checks passed! mypy: Success: no issues found in 1 source file ``` The warning is the existing Anthropic tokenizer approximation in an unchanged Hermes test. The scoped suite comprises the direct retrieval exemption tests plus Hermes and orchestrator-wrapper safety coverage; this is not a full-Python-suite claim. Normal commit/push hooks completed without bypass: Rust fmt, workspace Clippy with `-D warnings`, workspace Rust tests, native extension build/install/import verification, critical Python suite (**177 passed, 4 skipped, 1 warning in 25.48s**) and commitlint. The ordinary push succeeded with the genuine installed ORT library explicitly resolved. No native backend or gate threshold changed. ### Review follow-up verification The non-string-name caller correction is in `40fa055854b1c4b60da84f78d534976bc3fd40fb`; the reviewed thread is resolved. Verification against that exact published Python source: **83 passed, 1 existing Anthropic approximation warning in 16.44s**, across direct retrieval, Hermes and orchestrator safety suites. Changed-file Ruff and scoped mypy passed. A separate real native API smoke reused the genuine unchanged Rust extension and real OpenAI tokenizer: all four wire/wrapper cases returned exactly 27,490 recovery bytes, and all ordinary controls still compressed. An OpenAI `name=["Bash"]` returned its original `"ok"` result. Anthropic malformed names are tested at the identity-index boundary: the bundled tokenizer rejects such malformed tool-use names earlier, and this PR does not change or claim that tokenizer behavior. No mocked native backend, live provider request, or live retrieval-store claim. ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, latest-main base `976aa714ef3e19174dc0cb586cee7b20d064e4ef`, freshly built `headroom/_core.abi3.so`, real OpenAI `gpt-4o` tokenizer. Resolve the installed genuine `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`; no native fallback, mocked compressor or provider request. - Exact command / steps: run `python -m pytest -q tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py tests/test_hermes_tool_call_unwrap.py tests/test_orchestrator_wrapper_ccr_retrieve.py`, Ruff on the two changed files and mypy on `headroom/transforms/smart_crusher.py`. Separately call the production native-backed `SmartCrusher.apply()` with a 27,490-byte synthetic recovery result and an equally sized ordinary result; originating wrapper arguments invoke retrieval only for the recovery call. Exercise both wire formats and both wrappers. The isolated reproduction uses the existing test configuration `min_tokens_to_crush=0` unchanged before/after; production/native thresholds are not changed. - Observed result: before, all four recovery payloads became CCR-marked 505/509-byte outputs. After, all four returned exactly 27,490 bytes, matching the original byte for byte, with no new recovery CCR marker. Every ordinary wrapper result still compressed. Copied runtime results are below. - Not tested: a live provider session, proxy HTTP route, real client integration, or a complete retrieval loop. This is a real direct-library reproduction using synthetic returned recovery bytes, not a claim that a synthetic hash was retrieved from a live store. | Wire | Wrapper | Recovered bytes before | Recovered bytes after | Exact after | Ordinary output compressed after | |---|---|---:|---:|---|---| | OpenAI | `tool_call` | 505 | 27490 | Yes | Yes | | OpenAI | `functions.exec` | 505 | 27490 | Yes | Yes | | Anthropic | `tool_call` | 509 | 27490 | Yes | Yes | | Anthropic | `functions.exec` | 509 | 27490 | Yes | Yes | ## Runtime Rollout Safety - Rollout-managed feature(s): None; existing originating-call identity and CCR exemption only. - Minimum rollout channel: Stable; existing rollout controls unchanged. - Stable/default behavior changed: Direct `apply()` no longer re-compresses recognized wrapped recovery results. Unrelated wrapper output still compresses; malformed/ambiguous wrappers retain normal shared-parser behavior. - Kill switch / disable path: No new switch; revert this identity-map correction to restore prior direct-transform behavior. - Unsafe override required: None; runtime qualification used only synthetic local messages. - Qualification impact: Both wire formats, both wrapper families, real native output/control behavior and shared parser/router safety coverage qualified. Live-client/provider/proxy limits remain explicit above. - Rollback path: Revert the code/tests/docs to base `976aa714ef3e19174dc0cb586cee7b20d064e4ef`; no data/schema migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Shared wrapper parsing reused, no second identity convention - [x] Documentation updated - [x] Native and existing wrapper safety regressions pass - [x] No new source warnings, dependencies or native policy changes - [x] `CHANGELOG.md` untouched; Conventional Commit PR title used --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
e4a1ec19be |
refactor(transforms): deprecate unused compress_with_stats, extract_batch and align_for_cache (#4098)
## Description
`DiffCompressor.compress_with_stats`, `HTMLExtractor.extract_batch` and
`cache_aligner.align_for_cache` are public. The first two are methods on
classes exported from `headroom.transforms`, and the third's docstring
called it a stable public API. None is called anywhere in Headroom, the
plugins or the consumer repos; only tests call them. Each now emits a
`DeprecationWarning` that names the replacement.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `DiffCompressor.compress_with_stats` warns and points to
`DiffCompressor.compress`. Unlike `compress()`, it never wrote its CCR
entry to the compression store, so a retrieval marker in its output
could dangle. The docstring now says so.
- `HTMLExtractor.extract_batch` warns and points to calling
`HTMLExtractor.extract` per page.
- `align_for_cache` warns and points to
`CacheAligner(config).apply(messages, tokenizer)` and the
`stable_prefix_hash:` marker.
- The existing tests now assert the warnings. A new test checks that
`align_for_cache` warns at the caller and still returns the messages and
the hash.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_cache_aligner_detector_only.py tests/test_cache_aligner_prefix_stability.py tests/test_transforms/test_html_extractor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py
118 passed in 3.87s
$ ruff check / format: All checks passed; mypy: Success
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'align_for_cache(...);
DiffCompressor().compress_with_stats(...)'`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. Each deprecated call emits one DeprecationWarning
attributed to the caller and still returns its result. Output below.
- Not tested: Third-party callers; none is known.
```text
branch: cleanup/transforms-deprecate-unused-methods @
|
||
|
|
3db1a94443 |
fix(smart-crusher): keep string cells in json columns from decoding as other types (#4053)
## Description
The SmartCrusher lossless table (`csv-schema`) types a column `json`
when its cells mix types, for example `[26]{id:int,v:json?}`.
`_decode_cell` in `headroom/transforms/compaction_codec.py` reads every
cell of a `json` column with `json.loads` and only keeps the raw text
when that fails. The Rust encoder did not know the column type when it
rendered a string cell, so strings that spell a JSON value came back as
that value:
- the string `null` was written `"null"` and decoded as `None`
- the string `true` was written `true` (byte-identical to the bool) and
decoded as `True`
- the string `1` was written `1` and decoded as the int `1`
- `NaN` decoded as a float nan, and the string `"q"` (a string that
carries its own quotes) decoded as `q`
A string that parses as a JSON array or object (`[1]`, `{"a":1}`) never
reaches the formatter as a string: the classifier already un-escapes it
(`CellClass::StringifiedJson`), which is existing behavior and is not
changed here.
This changes the encoder only. In a column whose type tag is `json`, a
string cell is now written as a JSON string literal (then CSV-quoted)
when `json.loads` could parse it into something else. The decoder
already returns the string for that form, so it is not touched.
Everything else renders as before.
"Could parse into something else" is a conservative superset: after
trimming JSON white space (space, tab, newline, carriage return), the
text starts with `[`, `{`, `"`, `-` or an ASCII digit, or is exactly
`true`, `false`, `null`, `NaN` or `Infinity`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
-
`crates/headroom-core/src/transforms/smart_crusher/compaction/formatter.rs`:
`write_table` works out once which columns have type tag `json` and
passes that flag to `format_cell`. For a `Value::String` cell in such a
column, `is_ambiguous_json_text` decides whether it is rendered as
`csv_quote(serde_json::to_string(s))`. Non-string cells, the
`string`/`int`/`float`/`bool` columns, the JSON formatter and the
Markdown-KV formatter are unchanged. Bucketed tables go through
`write_table` too, so they get the same treatment.
- Rust tests in the same file:
`csv_formatter_json_column_quotes_strings_that_parse_as_json` (13
strings, exact wire output, with a `string` column as control),
`csv_formatter_json_column_leaves_other_strings_alone`, and
`csv_formatter_json_column_tells_strings_from_values`, which goes
through `compact` + `CsvSchemaFormatter` so the column type really comes
out as `json`, and checks that the string `"true"` and the bool `true`
no longer share a wire form.
- `tests/test_transforms/test_compaction_codec.py`: one test that
decodes a hand-written block using the new wire form next to the bare
forms for real values. It documents the contract the Rust change relies
on and passes with or without this change, since the decoder is not
changed.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` on the touched test file; `cargo fmt`
and `cargo clippy -D warnings` in CI)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# Rust, GitHub Actions run on our fork (toolchain 1.95.0)
https://github.com/SulimanAbdulrazzaq/headroom/actions/runs/37829441316
cargo fmt --all -- --check ok
cargo clippy -p headroom-core --all-targets -- -D warnings ok
cargo test -p headroom-core smart_crusher 345 passed; 0 failed
(342 on main in run 37827119554; the 3 new tests are the difference)
csv_formatter_json_column_leaves_other_strings_alone ... ok
csv_formatter_json_column_quotes_strings_that_parse_as_json ... ok
csv_formatter_json_column_tells_strings_from_values ... ok
# Python, 3.12
$ pytest tests/test_transforms/test_compaction_codec.py -q
17 passed (16 existing + 1 new)
$ ruff check tests/test_transforms/test_compaction_codec.py All checks passed!
$ ruff format --check tests/test_transforms/test_compaction_codec.py 1 file already formatted
```
## Real Behavior Proof
- Environment: Rust core built and tested on `ubuntu-latest` in GitHub
Actions with Rust 1.95.0 (the repository pin). Python 3.12.13 on Windows
11 for the decoder test and the reproducer.
- Exact command / steps: a script crushes 26 records `{"id": i, "v":
...}` whose `v` cycles through the strings `null`, `true`, `1`, `NaN`
and other values, decodes the result with `expand_compacted`, and
compares each row with its input. The encoded header is
`[26]{id:int,v:json?}`. In CI: `cargo test -p headroom-core
smart_crusher`.
- Observed result: before the change (the Rust core of the installed
`headroom-ai` 0.40.0 wheel), 8 of 26 rows differ, for example `'null'
(str) -> None`, `'true' (str) -> True`, `'1' (str) -> 1`, `'NaN' (str)
-> nan`. After the change the new Rust test shows the string `null` in a
`json` column is written `"""null"""` and the string `hello` is still
written `hello`; in the real-path test (through `compact` and
`CsvSchemaFormatter`) the rows are `1,"""null"""`, `2,null`,
`3,"""true"""`, `4,true`, `5,"""1"""`, `6,1`, `7,hello`, so each string
and each value has its own wire form. The Python test shows the decoder
returns `'null'`, `None`, `'true'`, `True`, `'1'`, `1`, `'NaN'` for such
a block.
- Not tested: the 26-row Python reproducer against a freshly built core
(the installed wheel on my machine is older and cannot see the Rust
change), and the Python wheel end to end. The Python decoder was not
changed.
### Maintainer native qualification (2026-10-08)
- Environment: Windows 11, Rust/MSVC, Python 3.12.13. This head contains
current `main`; contributor formatter logic was not modified.
- Exact commands: `cargo test -p headroom-core --lib
transforms::smart_crusher::compaction --locked`; `cargo build -p
headroom-py --features extension-module --locked`; then the PR's
`tests/test_transforms/test_compaction_codec.py` suite using that newly
built extension. A standalone native SmartCrusher/Python decoder probe
also encoded and decoded 100 records containing mixed
string/null/bool/numeric cells.
- Observed: **73 Rust compaction tests and 17 Python codec tests
passed**. The 100-record probe preserved every value and Python type,
including literal strings `null`, `true`, `false`, `1`, `NaN`,
`Infinity`, `-Infinity`, whitespace, quotes, empty strings, and embedded
newlines. No native encoder/decoder boundary was mocked.
- Scope: JSON-container-looking strings (`[1]`, `{"a":1}`) retain the
classifier's pre-existing `StringifiedJson` behavior. They are
explicitly outside this formatter repair; the author clarified that
limitation in the test. This PR fixes mixed-column primitive-literal
strings that actually reach the formatter, and does not claim a new
container-string round-trip contract.
- Hosted CI is tracked by the maintainer at this exact head; approval
remains pending until all applicable current checks pass.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: the lossless table text changes only
for string cells that look like JSON literals inside `json`-typed
columns. All other cells and all other column types render as before.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit. Older decoders read the new form
correctly, because it is a plain quoted JSON string, so nothing else
needs to change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Rust tests ran in CI on a fork, not locally, because the machine used
for development has no linker for the Rust toolchain. `mypy` and the
full Python suite were not run, only the touched test file. No other
test or fixture in the repository asserts the wire form of a string in a
`json` column (searched `crates`, `tests`, `docs` and `headroom` for
`json?` and `:json`).
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
f37980f7f7 |
fix(router): bound ML compression per request, not just per block (#3721)
## Description > Revised 2026-10-05 after review: the first ML call of a request is now capped by the remaining ML budget through Kompress's existing chunk-boundary time budget (no seed estimate), and budget-refused blocks pass through unchanged so their bytes stay stable on later turns. Third review: the same remainder now caps `RemoteKompressCompressor` too, over the whole HTTP request, and the remote call is declined when 50 ms or less is left. Closes #3711. Thanks to @gaaaork for the report — and for the follow-up correction, which is what made this findable. The corrected finding (it *is* the Kompress stage; `kompress` alone reproduces it) was right, and this PR answers the open question at the end of that comment. ### The question was: why did the size gate never fire? > `headroom_kompress_size_gate_total{outcome="within"} 2584` — only `within`. Either the gate is evaluated per chunk (each chunk under the ceiling, while their sum takes 70s), or it is not applied on this path. **It is the first.** The gate lives inside `_try_ml_compressor` (`content_router.py:4152`), and `_compress_mixed` calls that function **once per section** in a serial loop (`content_router.py:2815-2841`). So the gate only ever sees one section. It bounds **per-call size**; nothing bounded the **request**. Reproduced with the reported shape — a `tool_result` of prose blocks separated by small JSON objects — instrumented through the real router: ``` strategy : router:tool_result:mixed _try_ml_compressor calls : 8 size-gate outcomes : {'within': 8} <- no 'exceeded', exactly as reported ``` Eight separate model entries, eight independent gate decisions, no aggregate accounting. The 30s `compression_first_stage` budget is per-request while the only guard was per-section — a granularity mismatch. And there was no cumulative deadline either: the one existing helper (`_compression_deadline_seconds`) is used at a single site and only when `len(pending_tasks) == 1`, so the section loop ran unbounded. One note on the payload shape, since it cost me a false start: plain user text never reaches the ML stage at all (`router:protected:user_message`). The content has to arrive as a `tool_result` to reproduce — which is exactly what was reported. ### Why not just gate on the whole request That was the suggestion in the issue, and on its own it is not enough. It would fix the 1.4MB case while leaving a request of 100 sections at 40k tokens each equally able to hang — every section still passes any size ceiling. The guard needs to match the budget's *granularity*, not just its scope. So this adds a **request-scoped wall-clock ceiling**. Time is also the honest unit here: the estimator is already known to misjudge dense payloads (see the `#1171` comment above the gate), and the symptom being protected against is wall-clock, not tokens. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_ml_stage_deadline_seconds()` — `HEADROOM_ML_STAGE_DEADLINE_SECONDS`, default **15s**, deliberately under the 30s `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` so we degrade to a *partial* compression **before** the executor abandons a non-preemptible worker. Losing some savings beats losing the request and then quarantining the stage for everything behind it. `0` restores the previous unbounded behaviour. - Enabled by `apply()` — the one place that owns a whole request — and carried on `_PerRequestRuntimeState`, so the existing `copy_context()` fan-out (documented under #3486/#3556) propagates it to worker threads for free. The budget counts only time spent **inside** the ML compressor (cumulative, charged in a `finally`), and is clamped to half of `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`. - Admission: a block is refused when its projected cost (this request's worst measured seconds-per-token) exceeds what is left. Every admitted call — **including the first, which has no measurement** — is also **capped**: the remainder goes to Kompress as `_time_budget_cap_seconds`, lowering its existing per-call time budget, which Kompress already checks at every chunk boundary. So no call can run more than one chunk past the budget, and parallel workers each end by their admission time plus the remainder. - Remote Kompress (`HEADROOM_KOMPRESS_ENDPOINT`) gets the same cap. httpx timeouts are per phase, so the cap is applied to the request as a whole: the POST runs on a small worker pool (16) and is awaited with one overall deadline. Admission is bounded: a permit per worker is taken before submit, waiting at most until the caller's deadline, so a saturated call fails open unchanged and its payload is never enqueued. The worker holds its permit until it actually finishes, and it finishes by an absolute deadline: per-phase timeouts are capped to the time left and the body is streamed and cut off at the deadline, so a slow-drip response cannot hold a worker. At or below 50 ms left, no request is sent and the block passes through unchanged. With no request budget (direct callers) the call and its 20s timeout are exactly as before. - Blocks refused by the budget (or cut off by the cap) **pass through unchanged**. Not the size gate's lossy fallback: the refusal depends on the request, not the content, and the router caches that verdict, so raw bytes are the only ones that stay stable on later turns (prompt-cache safety). - Recorded as `headroom_kompress_size_gate_total{outcome="deadline"}`, so this is visible in the metric you were already reading, next to `within`/`exceeded`. - Direct `compress()` callers (tests, `/v1/compress`) stay unarmed and keep the old behaviour. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed The original four tests, using a stub in place of Kompress so they assert the *budget* rather than ONNX latency (the real model is not available in CI, and the bug is about how many times a slow stage is entered): - `test_size_gate_alone_cannot_bound_a_request` — pins the precondition: many `within`, zero `exceeded`, repeated model entry. Passes on `main`; it *is* the bug. - `test_ml_deadline_stops_further_ml_work_once_the_budget_is_spent` — fails on `main`. Measures the saving by running the same payload with the ceiling disabled and asserting strictly more model calls. - `test_deadline_zero_restores_previous_unbounded_behaviour` - `test_direct_compress_callers_stay_unarmed` — fails on `main`. Verified fail-before/pass-after by reverting only `content_router.py` and keeping the tests. Regression sweep `-k "router or kompress or mixed or compress"`: **2555 passed**, 10 failed — and those same 10 fail identically on `origin/main` with this file reverted, so **zero new failures**. Ruff 0.16.7 (the CI pin) clean. Added after review (each fails with the source reverted and the tests kept, passes with it): - `test_non_ml_prework_does_not_consume_the_ml_budget`, `test_a_slow_call_cannot_be_admitted_against_a_budget_it_will_overrun`, `test_budget_is_clamped_below_the_timeout_that_quarantines_the_stage`, `test_a_failing_ml_call_still_draws_down_the_budget` (first review). - `test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against` — first block's worst case is 3s under a 1s executor timeout (0.5s ML budget); asserts the first call is handed a cap ≤ 0.5s and total ML time stays within budget + one chunk. - `test_blocks_refused_by_the_budget_pass_through_unchanged` — refused block is byte-identical even with TextCrusher available. - `test_time_budget_cap_stops_a_call_and_returns_it_unchanged`, `test_time_budget_cap_bounds_the_batched_path_too` (real `KompressCompressor`, fake model/clock) — the cap stops at the first chunk boundary past it and returns the block unchanged, on both the single and batched paths. - `test_cold_remote_call_cannot_overrun_a_tiny_ml_budget` (third review, mirrors the reviewer's probe): real `RemoteKompressCompressor` over `httpx.MockTransport` with a 100 ms endpoint, 10 ms ML budget, 40 ms outer timeout, via public `apply()`. Asserts ML time < 40 ms, no HTTP request sent, prose blocks byte-identical. - `test_cold_remote_call_is_bounded_as_a_whole_by_the_ml_budget`: 200 ms budget, endpoint that hangs for 5 s. Asserts the first request's per-phase timeouts are lowered to the budget, total ML time ≤ budget + 0.1 s, blocks unchanged. - `test_remote_kompress_keeps_its_default_timeout_without_a_budget`: uncapped call keeps the 20s timeouts and does not hop threads. - `test_saturated_remote_calls_fail_open_without_queueing_their_payloads` (fourth review, the reviewer's repro): 16 workers held by requests whose 200 ms callers timed out, then 5 calls with 60 ms caps. Each fails open unchanged within its cap, the executor queue stays empty, none reaches the endpoint, and capacity returns once the held requests end. - `test_slow_drip_response_is_cut_off_at_an_absolute_deadline`: a body dripping a chunk every 50 ms for 2 s under a 200 ms cap; the worker stops at the deadline and returns its permit well before the drip ends. - `test_permit_is_held_until_the_worker_finishes_not_the_caller`. ### Test Output ```text $ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest -q \ tests/test_content_router*.py tests/test_transforms/test_content_router*.py \ tests/test_transforms/test_kompress*.py tests/test_kompress*.py tests/test_force_kompress_all.py \ tests/test_remote_kompress_dropin.py tests/test_transforms_content_router.py tests/test_router_*.py \ tests/test_onnx_router_threads.py tests/test_proxy_disable_kompress.py tests/test_proxy_per_provider_kompress.py 486 passed, 2 skipped, 1 warning in 19.42s # with only the source hunks reverted: FAILED tests/test_content_router_ml_stage_deadline.py::test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against FAILED tests/test_content_router_ml_stage_deadline.py::test_blocks_refused_by_the_budget_pass_through_unchanged FAILED tests/test_transforms/test_kompress_deadline.py::test_time_budget_cap_stops_a_call_and_returns_it_unchanged FAILED tests/test_transforms/test_kompress_deadline.py::test_time_budget_cap_bounds_the_batched_path_too ``` ## Real Behavior Proof - Environment: local macOS, Python 3.14, branch merged with current `main`. - Exact command / steps: the pytest invocation above; the reporter's payload shape (prose blocks separated by small JSON objects inside a `tool_result`) driven through the real `ContentRouter.apply()` with a stub in place of the ONNX model. - Observed result: ML time stays within the budget (plus at most one chunk) even when the first block's worst case exceeds the executor timeout; refused blocks are byte-identical. - Not tested: real Kompress ONNX latency on the reporter's hardware (model not downloaded locally); a real remote Kompress endpoint (covered network-free via `httpx.MockTransport`). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: stable. - Stable/default behavior changed: yes — a request now spends at most 15s (or half of `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`, if lower) inside ML; blocks past that are sent uncompressed. - Kill switch / disable path: `HEADROOM_ML_STAGE_DEADLINE_SECONDS=0` restores the previous unbounded behaviour. - Unsafe override required: no. - Qualification impact: none expected; only requests whose ML stage exceeds the budget see less compression. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes ### What this does not claim to fix The ~70s figure is still the reporter's measurement, not mine — I could not reproduce it locally because the Kompress model is not downloaded on this machine, so I discarded my own latency numbers as meaningless. What this PR fixes is the *consequence*: a request that spends too long in ML now degrades to partial compression instead of blowing the budget and quarantining the stage. @gaaaork — you offered to test a patch. If you can run this with `HEADROOM_ML_STAGE_DEADLINE_SECONDS` at the default 15, the stage should return in ~15s with partial compression and no `CompressionQuarantinedError`, and `headroom_kompress_size_gate_total` should start showing `deadline`. The separate question of why one 40k-token section takes seconds on your hardware is still worth chasing — this bounds the blast radius, it does not make the model faster. Documentation: no docs page covers `HEADROOM_ML_STAGE_DEADLINE_SECONDS` yet; mypy not run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3721" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
84f0e84dcc |
feat(compress): live-agent densify mode, lossless and cache-safe (#1402)
## Description Revised 2026-10-05 after review: the codec now decodes main's native null encoding (bare `null` = null, empty cell = missing key) with no formatter change, and agent-mode `tokens_after` is recounted with `count_messages` after factoring and reverts. Adds an opt-in agent densification mode that only shrinks content, verifies reversibility message by message, and reverts any output that cannot be proven lossless. It also adds a pure-Python inverse decoder and reversible low-cardinality value factoring for the existing `csv-schema` format. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `compress(mode="agent")` and `densify()` with verified-lossless fallback. - Add public `expand_compacted()` and `is_compacted()` helpers. - Decode `csv-schema` cells exactly as main's native formatter (#3665) encodes them: empty cell = missing key, bare `null` = null, quoted `""` / `"null"` = strings. This PR no longer changes the Rust formatter. - Recount `tokens_after` with the pipeline tokenizer's `count_messages` after agent-mode value factoring and lossless reverts. - New `CompressConfig` / `CompressResult` fields sit after `diagnostics`, so existing positional construction is unchanged. - Add reversible value factoring only when it reduces output size. - Preserve current `main` dependency placement, security pins, CCR gating, and Windows ONNX initialization. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_transforms/test_compaction_codec.py + tests/test_densify_agent_mode.py: 30 passed (native extension built from this branch) tests/test_compress_*.py + tests/test_transforms/ + agent-mode suite: 577 passed, 67 skipped cargo test -p headroom-core --lib compaction: 73 passed ruff check / ruff format --check on touched files: passed ``` ## Real Behavior Proof - Environment: macOS, locked `uv` environment, native extension built from this branch, `HEADROOM_*` env scrubbed - Exact command / steps: public `headroom.densify()` on a tool result of 40 records with three repeated long paths, some `null` paths and some literal `"null"` strings - Observed result: `factored: True | reverted: 0 | lossless: True`; `tokens_before: 1269 | tokens_after: 407 | count_messages(result.messages): 407`; `expand_compacted(content) == records: True` - Not tested: live multi-turn prompt-cache hit rate and proxy wiring; agent mode remains an opt-in library API ## Runtime Rollout Safety - Rollout-managed feature(s): none; `mode="agent"` / `densify()` is an opt-in library API - Minimum rollout channel: not applicable (library opt-in, not wired into the proxy) - Stable/default behavior changed: no; default `compress()` and native formatter output are unchanged - Kill switch / disable path: do not pass `mode="agent"` / do not call `densify()` - Unsafe override required: no - Qualification impact: none for the proxy path - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable: this PR does not change a user interface. ## Additional Notes The branch was rebuilt on current `main`. The older dependency-relocation portion was intentionally omitted: current `main` keeps those dependencies available and includes a security exclusion for the compromised `ast-grep-cli` release. The current CCR gate already supersedes the PR's older gate implementation and was retained. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/1402" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ffc359976b |
fix: protect file reads in chained shell commands (#2668)
## Description
`_is_read_command` decides whether a shell command's output is file
content the agent will patch from. Output classified as a read is passed
verbatim; everything else is eligible for lossy Kompress compression.
It inspected only the **first program**, after peeling `cd`/`sudo`/`rtk`
wrappers. Agents routinely batch a read behind other work, so every
chained read fell through to the compression path despite
`HEADROOM_PROTECT_READS=1`:
```
wc -l a.py b.py && sed -n '1,60p' c.py # parsed as `wc` -> not a read
grep -n X f.py | head -40; head -80 g.py # parsed as `grep` -> not a read
ls -la || cat f.py # parsed as `ls` -> not a read
```
Kompress operates on words and rejoins the survivors, so file content
arrives with function words missing. That is precisely the failure read
protection exists to prevent: the agent either re-reads to recover
detail (turn inflation) or builds an exact-match `old_string` from text
that no longer matches the file on disk.
Reported by a Headroom desktop user who saw source content garbled while
the `Read` tool path stayed byte-exact — which is what pointed at the
shell-command classifier rather than the tool exclusion list.
No linked issue; reported directly.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_is_read_command` now scans every `;` / `&&` / `||` segment and
protects if **any** segment is a read, instead of matching only the
first program.
- Extracted the existing single-command logic into `_segment_is_read`,
unchanged in behavior for a non-chained command.
- Added `_command_segments`, which splits on `;` / `&&` / `||` and
reduces each segment to its first pipeline stage.
- Pipelines are deliberately **not** split: a downstream stage consumes
the previous stage's output rather than a file, so `grep … | head -40`
stays compressible while `cat f.py | head -40` is still caught via its
first stage.
- The whole-string write/heredoc guard is unchanged, so a heredoc body
containing `;` cannot be split into a bogus read segment.
- Two latent false positives close as a consequence of per-segment
evaluation: a `-n` flag in a sibling segment no longer qualifies a bare
`sed`, and the lockfile carve-out now applies to the segment naming the
lockfile rather than the whole command.
- New test file `tests/test_transforms/test_read_command_detection.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — see Additional Notes
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_transforms/ \
tests/test_observed_wire_shapes.py tests/test_content_router_exclude_tools.py -q
======================= 408 passed, 62 skipped in 13.46s =======================
$ uv run --frozen --extra dev pytest tests/test_transforms/test_read_command_detection.py -q \
--cov=headroom.transforms.content_router --cov-report=term-missing --cov-branch
30 passed in 0.45s
# no uncovered lines or partial branches remain in the changed range (614-654)
$ uvx ruff check headroom tests --output-format concise
All checks passed!
$ uvx ruff format --check headroom/transforms/content_router.py \
tests/test_transforms/test_read_command_detection.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom
Success: no issues found in 512 source files
```
## Real Behavior Proof
- Environment: macOS (Darwin 24.6.0), Python 3.10.18, worktree branched
from `upstream/main` @ `28aa53dc`; plus a live Headroom desktop proxy
running `headroom-ai` 0.32.1 with Claude Code pointed at
`127.0.0.1:6767`.
- Exact command / steps: (1) pre-fix on the live proxy, ran `grep -n
"Read" transforms/compression_policy.py | head -40; echo "=== POLICY
HEAD ==="; head -80 transforms/compression_policy.py` through Claude
Code's Bash tool; (2) ran `ContentRouter().apply(...)` with
`HEADROOM_PROTECT_READS=1` over a `role: tool` bash observation produced
by `wc -l app/services/inventory.py && sed -n '1,200p'
app/services/inventory.py` (11,007 chars of plain Python source, aged
behind 20 follow-up turns so the positional "recent code" guard no
longer covers it), against both the pre-fix and post-fix
`content_router.py`.
- Observed result: step 1 returned the file's own docstring with
function words dropped and a retrieval marker appended; step 2 went from
no protection transform at all (pre-fix — the observation enters the
compression path) to `router:read_protected` (post-fix). Verbatim output
below.
- Not tested: byte-level Kompress corruption inside the repo venv (the
ML model is not downloaded there — `Kompress model not ready; requests
will not be compressed` — so step 2 shows the routing change and step 1
shows the corruption); `magika` unavailable in that venv (`no pip
onnxruntime native library was found`) so detection fell through to the
regex tier during step 2, which affects which compressor a released read
routes to but not whether a read is protected; Windows shell command
strings; fleet-scale traffic replay, so the savings impact of
newly-protected reads is unquantified.
### Proof detail
Step 1 — marker appended to the returned file content:
```
[478 items compressed to 305. Retrieve more: hash=5d959a193f8641579c741ad5]
```
On disk:
> Hand-mirrored port **of**
`headroom_core::compression_policy::CompressionPolicy` (Rust). **The**
Rust crate **is the** source **of** truth; **this** module exists **so
the** Python proxy's `TransformPipeline` (which still runs
`CacheAligner` **and other** detector-only transforms) **can** read
**the** same per-mode flags **the** Rust dispatcher does.
As received by the model:
> Hand-mirrored port
`headroom_core::compression_policy::CompressionPolicy` (Rust). Rust
crate source truth; module exists Python proxy's `TransformPipeline`
(which still runs `CacheAligner` detector-only transforms) read same
per-mode flags Rust dispatcher does.
Step 2 — routing branch taken by the tool observation:
```text
##### BEFORE (upstream/main content_router.py) #####
command: wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars: 11007
file content verbatim: True
transforms: ['router:protected:user_message', ... x10] <- no read protection
##### AFTER (this PR) #####
command: wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars: 11007
file content verbatim: True
transforms: ['router:read_protected', 'router:protected:user_message', ... x10]
```
Before the fix the observation carries no protection transform at all
and enters the compression path. After the fix it is
`router:read_protected`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs describe this internal classifier; its docstrings are
updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — no UI surface.
## Additional Notes
- `ruff check .` reports 4 errors, all pre-existing on `upstream/main`
and all in `plugins/headroom-oauth2/` (`RUF022`, `BLE001` x2, `S110`).
Untouched by this PR, so the box is left unchecked rather than claimed.
`ruff check headroom tests` is clean.
- Trade-off: the write/heredoc guard stays whole-string rather than
per-segment. That keeps `cat a.py && echo x > b` unprotected, which is
the pre-existing behavior — chosen so a heredoc body containing `;` can
never be split into a segment that looks like a read. Narrowing it
further is a separate change.
- Follow-up for the reporter's second issue (not in scope here): passive
traffic learning has no off switch, and the desktop starts the proxy
with `--learn` unconditionally.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
fef99cc5b4 |
feat(compress): add per-message compression diagnostics (#3058)
## Description Adds `CompressConfig.diagnostics` (bool, default `False`) and a companion `HEADROOM_DIAGNOSTICS=1` environment variable that enable per-message decision logging on every call to `compress()`. When active, `CompressResult.diagnostics` is a list of `MessageDecision` objects — one per input message — each describing: | Field | Description | |---|---| | `message_index` | Zero-based position in the original messages list | | `role` | `"user"` / `"assistant"` / `"tool"` / `"system"` / … | | `tokens_before` | Token count before the ContentRouter acted on this message | | `tokens_after` | Token count after (equals `tokens_before` for protected/passthrough) | | `action` | What was done: `"compressed:kompress:0.72"`, `"protected:user_message"`, `"passthrough:frozen"`, `"passthrough:small"`, etc. | Closes #2855. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/config.py`** — add `MessageDecision` dataclass; add `message_decisions: list[MessageDecision]` field to `TransformResult` - **`headroom/compress.py`** — add `CompressConfig.diagnostics`; add `CompressResult.diagnostics`; derive `collect_diagnostics` from config + env var; pass to pipeline; collect from `TransformResult` - **`headroom/transforms/pipeline.py`** — accumulate `message_decisions` from each transform result - **`headroom/transforms/content_router.py`** — instrument every decision branch in `apply()` with `_diag[i] = "action:..."` strings when `collect_diagnostics=True`; build `MessageDecision` list after Pass 3 (zero overhead when flag is off) - **`headroom/__init__.py`** — export `MessageDecision` in `__all__` and lazy-import map - **`tests/test_compress_diagnostics.py`** — 14 tests covering the full `compress() <-> pipeline` contract via a mock pipeline ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_compress_diagnostics.py -v ========================= test session starts ========================== collected 14 items tests/test_compress_diagnostics.py::test_message_decision_dataclass PASSED tests/test_compress_diagnostics.py::test_message_decision_importable_from_headroom PASSED tests/test_compress_diagnostics.py::test_transform_result_has_message_decisions_field PASSED tests/test_compress_diagnostics.py::test_transform_result_message_decisions_populated PASSED tests/test_compress_diagnostics.py::test_compress_config_diagnostics_defaults_false PASSED tests/test_compress_diagnostics.py::test_compress_config_diagnostics_can_be_set PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_none_by_default PASSED tests/test_compress_diagnostics.py::test_compress_passes_collect_diagnostics_when_config_true PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_list_when_config_true PASSED tests/test_compress_diagnostics.py::test_compress_result_diagnostics_contains_decisions PASSED tests/test_compress_diagnostics.py::test_env_var_enables_diagnostics PASSED tests/test_compress_diagnostics.py::test_env_var_absent_keeps_diagnostics_none PASSED tests/test_compress_diagnostics.py::test_env_var_overrides_config_false PASSED tests/test_compress_diagnostics.py::test_no_diagnostics_when_optimize_false PASSED ========================== 14 passed in 0.42s ========================== $ ruff check . All checks passed! ``` ## Real Behavior Proof - Environment: Python 3.12, headroom dev install - Exact command / steps: `python -m pytest tests/test_compress_diagnostics.py -v` - Observed result: 14 tests pass; diagnostics list is populated when `CompressConfig(diagnostics=True)` or `HEADROOM_DIAGNOSTICS=1` - Not tested: ContentRouter integration tests (require compiled `headroom._core` extension — run in CI) ## Runtime Rollout Safety - Rollout-managed feature(s): None — diagnostics are opt-in via `CompressConfig.diagnostics=True` or `HEADROOM_DIAGNOSTICS=1` - Minimum rollout channel: N/A (additive, opt-in only) - Stable/default behavior changed: No — `CompressConfig.diagnostics` defaults to `False`; `CompressResult.diagnostics` is `None` unless explicitly requested - Kill switch / disable path: Simply omit `diagnostics=True` / unset `HEADROOM_DIAGNOSTICS` - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this PR; no schema or data migration involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — no UI changes. ## Additional Notes The `collect_diagnostics` flag incurs zero overhead in the hot path when not set: the `_diag` dict in `ContentRouter.apply()` is never allocated and no `MessageDecision` objects are constructed. Signed-off-by: Radha Krishnan P <gingeekrishna@gmail.com> --------- Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech> Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com> |
||
|
|
8f3d677354 |
fix(kompress): degrade when a native dep is installed but unloadable (#3133)
## Description `_is_pytorch_available()` and `_is_onnx_available()` catch only `ImportError`. An installed-but-unloadable native dependency raises `OSError` instead: on Windows without the MSVC redistributable, `import torch` fails with `WinError 126` loading `c10.dll`. So a probe whose contract is to return a bool raises instead, and `is_kompress_available()` raises with it. `content_router._get_kompress()` guards its call with `except ImportError` too, so the `OSError` propagates past both guards into the request path rather than degrading to "kompress unavailable". Because the compressor is only cached on success, this repeats on every request, and Python drops failed modules from `sys.modules`, so each one pays a full `torch` import that ends in the same error. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_is_pytorch_available()` / `_is_onnx_available()` now catch `(ImportError, OSError)`, so an installed-but-unloadable native dependency degrades to "unavailable" instead of raising out of a bool-returning probe. - Both probes are `lru_cache`d. The answer cannot change within a process, and nothing else memoises a *failing* import, so without this the full failing `torch` import is retried on every request. - Added `test_unloadable_native_dep_degrades_instead_of_raising`, which simulates the Windows condition (onnxruntime absent so the `or` cannot short-circuit, torch raising `OSError`) and asserts both the graceful `False` and the single import attempt. - `test_module_import_without_torch` now calls `cache_clear()` around its assertion, which memoising makes necessary. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev pytest tests/test_transforms/ -q ======================= 384 passed, 67 skipped in 18.13s ======================= $ uvx ruff check . All checks passed! $ uv run --extra dev mypy headroom/transforms/kompress_compressor.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Reproduced from a Windows Server 2022 (10.0.20348) box where all three of vcruntime140.dll, vcruntime140_1.dll and msvcp140.dll are absent from System32, so the installed torch cannot load its DLLs. The behaviour was then isolated on macOS 26.5 / CPython 3.10.18 by simulating that condition, since the failure is an exception type and not platform-specific logic. - Exact command / steps: On the Windows box, the packaged proxy logged `OSError: [WinError 126] ... Error loading "...\torch\lib\c10.dll"` escaping `_is_pytorch_available` at kompress_compressor.py:489 via `_load_kompress` at :1002. To isolate, patch `builtins.__import__` so `onnxruntime`/`optimum` raise `ImportError` (absent) and `torch` raises `OSError` (installed, unloadable), then call `is_kompress_available()` directly and through the same `try/except ImportError` guard `content_router._get_kompress()` uses, counting torch import attempts across 3 requests. - Observed result: Before the change, `is_kompress_available()` raised `OSError` rather than returning a bool, the error escaped the caller's `except ImportError` on 3/3 requests, and torch was imported 4 times. After the change, it returns `False`, 0/3 requests raise, and torch is imported once. This is covered by the new test. - Not tested: The fixed build has not been run end-to-end on the Windows box, because installing the MSVC redistributable there would remove the very condition under test. Whether kompress then loads successfully on Windows once the redistributable is present is out of scope for this PR, which only makes its absence degrade correctly. ## Runtime Rollout Safety - Rollout-managed feature(s): None. This is an availability probe on an existing optional dependency, not a gated feature. - Minimum rollout channel: N/A - Stable/default behavior changed: No behaviour change where the dependencies import cleanly. The probe returns the same value; only the installed-but-unloadable case changes, from raising to returning False. - Kill switch / disable path: N/A. Existing kompress controls (HEADROOM_KOMPRESS_ENDPOINT, the router's own gates) are untouched. - Unsafe override required: No - Qualification impact: None expected. The one existing test that changed did so because memoising the probe makes an explicit cache_clear() necessary; its assertion is unchanged. - Rollback path: Revert the commit. There is no persisted state or migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
c7ebaedd74 |
fix(proxy): preserve OpenCode Headroom retrieve results (#3782)
## Description When OpenCode reports the retrieve tool as `headroom_headroom_retrieve`, its result could miss the existing retrieve exclusion and be compressed again. This adds that exact name to the shared alias mapping so the existing ContentRouter pass-through guard can preserve the retrieved content. Fixes #3780 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Map `headroom_headroom_retrieve` to the existing `headroom_retrieve` exclusion. - Add alias coverage and a ContentRouter regression test for the OpenCode name. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest -q tests/test_tool_name_aliases.py: 5 passed Changed-file Ruff checks, format check, compileall, and git diff --check: passed python -m pytest -q tests/test_transforms/test_content_router_ccr_retrieve_exemption.py: collection blocked by ModuleNotFoundError: headroom._core Full pytest, repository-wide Ruff, mypy, and live OpenCode/proxy verification were not run. ``` ## Real Behavior Proof - Environment: local submitter checkout; the native `headroom._core` extension is unavailable. - Exact command / steps: `python -m pytest -q tests/test_tool_name_aliases.py` after the alias change. - Observed result: 5 tests passed, including the exact OpenCode tool name and negative alias checks. - Not tested: the ContentRouter regression could not be collected locally; the issue-reported cache/token OpenCode runtime was not independently reproduced after this change. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: N/A. - Stable/default behavior changed: the matching OpenCode retrieve name now follows the existing exclusion behavior. - Kill switch / disable path: None. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The focused alias test passes, but the end-to-end ContentRouter test and live runtime behavior remain unverified in this environment. |
||
|
|
21d166b77a |
test: stop three sources of cross-test pollution on main (#3774)
## What
Three tests fail on `main` in a full-suite run and pass when their file
is run alone. Each is state one test leaves behind that a later test
reads. This PR isolates all three. **No production code changes.**
| Failing test | Left behind by | Fix |
|---|---|---|
| `/readyz` readiness tests reporting kompress `ready: true` |
`test_kompress_failsafe.py::test_canary_disabled_via_env` calling
`compressor.preload()`, which writes the model into the module-level
`_kompress_cache` | autouse fixture snapshots/restores `_kompress_cache`
|
| `test_runtime_log_refuses_a_symlinked_path` |
`tests/gateway/test_compress_turn_seam.py` leaking an
`_OwnerOnlyRotatingFileHandler` on the `headroom` logger | autouse
fixture detaches + closes any handler named `_PROXY_LOG_HANDLER_NAME` |
| 2 × `test_dense_line_elider.py` | the optional `html` extra
(`trafilatura`) being installed, so `HTMLExtractor` wins routing instead
of the elider | both tests pin `enable_html_extractor=False` |
The third is not pollution between tests but pollution from the
environment: CI does not install the `html` extra, so CI has never seen
these two fail. Anyone with `pip install headroom[html]` in their venv
does.
## How to verify
Before:
```
pytest tests/test_kompress_failsafe.py tests/test_proxy_health.py # health tests fail
pytest tests/gateway/test_compress_turn_seam.py tests/test_install/test_runtime.py # symlink test fails
pytest tests/test_transforms/test_dense_line_elider.py # fails iff trafilatura installed
```
After: all pass, and each file still passes alone.
Slices run locally (venv `bin` on PATH):
- `-k "kompress or health or elider or breakpoint or transform"` → 1040
passed, 17 skipped
- `-k "install or savings or stats or memory or learn or router"` → 2960
passed, 59 skipped
## Behaviour change
None for users. For test authors: the kompress model cache and the proxy
log handler no longer survive a test, so a test that *relied* on a
previous test having preloaded the model would now have to preload it
itself. No test in the suite does.
## Two defects found while debugging this, deliberately not fixed here
1. **`HTMLExtractor` emits lossy output with no CCR retrieval marker.**
A 3232-char HTML tool result is reduced to `"hi there"` —
`<script>`/`<style>` stripped — and booked as 99.8% savings with no
marker, so the original cannot be retrieved. Present since
|
||
|
|
c1fc84ade6 |
fix(smart-crusher): disclose dropped scalar-array items with a retrievable ccr sentinel (#3692)
## Description
On default config, `SmartCrusher` silently truncated long
string/number/mixed arrays down to `max_items_after_crush` and emitted
no disclosure at all — valid JSON with no marker, no sentinel, and
untouched sibling count fields. This contradicts the repo's own "Safety
first: never drop user/assistant content" principle.
The dict-array path already disclosed drops via a `{"_ccr_dropped":
...}` sentinel; scalar arrays had no equivalent. This PR gives them one:
when the `StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, it appends `"… N more items <<ccr:HASH
N_items_offloaded>>"` and stores the full original array in the CCR
store (same SHA256-12 hashing the Python bridge and retrieval use), so
the marker round-trips through `headroom_retrieve`. No-op when nothing
is dropped or the marker gate is off.
Closes #3650
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
| File | What changed |
|---|---|
| `crates/headroom-core/src/transforms/smart_crusher/crusher.rs` | New
`append_scalar_drop_sentinel()`: when
`StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, append the sentinel and store the full
original array in the CCR store. No-op when nothing is dropped or the
marker gate is off. |
| `headroom/transforms/smart_crusher.py` | `is_ccr_sentinel()` now
recognizes the string sentinel shape (previously only the dict shape),
so downstream code can filter it like the dict sentinel; docstring
updates. |
| `tests/test_transforms/test_smart_crusher_issue_3650.py` | New
regression tests: sentinel is emitted with the right drop count, carries
a retrievable CCR marker whose hash resolves to the full original array,
and is suppressed when the marker gate is off. |
| `crates/.../smart_crusher/crusher.rs` (tests module) | Rust regression
tests mirroring the above (sentinel emission, marker retrieval of the
full 120-item array, gate-off suppression). |
| `tests/parity/fixtures/smart_crusher/*.json` (3 files) | Recorded
outputs updated — strategy strings now reflect the trailing sentinel. |
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
cargo test --lib transforms::smart_crusher → 337 passed, 0 failed
(includes the new #3650 Rust regression tests)
pytest tests/test_transforms/test_smart_crusher_issue_3650.py \
tests/test_transforms/test_smart_crusher_bugs.py \
tests/test_transforms/test_smart_crusher_ccr_roundtrip.py \
tests/test_transforms/test_smart_crusher_rust_parity.py \
tests/test_transforms/test_content_router.py → 106 passed
(includes 18 Rust-parity fixture tests)
ruff check and ruff format --check on the touched Python files → clean
```
## Real Behavior Proof
- Environment: Linux container, Python 3.12, editable install from
`main` with the maturin-built native `_core` module (no ONNX model
involved — this path is pure Rust crusher).
- Exact command / steps: the repro snippet from #3650, verbatim
(`ContentRouter(ContentRouterConfig())`, 120 slugs + `"total": 120`).
- Observed result: **before** → 16 of 120 kept, `"<<ccr:" in out`
False; **after** → 16 kept + `'… 104 more items <<ccr:4f03caae30bb
104_items_offloaded>>'`, marker present, and the marker's hash
retrieves all 120 originals from the CCR store. The `total` sibling
field is left untouched — the truncation is now disclosed in-band next
to it rather than silently contradicting it.
- Not tested: the ONNX/Kompress model path from #3673 (a separate issue;
this fix is model-independent), Windows/macOS install flows, and the
`lossless_only=True` path (unchanged: nothing dropped there, so no
sentinel is emitted).
## Runtime Rollout Safety
- Rollout-managed feature(s): CCR marker gate (`enable_ccr_marker`)
- Minimum rollout channel: n/a (open-source library; applies to default
config where the marker gate is on)
- Stable/default behavior changed: Yes — truncated scalar arrays now
carry a trailing sentinel string instead of being silently shortened;
sibling count fields untouched
- Kill switch / disable path: set `enable_ccr_marker=False` — no
sentinel emitted, gate-off path unchanged
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — library behavior; observed output captured under Real Behavior
Proof.
## Additional Notes
Raphael Malikian — rtmalikian@gmail.com
If you'd like similar help with your own project or codebase, I'm open
to collaborate — feel free to reach out.
*This PR was prepared with AI assistance and reviewed by the author
before submission.*
---------
Signed-off-by: Raphael Malikian <rtmalikian@gmail.com>
|
||
|
|
6feb1fb22a |
fix(content_detector): detect CMTrace (SCCM/Intune) logs as BUILD_OUTPUT (#3712)
## Description
CMTrace-format logs — what SCCM, MDT and Intune Win32 app/script
deployments
write on Windows — are never detected as `BUILD_OUTPUT`. Two independent
causes,
and the second shadows a fix to the first, so both are fixed together.
Measured on a corpus of 253 real Intune/SCCM deployment logs,
`detect_content_type`
returned `BUILD_OUTPUT` for **zero** of them.
Closes #<!-- issue number if you filed one; otherwise delete this line
-->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- **Both implementations** — added `^<!\[LOG\[` to `_LOG_PATTERNS` /
`LOG_PATTERNS`. CMTrace puts its timestamp in an attribute *after* the
message, so the anchored `^\s*\d{4}-\d{2}-\d{2}`,
`^\s*\[\d{2}:\d{2}:\d{2}\]` and `^={3,}` patterns cannot fire, and a
record
need not contain ERROR/WARN/INFO. `_try_detect_log` returned `None` at
its
`pattern_matches == 0` guard.
- **Rust only** — ported Python's `_prefix_looks_like_path` as
`prefix_looks_like_path` and applied it in `try_detect_search`. A
CMTrace
divider contains no whitespace, so `SEARCH_RESULT_PATTERN` runs from
`<![LOG[` through `]LOG]!><time="12`, then `:`, then `00`, then `:` — a
clean
`file:line:` match. Python rejects it via `_is_search_result_line`'s
path-shape guard (no `<`, `>`, `=`); Rust applied the regex bare. Since
search is dispatched before logs, Rust would otherwise shadow the fix
above.
Porting the guard, rather than tightening the Rust regex, keeps
`SEARCH_RESULT_PATTERN` byte-identical across the two and converges the
logic on Python's.
- Tests on both sides: CMTrace multi-line → `BUILD_OUTPUT`, CMTrace
single-line
→ `BUILD_OUTPUT`, the divider is rejected as a search line, plus
regression
guards that ordinary and extensionless grep output still detect as
`SEARCH_RESULTS`. The Rust suite additionally pins
`prefix_looks_like_path`
against Python's rules directly.
### Note on the parity harness
The Rust/Python divergence above was **not** caught by
`parity-run --only content_detector`, which reports `matched=21
diffed=0` both
before and after this change. None of the 21 recorded fixtures exercise
a
`<`-prefixed `file:line:` shape, so the drift was invisible to it. Worth
knowing
if the harness is being relied on as the parity guarantee.
### Two further Rust divergences, reported not fixed
Found while porting the guard; left alone to keep this PR to one logical
change.
Happy to open a separate issue or PR:
- Rust has no equivalent of `_GREP_COLON_DASH_PATTERN` or
`_is_grep_context_line` — the `grep -A/-B/-C` context-line handling
added in
#3580.
- Rust returns on `matching_lines == 0`; Python uses an absolute floor
of
`matching_lines < 2`, added deliberately because the search compressor
drops
non-matching lines, so a single coincidental match is data loss.
### Why a second, single-line test
Real Intune logs are emitted as one continuous stream with **no line
separators** — a 133 KB production sample held 571 records and 2 line
feeds.
That form never reaches `_try_detect_search` (it exits at
`matching_lines < 2`),
so it fails on Bug 1 alone and lands in `PLAIN_TEXT`. The multi-line
form is
what exercises Bug 2. Both shapes occur, so both are tested.
## Testing
- [x] Unit tests pass (`pytest`) — 12,680 passed; see note on
pre-existing Windows failures below
- [x] Linting passes (`ruff check .`) — `All checks passed!`
- [x] Type checking passes (`mypy headroom`) — 1 pre-existing error in
an unrelated file, none in the files this PR touches
- [x] New tests added for new functionality — 5 Python, 5 Rust
- [ ] Manual testing performed
### Test Output
Python, on 3.13.15 / pytest 9.0.3 — **5 passed:**
```text
tests/test_transforms/test_content_detector_cmtrace.py::test_multiline_cmtrace_detected_as_build_output PASSED [ 20%]
tests/test_transforms/test_content_detector_cmtrace.py::test_single_line_cmtrace_detected_as_build_output PASSED [ 40%]
tests/test_transforms/test_content_detector_cmtrace.py::test_cmtrace_divider_is_not_treated_as_a_search_result_line PASSED [ 60%]
tests/test_transforms/test_content_detector_cmtrace.py::test_grep_output_still_detected_as_search_results PASSED [ 80%]
tests/test_transforms/test_content_detector_cmtrace.py::test_extensionless_grep_paths_still_detected PASSED [100%]
================================================== 5 passed in 0.71s ==================================================
```
Rust, `cargo test -p headroom-core content_detector` — **27 passed** (21
pre-existing + 6 new). The 21 pre-existing `content_detector` tests all
still
pass, including `search_results_detected` and `build_output_detected`,
which is
the direct evidence that adding the path-shape guard did not disturb
existing
detection:
```text
running 27 tests
test transforms::content_detector::tests::cmtrace_divider_is_rejected_by_the_path_shape_guard ... ok
test transforms::content_detector::tests::path_shape_guard_matches_python_rules ... ok
test transforms::content_detector::tests::multiline_cmtrace_detects_as_build_output ... ok
test transforms::content_detector::tests::single_line_cmtrace_detects_as_build_output ... ok
test transforms::content_detector::tests::grep_output_still_detects_as_search_results ... ok
test transforms::content_detector::tests::extensionless_grep_paths_still_detected ... ok
test transforms::content_detector::tests::search_results_detected ... ok
test transforms::content_detector::tests::build_output_detected ... ok
[...19 further pre-existing content_detector tests, all ok...]
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 915 filtered out; finished in 0.07s
```
Rust↔Python parity harness over all 21 recorded `content_detector`
fixtures —
**already run, zero divergence**:
```text
$ cargo run -p headroom-parity --bin parity-run -- run --only content_detector
[content_detector] total=21 matched=21 skipped=0 diffed=0
```
Full Python suite — **already run**:
```text
45 failed, 12680 passed, 603 skipped, 6141 warnings, 81 errors in 2743.50s (0:45:43)
```
**The 126 failures/errors are pre-existing and platform-specific, none
related
to this change.** Evidence:
- They span 26 distinct test files. **Not one of them references
`content_detector` or `detect_content_type`** (checked by grep across
every
failing file).
- 95 of 126 are `PermissionError: [WinError 32] The process cannot
access the
file because it is being used by another process` — Windows SQLite file
locking, concentrated in `test_sqlite_graph_store.py` (37),
`test_sqlite_vector_index.py` (18) and `test_memory/` (36).
- The remainder are Windows-platform assumptions: `module 'signal' has
no
attribute 'SIGHUP'`, `WindowsPath` vs POSIX path comparisons, CRLF-vs-LF
in
`test_owned_asset_encoding.py`, and two missing external binaries
(`ast-grep`, `difftastic`).
Per CONTRIBUTING ("Test/CI-only PR chasing a known `main` failure —
Don't.
We're tracking it"), these are left alone. Happy to confirm against a
clean
`main` run on request.
Lint — **already run, clean**:
```text
$ py -m uv run ruff check .
All checks passed!
```
Type check — **already run**:
```text
$ py -m uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function]
Found 1 error in 1 file (checked 538 source files)
```
**That error is pre-existing and unrelated.**
`headroom/integrations/langchain/chat_model.py` is not touched by this
PR and contains no reference to
`content_detector` or `detect_content_type`. The two files this PR
changes
produce no mypy errors.
## Real Behavior Proof
- Environment: Windows 11 Enterprise 10.0.26200; Python 3.13.15 with
dependencies resolved from `uv.lock` (`py -m uv sync --extra dev`, uv
0.12.17); cargo/rustc 1.95.0; branch `fix/cmtrace-content-detection`
based on `94206e26` (release 0.38.0). Full detail under *Environment*
below.
- Exact command / steps: `detect_content_type` was run over a 40-record
synthetic CMTrace log three times — with the patch present, with `git
checkout HEAD~1 -- headroom/transforms/content_detector.py` reverting
only the patched file, and with it restored — each run printing whether
the fix was present in the module it actually imported. Full command
list under *Exact command / steps* below.
- Observed result: on identical input, `ContentType.BUILD_OUTPUT` with
the fix present and `ContentType.PLAIN_TEXT` with it reverted, toggled
in both directions and back; classification tracks the file state
exactly. Transcript under *Observed result* below.
- Not tested: end-to-end through the CLI or proxy (detection was
exercised directly, `Manual testing performed` is unchecked above); the
Rust half has unit and parity coverage but no before/after toggle of a
built binary; UTF-16 or BOM-prefixed CMTrace input; non-Windows CMTrace
producers; the 253-log corpus was measured before the patch and not
re-measured after. Detail under *Not tested* below.
### Environment
- Windows 11 Enterprise 10.0.26200
- Python 3.13.15, from `C:\Program Files\Python313`
- uv 0.12.17 (x86_64-pc-windows-msvc), installed via `py -m pip install
uv`
- Dependencies from `uv.lock` via `py -m uv sync --extra dev`
- cargo 1.95.0 / rustc 1.95.0
- Branch `fix/cmtrace-content-detection`, based on `94206e26` (release
0.38.0)
### Exact command / steps
```bash
# environment
py -m pip install uv
py -m uv sync --extra dev
# confirm the suite imports the working tree, not a built copy
py -m uv run python -c "import headroom.transforms.content_detector as m; print(m.__file__)"
# -> <repo>\headroom\transforms\content_detector.py
# tests
py -m uv run pytest tests/test_transforms/test_content_detector_cmtrace.py -v
py -m uv run pytest
py -m uv run ruff check .
py -m uv run mypy headroom
cargo test -p headroom-core content_detector
cargo run -p headroom-parity --bin parity-run -- run --only content_detector
# before/after on the same input, reverting only the patched file
py -m uv run python -c "<detect command, below>"
git checkout HEAD~1 -- headroom/transforms/content_detector.py
py -m uv run python -c "<same command>"
git checkout HEAD -- headroom/transforms/content_detector.py
py -m uv run python -c "<same command>"
```
The `git checkout` toggle only proves anything if the import resolves to
the
working tree rather than an installed copy, which is why the detect
command
below prints the provenance of the module it imported alongside the
result.
### Observed result
Same input, same command, toggling only the patched file, on a synthetic
40-record CMTrace log containing no real data:
```text
$ py -m uv run python -c "import headroom.transforms.content_detector as m; src=open(m.__file__, encoding='utf-8').read(); print('fix_present =', 'CMTrace record opener' in src); print('result =', m.detect_content_type(open(r'...\cmtrace-sample.log').read()).content_type)"
fix_present = True
result = ContentType.BUILD_OUTPUT
$ git checkout HEAD~1 -- headroom/transforms/content_detector.py
$ py -m uv run python -c "...same command..."
fix_present = False
result = ContentType.PLAIN_TEXT
$ git checkout HEAD -- headroom/transforms/content_detector.py
$ py -m uv run python -c "...same command..."
fix_present = True
result = ContentType.BUILD_OUTPUT
```
Toggled in both directions and back, and the classification tracks the
file
state exactly.
Note the "before" value is `PLAIN_TEXT`, not `SEARCH_RESULTS`: Python's
existing
path-shape guard already keeps this content off the search path, so it
was
falling through to prose rather than being misrouted. That is the whole
of the
Python-side bug — the log detector simply never fired. The Rust port,
lacking
that guard, *would* have returned `SEARCH_RESULTS` here, which is why
this PR
ports it.
### Supporting analysis, gathered before the patch
Impact analysis against all 21 recorded parity fixtures in
`tests/parity/fixtures/content_detector/`. Each fixture input was
extracted and
run against the search pattern and against the new CMTrace pattern. **No
fixture
changes classification** — the only `search` fixture matches 4 of 4
lines either
way:
```
$ printf '%b\n' "$(...79e6b68a05a1c639.json input...)"
src/main.py:42:def process():
src/util.py:13: return None
lib/x.py:7:class X:
tests/test_a.py:3: assert True
matches: 4 of 4
```
Why the bare `file:line:` shape fires on this format at all — on the
same
synthetic log (40 records, 20 dividers):
```
$ grep -cP '^[^\s:]+:\d+:' cmtrace-sample.log
20
$ grep -oP '^[^\s:]+:\d+:' cmtrace-sample.log | head -1
<![LOG[===========================================================================]LOG]!><time="10:00:
```
20 of 40 lines matching gives ratio 0.50 and confidence `0.4 + 0.50*0.6
= 0.70`,
over the 0.6 cutoff. Python discards those matches at the path-shape
guard, so
it lands in `PLAIN_TEXT`; Rust, which had no such guard, is where that
0.70
would have been returned as `SEARCH_RESULTS` — the shadowing this PR
removes.
None of `_LOG_PATTERNS`' existing signals match that same file, each
checked
individually:
```
^\s*\d{4}-\d{2}-\d{2} 0 matches
^\s*\[\d{2}:\d{2}:\d{2}\] 0 matches
^={3,}|^-{3,} 0 matches
\b(ERROR|FAIL|FAILED|FATAL|CRITICAL)\b 0 matches
\b(WARN|WARNING)\b 0 matches
\b(INFO|DEBUG|TRACE)\b 0 matches
```
**Downstream real-world measurement:** a tool built on `headroom-core`
processed
253 real Intune/SCCM deployment logs. `LogCompressor` was dispatched **0
times**
across the whole corpus; every log went to the prose path.
### Not tested
- End-to-end through the CLI or proxy. Detection was exercised directly
via
`detect_content_type`; `Manual testing performed` is left unchecked
above.
- A before/after toggle of a built Rust binary. The Rust half is covered
by its
6 new unit tests and by `parity-run --only content_detector`.
- UTF-16 or BOM-prefixed CMTrace input — out of scope here, detection
operates
on already-decoded text.
- Non-Windows CMTrace producers.
- Re-measurement of the 253-log corpus after the patch; that figure is a
pre-patch measurement.
## Runtime Rollout Safety
- Rollout-managed feature(s): none — pure detection logic in
`content_detector`, behind no flag, channel or rollout gate.
- Minimum rollout channel: N/A — not rollout-managed; it ships with the
ordinary release.
- Stable/default behavior changed: yes, narrowly — content beginning
`<![LOG[` now classifies as `BUILD_OUTPUT` instead of `PLAIN_TEXT`
(Python) or `SEARCH_RESULTS` (Rust), and in Rust a `file:line:` prefix
containing `<`, `>` or `=` no longer counts as a search line, which is
what Python has always done. All 21 recorded parity fixtures are
unchanged: `matched=21 diffed=0`.
- Kill switch / disable path: none — there is no runtime toggle;
disabling means reverting the commit.
- Unsafe override required: no.
- Qualification impact: `BUILD_OUTPUT` routes to `LogCompressor` rather
than the prose path, so measured compression ratio on CMTrace input
changes — see Additional Notes, which quantifies it and explains why it
is still the correct classification.
- Rollback path: single-commit revert; no migration, persisted state or
cache format is involved.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs cover detector internals
- [x] My changes generate no new warnings — `ruff check .` clean; `cargo
test` compiled without warnings on the touched crate
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally — 5 Python + 27 Rust;
parity `matched=21 diffed=0`
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
**One thing worth flagging to maintainers, because it changed our own
conclusion.** Correcting detection is necessary but, for this format,
not
sufficient.
We first patched this downstream by routing CMTrace to `BUILD_OUTPUT`.
On
measuring it, we reverted that change: `LogCompressor` returns this
content
**unchanged**. Real CMTrace logs have no line separators, so a
line-oriented
compressor sees one enormous line and has nothing to deduplicate.
| Route | Time | 266 KB CMTrace sample |
|---|---|---|
| `BUILD_OUTPUT` → `LogCompressor` | 0.10 s | 133 KB → 133 KB (**0%**) |
| prose/embedding path | 17.9 s | 266 KB → 9.5 KB (**96%**) |
4 of 5 sampled files saved exactly zero bytes on the log path; across
152 real
CMTrace files the prose path averages 71%.
This PR fixes detection only, because that is the actual bug and it is
correct
on its own terms — CMTrace *is* build output. But `LogCompressor` would
need
record-boundary awareness (splitting on `]LOG]!>` rather than `\n`)
before these
files genuinely benefit. Happy to open a separate issue or PR for that
if
maintainers want it; flagging it here so this change doesn't land and
look like
a regression on compression ratio.
**Not a performance report.** On our corpus these files average 2.7 s;
the slow
files are simply the large ones, regardless of format.
Co-authored-by: PRokicki <prokicki@hemmersbach.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
d971f7c398 |
fix(transforms): stop routing timestamped log rows as grep matches (#3748)
## Description
`_is_search_result_line` now rejects any line that `_TIMESTAMP_ROW_RE`
(from `lossless_compaction.py`, #3419) recognizes as a timestamped log
row, before the grep-shape patterns are tried.
The colon branch accepts `^[^\s:]+:\d+:` whenever the prefix passes
`_prefix_looks_like_path`, and that guard only rules out `<`, `>` and
`=`. A bare ISO date-hour prefix (`2026-09-23T10`) passes, so every row
of a timestamped log counts as a grep match at ratio 1.0 and the payload
routes to `SEARCH_RESULTS`.
Since #3419 the lossless fold skips timestamp rows, so the payload falls
through to the lossy `SearchCompressor`: 2,000 rows collapse to 5, the
note misreads them as "matches in `2026-09-23T10`", and the minute is
printed back as an integer (`10:00:00Z` becomes `10:0:00Z`, a time that
does not exist in the input).
Reusing the lossless module's regex keeps the two guards in lockstep: a
row the fold refuses to treat as grep output can no longer be classified
as grep output here either. The context-line branch
(`_is_grep_context_line`) already kept dates out via its `/`-or-`.`
requirement; this closes the same hole in the colon branch.
Closes #3736
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/content_detection.py`: `_is_search_result_line`
checks `_TIMESTAMP_ROW_RE.match(line)` (imported from
`lossless_compaction`) before the colon and context branches, so
timestamped log rows are never counted as grep matches.
- `tests/test_transforms/test_content_detector_timestamp_search.py`: 6
new regression tests (2 pinning the bug, 4 controls for real grep
output, date-named files like `logs/2026-09-23.log:42:`, and
extensionless `Makefile:12:`).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Red-green on base
|
||
|
|
7f2766ca4b |
fix(kompress): bound the inference deadline to the request, not each block (#3693)
## Description `HEADROOM_COMPRESSION_DEADLINE_MS` (default 20s) is a budget on kompress inference, checked at chunk boundaries against a clock that `compress()` starts itself when the caller passes no `_deadline_started_at`. Nothing outside `kompress_compressor.py` ever passes one, so the budget bounds a **block**, not a request: a request with a dozen compressible blocks gets a dozen full deadlines. That is how one request runs past the pipeline's own `COMPRESSION_TIMEOUT_SECONDS` (default 30s). The overrunning worker cannot be preempted, so it becomes timeout debt and opens the quarantine, and every request behind it forwards with NO compression until that worker exits or the 60s cap lapses. Same cascade as #2360, reached on the Anthropic path by ordinary large agentic requests rather than by a Codex WS 5s deadline. `apply()` now stamps one `time.perf_counter()` origin into the existing per-request `_PerRequestRuntimeState` (#3486) and `_try_ml_compressor` hands it to every kompress call for that request, so a big request degrades to partial compression inside its budget instead of overrunning and starving the requests queued behind it. Related: #3653 (the thread/fd leak each quarantine event also causes) and #2451 (the concurrency-1 execution slot that makes blocks serialize). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `_PerRequestRuntimeState` carries `kompress_deadline_started_at`, stamped once per `apply()` alongside the other per-request fields. - `_try_ml_compressor` forwards that origin as `_deadline_started_at` on every kompress call. - `KompressCompressor.shares_request_deadline = True` gates the kwarg; `RemoteKompressCompressor.compress()` has no such parameter, so it is not handed one. - Three tests in `tests/test_transforms/test_kompress_deadline.py`. ## Testing - [x] Unit tests pass (`pytest`) ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_router_registry_smartcrusher.py -q 404 passed, 67 skipped in 32.37s $ uv run --frozen --extra dev pytest tests/test_transforms/test_kompress_deadline.py -q 6 passed in 2.00s # Against the unfixed source, the two behavioural tests fail: FAILED test_every_block_of_one_request_draws_down_the_same_deadline FAILED test_a_second_request_gets_a_fresh_deadline 2 failed, 4 passed ``` ## Real Behavior Proof - Environment: macOS 24.6 arm64, Python 3.12, headroom-ai 0.37.0 installed wheel, ONNX kompress (`chopratejas/kompress-v2-base`), backend anthropic, one heavy Claude Code user over 2026-09-17..21. - Exact command / steps: Measured the installed runtime directly. Timed `KompressCompressor.compress()` on a real 20 KB block pulled from `/transformations/feed` (2622 words, 8 chunks), four calls. Parsed 4100 `PERF` lines and every `CompressionQuarantinedError` request id out of `~/.headroom/logs/proxy.log*`, bucketed by `tok_before`, and priced the quarantined requests at the savings rate achieved by non-quarantined requests in the same size bucket. - Observed result: Inference costs ~1.6 ms/word: the 20 KB block takes 4.2s per call, reproducible (4548/4162/4392/4237 ms). `opt_ms` by size: 0-50k tokens n=376 avg 113 ms; 200-250k tokens n=101 avg 5192 ms, max 67247 ms. One day carried 266 "Kompress slow compress" markers totalling 682s of inference, max single inference 21.9s. 250 requests were refused compression by the quarantine, costing ~3.56M tokens, 8.7% of that day's achievable savings (per-day: 2.2%, 19.4%, 3.7%, 8.7%). With the deadline shared, the same request stops kompressing at the budget and returns instead of overrunning the 30s pipeline timeout. - Not tested: The fleet-wide prevalence of the quarantine cascade, which no counter reports today; and the CUDA/MPS batched path, which shares one deadline inside `compress_batch` already and was not re-measured on this change. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes. A request whose kompress inference exceeds `HEADROOM_COMPRESSION_DEADLINE_MS` in total now stops compressing further blocks, where before each block restarted the budget. That is a reduction in per-request compression for requests that were previously overrunning the pipeline timeout, and an increase for every request that was being quarantined behind them. - Kill switch / disable path: Set `HEADROOM_COMPRESSION_DEADLINE_MS` high (for example `600000`) to restore effectively unbounded per-request inference. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit; the deadline returns to being per-block. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review |
||
|
|
9263b42099 |
feat(router): elide dense machine-generated lines no compressor can shrink (#3685)
## Description Tool outputs that dump a fetched web page, a bundled asset or an encoded blob (minified JS/CSS, base64, RSC payloads) arrive as a few very long lines with almost no whitespace. No structural compressor can do anything with them: `code_aware` is disabled, `log`/`text` return them unchanged, and the HTML extractor returns either the whole thing or `""` (which the unit layer then rejects as an empty block and keeps all of it verbatim). The block rides through the router at ratio 1.0. On a Codex web-research session these were the four largest tool outputs (75k tokens) and the whole session compressed at 1.3% of new input. This adds a last-resort pass in `_apply_strategy_to_content`: after the strategy chain (and on the passthrough path), every line that is >= 300 chars with < 6% spaces keeps its head and tail and the middle becomes a marker. Prose, code, logs, indented JSON, CSV and markdown all have far more spaces and are never touched; JSON-shaped lines are left to SmartCrusher. The pre-elision block goes into the CCR store and a `Retrieve original: hash=` marker is appended, so the messages path's marker-less lossy gate (#1307) keeps the result and the agent can recover exact bytes. Runs only on strategies whose result is still plain text (`html`, `log`, `text`, `code_aware`, `search`, `passthrough`), never after SmartCrusher/tabular/config/diff/Kompress, and never in lossless mode. New config flag `enable_dense_line_elision`, default on. Also: an HTML extraction that comes back `""` on non-empty input is now treated as "nothing extracted" (fall through) rather than "compressed to zero tokens". ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Performance improvement ## Changes Made - `headroom/transforms/dense_line_elider.py`: `is_dense_line` / `elide_dense_lines`, byte-identical no-op when nothing is dense. - `headroom/transforms/content_router.py`: `enable_dense_line_elision` config, `_DENSE_ELIDE_AFTER` strategy set, `_elide_dense` (CCR store + marker), hook after the strategy chain and on the passthrough path, empty-HTML-extraction fallthrough. - `tests/test_transforms/test_dense_line_elider.py`: 8 tests (prose/code/JSON untouched, elision shape, router hook on/off, marker retrievable from the CCR store, survives the messages-path lossy gate, lossless mode never elides, search output keeps line structure). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --frozen --extra dev pytest tests/test_transforms/test_dense_line_elider.py -q -> 8 passed uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_no_ccr_lossy.py tests/test_transforms_content_router.py tests/test_openai_responses_compression_units.py tests/test_code_aware_regressions.py tests/test_router_registry_dispatch.py tests/test_router_external_dispatch.py tests/test_compression_observability.py tests/test_openai_responses_context_compaction.py tests/test_agent_savings.py tests/test_openai_responses_t3_replay_regression.py -q -> 588 passed, 72 skipped uv run --frozen --extra dev pytest tests/ --ignore=tests/e2e -q -> 9 failed, 12619 passed, 597 skipped (see note below) uvx ruff check headroom/transforms tests/test_transforms/test_dense_line_elider.py -> All checks passed! uv run --frozen --extra dev mypy headroom/transforms/content_router.py headroom/transforms/dense_line_elider.py -> Success: no issues found in 2 source files ``` The 9 full-suite failures are unrelated to this change: `tests/test_provider_grok.py` (2) and `tests/test_cli/test_wrap_grok.py` fail because this shell exports `GROK_CLI_CHAT_PROXY_BASE_URL` (all pass with it unset); the other 7 (`test_proxy_health.py` readyz/kompress x4, `test_read_maturation_handler_nobust.py` x2, `test_providers/test_deepseek.py` x1) are order-dependent, pass when run in isolation, and fail identically with `upstream/main`'s `content_router.py` swapped in. ## Real Behavior Proof - Environment: this branch, `HeadroomProxy(ProxyConfig(stateless=True, mode="token"))`, `HEADROOM_DISABLE_KOMPRESS=1`, two real `~/.codex/sessions/*.jsonl` rollouts replayed through `_compress_openai_responses_payload` one request per tool-output item (`input=items[:i+1]`), novel `tokens_saved` (this pass minus previous pass) over tiktoken of the items added since the previous request. Same script with `elide_dense_lines` monkeypatched to a no-op for the "off" arm. - Exact command / steps: `HEADROOM_ELIDE_TEST=off|on uv run --frozen --extra dev python replay_onoff.py <rollout.jsonl> <n_outputs> <model>` - Observed result: see the block below (web-research session 1.3% -> 15.3% of new input, coding session unchanged at 34.6%). ```text 2026-09-09 gpt-5.6-sol elision off: novel=4377 new_input=344023 rate=1.3% 2026-09-09 gpt-5.6-sol elision on: novel=52737 new_input=344023 rate=15.3% 2026-09-07 gpt-6-astra elision off: novel=51520 new_input=148924 rate=34.6% 2026-09-07 gpt-6-astra elision on: novel=51520 new_input=148924 rate=34.6% ``` The web-research session (curl'd pages, truncated minified bundles) goes 1.3% -> 15.3%; the coding session (code-mode JSON chunk outputs, no dense lines) is byte-for-byte unchanged. Across 14 local Codex sessions, dense lines are 13.5% of all tool-output characters (0% to 54% per session), so the fleet effect will be uneven and largest for research-heavy users. What remains un-elided on the research session is inside `embedded_json` spans (SSE `data: {...}` lines) and is excluded on purpose: eliding inside those would corrupt the JSON. - Not tested: live proxy traffic with a real Codex or Claude Code client (replay only); Kompress enabled alongside elision (replays ran with `HEADROOM_DISABLE_KOMPRESS=1`, matching the desktop's production config); a CCR `/v1/retrieve` round trip over HTTP (the store-and-retrieve check is in-process in the tests). Desktop-side: measured 2026-09-20 on the `user_daily_savings` rollup, Codex-dominant users sit at a median 6.5% strict input rate (n=341 user-days) vs 25.7% for Claude Code (n=478) at identical cache coverage; this is the largest single lever found for that gap. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes. `ContentRouterConfig.enable_dense_line_elision` defaults to on, so blocks with dense machine-generated lines that previously passed through at ratio 1.0 are now elided with a CCR retrieval marker. Blocks with no such lines are byte-identical to before. - Kill switch / disable path: `ContentRouterConfig(enable_dense_line_elision=False)`; `lossless=True` also disables it. - Unsafe override required: No. - Qualification impact: None; the elision is a deterministic post-pass and does not touch routing decisions, strategy selection or the frozen-prefix logic. - Rollback path: Revert this PR, or set `enable_dense_line_elision=False`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Documentation: no docs page lists `ContentRouterConfig` flags today, so there is nothing to update; happy to add one if you want the flag documented. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
58c7f72106 |
fix(smart-crusher): keep non-dict arrays and object keys whole under lossless_only (#3628)
## Description
`lossless_only` is documented as producing marker-free, byte-recoverable
output where rows are never dropped. Only the dict-array row-drop path
(`crush_array_with_source`) checked it, though. The string, number and
mixed-array crushers and the object-key crusher kept sampling in strict
mode, and they drop items without emitting any CCR marker. So `{"slugs":
[53 strings]}` came back with 15 slugs whether `lossless_only` was on or
off, and nothing in the output showed that items were removed.
Each of those four crushers now returns its input unchanged under
`lossless_only`, as `crush_array_with_source` already does for dict
arrays. Behavior with `lossless_only` off is unchanged.
Closes #3625
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crushers.rs`: `crush_string_array`, `crush_number_array` and
`crush_object` return their input under `lossless_only`, with strategies
`string:lossless_only`, `number:lossless_only` and
`object:lossless_only`. Each check comes after the crusher's existing
passthrough checks.
- `crusher.rs`: `crush_mixed_array` returns its items under
`lossless_only` (`mixed:lossless_only`).
- `config.rs`: the `lossless_only` doc comment now names these crushers.
- Tests:
- three unit tests in `crushers.rs`;
- two in `crusher.rs`, one of them end to end through `crush()` using
the #3625 payload;
- one Python test through the public `SmartCrusher` wrapper.
Each Rust test first asserts that the default config really drops items
on the same input, so the strict assertion cannot pass vacuously.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# Before the fix (origin/main
|
||
|
|
7bd4dbaf8f |
fix(kompress): append the CCR marker whenever the saving pays for it (#3484)
## Description Kompress previously added a CCR retrieval marker only below a fixed compression ratio. Smaller reductions could therefore produce lossy output without a retrieval marker, which the router discarded. The single, batch, and remote paths now compare the complete original payload with the complete candidate plus retrieval marker. A marked candidate ships only when it is smaller under the same `cl100k_base` estimate, and its result accounting uses those measured counts. Otherwise the original passes through. CCR-disabled behavior is unchanged. This is a fixed tokenizer estimate, not an exact billing count for every OpenAI model or other provider. If the encoder is unavailable, the fallback compares character counts rather than claiming a provider-token bound. ## Type of Change - [x] Bug fix ## Testing Latest local validation: 81 tests passed across marker, remote, deadline, compressor, size-gate, and CCR policy/resolution suites. Ruff check and format check pass for the five changed Python files. ## Real Behavior Proof - Environment: macOS, Python 3.13, deterministic compressor stubs with real source-derived CCR hashes and `cl100k_base` encoding. - Exact command / steps: Run the marker/remote/deadline suites and related compressor/CCR suites. Re-run the single/batch boundary probe using `" ".join(["alpha"] * 36 + ["bureaucratic"] + ["alpha"] * 63)` with the supplied drop-first-36 model stub. - Observed result: The formerly expanding candidate (100 original tokens, 103 with marker) now passes through unchanged, reporting 100 to 100 with no retrieval key. Tests also verify that candidates with net savings ship with marker-inclusive accounting, and partial deadline completion preserves the unprocessed tail. - Not tested: exact billing equivalence for other model tokenizers, live model/provider traffic, or long-term retrieval-rate effects. Current-head CI is pending. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: CCR-enabled Kompress uses full-payload net savings rather than a fixed ratio threshold for marker admission. Marker-inclusive successful result counts use the same tokenizer estimate as the gate. - Kill switch / disable path: `KompressConfig(enable_ccr=False)` disables CCR markers as before. - Unsafe override required: No. - Qualification impact: Compression accounting includes the marker on admitted CCR results; provider billing remains an estimate. - Rollback path: Revert this PR. ## Review Readiness - [x] Prior marker-accounting findings resolved. - [x] Focused and related local checks pass. - [ ] Current-head CI complete. - Final human review is required. ## Checklist - [x] Single, batch, remote, and deadline regressions covered. - [x] No manual CHANGELOG edit. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
|
8884d87378 |
fix(transforms): stop compression garbling mixed subagent output (#3286)
## The report
A user's model called compressed subagent output "too garbled to use"
and burned CCR retrievals to reconstruct it — **not** because it needed
more context. One retrieval returned nothing but the Claude Code harness
sanitizer banner, reported as `original_item_count: 33,
compressed_item_count: 25`.
Root-cause chain (verified by reproduction): the harness prepends a
bracket-delimited banner (`[harness: ... you.]` — exactly 33
whitespace-delimited words) and neutralizes `<` → `<\`. Headroom's
mixed-content splitter typed the banner as JSON (bracket balance, no
validation) → SmartCrusher couldn't parse it → the fallback chain fed it
to lossy Kompress → Kompress word-dropped the banner 33→25 and stored it
behind a retrieval hash. Meanwhile tabular sections rendered as
quote-wrapped JSON-string blobs with `\n` as two-character escapes, and
`ensure_ascii=True` boundaries turned the output's unicode (`→ └ ✓`)
into `\uXXXX` soup. The model reasonably concluded the output was
garbled.
## Fixes
1. **`split_into_sections` validates JSON before typing a block
`JSON_ARRAY`** — same validation its own mixed-content gate
(`_has_valid_json_block_with_text`) has always used. Tag-protection
placeholders, which self-isolated only by accident of that bug
(`{{HEADROOM_TAG_N}}` bracket-balances), are now isolated explicitly via
a new `isolate=` parameter fed by the router; contiguous prose fragments
re-coalesce so the `\n\n` reassembly stops doubling newlines in
uncompressed prose.
2. **Kompress gets a real floor: `min_input_words = 64`**
(config-tunable, clamped at the historical 10), applied on the
in-process, batch, apply, and remote paths. Below it, lossy
word-dropping is a net loss — the retrieval marker alone is ~20 words —
and short blocks are disproportionately instruction-like.
3. **The mixed path unwraps SmartCrusher's whole-array CSV render** when
it comes back as a bare JSON string, splicing raw readable lines into
the text instead of a quoted escape blob.
4. **`ensure_ascii=False` at model-visible boundaries**: MCP
retrieve/stats responses and the audit-safe splice reserialization
(which now also matches serde_json's non-escaping behavior).
5. **Kompress honesty**: the marker says `N words compressed to M`
(shared `ccr_retrieval_marker` helper, unit-tested), and
`store_kompress_in_ccr` no longer writes word counts into the store's
*item count* fields — token counts already carry the size story.
The upstream trigger (the harness's `<` → `<\` neutralization corrupting
JSON semantics) is not Headroom's to fix, but with #1 and #2 the banner
now passes through byte-intact and nothing lossy touches it.
## Testing
- New `tests/test_garbled_compression_fixes.py` (12 tests) pins every
fix, including an end-to-end router pass over a reconstructed
harness-sanitized fixture asserting the banner survives byte-identical
and no `\uXXXX` appears.
- Existing small-fixture kompress/router tests updated to set
`min_input_words=10` explicitly (they test other mechanics; fixtures sit
under the new production floor by design).
- Affected sweep (`-k "compress or ccr or crusher or router or mixed or
kompress or hermes"`, ~2.9k tests): green apart from order-dependent
flakes that shift identity between runs (deepseek tokenizer `AutoConfig`
import, hermes/proxy-ccr) — each passes standalone and in direct
combination with the new tests; the full CI shards are the authoritative
check.
- ruff 0.16.3 `check` + `format --check` clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
a708c0571e |
fix(ci): prevent native detector from hanging test shards (#2996)
## Description CI shard 4 was not merely slow: after thousands of fast tests it parked indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The router watchdogged only the first native call and then permanently trusted direct calls via `_detect_native_verified`. Earlier suite activity can change ORT/native state after that first success, making a later call deadlock until GitHub cancels the job. This keeps every native call bounded by the existing watchdog, activates the process-wide pure-Python circuit breaker after a timeout, restores the test-job ceiling to 30 minutes, and removes a separate wall-clock scheduler assertion that generated false shard-1 failures despite the structural regression guards passing. No issue is auto-closed by this infrastructure repair. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed the unsafe process-lifetime `_detect_native_verified` fast path. - Kept every native detection call behind the existing bounded watchdog. - Preserved the process-wide fallback circuit breaker so only the first wedged call consumes the watchdog budget. - Added a success-then-hang regression test. - Isolated native circuit-breaker state in fallback exception tests. - Restored the CI test timeout from the temporary 90-minute diagnostic ceiling to 30 minutes. - Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with its meaningful absolute regression ceiling while retaining source-level guards against the removed semaphore and nested executor. - Corrected import order and formatting defects inherited from current main so the synthetic merge commit passes repository-wide lint. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Exact local shard-4 command with coverage: 2723 passed, 172 skipped, 8661 deselected in 108.40s Focused detector/router suite: 62 passed Codex scheduler suite: 3 passed, 1 skipped ruff check . All checks passed! ruff format --check . 1411 files already formatted mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` Exact-head GitHub CI on `28f284c7a156d5be2fac2d21ba904a19d3389e6d` is entirely green. Test jobs 1–4, test-extras, test-agno, build, wheel, lint, CodeQL, dependency audit, secret scan, smoke, governance, and conflict checks all passed. Remaining skips are path-filtered jobs not applicable to this diff. ## Real Behavior Proof - Environment: macOS arm64/Python 3.13 locally; GitHub-hosted Ubuntu/Python 3.12 using the production CI workflow and prebuilt wheel. - Exact command / steps: reproduced `pytest tests scripts/tests --splits 4 --group 4 ...` hanging in native detection; sampled the parked process; reran with `pytest-timeout` to locate `_rust_detect`; applied the correction; reran the exact shard locally and all four CI shards remotely. - Observed result: local shard 4 completed in 1:48. GitHub shard 4's pytest step completed in 5:45 and its full job in 8:06 under the restored 30-minute ceiling. All four shards passed on the same head. - Not tested: deliberately wedging a real production ORT runtime outside the deterministic mocked regression; the watchdog behavior is covered with a native-call fake that succeeds once and then never returns. ## Runtime Rollout Safety - Rollout-managed feature(s): native content detection watchdog and fallback only. - Minimum rollout channel: normal patch release; no staged feature flag required. - Stable/default behavior changed: every native detection call remains watchdog-bounded instead of only the first successful call. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` bypasses native detection; `HEADROOM_DETECT_TIMEOUT_SECS` controls the watchdog budget. - Unsafe override required: none. - Qualification impact: full Python CI matrix must remain green; exact shard-4 completion is the primary qualification evidence. - Rollback path: human revert of this PR if bounded calls cause an unexpected regression; setting the Python backend provides an immediate operational fallback without code rollback. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation — inline lifecycle documentation and PR operational notes; no user-facing docs change is needed - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; no UI change. ## Additional Notes Human review only. No merge or auto-merge action has been configured. The branch includes current main and preserves the MCP SDK compatibility cap `mcp>=1.28.1,<2.0.0`. |
||
|
|
a3fe5cb65b |
fix(onnx): enforce Rust API-24 runtime compatibility (#2979)
## Description Rust fastembed enables ORT C API 24, but the Python dependency allowed ONNX Runtime 1.23.2. Entering ort's initializer with that library deadlocks permanently instead of returning an error. Align dependency resolution where compatible wheels exist and preflight native detection where they do not. Closes #2960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Require ONNX Runtime 1.24+ for Python 3.11+ in the proxy and voice extras. - Keep the available pre-1.24 runtime on Python 3.10 for Python ONNX consumers. - Refuse to auto-pin an incompatible runtime into the Rust extension. - Bypass native detection immediately when API 24 is unavailable, preserving Python fallback without a five-second watchdog delay or stuck native thread. - Add dependency, pinning, override, and router regression coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py tests/test_onnx_runtime.py tests/test_transforms/test_content_router.py 88 passed in 9.31s $ uv run ruff check headroom/_ort.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS arm64; Python 3.13.14 and uv-managed Python 3.10.20. - Exact command / steps: run the issue's direct `headroom._core.detect_content_type` call in a subprocess with a 12-second timeout on Python 3.13; run `_detect_content` on Python 3.10 after resolving the proxy extra. - Observed result: Python 3.13 resolves ORT 1.26.0 and native detection returns `json_array`; Python 3.10 resolves ORT 1.23.2, leaves `ORT_DYLIB_PATH` unset, reports compatibility false, and immediately returns the Python `json_array` fallback. - Not tested: Linux-specific shared-object execution locally; CI's existing Linux Rust job already preflights ORT 1.24+ and exercises native tests. ## Runtime Rollout Safety - Rollout-managed feature(s): Native Rust content detection. - Minimum rollout channel: Stable/default; this is a deadlock prevention guard. - Stable/default behavior changed: Python 3.11+ installs a compatible ORT; Python 3.10 skips incompatible native detection. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` remains available; an explicit `ORT_DYLIB_PATH` remains an operator override. - Unsafe override required: No. - Qualification impact: Native detection stays enabled only with API-24-compatible ORT. - Rollback path: Revert this PR, which restores the old watchdog-only degradation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The large lockfile diff is dependency resolution: Python 3.10 keeps ORT 1.23.2 while 3.11+ resolves 1.26.0. The functional Python change is intentionally small and keeps explicit `ORT_DYLIB_PATH` overrides working. |
||
|
|
3bb02f8f75 |
fix(transforms/smart_crusher): don't crash on a tool call with a null function (#2232)
## Description
A tool call whose `function` field is explicitly `null` crashes
SmartCrusher's per-request context extraction.
`_extract_context_from_messages` (called at the top of `apply()`) walks
recent assistant tool calls:
```python
for tc in msg.get("tool_calls", []):
if isinstance(tc, dict):
func = tc.get("function", {})
args = func.get("arguments", "")
```
`dict.get("function", {})` only substitutes `{}` when the key is
**missing**. When the key is present but `null` — `{"id": "1", "type":
"function", "function": null}`, which clients emit for a partial or
streamed tool call — `func` is `None`, and `None.get("arguments")`
raises `AttributeError`. That propagates out of
`_extract_context_from_messages` and crashes `apply()` for the entire
request, so the request either errors or has to fail open to
uncompressed with a logged traceback.
The sibling `_build_tool_name_index` in the same file already guards
this exact shape with `(tc.get("function") or {})` — this call site just
wasn't updated to match.
## Fix
Use the same null-safe form:
```python
func = tc.get("function") or {}
```
`None` (and any other falsy value) now collapses to `{}`, the null tool
call contributes no context, and extraction continues to the next call.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/smart_crusher.py`: `tc.get("function", {})` →
`tc.get("function") or {}` in `_extract_context_from_messages`.
- `tests/test_transforms/test_smart_crusher_bugs.py`: new test asserting
a `{"function": null}` tool call doesn't crash extraction and later
calls are still read.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_bugs.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/transforms/smart_crusher.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the extraction loop with a dependency-free script and left
the full pytest to CI.
- Exact command / steps: ran an assistant message with tool calls
`[{"function": null}, {"function": {"arguments": "keep-me"}}]` through
the OLD `get("function", {})` loop and the NEW `get("function") or {}`
loop.
- Observed result: OLD raises `AttributeError` on the null function; NEW
skips it and returns `"keep-me"` from the following call.
- Not tested: a live proxy request carrying a null-function tool call;
full local `pytest` deferred to CI (OOM).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new test uses the
existing `_make_crusher` helper in
`tests/test_transforms/test_smart_crusher_bugs.py`, so it runs under the
normal CI pytest job; behaviour is additionally verified by the
standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
d7bc1e275f |
fix(content-router): protect custom-tag blocks before mixed-content section split
Protect custom-tag blocks during mixed-content routing. |
||
|
|
677e09735a |
fix(transforms): stop ContentRouter recompressing headroom_retrieve results (#2654)
## Description `ContentRouter` (the transform actually registered in the default/proxy compression pipeline -- see `transforms/pipeline.py`) recompresses the output of its own `headroom_retrieve` tool. That tool's entire contract is returning already-retrieved, original content verbatim; recompressing it produces a new `<<ccr:hash>>` marker the caller can never redeem -- an unresolvable retrieval loop. `SmartCrusher` already has a guard against this exact failure mode (#1077), but only on its `apply()` entry point. `ContentRouter` calls the lower-level `SmartCrusher.crush()` directly, bypassing that guard entirely, since `crush()` takes a raw content string with no tool identity at all. Closes #1077 (reopens the same failure mode ContentRouter's own call path, which #1077's original fix did not cover). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `transforms/content_router.py`: adds an unconditional guard to all three of the places `ContentRouter` can hand a `headroom_retrieve` result to compression: the OpenAI-shape `role:"tool"`/legacy `role:"function"` string-content loop, the Anthropic-shape `tool_result` block loop, and a third, distinct shape -- top-level `{"type": "text"}` blocks under a `role:"tool"`/`"function"` message that never go through a `tool_result` wrapper (a real, already-tested wire shape in this codebase; see `test_tool_role_text_blocks_compressed_by_default`). All three use `is_tool_excluded()` (not a bare comparison) because MCP-served tools appear here under their qualified form, e.g. `mcp__headroom__headroom_retrieve`. Legacy `role:"function"` messages carry no call id in that shape, so the tool name is read directly off the message's `name` field instead of through the id-keyed `tool_name_map`. - Hoisted the per-iteration `is_tool_excluded(..., ("headroom_retrieve",))` calls into a single precomputed `ccr_retrieve_tool_ids` set, computed once alongside the existing `excluded_tool_ids` set, rather than recomputing aliases on every message/block. - `config.py`: adds `"headroom_retrieve"` to `DEFAULT_EXCLUDE_TOOLS` and `DEFAULT_VERBATIM_EXCLUDE_TOOLS` -- this also covers a third path (cross-turn message dedup, `_cross_turn_dedup_messages`) that consults the same frozensets and has no dedicated guard of its own. Also hardens `_tool_name_aliases()` against a non-string tool name (pre-existing fragility, not introduced by this PR, but shares the same call path) by returning no aliases instead of crashing on `.lower()`. - Documentation: updated `ContentRouterConfig.exclude_tools`'s field comment (was stale -- didn't mention this override is unconditional even when a caller explicitly empties `exclude_tools`), and added a comment on `DEFAULT_VERBATIM_EXCLUDE_TOOLS` noting all three real consumers. - Kept `"headroom_retrieve"` as a literal string (matching every other entry in those frozensets) rather than importing the existing `CCR_TOOL_NAME` constant from `ccr.tool_injection` into `content_router.py` -- that module is imported eagerly by `pipeline.py` (unlike `smart_crusher.py`, which imports the same constant lazily), so pulling in `headroom.ccr` there would add a new eager-import edge to a hot module for a one-line DRY win. Happy to change this if a maintainer prefers the constant. **Known, accepted tradeoff:** `is_tool_excluded()`'s alias matching strips any `mcp__<server>__` prefix before comparing, so a third-party MCP server exposing a tool literally named `headroom_retrieve` would also match. Narrowing this to headroom's own server specifically would need a bespoke check inconsistent with how every other excluded-tool entry is matched in this codebase; given how specific the name is, the collision risk is accepted rather than special-cased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_transforms/ tests/test_transforms_content_router.py -q 1 failed, 420 passed, 62 skipped in 12.50s FAILED tests/test_transforms/test_kompress_compressor.py::...test_onnx_session_options_read_thread_caps (pre-existing, unrelated to this diff -- confirmed via `git stash` that it fails identically against unmodified upstream/main; an ONNX thread-cap assertion, not a compression-routing test) $ uv run ruff check headroom/config.py headroom/transforms/content_router.py \ tests/test_transforms/test_content_router_ccr_retrieve_exemption.py \ tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py All checks passed! $ uv run ruff format --check <same files> 5 files already formatted $ uv run mypy headroom/config.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` - `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`: 10 tests -- MCP-qualified name (Anthropic + OpenAI shape), bare name, unconditional-even-with- `exclude_tools=frozenset()`, negative control (normal tools still compressed, asserted via the absence of the `router:excluded:ccr_retrieve` marker), the top-level-text-block shape, legacy `role:"function"`, litellm list-form content nested in a `tool_result` block, mixed retrieve+normal blocks in one turn, and a content well below the compression floor (proving the guard is size-independent). - `tests/test_transforms/test_content_router.py`: `test_anthropic_mcp_bare_tool_alias_exclude_tools` (#1822) updated to assert the new, stronger byte-verbatim guarantee for `headroom_retrieve` specifically; `test_anthropic_mcp_bare_tool_alias_exclude_tools_generic` added to keep the original #1822 general-mechanism coverage (bare-alias matching for an arbitrary, non-exempt tool). - `tests/test_transforms_content_router.py`: updated 10 pre-existing `_process_content_blocks()` unit tests for the new `ccr_retrieve_tool_ids` parameter (all pass empty sets -- none of those tests involve `headroom_retrieve`). - Verified the local installed package copy (a separate, drifted internal version) with a standalone repro script exercising the two new shapes directly against `ContentRouter.apply()` -- both correctly report `router:excluded:ccr_retrieve`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.7, `uv sync --extra dev` on this branch. - Exact command / steps: standalone repro building an assistant `tool_use` for `mcp__headroom__headroom_retrieve` paired with a large-JSON `tool_result`, through `ContentRouter().apply()`; repeated for the top-level-text-block and legacy-`function`-role shapes. - Observed result: unpatched (Anthropic `tool_result` shape, `git stash` to `upstream/main`), the retrieve output was rewritten 3680 -> 1861 bytes (mangled into a compact tabular form); patched (this branch), it is forwarded 3680 -> 3680 bytes byte-identical, no `<<ccr:` marker present. The two additional shapes fixed in this PR's second commit -- top-level text block under `role:"tool"`, and legacy OpenAI `role:"function"` -- both report `excluded=True` (protected) against this branch, where they reported `excluded=False` (recompressed) before the second commit. - Not tested: the actual `headroom mcp serve` + `headroom wrap` proxy end-to-end over a live Anthropic API call (would need API credentials); the OpenAI-chat-completions `CompressionUnit` path (out of scope, see #1176 below); the opt-in `ToolResultInterceptorTransform` path. ## Relationship to other issues/PRs - Issue #1077 (closed) is this exact bug; PR #1323 fixed it only for `SmartCrusher.apply()`'s own call path (the "legacy" pipeline path, per `smart_crusher.py`'s own comment), not `ContentRouter`, which is what the default/proxy pipeline actually uses. - Open PR #1176 addresses an adjacent, non-overlapping gap: the `CompressionUnit`-based OpenAI chat-completions path (`router.compress()` calls in `transforms/compression_units.py`/`compression_batches.py`), which has no tool-identity context at all and needs its own capture/restore mechanism. This PR does not touch that path. - Filed #2656 as a follow-up: code review on this PR found the same bug class still reachable through `SmartCrusher.apply()`'s own bare-name guard (not alias-aware, so it misses the MCP-qualified form) and through two unguarded direct `.crush()` calls in the LangGraph and Strands integrations. Both are pre-existing, narrower/separate call paths from `ContentRouter`'s primary proxy pipeline, so tracking them separately keeps this PR reviewable as one logical change. - Also not covered by this PR (flagging rather than silently omitting): `proxy/system_compaction.py`'s `router.compress(text, context="")` call, and the opt-in `ToolResultInterceptorTransform` (`HEADROOM_INTERCEPT_ENABLED=1`) -- neither was checked for CCR-awareness. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` -- it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A -- this is a backend compression-routing fix with no UI surface. ## Additional Notes This PR is two commits: the first commit added the initial two-loop guard; a second commit followed after code review found the guard was incomplete for two additional wire shapes (top-level text blocks, legacy `role:"function"`) and added the missing test coverage plus a few cleanup items (deduplicated guard logic, comment accuracy, a pre-existing non-string-tool-name fragility). See `Changes Made` above for the full list. Filed #2656 for the remaining out-of-scope gaps found during that same review. --------- Co-authored-by: Michael Tarleton <mtarleton@istation.com> |
||
|
|
3e348f327f |
fix(ccr): stop persisting retrieval markers as original content (#2694) (#2703)
## Description CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at all — where the original bytes belonged, so `headroom_retrieve(hash)` answered with the very placeholder the caller was trying to resolve. For a base64/credential field that is permanent, silent data loss: the inner marker's hash is the only handle on the real payload, and it disappears from anywhere the model can see. Four sites, one root cause — **a compressed intermediate (or nothing) was stored in place of the source**, the same defect class as #1209 (tag placeholders persisted as originals). Closes #2694 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`compaction/walker.rs`** — `walk_array` compacted through the store-LESS `compact()`. Opaque cells inside a compacted table got a marker whose payload was **never written**, so retrieval 404'd forever. Now uses `compact_with_store` so the emitted hash resolves. - **`compaction/classifier.rs`** — nothing stopped an already-marked string from being offloaded a second time, which stashed the MARKER as the new entry's "original". Marker-bearing text is our own output, not source content, so it is never classified opaque. One guard at the choke point both the walker and the table compactor share. - **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop marker hashed and stored rows whose leaves were **already** rewritten (prose compressed, blobs marker-substituted), so retrieving dropped rows returned compressed output. Now hashes and stashes the pre-processing array via `crush_array_with_source`. - **`content_router.py`** — compression pinning matched only `Retrieve more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so opaque-blob output was readmitted to the compressor on a later turn — the path that feeds the corruption above. Consolidated into `_is_already_compressed()` and applied at all three pinning sites. - **`cache/compression_store.py`** — store-level guard: refuse to persist a *bare* marker as `original_content` and log at ERROR, so a future producer regression surfaces loudly instead of silently converting "retrievable" into "gone". Deliberately narrow — originals may legally *contain* markers (nested offloads); only a bare marker is rejected. - **Regression tests** — `test_nested_table_markers_resolve_to_source_bytes` (asserts payloads are verbatim-retrievable, not merely that a marker was emitted) and `test_already_marked_content_is_not_re_offloaded`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo build -p headroom-core Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s $ cargo test -p headroom-core --lib smart_crusher test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s $ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q 16 passed in 0.68s $ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q 50 passed in 12.52s $ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q 100 passed in 13.14s $ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \ tests/test_transforms/test_smart_crusher_ccr_roundtrip.py All checks passed! $ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai 0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt via `maturin develop --release`. - **Exact command / steps:** compact a nested document — 5 rows whose `detail` field is a stringified sub-array of 6 base64 blobs (1600 B each) — then, for every `<<ccr:HASH>>` marker in the output, call `ccr_get(HASH)` and check the payload is the verbatim source rather than a marker. ```python inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)] doc = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]} out = SmartCrusher().compact_document_json(json.dumps(doc)) for h in re.findall(r"<<ccr:([0-9a-f]+)", out): payload = crusher.ccr_get(h) # must be real bytes, not a marker ``` - **Observed result — BEFORE (on `main`):** all six payloads collapsed into a single dead marker. The rendered sub-table was re-classified opaque (`html`, because `<<` reads as a tag), offloaded again, and its payload never stored — so the six inner hashes were erased from the visible text *and* the outer hash resolved to nothing. ```text {"rows":"[5]{detail:string,id:int,note:string} <<ccr:3fb1d44933da,html,289B>>,0,x <<ccr:3fb1d44933da,html,289B>>,1,x ..."} 3fb1d44933da -> RUST MISS # unrecoverable — 6 × 1600 B gone ``` - **Observed result — AFTER (this branch):** the sub-table stays inline, each blob keeps its own marker, and every marker resolves to verbatim source. ```text {"rows":"[5]{detail:string,id:int,note:string} \"[6]{k:string,tok:string,v:int} key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."} 6ad5846997f4: resolves, len=1600, is-verbatim-source=True 78a0bd9364a7: resolves, len=1600, is-verbatim-source=True 955b1fed2ef7: resolves, len=1600, is-verbatim-source=True a0cef69da7f0: resolves, len=1600, is-verbatim-source=True dfcde5e940c0: resolves, len=1600, is-verbatim-source=True e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True RESULT: PASS — every marker resolves to real source bytes ``` ## Notes for reviewers - The issue also reports **function words dropped from retained prose** (`is`, `a`, `the`) and **interleaved log output corrupting `headroom doctor`'s table borders**. Those are separate defects on different paths (extractive prose compression and log-handler buffering respectively) and are **not** addressed here — this PR is scoped to the CCR store/retrieve corruption. They should be tracked separately; the prose one overlaps #2586. - The `crusher.rs` prose-hook fix is on the Rust pipeline (`json_offload`) rather than the Python proxy path, but it is the same store-the-intermediate bug and was cheap to close while in the file. |
||
|
|
1a2688b57f |
test(kompress): close patch-coverage gaps from #2716 (#2721)
## Description Codecov flagged 9 uncovered lines on #2716 after it merged: `hf_entry_known_absent`'s own body in `headroom/onnx_runtime.py` was only ever exercised indirectly (every existing test in `tests/test_transforms/test_kompress_compressor.py` monkeypatched it away rather than calling the real implementation), and `_load_pytorch_weights` / `_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py` had three untested branches: the double cache-miss under `allow_download=False` (merged.pt confirmed absent AND the plain fallback also not cached), a genuine non-404 download failure propagating instead of silently falling back, and the already-cached fast path in `_load_kompress_pytorch`. ## Type of Change - [ ] Bug fix - [ ] New feature - [x] Test coverage improvement, no production code change ## Changes Made - `tests/test_onnx_runtime.py`: added `_write_fake_hf_cache` (builds a minimal on-disk HF hub cache layout, including the `.no_exist/<hash>/<filename>` marker huggingface_hub writes after a real 404) and three direct tests of `hf_entry_known_absent` against the real `huggingface_hub.try_to_load_from_cache`, not a mock of it. - `tests/test_transforms/test_kompress_compressor.py`: added `test_cache_only_raises_when_confirmed_absent_but_plain_also_missing`, `test_genuine_download_failure_propagates_instead_of_falling_back`, and a new `TestLoadKompressPytorchCaching` class covering the already-cached fast path. ## Testing ```text $ .venv/bin/python3 -m pytest tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py -q 51 passed $ .venv/bin/python3 -m pytest tests/ -k "kompress or onnx_runtime" -q --cov=headroom.transforms.kompress_compressor --cov=headroom.onnx_runtime --cov-report=term-missing # before: onnx_runtime.py Missing includes 132-136 (hf_entry_known_absent's entire body); # kompress_compressor.py Missing includes 805-806, 818, 836 # after: none of those lines appear in Missing anymore 191 passed, 7 skipped $ .venv/bin/python3 -m ruff format --check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py 2 files already formatted $ .venv/bin/python3 -m ruff check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py All checks passed! ``` ## Review Readiness - Test-only, additive diff (113 insertions, 0 deletions, 0 lines touched outside the two test files). No behavior change possible. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes Not closing: the remaining branch-partial on the `device == "auto"` cuda/mps/cpu selection in `_load_kompress_pytorch` (would need mocking `torch.cuda.is_available()` / `torch.backends.mps.is_available()` for marginal benefit); left as-is. |
||
|
|
46da91b2f1 |
fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors (#2716)
# PR draft: fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors Branch: `rnoz/fix-kompress-merged-checkpoint` (off `upstream/main`). Two commits. Issue: https://github.com/headroomlabs-ai/headroom/issues/2714 (filed, open). --- ## Description `_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py` downloaded `model.safetensors` from the default model repo `chopratejas/kompress-v2-base` and loaded it with `strict=False`, discarding the missing/unexpected key report. That file is the unmerged PEFT checkpoint (encoder keys prefixed `encoder.base_model.model...`), which never matches `HeadroomCompressorModel`'s plain `encoder.*` keys. The LoRA-adapted encoder weights were silently dropped while `token_head`/`span_conv` happened to match and loaded fine, so the model ran with a stock, non-adapted `answerdotai/ModernBERT-base` encoder feeding correctly trained decision heads, with no error and a healthy status reported everywhere. `scripts/export_kompress_v2_onnx.py` already documents this exact mismatch and loads the correct `merged.pt` sub-state-dicts for its own export path. This PR mirrors that same loading logic into the runtime PyTorch loader, with a fallback to the plain `model.safetensors` format for repos that never shipped a `merged.pt` (verified against the v1 `chopratejas/kompress-base` repo via the public HF API, which has no `merged.pt`). Both paths now check the missing/unexpected key report and raise instead of silently proceeding on a mismatch. A second commit fixes a gap an adversarial review caught in the first: the cache-only (`allow_download=False`, startup preload) path could not tell "this repo genuinely has no merged.pt" apart from "merged.pt exists but is not downloaded yet", so it would have fallen back to a stale `model.safetensors` left over from before this fix on exactly the upgrade path this PR is meant to close. It now uses `huggingface_hub`'s own `.no_exist` cache marker (via a new `hf_entry_known_absent()` helper in `headroom/onnx_runtime.py`) to make that distinction without a network call, and only falls back when absence is confirmed; otherwise it raises `KompressModelNotCached` so the caller defers instead of guessing. Closes #2714 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/kompress_compressor.py`: added `_load_merged_state_dict`, `_load_plain_state_dict`, and `_load_pytorch_weights`, replacing the inline `model.safetensors` download + `load_state_dict(strict=False)` call in `_load_kompress_pytorch`. `merged.pt` is tried first; the plain format is only used when its absence is confirmed. - `headroom/onnx_runtime.py`: added `hf_entry_known_absent()`, a thin wrapper around `huggingface_hub.try_to_load_from_cache()` that reads the on-disk `.no_exist` marker HF writes after a real 404, so cache-only code can distinguish "confirmed absent" from "never checked" without hitting the network. - `tests/test_transforms/test_kompress_compressor.py`: added `TestPytorchWeightLoading` (8 tests) covering the merged-checkpoint happy path, missing-section and key-mismatch failures, the plain-format fallback for repos without `merged.pt`, and the cache-only ambiguity fix (confirmed-absent vs unconfirmed). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] Manual testing performed (real download against the live `chopratejas/kompress-v2-base` repo, not just mocks) ### Test Output ```text $ .venv/bin/python3 -m pytest tests/ -k kompress -q 178 passed, 7 skipped in 22.39s $ .venv/bin/python3 -m ruff check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py All checks passed! $ .venv/bin/python3 -m ruff format --check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py 3 files already formatted $ .venv/bin/python3 -m mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py Success: no issues found in 2 source files ``` ## Real Behavior Proof Ran the actual fixed loader against the live `chopratejas/kompress-v2-base` HF repo (not a mock), before and after each fix: ```text # BEFORE (parsed the real cached model.safetensors header by hand, no safetensors lib needed): total tensors: 316 encoder.base_model.model.embeddings.norm.weight encoder.base_model.model.embeddings.tok_embeddings.weight ...(all 310 encoder tensors share this prefix)... span_conv.0.bias / span_conv.0.weight / span_conv.2.bias / span_conv.2.weight token_head.bias / token_head.weight exact prefix match count with plain "encoder.<rest>": 0 # This confirms the pre-fix code's model.load_state_dict(state_dict, strict=False) # silently dropped every encoder weight (0 keys match HeadroomCompressorModel.encoder), # while token_head/span_conv happened to match and loaded. # AFTER (commit 1, real merged.pt download + load): $ .venv/bin/python3 -c " import headroom.transforms.kompress_compressor as kmod model = kmod._get_model_class()() kmod._load_pytorch_weights(model, 'chopratejas/kompress-v2-base', allow_download=True) print('SUCCESS: 0 missing/unexpected keys across all three sections') " SUCCESS: 0 missing/unexpected keys across all three sections # AFTER (full pipeline, real end-to-end compression through the public API): $ .venv/bin/python3 -c " import headroom.transforms.kompress_compressor as kmod compressor = kmod.KompressCompressor() result = compressor.compress(sample_traceback_plus_boilerplate_text) print(result.original_tokens, result.compressed_tokens, result.tokens_saved) print('ValueError: bad input' in result.compressed) " 497 454 43 True # must-keep line (the actual error) survived compression # AFTER (commit 2, cache-only ambiguity): unit tests # test_cache_only_defers_instead_of_using_stale_plain_checkpoint: PASSED # test_cache_only_uses_plain_checkpoint_when_merged_pt_confirmed_absent: PASSED ``` ## Review Readiness - [x] I have performed a self-review - [x] An independent adversarial review pass was run on both commits before this PR was opened; its one finding (the cache-only ambiguity) is fixed in commit 2, verified with new regression tests - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal loader behavior, no public API or config surface changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective (regression tests for both the original silent-drop bug and the cache-only ambiguity found in review) - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes - Both `merged.pt` and the plain `model.safetensors` fallback now raise loudly on any state-dict mismatch instead of proceeding with partially-loaded weights, closing the general silent-failure class this bug belonged to, not just this one instance of it. - No other call sites of `_load_kompress_pytorch` or its removed inline code exist; its public signature is unchanged. |
||
|
|
6d5516dcb8 |
feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already *detected* as code (Magika labels in `headroom/compression/detector.py` include `php`, and the Rust `magika_detector.rs` lists it too) but there was no PHP `LangConfig`, so PHP content silently passed through uncompressed. This wires PHP through the tree-sitter compression path following the C# pattern (the most recently added, fully functional language — deliberately not the quarantined Perl path). A secondary detection bug is fixed along the way: PHP's `$variables` match Perl's prefilter regex, and the existing Perl-dominance guard in `detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php` open tag — which no Perl source contains — now drops Perl from the candidate set before that guard runs. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` + `phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the actual tree-sitter-php grammar (node names verified by parsing samples): `namespace_use_declaration` imports, `function_definition`/`method_declaration` functions, `class_declaration`/`interface_declaration`/`trait_declaration` classes, `enum_declaration` types, `declaration_list` class bodies, `compound_statement` function bodies. `namespace_definition` maps to `package_node` so statement-scoped `namespace App;` hoists ahead of the `use` imports (required PHP ordering); the rare block-scoped `namespace A { }` form takes the same path and is preserved verbatim — valid output, just no compression inside the block. PHP prefilter regexes added; supported-languages error message updated; `<?php`-tag Perl disambiguation in `detect_language`. - `headroom/transforms/content_detector.py`: `php` entry in `_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the code-aware route. - `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport` mirroring `TestCSharpSupport` — signatures preserved / bodies elided, `<?php` → `namespace` → `use` → declarations ordering, auto-detection despite the Perl sigil overlap, alias coercion, malformed passthrough. - `tests/test_code_compressor_language_alias.py`: `php` in the canonical list, `phtml` in the alias table. - `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2 supported-languages row. No new dependency: `tree-sitter-language-pack` (the existing `[code]` extra) already ships the PHP grammar. No Rust changes needed. ## Testing - [x] New unit tests added and passing - [x] Full affected test suites pass locally **Test Output** ``` $ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q ============================= 120 passed in 7.81s ============================= $ python -m pytest tests/test_transforms/ -q 3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate, # text_crusher unicode parity) reproduce identically on a clean # upstream/main checkout in this environment — pre-existing local # ONNX runtime quirks, unrelated to this change $ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean $ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, tree-sitter + tree-sitter-language-pack (<1.0) installed, branch `feat/201-php-code-compression` off `upstream/main`. - Exact command / steps: parsed PHP samples (namespaced class w/ methods, block-scoped namespace, mixed HTML+PHP) with `tree_sitter_language_pack.get_parser('php')` to verify every node name used in the config; then ran `CodeAwareCompressor().compress(php_code, language="php")` and `compress(php_code)` (auto-detection) on a 48-line realistic service class. - Observed result: explicit and auto-detected paths both return `language=CodeLanguage.PHP`, `compression_ratio=0.64`, `syntax_valid=True`; method bodies elided to `// [N lines omitted]` while `<?php`, `namespace`, `use` lines, class header, and all signatures are preserved verbatim in the original order. Before the detection fix, auto-detection returned `UNKNOWN` (Perl prefilter dominance) — reproduced and then verified fixed. - Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]` on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic mixed case); these fall back to verbatim preservation via the uncaptured-node pass or malformed-passthrough, both of which are covered by tests for the simple cases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c811007f81 |
fix(kompress): match all ONNX backends with startswith, not exact "onnx" (#2448)
## Description With `HEADROOM_KOMPRESS_BACKEND=onnx_coreml`, every Kompress compression call and the startup canary crash with `'_OnnxModel' object has no attribute 'parameters'`, so Kompress silently degrades to passthrough and `/health` reports `kompress: unhealthy, backend: null`. Root cause: `headroom/transforms/kompress_compressor.py` gated the ONNX-vs-PyTorch branch with an exact string match `backend == "onnx"`. But `_load_kompress_onnx` returns `onnx_coreml` (CoreML) or `onnx_cpu` — never the bare string `onnx`. So under `onnx_coreml` the code built PyTorch tensors and dispatched to a device via `next(model.parameters())`, which the `_OnnxModel` wrapper doesn't implement. This is the accelerated backend Apple Silicon users reach for, so the fast path is exactly the broken one. Fixes #2442 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Change the four exact-match `backend == "onnx"` sites in `headroom/transforms/kompress_compressor.py` to `backend.startswith("onnx")`, matching the convention already used by `_model_device_type`: `_timed_canary`, `compress`, `compress_batch`, and the batch-parallelism guard in `_should_use_sequential_fallback`. - Update the guard comment ("ONNX CPU provider" → "ONNX EPs") since it now covers all ONNX execution providers. - Add regression tests exercising `_timed_canary` on `onnx_coreml` (must take the numpy path and never touch `.parameters()`) with a negative control proving the PyTorch branch still dispatches to a device. - Leave `CHANGELOG.md` untouched — release-please generates it from conventional commits. - Out of scope: the secondary `/health` under-reporting the issue flags as informational (deferred-preload warmup object never flips to `loaded`). ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_kompress_compressor.py::TestOnnxBackendPrefixGating -q`) - [x] Linting passes (`ruff check`, `ruff format --check` on the two changed files) - [x] Type checking passes (`mypy headroom/transforms/kompress_compressor.py --ignore-missing-imports`) - [x] New tests added for new functionality ### Test Output ```text $ python -m pytest tests/test_transforms/test_kompress_compressor.py::TestOnnxBackendPrefixGating -q collected 2 items tests\test_transforms\test_kompress_compressor.py .. [100%] 2 passed in 2.20s $ ruff check headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, local dev checkout on a branch off upstream/main (no Apple Silicon / CoreML hardware available) - Exact command / steps: Ran the new `TestOnnxBackendPrefixGating` regression; then temporarily reverted one site back to `backend == "onnx"` and re-ran to confirm the test discriminates. - Observed result: With the fix, `_timed_canary(model, tokenizer, "onnx_coreml")` returns a float and never touches `.parameters()`. Reverting one site makes the onnx_coreml test fail (it takes the `pt` tensor path and hits the paramless model), proving the test catches the exact bug. The issue reporter separately verified the fix on real Apple Silicon hardware (onnxruntime 1.27.0, CoreMLExecutionProvider): zero occurrences of the error afterward and compression completing on the CoreML session. - Not tested: End-to-end run on real CoreML hardware from this environment — reproduced via the unit-level device-dispatch seam instead; hardware confirmation is in the issue. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
b75999017f |
fix(transforms/kompress-remote): keep compress fail-open on malformed 200 (#2320)
## Description
`RemoteKompressCompressor` (the opt-in `HEADROOM_KOMPRESS_ENDPOINT`
remote compression client) documents a fail-open contract in its own
docstring:
> Fails OPEN: any network/HTTP error returns the content verbatim so a
flaky endpoint degrades compression rather than breaking the proxy.
But only the network call and the `compressed` field check actually run
inside the fail-open guard. The metadata coercions run **after** the
`except`, outside it:
```python
try:
resp = self._client.post(...)
resp.raise_for_status()
data = resp.json()
compressed = data["compressed"]
if not isinstance(compressed, str):
raise TypeError("...")
except Exception as e: # fail OPEN
logger.warning("Remote Kompress failed (%s); passing through", e)
return self._passthrough(content, n_words)
result = KompressResult(
compressed=compressed,
original=content,
original_tokens=int(data.get("original_tokens", n_words)),
compressed_tokens=int(data.get("compressed_tokens", len(compressed.split()))),
compression_ratio=float(data.get("compression_ratio", 1.0)), # <-- outside the guard
model_used=str(data.get("model_used", self.config.model_id)),
)
```
So a hosted `/compress` endpoint that returns a 200 with a valid
`compressed` string but a malformed metadata field escapes the guard and
raises out of `compress`, breaking the proxy request instead of passing
through. The most realistic trigger is an explicit JSON `null`:
`data.get("compression_ratio", 1.0)` returns `None` for a **present**
key (the default only applies to a missing key), and `float(None)`
raises `TypeError`. A non-numeric string like `"original_tokens":
"lots"` raises `ValueError` the same way. Since the whole point of the
flag is to support arbitrary self-hosted endpoints, a slightly-off but
well-meaning endpoint (sending `null` for a field it could not compute)
takes down the request path this class exists to protect.
## Fix
Move the response parsing (the `KompressResult` construction with its
`int`/`float`/`str` coercions) inside the fail-open `try`, so any
malformed field degrades to verbatim passthrough like every other
bad-response case:
```python
try:
...
compressed = data["compressed"]
if not isinstance(compressed, str):
raise TypeError("...")
result = KompressResult(
compressed=compressed,
original=content,
original_tokens=int(data.get("original_tokens", n_words)),
compressed_tokens=int(data.get("compressed_tokens", len(compressed.split()))),
compression_ratio=float(data.get("compression_ratio", 1.0)),
model_used=str(data.get("model_used", self.config.model_id)),
)
except Exception as e: # fail OPEN
logger.warning("Remote Kompress failed (%s); passing through", e)
return self._passthrough(content, n_words)
```
No behavior change on a well-formed response; only the malformed-200
path changes (raise to passthrough).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/kompress_remote.py`: move the `KompressResult`
construction and its field coercions inside the fail-open `try`.
- `tests/test_transforms/test_kompress_remote.py`: add
`test_remote_kompress_null_numeric_field_fails_open` (explicit JSON
`null`) and `test_remote_kompress_non_numeric_field_fails_open`
(non-numeric string), both asserting verbatim passthrough.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/transforms/kompress_remote.py tests/test_transforms/test_kompress_remote.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/transforms/kompress_remote.py tests/test_transforms/test_kompress_remote.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/transforms/kompress_remote.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the control flow with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (coercions outside the `try`)
and NEW (inside the `try`) parsing against a 200 body `{"compressed":
"short result", "compression_ratio": null}` and against a well-formed
body.
- Observed result: OLD raised `TypeError` on the null field (proxy
request breaks); NEW returned passthrough; a well-formed body still
compressed under NEW. The added tests assert both malformed cases
(`null` and non-numeric string) return the original content with
`compression_ratio == 1.0`.
- Not tested: a live remote Kompress endpoint; the added tests drive
`RemoteKompressCompressor` through an `httpx.MockTransport`, matching
the existing test harness in this file.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added tests reuse the
existing `httpx.MockTransport` harness in `test_kompress_remote.py` and
run under the normal CI pytest job, and the behavior is corroborated by
the standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|