Commit Graph
143 Commits
Author SHA1 Message Date
zhengqiuyangandJD Davis 303f521812 fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description

A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.

This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.

Closes #4874

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
  string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored

$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored

# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
  `fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib

compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds

`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
  it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
  it the string renders as the JSON string literal (`"""[]"""`), which
  `expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
  included for CI, where the wheel is built normally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:50:47 -06:00
f4e3ad4b11 fix(content-router): keep list-form tool_result text blocks on separate lines (#4739)
## Description

A `tool_result` whose `content` is a list of several `text` blocks gets
flattened with `"".join(...)` in `ContentRouter` before the
read-protection check and before compression. MCP tools that return more
than one `TextContent` and litellm-style clients send this shape. When a
block does not end with a newline, the last line of one block and the
first line of the next one become a single line. The compressed result
is written back as one block, so the model sees a line the tool never
produced.

This PR joins the blocks with a newline unless the block already ends
with one, at both flatten sites.

Fixes #4729

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: new helper
`_join_tool_result_text_blocks()` that joins list-form `text` blocks
with `\n`, skipping the separator when a block already ends with a
newline. Empty blocks add no extra line.
- Used at both places that flattened with `""`: the read-protection
check and the compression path. The wire shape does not change (the
existing `[{"type": "text", ...}]` re-wrap is kept), and results that
are not compressed keep their original blocks.
- `tests/test_transforms/test_content_router.py`: one `router.apply`
test with a two-block log `tool_result` that asserts that the output was
compressed and that every log line in it is a line the tool actually
returned, plus four parametrized cases for the helper.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Exact source: head `5305d8acb5f505e635d1287cae30ed7f8d9eb595`, base
`c05fbdddc873078dd97cb9ede841e75ef7ed0bef`. The PR has three commits:
`f273fbb9` (the fix, written on `1da8930a`) and two merges of `main`
(`2cba6c5b`, `5305d8ac`), both without conflicts. `git diff
c05fbddd...5305d8ac` still touches only the two files listed above.

Upstream CI on `5305d8ac`, all `completed` / `success`: CI
[43338122484](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122484),
Merge Conflicts
[43338122336](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122336),
Changelog Guard
[43338122370](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122370),
rust
[43338122318](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122318),
Evaluation Suite
[43338122592](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122592),
Security
[43338122460](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122460).

Local runs below were built from `git archive 5305d8ac` in a
`python:3.13-slim-bookworm` image with `uv sync --frozen --extra proxy
--extra dev`. ruff ran on the two touched files and mypy on
`content_router.py`. pytest ran on the content router test modules. For
the reverse check, `content_router.py` was replaced with the copy from
`c05fbddd` and the tests kept.

A wider run over the 122 test files that touch the content router gave 9
failed / 2669 passed on `5305d8ac` and 9 failed / 2664 passed on
`c05fbddd`. The 9 failures are the same set on both (memory_bridge,
offline_egress docs, tabular; they come from the test image, not this
change), and the extra 5 passes are the new tests.

### Test Output

```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new tests, 5 consecutive runs, all identical:
======================= 5 passed, 66 deselected in 5.01s =======================
# test_content_router.py + test_content_router_ccr_retrieve_exemption.py + test_content_router_shell_small_results.py:
============================= 125 passed in 5.61s ==============================
# with content_router.py restored to c05fbddd (fix removed):
E   AssertionError: assert ['2026-11-10T...er-0 retry 0'] == []
FAILED tests/test_transforms/test_content_router.py::test_list_form_tool_result_blocks_are_not_fused
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts0-a\nb\nc]
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts1-a\nb]
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts2-a\nb]
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts3-only]
================== 5 failed, 1 passed, 65 deselected in 1.88s ==================
```

The four `test_join_tool_result_text_blocks` cases fail on the base
because the helper does not exist there. The passing test in the last
run is the control
`test_anthropic_tool_result_lossy_without_marker_stays_verbatim`.

## Real Behavior Proof

- Environment: Linux container, Python 3.13, headroom installed from
this branch with `uv sync --frozen --extra proxy`. Upstream is a local
stdlib HTTP server on 127.0.0.1:9911 that records the `/v1/messages`
body it receives and returns a fixed `ok` message. Dummy API key, no
real provider.
- Exact command / steps: `python -m headroom.cli proxy --host 127.0.0.1
--port 8799 --no-telemetry --no-rate-limit --anthropic-api-url
http://127.0.0.1:9911`, run once with the default mode and once with
`--mode token`. Then POST `/v1/messages` with a `tool_result` whose
content is two `text` blocks of 150 log lines each, the first block not
ending in a newline. Same steps against `main` for comparison.
- Observed result: the proxy compressed the tool result in all four runs
(`content_router: 3 msgs — 1 compressed (tool:log:0.03)`) and the client
got `200` with `ok`. On `main` the body forwarded upstream had 1 fused
line, `2026-11-10T10:29:00Z INFO worker-2 processed batch 149
ok2026-11-10T11:00:00Z WARN worker-0 retrying batch 0`. With this patch
it had 0 fused lines in both modes.
- Not tested: a real Anthropic or OpenAI endpoint, SSE streaming
responses, Bedrock, the OpenAI `/v1/chat/completions` handler (it goes
through the same `ContentRouter`), Windows and macOS.

## Runtime Rollout Safety

- Rollout-managed feature(s): N/A, this is a bug fix in the existing
compression path with no flag.
- Minimum rollout channel: N/A, no rollout-managed feature.
- Stable/default behavior changed: Yes, only for list-form `tool_result`
content with more than one `text` block: the blocks are now joined with
a newline instead of nothing before compression.
- Kill switch / disable path: N/A, no flag. Disabling compression for
the tool, as today, skips this path.
- Unsafe override required: No.
- Qualification impact: None expected. Single-block and string
`tool_result` content takes the same path as before.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`; it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation is N/A, there is no user-facing change to document. I
checked the other provider paths: OpenAI `role: "tool"` messages with
list content are handled block by block and are not joined, so they do
not have this bug. "New and existing unit tests pass locally" refers to
the content router test modules listed above, run in the container.

Written with help from Claude Code.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4739"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:25:28 -06:00
3aafdf2dcc fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description

When the code compressor declined a block of source code, the router
passed the block to Kompress. Kompress is a prose model. It drops
low-salience words, and in code those words are keywords such as `for`,
`in`, `if` and `import`. The result no longer parses. #4214 shows a
fenced Python script that comes back with `for` and `in` removed.

This PR stops that in the Python router and in the Rust live-zone
dispatcher. When the code compressor declines, the code is forwarded
unchanged.

Closes #4214

Credit: the Python change is oxura's work from
https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018
(277ca1f). Their commit is the first commit here, rebased onto `main`,
and oxura is its author. The test, docs and Rust commits were added on
top. oxura asked us to open this PR from our fork
([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)),
and said they have not run this branch, so the model and native results
below are ours, not theirs.

## Behaviour change to review

With `enable_code_aware=False` (the default), source code now stays
unchanged. Before this PR, it was sent to Kompress and lost keywords.
With `enable_code_aware=True`, code that the code compressor declines
also stays unchanged. Before, Kompress ran on it.
A code block the code compressor does compress is unaffected. Prose,
logs, JSON and tables are unaffected.

Side effect: a code block that Kompress used to shrink (and break) now
keeps its full size. This is the intended trade, but it means fewer
tokens saved on code when `enable_code_aware=False`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
- [ ] New feature
- [ ] Breaking change (the default-config behaviour change above is
intended, and it only stops damage to code)

## Changes Made

Five commits:

- `596764c3` `fix(router): keep source code out of prose fallbacks`.
Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in
`headroom/transforms/content_router.py`. This is 277ca1f rebased onto
`main`, and it differs from 277ca1f in two ways only: the two test-file
conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because
`AGENTS.md` says not to add content under `wiki/`.
- `7705a4e6` `test(router): repin the code-to-Kompress tests and add the
#4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by:
oxura` trailer. Five tests asserted the old fallback. They are rewritten
to assert the new behaviour (code unchanged, Kompress never sees it)
instead of being deleted. Adds a regression test for the script from
#4214, with `enable_code_aware` both off and on. Updates
`docs/content/docs/code-compression.mdx` and
`docs/content/docs/how-compression-works.mdx`, which said that detected
code goes to Kompress.
- `6631668a` `fix(live-zone): leave source code unchanged when the code
compressor declines`. Author: Tao Hansen. Same fix in
`crates/headroom-core/src/transforms/live_zone.rs`: a declined
`SourceCode` block is now a NoOp, not a Kompress input. Tests:
`tests/live_zone_source_code_decline.rs` (single test, needs no model),
and new cases in `tests/live_zone_dispatch.rs`.
- `d8776283` `fix(router): turn off the code compressor's internal
Kompress fallback`. Author: Tao Hansen. Follows review: the router's
code compressor was still built with
`fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown
languages, Perl and AST failures could build a `KompressCompressor`
inside `CodeAwareCompressor`. The router now always sets
`fallback_to_kompress=False`. The #4719 test that pinned `True`
(`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as
`test_enabled_kompress_still_turns_off_code_aware_fallback`, and the
unknown-language, Perl (by language hint, since `detect_language()`
reports Perl-dominant code as unknown) and AST-failure paths are tested
with spies on `is_kompress_available` and `KompressCompressor`, with
`enable_kompress` both off and on.
`docs/content/docs/code-compression.mdx` notes that the router always
turns the option off.
- `c1f8464e` `fix(router): keep passthrough code from external
compressors and stale caches`. Author: Tao Hansen. Follows review:
`_try_external_compressor` skips `PASSTHROUGH` blocks (code with
`prefer_code_aware_for_code=False`, whole or as a mixed section), which
would otherwise be offered to a prose-only external compressor as
`text/plain`. The disk cache namespace now also hashes
`_ROUTING_POLICY_REVISION`, so a same-version restart does not replay
code that Kompress damaged before this change. Tests: two
external-dispatch cases with a `text/plain` external, and one namespace
test. All three fail on the previous head.

I did not touch `CHANGELOG.md`.

## Testing

- [x] Unit tests pass (`pytest`): see below, the failures are
environment failures that also fail on `main`
- [x] Linting passes (`ruff check .`, `ruff format --check .`)
- [x] Type checking passes (`mypy
headroom/transforms/content_router.py`)
- [x] New tests added for new behaviour
- [x] Manual testing performed (real model replay, below)

### Test Output

macOS, Python 3.14. Head is now `c1f8464e`.

```text
After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean.
After the rebase onto main c05fbddd and the review fix (d8776283):
  router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped
  tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail
    (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure)
  ruff check / ruff format --check headroom tests: clean
  cargo test -p headroom-core: 1151 passed, 0 failed
Earlier run on 9df60c7c (before the rebase):
router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed:
  test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget,
  a timing test the branch does not touch; it passes 3 of 3 times when rerun alone
ruff check .:           All checks passed!
ruff format --check headroom tests: 1714 files already formatted
mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
cargo test -p headroom-core:   22 test binaries, 1151 passed, 0 failed
cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean
pytest tests (full):     31 failed, 19208 passed, 513 skipped
```

Rust bug-fix tests, fail before and pass after (the router-test commit's
source with the new Rust tests, then the live-zone commit; measured
before the rebase as `9df60c7c`, and the Rust code is unchanged since):

```text
before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on 7705a4e6)
        source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed)
after:  all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok
```

The dispatch fail-before was measured on the earlier, code-identical
branch (only documentation files differ). The dispatch test only fails
before the fix when the Kompress model is loaded (`HF_HOME` with the
model, `ORT_DYLIB_PATH` set). The single-test binary fails before the
fix on any host.

Python regression: with the new tests run against `origin/main`'s router
(no fix), 9 test cases fail (the #4214 fence test in both configs, the
repinned tests, and the fenced-code keyword test). With the fix they
pass.

### Failures in the full `pytest tests` run

The full run on the branch had 31 failures, none in router or
compression code. I compared them with pristine `origin/main`
(200c838c):

- 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path
is too long under this checkout directory (`OSError`). They fail the
same way in isolation on `main`.
- 17 other failures are in `test_proxy_health.py`,
`test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`,
`test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`,
`test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A
full run on `main` fails 29 tests, and the set differs from run to run
(the two sets overlap by 17 of 31). I re-ran the 14 failures that were
not in the `main` list in isolation on the branch: 12 pass, the other 2
are the UDS tests above.
- Known flaky test:
`tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`
failed once in a loaded run and passes on rerun. File repeated 3 times
on `596764c3` and on `origin/main`: 16 passed each time. It is timing
based and not caused by this change. It passed in the 558-test router
run above.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`,
real `kompress-v2-base` int8 ONNX model from an offline copy of the
public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the
venv's onnxruntime.
- Fence: a 1324-character valid Python script. The issue quotes only
four of its lines, so the rest is a reconstruction in the same style,
not the reporter's bytes.
- Exact command / steps: `ContentRouter.compress` on the fence followed
by about 40 lines of prose, for four configs (default;
`enable_code_aware=True`; `prefer_code_aware_for_code=False`;
`enable_code_aware=True` with lossless-then-lossy). Also
`headroom.compress` on a full Anthropic message list with the fence in a
`tool_result`.
- Observed result: Before (v0.44.0): in the default and code-aware
configs the fence comes back as 1144 characters and does not parse
(`import json, collections` becomes `json, collections`; `[s for s in
...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs
and in `headroom.compress`, the fence is byte-identical, 1324
characters, and parses. Prose control: the same router on prose still
compresses. A prose-only block went from 5709 to 5357 characters
(strategy TEXT). A prose / fence / prose block went from 12760 to 12056
characters (strategy MIXED), and the fence in it is byte-identical and
parses.
- Not tested: the reporter's exact 1324-character fence (only four lines
are public, so the tests use the reconstructed fence described above);
the Rust live-zone path end to end through the proxy with a real request
(covered by `cargo test` only); platforms other than macOS; the full
proxy on a live deployment.

## Other places where code can still reach a prose model

I checked every path from code to a prose or ML compressor, in Python
and Rust.

| Path | State |
| --- | --- |
| Python `CODE_AWARE` branch, code compressor declines or is disabled |
Fixed here: the block stays unchanged |
| Python post-strategy fallback after "no savings" (`content_router.py`,
`fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only
applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so
unchanged |
| Python lossless-then-lossy (opt-in) after the lossless fold |
`CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged |
| Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in)
| Prose-like output, not code. Unchanged |
| Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged |
| Python mixed content with fenced code (`_compress_mixed`) | Each fence
goes through the `CODE_AWARE` branch, so it is fixed by this change.
Covered by the new fence tests |
| Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before
this PR |
| Python `_experimental_compress_read`
(`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment
that runs Kompress on file reads, which can be code. Off by default.
Left as is |
| Python size gate and deadline fallbacks in `_try_ml_compressor` | Run
only after a caller chose Kompress, so code no longer reaches them |
| Python `CodeAwareCompressor` internal fallback
(`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST
failure) | Fixed in commit 4: the router always builds it with
`fallback_to_kompress=False`. Covered by Kompress-spy tests |
| Python external compressor dispatch for `PASSTHROUGH` code | Fixed in
commit 5: never offered to an external compressor |
| Disk result cache made before this change | Fixed in commit 5: a new
namespace through `_ROUTING_POLICY_REVISION` |
| Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) |
| Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It
is the single remaining call to `kompress_or_noop` |
| Rust: prose text with an embedded fenced code block | **Not
reproduced.** I built synthetic tool results of prose around the fenced
Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher
classified each one as `source_code`, so commit 3 leaves it unchanged,
and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has
no fence-aware splitting, but I found no input where that sends code to
Kompress, so this PR files no follow-up. |
| Python `config_compressor.py` docstring says YAML/TOML/INI "fall
through to the lossy prose path" | Documentation only: a stale comment,
not a code path. Config content has its own `CONFIG` strategy. Can be
refreshed in a later change |

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, in the default config code is no
longer sent to Kompress (see "Behaviour change to review"). Tokens saved
on code drop in return for code that stays valid.
- Kill switch / disable path: none added. Revert the PR to restore the
old fallback.
- Unsafe override required: no.
- Qualification impact: none expected. Compression of code by the code
compressor is unchanged.
- Rollback path: revert the three commits. No data, schema or config
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines (`ruff`, `cargo
fmt`, `clippy` clean)
- [x] I have made corresponding changes to the documentation
(`docs/content/docs/code-compression.mdx`,
`docs/content/docs/how-compression-works.mdx`)
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
- [ ] New and existing unit tests pass locally: the only failures are
environment failures that also fail on `main`, listed above

## Additional Notes

- Rust has no fence-aware splitting like the Python router has. A
synthetic prose-plus-fence repro did not reach Kompress (see the table
above), so no follow-up issue is filed.
- One stale comment remains in `content_router.py`
(`_try_ml_compressor`, the Phase 0 size-gate note lists
`CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not
changed here to keep the review commits as they are. Say if you want it
fixed in this PR.
- Reviewers: the Python change is easiest to review commit by commit.
Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and
the Rust fix.

---------

Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:25:06 -06:00
c21b0813bd fix(compress): stop adding markers the caller did not ask for (#4678)
`/v1/compress` returns a body that someone else's client forwards
unchanged, so anything Headroom adds there reaches the provider. Two
things got added that the caller never asked for.

## Stale-Read retrieval hashes in marker-free modes

Read lifecycle replaces a Read that a later edit made stale with a short
notice. That notice always ended in `Retrieve original: hash=…` and
wrote the original to the CCR store, even in the default and
`lossy_inline` modes. The docs call those modes marker-free, and their
callers have no retrieval tool, so the hash pointed nowhere.

- Marker-free modes keep the notice ("re-read the file for current
content") but drop the hash and skip the store write.
- Lossless routers (proxy `--lossless`,
`ContentRouterConfig(lossless=True)`) skip read lifecycle entirely,
because the notice drops the file's bytes.
- Read maturation now recognises the hashless notice. Otherwise it would
replace the notice with a marker whose hash points at the notice itself.

## Cache breakpoints the client didn't send

On the session path, `normalize_message_cache_control` fell back to
placing `{"type": "ephemeral"}` on the last block message when the
client marked nothing. A caller that chose not to cache got billed cache
writes. `/v1/compress` now passes `place_unmarked=False`: if the client
marked no message, markers are stripped and none are placed. Proxy
handlers keep today's behaviour, and client markers in a misaligned list
still go through the old path.

## Tests

Each new test fails on main.

- `test_marker_free_modes_mark_a_stale_read_without_a_retrieval_hash`,
for the default mode and `lossy_inline`
- `test_lossless_router_keeps_a_stale_read`
- `test_session_turns_add_no_marker_the_caller_did_not_send`
- `test_respects_lifecycle_markers`, extended to the hashless notice

The related suites (gateway, transforms, read maturation, prefix
tracker, cache_control, compress endpoint and session) pass: 1371 passed
and 9 skipped before the review fixes, then 222 of 222 on the touched
files after them. Lint is clean on CI's ruff 0.17.0, and mypy is clean
on the touched modules.

## Not in this PR

- Read maturation (`handlers/anthropic.py`) is not gated on `--lossless`
or `--no-ccr` either. It's the same kind of bug on the proxy path, so it
belongs in a separate fix.
- The "lossless densify only" pipeline in `compress.py` sets
`ccr_enabled=False` without `lossless=True`, so read lifecycle still
runs there.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4678"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Real Behavior Proof

Maintainer qualification on Windows, Python 3.12.13, using an actual
Uvicorn server bound to an owned loopback socket and HTTPX over real
HTTP. The normal `/v1/compress` endpoint and compression pipeline ran
with offline mode and Kompress disabled; no handler, compression stage,
HTTP response or store was mocked. Workspace, configuration, savings and
TOIN paths were isolated. All 243 native inputs and the reused artifact
SHA-256
`5be63a4f31d3e38e7427d28380ecb977616e3f6a3dd6a1c59af5ff6361a05978` were
verified; this was not a fresh native build.

- Exact command / steps: the committed probe is `python -m pytest -q -s
tests/test_compress_runtime_http.py`. The qualifying run used the
maintainer loopback-only launcher with an explicit verified native path
and SHA-256, followed by the four focused
session/API/maturation/lifecycle suites. All 138 tests passed. The probe
starts the actual server, GETs `/health`, and sends nine POSTs to
`/v1/compress`: a Read followed by Edit in default, lossy-inline and
lossless modes, then three session turns without caller cache markers
and three turns with a caller marker. It stops and joins the server in
`finally`.
- Observed result: health and all nine compression requests returned
HTTP 200. Default and lossy-inline emitted stale notices with no
retrieval hash or CCR hashes, saving 3,576 tokens per request. Lossless
preserved the original Read bytes. All three unmarked session turns
returned zero cache markers; all three marked controls retained exactly
one. Session compression saved 7,350 / 15,901 / 25,958 tokens. Server
shutdown joined successfully.
- Before-fix control: the same real-server requests with all six
affected production modules replaced by current main
`200c838c7393d20ae56a0c338f832ed27133c398` failed five assertions:
default and lossy-inline included retrieval hashes, and each unmarked
session turn added a cache marker. Lossless and the three marked
controls passed on main and are not claimed as reproduced regressions.
Candidate source was restored byte-for-byte.
- Not tested: provider forwarding or billing, a live model, deployment
behavior, or the separate proxy read-maturation scope already listed
above. External startup connection attempts were blocked by the
qualifying launcher; this is not a zero-attempt network claim. No author
readiness or self-review declaration is added by this maintainer
evidence.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-13 09:39:10 -06:00
8cdce172ea fix(router): ignore shell operators inside quoted arguments in read detection (#4646)
## Description

Item 3 of #4478, the quoted-pattern case that #4521 left open. Read
detection scanned the raw command text for shell syntax without regard
to quoting: `_segment_is_read` treated any `>` as a redirect
(`content_router.py:893` on `e4a60ed7`) and split the pipeline at the
first `|`, and `_command_segments` split on `;`, `&&` and `||`. A quiet
sed whose script holds one of these characters, such as `sed -n '/if x >
0/,/return/p' a.py` or the GNU alternation `sed -n '/def \|class /p'
a.py`, was therefore not detected as a file read and lost read
protection.

The three scans now run on a copy of the command in which quoted,
backslash-escaped and commented characters are masked, and the match
positions are mapped back onto the original text. Two shell rules keep
word boundaries intact (review round 2): a backslash-newline is a line
continuation and is masked as two spaces, so a redirect or `tee` on the
next line still counts as a write; and an unquoted `#` at the start of a
word masks the rest of its line without changing quote state, so a quote
mark inside a comment cannot hide the next line's separators. Unquoted
operators behave as before.

Refs #4478

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: new `_mask_quoted(command)`,
which returns a same-length string with characters inside `'...'` /
`"..."`, backslash-escaped characters and comment text replaced by `_`
(the quote characters are kept; a backslash-newline becomes two spaces;
a comment's newline stays visible). `_segment_is_read` runs its
redirect/`tee` check and its pipeline split on the masked string;
`_command_segments` finds `;` / `&&` / `||` on the masked string and
slices the original.
- `tests/test_transforms/test_read_command_detection.py`:
`test_quoted_shell_operators_are_arguments`, 25 cases. Quoted `>`, `>>`,
`|`, `;`, `&&` and `tee` inside a sed script or a file name are reads; a
real redirect, pipe-to-`tee` or separator next to a quoted script still
counts; `bash -lc "cat foo.py > copy.py"` is still not a read (the inner
command is checked on its own, as before). Round 2 adds: continued `>`,
`>>` and `| tee` are writes; a comment with an unmatched `"` or `'`
followed by `ls; cat foo.py` / `true; cat foo.py` on the next line is a
read; `cat foo.py # > not a redirect` is a read; a `#` inside a word is
not a comment.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) (run on
`headroom/transforms/content_router.py` only, see below)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (source at e4a60ed7, new test kept)
tests/test_transforms/test_read_command_detection.py: 9 failed, 104 passed

# round 2: the new review cases against the round-1 head 639bfa9e
7 failed, 115 passed

# pass after (head 50798e5a)
tests/test_transforms/test_read_command_detection.py: 122 passed

# wider set, same environment:
# tests/test_transforms + tests/test_*content_router*.py + tests/test_router*.py
e4a60ed7:     27 failed, 930 passed, 96 skipped   (round 1 comparison)
639bfa9e:     25 failed, 941 passed, 96 skipped
50798e5a:     18 failed, 948 passed, 96 skipped
# the 18 are the same tests on every side (the extras are the new cases); they
# come from the local `_core` build not matching this tree (Rust parity
# fixtures, minification), not from this change

# ruff check / ruff format --check: All checks passed! / 2 files already formatted
# mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2) on Windows 11, Python 3.12.15, `headroom`
Python source at `e4a60ed7` and at this branch, offline.
- Exact command / steps: `ContentRouter.apply` on a three-message
transcript (user, an assistant `Bash` tool call, a 3,890-byte Python
tool result), with `HEADROOM_PROTECT_READS=1`,
`HEADROOM_SHELL_RESULT_MIN_CHARS=2000` and the router config used by
`test_quiet_sed_output_reaches_read_protection`
(`enable_kompress=False`, `enable_code_aware=False`). Checked whether
`router:read_protected` is in `transforms_applied` for each command.
- Observed result: on `e4a60ed7` the three quoted-operator commands lose
read protection; on this branch they keep it, and both controls are
unchanged (table below). For the review-round cases, real `bash -c` in a
temporary directory agrees with `_is_read_command` on this head for all
7 (continued `>`, `>>` and `| tee` write `copy.py`; the three
multiline-comment commands and `cat foo.py # > not a redirect` print the
file); on `639bfa9e` 6 of the 7 disagreed with bash.

  | command | `e4a60ed7` | this branch |
  |---|---|---|
  | `sed -n '1,120p' foo.py` (control) | protected | protected |
| `sed -n '/if value > 0/,/return 0/p' foo.py` | not protected |
protected |
  | `sed -n '/def \|class /p' foo.py` | not protected | protected |
  | `sed -n '/x;y/p' foo.py` | not protected | protected |
| `sed -n '1,120p' foo.py > out.txt` (control, real redirect) | not
protected | not protected |

- Not tested: a run in which an unprotected read is actually compressed.
With Kompress and code-aware compression off, the tool result came back
byte-identical on both sides, so this proof shows the classification
only. Not tested: the full test suite and `mypy headroom` as a whole
(only `content_router.py`). Heredoc handling (`<<`) is unchanged and
still a whole-command bailout.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: commands whose only
`>`, `|`, `;`, `&&` or `||` sits inside quotes, after a backslash or
inside a comment are now classified by the rest of the command, so a
quiet sed or `cat` of that kind gets read protection when
`HEADROOM_PROTECT_READS` is on.
- Kill switch / disable path: None added; read protection as a whole is
still controlled by `HEADROOM_PROTECT_READS`.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

#4478 had four items. Items 1 and 2 were fixed by #4488, item 4 by
#4583, and the quiet-option half of item 3 by #4521, whose description
notes that quoted redirect parsing remained separate. This PR covers
that remainder. The `|` and `;` cases are the same cause as the `>` case
reported in the issue (a quote-blind scan of the same command), so they
are fixed together rather than left for a later PR.

Review round 2 (head `50798e5a`) addresses the two masking regressions
found by Devin and Copilot and confirmed by JerrettDavis:
backslash-newline continuations and quote marks inside comments.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


### Maintainer requalification

Re-reviewed the author update against upstream main
e4a60ed72f. The two earlier actual Git
Bash counterexamples now classify correctly: continued redirect writes
an exact copy and is not a read; a quote inside a comment no longer
hides the next-line source read. All 132 focused read-detection and
Gemini tests pass; touched-source Ruff/format and whitespace checks
pass. The only maintainer change is the shared Gemini ASGI request test
fixture from #4644; hosted failure was the same four independently
verified main failures, with no contributor production logic changed.
Addressed Devin/Copilot threads resolved after inspecting author fixes.
Exact-head hosted CI remains pending.


### Maintainer current-main requalification

Exact head `2461a910e31c5140d160942927c1c7b3fa203cc1`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Cleanly integrated upstream/main
ea59ac3eef and re-reviewed the additive
content-router overlap with main's confident HTML classification. Actual
isolated tree-sitter0.26.0:133 read-command/HTML tests pass,94 unrelated
cases deselected,1 HTML assertion fails. A control with byte-identical
main Python/transforms/test/lock inputs reproduces that same
unrecoverable_sections assertion using the same existing development
native extension, so it is not a finding against this PR. Read-command
cases pass; prior actual Git Bash continued-redirection/comment-quote
evidence remains scoped to unchanged shell parser inputs. Ruff and
whitespace pass. No contributor behavior rewritten, fresh native build
or cloud runtime claim. No new PR correctness finding; final approval
awaits exact-head hosted checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `69a6bd592932467162d9433601b0c2bd1c4c4728`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 134 passed, 94 deselected in 5.40s.
Selected real dependency overlays and native DLL paths are recorded in
final4646-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification. The earlier HTML assertion with
the old primary extension now passes with the fresh combined DLL; the
same case also passes on a byte-identical-main Python-source control. No
production workaround or assertion suppression was required.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:20:15 -06:00
f333edf20f fix(router): keep tagged blocks out of the lossy size-gate fallback (#4543)
## Description

When a text block is over the Kompress size gate
(`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000) and contains protected
custom tags, `ContentRouter._try_ml_compressor` passed the
`{{HEADROOM_TAG_N}}` placeholder text to the TextCrusher / LogCompressor
fallback. Two things went wrong:

1. Neither fallback takes a `ccr_original` argument, so CCR stored the
placeholder text. `headroom_retrieve` returned `{{HEADROOM_TAG_0}}`,
`{{HEADROOM_TAG_1}}`, ... instead of the tagged content.
2. When the fallback dropped the lines holding the placeholders,
`restore_tags` discarded each lost wrap (by design,
`tag_protector_placeholder_lost`). The tagged content was then gone from
the output sent upstream too.

This is the same defect class as #1209 (Kompress stored the placeholder
intermediate) and #2694 (SmartCrusher, `ccr_source`), on the size-gate
branch those fixes did not reach.

The fix: when tags were protected, the size-gate branch returns the
original `content` unchanged instead of running the lossy fallback. That
is what the request-budget branch directly above it already does for the
same reason (the bytes must stay stable and intact). I chose this over
threading a `ccr_original` through the fallbacks because
`TextCrusher.compress` and `LogCompressor.compress` have no such
parameter, and the output would still lose the tagged lines through
`restore_tags`. Passing through is the smallest change and loses
nothing; the cost is that a gated block with custom tags is no longer
shrunk by the fallback.

Fixes #4542

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: in the Kompress size-gate
branch of `_try_ml_compressor`, return `content` unchanged when
`protected` is non-empty. The now-unreachable `restore_tags` call at the
end of that branch is removed (the fallback only runs when nothing is
protected).
- `tests/test_transforms/test_kompress_size_gate.py`: new regression
test `test_gate_keeps_tagged_blocks_out_of_lossy_fallback`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Fail before (fix reverted, new test kept):

```text
FAILED tests/test_transforms/test_kompress_size_gate.py::test_gate_keeps_tagged_blocks_out_of_lossy_fallback
========================= 1 failed, 5 passed in 2.19s ==========================
```

Pass after:

```text
tests/test_transforms/test_kompress_size_gate.py ......                  [100%]

============================== 6 passed in 1.70s ===============================
```

Related suites (`tests/test_transforms` plus
`tests/test_ccr_tag_placeholder_regression.py`, excluding
`test_text_crusher_cjk_eval.py`, which imports `benchmarks/` that my
test copy does not include):

```text
FAILED tests/test_transforms/test_kompress_compressor.py::TestLazyImports::test_unloadable_native_dep_degrades_instead_of_raising
1 failed, 520 passed, 78 skipped in 24.66s
```

That one failure (`assert attempts["torch"] == 1`) is identical with my
change reverted (`1 failed, 30 passed, 11 skipped` on
`test_kompress_compressor.py` alone), so it is not caused by this PR.

```text
$ ruff check headroom/transforms/content_router.py tests/test_transforms/test_kompress_size_gate.py
All checks passed!
$ ruff format --check <same files>
2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2), Python 3.12.15, `main` at `a2d4fe72` with
the `_core` extension from the 0.42.0 wheel (no commits under
`tag_protector.rs`, `log_compressor.rs` or `crates/headroom-py` since
0.42.0), `HEADROOM_OFFLINE=1`, in-memory CCR.
- Exact command / steps: ran the script from #4542 (a short header, then
N lines of `<item>row i value</item> note text here for item number i`,
through `ContentRouter(config=ContentRouterConfig()).compress(text)`)
for N=3000 and N=4000, before and after the change, and retrieved the
CCR entry behind any `hash=` marker.
- Observed result: before the change, N=4000 became `[4006 lines
omitted]\n[4006 lines compressed to 0. Retrieve more:
hash=dda7b8a79b740b17a0a5461b]`, and the retrieved original was 229,839
bytes, not equal to the input, with 0 `<item>` and 4,000
`{{HEADROOM_TAG`. After the change, N=4000 comes back byte-identical
(253,839 chars, 4,000 `<item>`, 0 placeholders, no CCR marker). N=3000
is unchanged in both runs (189,839 chars, 3,000 `<item>`).
- Not tested: the Rust-side `tag_protector` and a freshly built `_core`
(I used the prebuilt 0.42.0 wheel's); a live proxy run; mypy; the case
where the gated block has no protected tags (unchanged code path,
covered by the existing gate tests); other platforms.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is a bug fix inside the
existing size-gate branch.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly. A block over the size
gate that contains protected custom tags is now returned unchanged
instead of being run through TextCrusher/LogCompressor with the tags
lost. Blocks with no protected tags take the same path as before.
- Kill switch / disable path: `HEADROOM_KOMPRESS_MAX_TOKENS=0` disables
the gate entirely (existing behavior).
- Unsafe override required: No
- Qualification impact: None expected. Token savings drop only for gated
blocks that carry custom tags, which previously lost the tagged content.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (one
unrelated failure, see Test Output)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Docs unchanged: no public API or documented behavior changes. An
alternative would be to teach TextCrusher/LogCompressor a `ccr_original`
and also keep the tagged lines; that touches two compressors (and the
Rust side) for a block shape that is rare, so I kept the change to the
router. Happy to do it that way if you prefer.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4543"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `38f4e6e6c37ae4b7b46bf97be50ad5fe320c2094`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically merged upstream/main e4a60ed7 and re-reviewed the combined
size-gate branch and existing review discussion. Protected tagged blocks
return original content before either lossy fallback can consume
placeholder text; unprotected fallback behavior remains intact. No
contributor production logic was rewritten. All 11 size-gate and CCR
tag-regression tests pass with the existing development native
extension; Ruff lint/format and whitespace checks pass. The existing
documented before/after router reproduction remains applicable; this is
not a fresh native-build or live-proxy qualification claim.

The shared test-only Gemini streamed-request fixture repair addresses
independently established main failures while preserving existing
assertions; all ten affected cases pass. Formal approval awaits
exact-head hosted qualification.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


Maintainer commit-message qualification repair: current head
addf0470b7. Corrected only the maintainer
main-sync commit message to Conventional Commit format. The complete
file tree is byte-identical to the previously tested head; exact-head
hosted checks are being refreshed. Nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 10:44:09 -06:00
db30d82dff fix(router): log the ONNX fallbacks at info instead of warning (#4535)
<!-- maintainer-current-main -->
## Maintainer latest-main checkpoint
Published `fe40ae66e3d0dd9e7df00be81716c42370139004` incorporates
upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 23
ORT/dylib/logging/Gemini tests pass; Ruff/format/diff pass. Main sync is
clean; expected-fallback source changes remain only the two INFO log
levels. Shared Gemini request-fixture repair is test-only. No new code
finding.

The inherited main shard-2 failure was confirmed in actual hosted logs
and locally (four Gemini failures). This head carries the maintainer
fixture repair: real ASGI Request/JSON bytes replace an unused
body-reader mock. All six test-function ASTs/assertions are preserved;
all 10 Gemini cases pass. Contributor production logic was not rewritten
by the sync/fixture follow-up. Earlier runtime captures retain their
original commit/platform scope. Fresh exact-head hosted qualification is
requested; nothing merged by this review.
<!-- /maintainer-current-main -->

## Description



A plain `pip install headroom-ai` has no ONNX Runtime (it comes with
`[proxy]`), and the pure-Python detector is the supported path there.
The first `compress()` call still logged "Native content detection
requires ONNX Runtime 1.24+; using pure-Python detection for this
process" as a WARNING. Every library user sees it on first import; both
coding agents in our sandbox runs saw it, and one spent a step
investigating it before continuing.


## Type of Change



- [x] Bug fix (non-breaking change that fixes an issue)



## Changes Made



- `headroom/_ort.py`: the sibling "onnxruntime x.y exposes an older C
API than Rust detection requires" message drops to INFO as well. It
fires on Intel macOS, where the `proxy` extra pins `onnxruntime <1.24`
by design (`docs/installation.mdx`: Rust ML detection stays disabled
there), so a supported `[proxy]` install warned on first `compress()`
too. Its test now asserts the level.


- `headroom/transforms/content_router.py`: `logger.warning` to
`logger.info` for the ORT-compat fallback, with a comment on why. The
existing circuit breaker (`_detect_native_unhealthy`) already makes it
fire once per process; unchanged.
-
`tests/test_transforms/test_ort_dylib.py::test_content_router_bypasses_native_detector_for_incompatible_ort`:
now captures at INFO, asserts the record's level is INFO and that a
second call logs nothing.


## Testing



- [x] Unit tests: the test fails before the fix (`assert 30 == 20`) and
passes after; `tests/test_transforms/test_ort_dylib.py` 10 passed.
- [x] `ruff check` / `ruff format --check`



### Test Output



```text

pytest tests/test_transforms/test_ort_dylib.py: 10 passed (fails before the fix: assert 30 == 20)

source build, default logging: 20031 -> 95 FATAL kept: True  (no fallback line on stderr)

source build, INFO enabled: INFO headroom.transforms.content_router: Native content detection requires ONNX Runtime 1.24+; using pure-Python detection for this process.

```



## Real Behavior Proof



- Environment: macOS arm64, Python 3.12, clean venv with the published
0.42.0 wheel (no onnxruntime).
- Exact command / steps: `HEADROOM_OFFLINE=1 python -c "from headroom
import compress; ..."` on a 3,000-line log.
- Observed result: the warning line on stderr on first import, then
`20031 -> 95`.
- After-fix, this branch built from source (`uv pip install -e .` with
maturin, fresh venv with tiktoken, pydantic, filelock,
opentelemetry-api; no onnxruntime):
  ```text

headroom from /Users/tejas/headroom-wt-onnx-info/headroom/__init__.py
  20031 -> 95 FATAL kept: True

  ```

No fallback line on stderr at default logging. With
`logging.basicConfig(level=INFO)` the same run prints
`INFO headroom.transforms.content_router: Native content detection
requires ONNX Runtime 1.24+; using pure-Python detection for this
process.` once.
- Not tested: the native (ORT present) path, which this change does not
touch.


## Runtime Rollout Safety



- Rollout-managed feature(s): none; two log records change level from
WARNING to INFO, no behaviour change
- Minimum rollout channel: stable (ships with the next release)

- Stable/default behavior changed: only log level; detection and
compression paths are untouched
- Kill switch / disable path: not needed; `HEADROOM_LOG_LEVEL`-style
logger config still surfaces the line at INFO
- Unsafe override required: no

- Qualification impact: none

- Rollback path: revert the commit



## Review Readiness



- [x] I have performed a self-review

- [x] This PR is ready for human review



🤖 Generated with [Claude Code](https://claude.com/claude-code)





## Maintainer post-merge revisit: main e9331b6b

Published `6d9b9a52ccb939fd6c404c1debe955ecda3af3a9` includes upstream
main `e9331b6b963b5b6554f45943abed2366aaa25ea6`. 13 ORT
runtime/dylib/logging tests passed. Clean main sync; the source change
remains limited to expected fallback log levels, preserving fallback
behavior and once-per-process behavior. Author source-built default/INFO
evidence and latest discussion reconsidered; no new finding.
Touched-file Ruff/format and whitespace checks pass. Contributor runtime
logic was not rewritten. Scoped Windows tests use the existing
development native extension; no fresh native build or installed-agent
qualification is claimed. Hosted CI must finish on this exact head
before reapproval. Nothing merged.


### Maintainer current-main requalification

Exact head `7b7b3a1624c3e644f67b2ad9707c7e222ed2771c`, upstream main
`ecbaf86d78cf52d185fbffa306157b9a4cc1b906`.

Mechanically updated against upstream/main
ecbaf86d78 and re-reviewed the two
expected ONNX-unavailable/older-API fallback records. Only severity
changes to INFO; fallback behavior and once-per-process suppression
remain intact. The test-only conflict retains main's two-chunk ASGI
request helper and all memory assertions. No contributor production
logic was rewritten.

All 23 ORT/logging/Gemini tests pass; Ruff0.17.3 lint/format and
whitespace checks pass. The author's source-built no-ORT default/INFO
after-fix replay and its macOS scope were read and remain applicable; no
new source-built runtime replay is claimed here. Local execution used
the existing native extension. Formal approval awaits exact-head hosted
qualification; maintainers handle those checks. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 10:39:25 -06:00
7c4af752ad fix(transforms): stop the dense-line elider cutting Chinese, Japanese and Thai prose (#4295)
## Description

`is_dense_line()` (`headroom/transforms/dense_line_elider.py`, #3685)
decides a line is minified or encoded output if it is at least 300
characters long and less than 6% of it is spaces. Chinese, Japanese and
Thai don't put spaces between words, so an ordinary paragraph in those
languages passes the same test: a 455-character Chinese paragraph has
zero ASCII spaces.

Once the dense lines in a block add up to 2,000 characters, which is
about five paragraphs of a fetched Chinese news or docs page,
`elide_dense_lines()` keeps the first 160 and last 80 characters of
**each paragraph** and replaces the middle with `...[N chars of dense
machine-generated content elided]...`. This runs on the router's text
fallback, so `headroom.compress()` and the proxy both cut readable prose
mid-sentence and tell the model it was "machine-generated content". The
original is CCR-retrievable, but the model has no reason to suspect it's
missing article text.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/dense_line_elider.py`: new `MAX_NON_ASCII_RATIO =
0.1`. `is_dense_line()` returns `False` when more than 10% of a line's
characters are non-ASCII. 10% rather than 20% (changed after review) so
that CJK prose inside scraped HTML, where long ASCII attributes or URLs
make up most of the line, is kept too. Minified JS/CSS, base64, JWT/RSC
payloads and hex blobs are almost entirely ASCII, so they are still
elided. The check is `not line.isascii() and n -
len(line.encode("ascii", "ignore")) > n * 0.1`, which runs in C: about
0.003 ms on a 1.1 MB ASCII line and 2 ms on a 1.1 MB line with some CJK
in it. `is_dense_line()` already spends hundreds of ms on such lines in
its JSON scan.
- `tests/test_transforms/test_dense_line_elider.py`:
- `test_space_free_scripts_are_prose_not_dense` (zh / ja / th): six long
paragraphs pass through `elide_dense_lines()` byte-identical.
- `test_cjk_prose_wrapped_in_long_ascii_markup_is_kept`: six `<p
data-article-url="…long URL…">` lines with 84 Chinese characters each
(10–20% non-ASCII) pass through unchanged;
- `test_dense_lines_still_elided_next_to_cjk_prose`: minified JS and
base64 next to a Chinese paragraph are still elided (2 lines), and the
paragraph is kept.

## Testing

- [x] Unit tests pass (`pytest`):
`tests/test_transforms/test_dense_line_elider.py`,
`tests/test_json_protection_boundaries.py`,
`tests/test_content_router*.py`,
`tests/test_transforms_content_router.py`,
`tests/test_transforms/test_content_router.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `main` (fix reverted, tests kept):

```text
$ python -m pytest tests/test_transforms/test_dense_line_elider.py -q
FAILED tests/test_transforms/test_dense_line_elider.py::test_space_free_scripts_are_prose_not_dense[ja]
FAILED tests/test_transforms/test_dense_line_elider.py::test_space_free_scripts_are_prose_not_dense[th]
FAILED tests/test_transforms/test_dense_line_elider.py::test_space_free_scripts_are_prose_not_dense[zh]
FAILED tests/test_transforms/test_dense_line_elider.py::test_cjk_prose_wrapped_in_long_ascii_markup_is_kept
FAILED tests/test_transforms/test_dense_line_elider.py::test_dense_lines_still_elided_next_to_cjk_prose
========================= 5 failed, 9 passed in 6.65s ==========================
```

With the fix:

```text
$ python -m pytest tests/test_transforms/test_dense_line_elider.py tests/test_json_protection_boundaries.py \
    tests/test_content_router*.py tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py -q
======================= 254 passed, 2 skipped in 18.97s ========================

$ ruff check headroom/transforms/dense_line_elider.py tests/test_transforms/test_dense_line_elider.py
All checks passed!
$ ruff format --check headroom/transforms/dense_line_elider.py tests/test_transforms/test_dense_line_elider.py
2 files already formatted
```

(On the first revision, one run of that command had a failure I didn't
capture. Two reruns passed. The machine had a load average of about 50
at the time, and the router's deadline tests are timing-sensitive.)

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`, public `headroom.compress()` API
- Exact command / steps: called `headroom.compress(messages,
model="gpt-4o", model_limit=128000, protect_recent=0)` on an
OpenAI-format conversation whose `fetch_page` tool result is a Chinese
article: a title plus six 455-character paragraphs (2,751 chars). I
counted elision markers in the compressed tool result and printed an
excerpt, on `main` and on this branch.
- Observed result: on `main` all 6 paragraphs are cut
(`router:text:0.59`, 2,751 → 1,918 chars), each with a `...[215 chars of
dense machine-generated content elided]...` marker in mid-sentence. On
this branch the article goes through unchanged (0 of 6 elided).

  ```text
  == main
  chars 2751 -> 1918 | transforms: [..., 'router:text:0.59']
  paragraphs elided: 6 of 6
excerpt: 能相似的方式做出反应的智能机器,该领域的研究包括机器人、语 ...[215 chars of dense
machine-generated content elided]... 一个分支,它企图了解
  == this branch
chars 2751 -> 2751 | transforms: ['router:protected:user_message',
'router:protected:user_message']
  paragraphs elided: 0 of 6
  ```
- Not tested: the live proxy. Minified bundles whose string literals
push non-ASCII above 10% (a synthetic i18n bundle at 14% is now kept)
are left alone rather than elided. That's the false negative the
architecture principles prefer, and ASCII-dominated bundles, base64 and
RSC are still elided. Other unspaced scripts (Lao, Khmer, Burmese) are
covered by the same rule, but I didn't test them.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: long lines that are more than 10%
non-ASCII are no longer elided. ASCII-dominated dense lines (minified
JS/CSS, base64, RSC) are elided exactly as before.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

This is not the same as the Kompress/ML text path. That path only runs
once the elider has declined a block, and I haven't checked how it
treats CJK.


Maintainer requalification (Windows/Python 3.12, current-main
integration, existing native extension): the public
`headroom.compress()` API was exercised with six paragraphs each of the
Chinese, Japanese, and Thai samples as a `fetch_page` tool result,
`model="gpt-4o", model_limit=128000, protect_recent=0`, offline. For
each language, current main produced six dense-elision markers and
changed the text; this branch produced zero markers and byte-identical
text. The broader router suite reported 306 passes and seven failures;
all seven exact failure IDs reproduced on actual unchanged main
(embedded-JSON/mixed-content assertions), so they are not attributed to
this patch. Changed-file lint, formatting, and whitespace checks passed.
No live provider/proxy or fresh native build is claimed; hosted sync
qualification remains pending.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:44:42 -06:00
XandJerrettDavis 54f48fb1c2 fix(code): keep semicolon-separated statements valid (#4630)
## Description

Python's grammar exposes the `;` between two statements on one physical
line as a root-level anonymous token. The leftover top-level sweep
captured it as a fragment, so assembly put a bare `;` on its own line:

```python
v=json.load(open(B+'v.json'))
;
m=json.load(open(B+'m.json'))
```

The result no longer parses, `compress()` discards it and returns the
original, and the proxy logs `Code compression produced invalid syntax
for python (...) returning original` for every such tool result. Two
live fenced blocks (220 and 117 tokens) hit this in the same proxy
session.

The line-based statement collection has a second symptom: when two
statements share one physical line (`x=1; y=2`), both AST nodes map to
the same row. Function bodies appended that row twice; class bodies
appended the full line twice, producing syntactically valid but wrong
output (`x=1; y=2` executed twice, compression ratio inflated to 1.429).

A third symptom surfaced once the output became valid: Python runs
top-level statements in order, but imports are always emitted first.
`import sys; sys.path.insert(0, 'vendor'); import plugin` now compressed
to valid code with `import plugin` hoisted above the path update, so the
plugin failed to load with `ModuleNotFoundError`.

Demoting such imports fixed ordering inside the top-level bucket, but
definitions are assembled before that bucket. `import sys; marker = 1;
import decimal` followed by `class Money(decimal.Decimal)` still moved
the class ahead of `import decimal` and raised `NameError` at definition
time.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Skip unnamed `;`/`,` children in the top-level leftover sweep so a
standalone separator can never become a code fragment.
- Merge overlapping row ranges in `_compress_function_ast` before the
budget loop. Taking the union (instead of dropping the later range)
preserves continuation lines of a multi-line statement that starts after
`;`.
- Track the last emitted row in `_compress_class_ast` and emit shared
rows once, trimming only rows already emitted so continuation lines
survive.
- For Python only, keep a root-level import behind executable top-level
code: the top-level sweep demotes such imports to the top-level code
stream (source order preserved) instead of hoisting them above the code
they depend on. Other languages are untouched because their imports/uses
are order-independent or must lead the file.
- For Python only, decline reassembly when executable top-level code
precedes any declaration. Definitions are emitted before the top-level
code bucket, so a class base, decorator, or default expression after
that code would be evaluated before the code or import it needs. Safe
reassembly cannot be guaranteed, so the original snippet is returned
unchanged. Snippets with no declarations keep compressing with demoted
imports in source order.
- Added `TestSemicolonSeparatedStatements` with seven regression tests:
top-level fallback, class-body duplication, class-body continuation,
function-body shared-line/continuation behavior, import-order
preservation that executes the compressed output, and executable
class-base and decorator/default ordering regressions.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches untyped helpers
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
109 passed in 5.22s

$ .venv/bin/ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ .venv/bin/ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted

$ # same new tests against unpatched origin/main 5c8772b7 in a detached worktree
3 failed, 92 deselected in 0.69s
```

## Real Behavior Proof

- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from the repo
`.venv`, head `d581720d` based on main `31dcaf41`, both synced by the
reviewer without changing compressor source.
- Exact command / steps: ran `pytest
tests/test_transforms/test_code_compressor.py -q -k Semicolon` on a
detached worktree of unpatched `origin/main` and on this branch. Then
compressed the two real fenced Python tool results captured from
`~/.headroom/logs/proxy-8788.log` (220- and 117-token blocks) with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1))`,
and parsed each output with `ast.parse`. For the import-order case,
wrote a module to a temp `vendor/` directory, compressed `import sys;
sys.path.insert(0, '<vendor>'); import <module>`, and executed the
output. For the cross-bucket case, compressed the reviewer's `import
sys; marker = 1; import decimal\nclass Money(decimal.Decimal):\n pass\n`
and a decorator/default variant using `functools.lru_cache` with
`value=marker`, then executed both. Re-ran both counterexamples on the
same 0.42.0 compressor build carrying the import-demotion patch without
the fallback to confirm they reproduced.
- Observed result: on unpatched `5c8772b7` all three original semicolon
tests fail — top-level compression returns the unchanged original after
`Code compression produced invalid syntax for python (35 tokens)`, the
class sample emits `x=1; y=2` twice, and the function sample emits the
shared line twice. On `d581720d` all seven pass; full file 109 passed.
For the 220-token live sample: before the fix it fell back to the
original after the warning; after, ratio `0.995`, no bare `;` line,
`ast.parse` succeeds. The 117-token sample is valid with no bare `;`
(ratio `1.0`, so the router still skips it for low savings). Class
sample: `x=1; y=2` count `2 -> 1`, ratio `1.429 -> 1.0`. Function
sample: shared line count `2 -> 1` and continuation line count `2 -> 1`.
Import-order sample: before the import fix the output was `import sys /
import semicolon_order_plugin / (blank) / sys.path.insert(...)` and
`exec` raised `ModuleNotFoundError: No module named
'semicolon_order_plugin'`; after, the order is `import sys`,
`sys.path.insert(...)`, `import semicolon_order_plugin` and `exec`
succeeds. Cross-bucket samples: before the fallback the class variant
emitted `import sys / class Money(decimal.Decimal) / marker = 1 / import
decimal` and `exec` raised `NameError: name 'decimal' is not defined`;
the decorator variant raised `NameError: name 'functools' is not
defined` at decoration time. After the fallback both return the original
snippet byte-for-byte and `exec` succeeds (`Money` is a
`decimal.Decimal` subclass, `compute()` returns `1`). A semicolon
snippet without declarations (`import sys; sys.path.insert(...); import
json`) still compresses, emitting `import sys`, then the path update,
then `import json`.
- Not tested: the Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`) has the same
structure and was not changed; end-to-end proxy traffic was not rerun on
this branch; non-Python fixtures for shared lines were not added (the
range merge is mechanical, but only Python exposes `;` as a root token,
and only Python demotes imports or applies the ordering fallback).

Function output after the fix (shared line and continuation each appear
once):

```python
def f():
    x=1; y=(2 +
            3)
    a=3
    b=4
    c=5
    # [3 lines omitted]
    pass
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: content with semicolon-separated
statements is compressed instead of falling back to the original; shared
lines are emitted once instead of duplicated; Python imports that follow
executable top-level code stay after it; Python snippets where
executable code precedes a definition are returned unchanged instead of
being reordered
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The top-level separator was only observed for Python: tree-sitter-python
exposes the assignments/calls on either side of `;` as direct module
children, with the `;` itself as an anonymous root sibling. The same
line-based collection is shared by the other grammars, so the range
merge also protects them, but no non-Python fixture was added.

Import demotion is Python-only and only applies after the sweep has
already emitted executable top-level code. Imports before any executable
code keep their place in the hoisted import block, so module docstrings
and license headers are unaffected. `_verify_syntax` also runs
`compile()`, which rejects misplaced `__future__` imports, so a demotion
that violated that rule would fall back to the original instead of
emitting broken ordering.

The ordering fallback is Python-only and is deliberately conservative:
any declaration after executable top-level code disables AST reassembly
for the snippet, even when the definition happens not to reference the
preceding code. The cost is lost compression for that shape; the
alternative is a semantically reordered result that syntax validation
cannot detect.

The Rust core port mirrors these functions (`extract_structure`, body
statement collection, class-body reconstruction). Parity there would
need its own fixtures and is out of scope for this PR.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 22:41:55 -06:00
XandJD Davis 7c150ec6b0 fix(code): keep Rust attributes attached to their items (#4624)
## Description

Rust outer attributes (`#[derive(...)]`, `#[inline]`, `#[cfg(...)]`) are
grammar siblings of the item they decorate, exactly like doc comments.
`_get_leading_comment_text` only collected comment node types, so
attributes fell through to the leftover top-level sweep and drifted away
from their item. On a mixed file the compressed output looked like this:

```rust
#[derive(Debug, Clone, PartialEq)]

pub struct Item {
    pub id: u64,
    pub name: String,
}

pub fn log_item(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [3 lines omitted]
}

/// Logs one item.
#[inline]
#[cfg(feature = "extra")]
```

The attribute block ends up after the last item, and `rustc` rejects the
file with `error: expected item after attributes`. Tree-sitter accepts
it, so the compressor's syntax check does not catch the corruption.

Rust line comments also include their trailing newline in the node text
(`/// doc\n`), so simply attaching them to the same preamble would
introduce blank lines.

Rust also allows blank lines between an outer attribute and the item it
decorates:

```rust
#[inline]

pub fn record(item: &Item) {}
```

The attribute is still attached to the function. Only ordinary comments
require row adjacency; attributes must survive the gap as well.

Comments can also sit between the attribute and its item:

```rust
#[inline]

// Keep with the function below.

pub fn record(item: &Item) {}
```

A blank line before that comment ended the preamble scan before the
attribute was reached. Detached comments are now skipped while scanning,
so the attribute behind one is still collected; the comments themselves
keep the ordinary adjacency policy.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_ATTACHED_NODE_TYPES = {"attribute_item"}` and made the
leading-sibling collector treat those nodes like comments.
- Renamed the collected list to `preamble`, since it now holds comments
and attributes, and updated the docstring.
- Strip the trailing newline from each collected sibling before joining
(Rust line comments carry one in their node text).
- Keep the blank-line adjacency rule for ordinary comments only;
attributes always attach to the following item, since Rust permits
whitespace between an outer attribute and its item.
- Skip detached comments while scanning the preamble so an outer
attribute behind one is still collected, without capturing those
comments.
- Added `TestRustAttributeAttachment` with four regression tests,
including blank-line-separated attributes and attributes behind an
intervening comment.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches an untyped helper
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_transforms/test_code_compressor.py -q
96 passed in 7.41s

$ uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ uv run ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
```

The new blank-line and intervening-comment tests fail on the unpatched
helper and pass with this change.

## Real Behavior Proof

- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from `uv sync
--extra dev --extra code`, `rustc 1.99.0`.
- Exact command / steps: compressed the samples below with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1,
max_body_lines=1, enable_ccr=False, semantic_analysis=False))`, wrote
each output to disk, then ran `rustc --edition 2021 --cfg
'feature="extra"' --crate-type lib`.
- Observed result: before the fix the compressed output ended with the
orphaned `/// Logs one item.` / `#[inline]` / `#[cfg(...)]` block and
`rustc` exited with `error: expected item after attributes`; with the
first fix ratio `0.808`, the preamble stayed attached, and `rustc` exit
0. For the blank-line sample, before the follow-up the attribute was
detached and `rustc` failed; after the follow-up the output emits
`#[inline]\npub fn record(...)`, ratio `0.795`, and `rustc` exit 0. For
the comment-between sample, before this change the attribute and comment
were emitted after the function and `rustc` failed; after the change the
output emits `#[inline]\npub fn record(...)`, ratio `0.746`, and `rustc`
exit 0. The detached comment keeps the ordinary policy and remains in
the leftover sweep.
- Not tested: attributes on `impl`/`trait`/`use` items (same sibling
mechanism, no dedicated fixture), other grammars (only Rust emits
`attribute_item`), and end-to-end proxy traffic.

Compressed output after the fix (compiled successfully):

```rust
#[derive(Debug, Clone, PartialEq)]
pub struct Item {
    pub id: u64,
    pub name: String,
}

/// Logs one item.
#[inline]
#[cfg(feature = "extra")]
pub fn log_item(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [3 lines omitted]
}
```

Blank-line attribute sample after the follow-up (compiled successfully):

```rust
pub struct Item { pub id: u64 }

#[inline]
pub fn record(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [2 lines omitted]
}
```

Comment-between sample after the follow-up (compiled successfully):

```rust
pub struct Item { pub id: u64 }

#[inline]
pub fn record(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [3 lines omitted]
}

// Keep with the function below.
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: Rust attributes are now emitted with
their item instead of being separated or orphaned
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The leftover path that caused this is shared by every language, but only
the Rust grammar emits `attribute_item` siblings today, so the behavior
change is scoped to Rust. The Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`) has no
equivalent preamble collection; parity there is out of scope for this
PR.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 09:14:51 -06:00
XandJD Davis 0d0cda22b4 fix(code): keep multi-line class headers' opening brace (#4623)
## Description

The code-aware compressor rebuilds a class/impl header from the lines
before the member container, then re-adds the container's start line
only when that line begins with `{`. A multi-line signature can put the
opening brace on a continuation line shared with the signature tail:

```rust
impl<T> Wrapper<T>
where T: Value + Debug + Clone {
```

```java
public class Calc
    extends Base
    implements Runnable {
```

In that case the rebuilt header loses its `{`, the rebuilt source no
longer parses, the syntax check rejects the output, and the compressor
returns the input unchanged — so these containers are never compressed.
On a live proxy this shows up as repeated runtime warnings while
compressing Rust code:

```text
Code compression produced invalid syntax for rust (614 tokens), returning original
Code compression produced invalid syntax for rust (366 tokens), returning original
```

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_compress_class_ast` keeps the member container's start line whenever
the container node begins with the opening brace, instead of requiring
the line to start with `{`.
- Colon-signature languages (Python) are unchanged: their block node
starts at the first body statement, which can itself begin with `{`, so
they stay on the colon path.
- Added `TestMultiLineClassHeaders` with a Rust `impl ... where ... {`
and a Java `extends`/`implements` header.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
adds a branch to existing untyped helper logic
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_transforms/test_code_compressor.py -q
94 passed, 2 warnings in 24.25s

$ uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ uv run ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
```

Both new tests fail on `main` (the compressor logs `invalid syntax ...
returning original` and `compression_ratio == 1.0`) and pass with this
change.

## Real Behavior Proof

- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from `uv sync
--extra dev --extra code`, base commit `7e0094d`; `rustc 1.99.0`, `javac
17.0.21`.
- Exact command / steps: compressed the samples with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1,
max_body_lines=2, enable_ccr=False, semantic_analysis=False))`, wrote
the compressed output to disk, then compiled it with `rustc --edition
2021 --crate-type lib <out>.rs` and `javac <out>/Calc.java`.
- Observed result: before the fix both samples logged `Code compression
produced invalid syntax ... returning original` and `compression_ratio`
was `1.0`; after the fix Rust ratio `0.943` with `where T: Value + Debug
+ Clone {` retained and `rustc` exit 0, Java ratio `0.803` with
`implements Runnable {` retained and `javac` exit 0.
- Not tested: C++, JavaScript/TypeScript and C# multi-line class headers
(same code path, but no dedicated fixtures were added), and end-to-end
proxy traffic.

Compressed Rust output (compiled successfully):

```rust
impl<T> Wrapper<T>
where T: Value + Debug + Clone {
    pub fn record(&self, sink: &mut Vec<u64>) {
        for item in &self.items {
            sink.push(item.id());
        }
        // [2 lines omitted]
    }
}
```

Compressed Java header:

```java
public class Calc
    extends Base
    implements Runnable {
    public void compute(int x) {
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: compression now succeeds for
containers that previously fell back to the original
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The same header logic exists in the Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`), which
currently keeps only `node_lines[..sig_end]` and has the same gap. This
PR intentionally stays scoped to the Python implementation; the Rust
port can follow separately if maintainers want parity.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 09:14:25 -06:00
RyrenzandJD Davis 1b9d296c19 fix(proxy): preserve Cline headroom__headroom_retrieve results from recompression (#4586)
## Description

Cline names MCP tools `<server>__<tool>`, so with the default `headroom`
server the retrieval tool reaches the proxy as
`headroom__headroom_retrieve`. `_tool_name_aliases()` in
`headroom/config.py` maps OpenCode's single-underscore
`headroom_headroom_retrieve` and the `mcp__` / `mcp_` wrappers to
`headroom_retrieve`, but not the double-underscore form.
`is_tool_excluded()` therefore returns `False` for it, so both the
ContentRouter `ccr_retrieve` guard and the OpenAI handler's
`headroom_retrieve_call_ids` miss the call. The retrieved content then
goes through normal compression again, which produces the retrieval loop
described in the issue.

This follows #3782: the double-underscore spelling now gets the same
exact-name alias as the single-underscore one. I kept it an exact match
on the default server name, not a generic `*__headroom_retrieve` suffix
rule, so the change doesn't widen matching for other exclusion entries.
The content-based guard suggested in the issue (detecting a `hash` +
`original_content` payload whatever the tool name is) is a larger
behavior change, and this PR leaves it out.

Closes #4585

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/config.py`: `_tool_name_aliases()` now maps
`headroom__headroom_retrieve` (case-insensitive) to `headroom_retrieve`,
alongside the existing OpenCode spelling.
- `tests/test_tool_name_aliases.py`: adds the Cline spelling to the
parametrized exclusion test.
- `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`:
new test that sends an OpenAI-shaped `headroom__headroom_retrieve` call
and result through `ContentRouter.apply()` and asserts the result passes
through unchanged with `router:excluded:ccr_retrieve`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Both new test cases fail on unmodified `main` (`2 failed, 16 passed`)
and pass with the fix. Lint, format and mypy were run on the changed
files only.

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_tool_name_aliases.py tests/test_transforms/test_content_router_ccr_retrieve_exemption.py tests/test_ccr_tool_injection_cache_stability.py tests/test_hermes_tool_call_unwrap.py tests/test_openai_responses_compression_units.py -q -p no:cacheprovider -o addopts=
125 passed, 1 warning in 6.16s

# same two files on unmodified main (headroom/config.py reverted):
FAILED tests/test_tool_name_aliases.py::test_retrieve_tool_aliases_are_excluded[headroom__headroom_retrieve]
FAILED tests/test_transforms/test_content_router_ccr_retrieve_exemption.py::TestContentRouterCcrRetrieveExemption::test_cline_server_tool_mcp_name_not_recompressed
2 failed, 16 passed in 3.38s

$ .venv/bin/ruff check headroom/config.py tests/test_tool_name_aliases.py tests/test_transforms/test_content_router_ccr_retrieve_exemption.py
All checks passed!
$ .venv/bin/ruff format --check headroom/config.py tests/test_tool_name_aliases.py tests/test_transforms/test_content_router_ccr_retrieve_exemption.py
3 files already formatted
$ .venv/bin/mypy headroom/config.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Ubuntu 26.04 x86_64, Python 3.12, offline, no provider
- Exact command / steps: `.venv/bin/python proof.py`, a standalone
script outside pytest. It prints `is_tool_excluded(name,
DEFAULT_EXCLUDE_TOOLS)` for the three retrieve spellings, then runs
`ContentRouter(ContentRouterConfig(min_section_tokens=10)).apply()` on
an OpenAI-shaped assistant `tool_calls` entry named
`headroom__headroom_retrieve` followed by a 3,680-char JSON tool result,
and prints the output size and the transforms that were applied.
- Observed result: on unmodified `main`, the Cline name is not excluded
and the retrieve result is crushed:

  ```text
  is_tool_excluded('headroom_retrieve') = True
  is_tool_excluded('headroom_headroom_retrieve') = True
  is_tool_excluded('headroom__headroom_retrieve') = False
input chars=3680 output chars=1861 unchanged=False new_ccr_marker=False
  transforms: ['router:smart_crusher:0.40']
  ```

  With this branch, the result passes through byte-for-byte:

  ```text
  is_tool_excluded('headroom_retrieve') = True
  is_tool_excluded('headroom_headroom_retrieve') = True
  is_tool_excluded('headroom__headroom_retrieve') = True
  input chars=3680 output chars=3680 unchanged=True new_ccr_marker=False
  transforms: ['router:excluded:ccr_retrieve']
  ```

- Not tested: a live Cline session and real provider traffic through the
proxy, and Cline's hashed-name fallback for tool names over 64
characters (not reachable with the default `headroom` server name, which
gives a 27-character tool name).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this is the default tool-exclusion
code path, with no flag
- Minimum rollout channel: stable; the only effect is that one more
tool-name spelling is excluded from compression
- Stable/default behavior changed: yes, but only for tools named exactly
`headroom__headroom_retrieve` (case-insensitive), whose results are now
kept verbatim instead of compressed
- Kill switch / disable path: none added; the change is a one-line alias
in `_tool_name_aliases()`
- Unsafe override required: no
- Qualification impact: none; other tool names keep their current
exclusion behavior
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4586"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:41:57 -06:00
EndidandJD Davis ac8423965d fix(code): preserve syntax in CCR retrieval comments (#4583)
## Description

`CodeAwareCompressor.compress()` validates the AST-compressed source
before appending its CCR retrieval marker. The marker always started
with `#`, so JavaScript, TypeScript, Go and other C-family outputs
became invalid after validation while still reporting
`syntax_valid=True`.

Use the detected language's existing `LangConfig.comment_prefix` for
that marker. Python keeps `#`; C-family languages use `//`.

Addresses **item 4** of #4478. This deliberately does not close the
umbrella issue or change its other three items.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Reuse the existing per-language comment configuration when appending
CCR metadata.
- Exercise real compression with CCR enabled for Python, JavaScript,
TypeScript and Go. Parse the final output and verify that the recovery
hash retrieves the exact original source.
- No dependency, public API, rollout flag or changelog changes.

## Testing

- [x] New regression tests fail before the fix and pass after it.
- [x] Six relevant compressor suites pass: **149 passed**.
- [x] `ruff check .` passes; both changed files pass `ruff format
--check`.
- [x] Project pre-commit checks (including isolated mypy) and commitlint
pass.
- [x] Manual compression + Node syntax check + real in-memory CCR
retrieval performed.
- The full development environment's `mypy headroom
--ignore-missing-imports` reports two existing errors in
`memory/adapters/embedders.py:389` and
`integrations/langchain/chat_model.py:380`. A baseline run with the
unmodified compressor via `mypy --shadow-file` reports exactly the same
errors. These files are not changed here.

### Test Output

```text
Before: pytest tests/test_transforms/test_code_compressor.py -k ccr_marker_preserves -q
3 failed, 1 passed, 92 deselected
(JavaScript, TypeScript and Go fail the final-source syntax assertion.)

After: same command
4 passed, 92 deselected

pytest tests/test_transforms/test_code_compressor.py \
  tests/test_code_compressor_language_alias.py \
  tests/test_code_compressor_syntax_breaker.py \
  tests/test_code_compressor_bash.py \
  tests/test_code_compressor_thread_safety.py \
  tests/test_transforms/test_code_compressor_cjk.py -q
149 passed in 12.19s
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.12.14, Node 24.16.0,
tree-sitter-language-pack 0.13.0. Python source based on `1fe8549f`;
development/code dependencies installed from `uv.lock`, with the
official headroom-ai 0.42.0 wheel also installed. No provider or model
API call.
- Exact command / steps: run the script below from the checkout with
`HEADROOM_OFFLINE=1`, then run `node --check
/tmp/headroom-ccr-check.mjs`. Run once on the base revision and once
with this patch.

```python
from pathlib import Path
from headroom.cache.compression_store import CompressionStore, set_request_compression_store
from headroom.transforms.code_compressor import CodeAwareCompressor, CodeCompressorConfig
from tests.test_transforms.test_code_compressor import generate_javascript_code

store = CompressionStore()
set_request_compression_store(store)
code = generate_javascript_code()
result = CodeAwareCompressor(
    CodeCompressorConfig(min_tokens_for_compression=10)
).compress(code, language="javascript")
Path("/tmp/headroom-ccr-check.mjs").write_text(
    result.compressed, encoding="utf-8", newline="\n"
)
print(result.syntax_valid, result.compressed.splitlines()[-1])
assert result.cache_key
assert store.retrieve(result.cache_key).original_content == code
```

- Observed result: Before the fix, Headroom reports `True`, but Node
exits 1 with `SyntaxError: Invalid or unexpected token` at `# [200
tokens compressed. Retrieve more: hash=... Expires in 5m.]`.
- After: marker starts `// [200 tokens compressed...`; Node exits 0; CCR
retrieval equals the input byte for byte.
- Not tested: live provider/proxy traffic, Windows/Linux, full
repository test suite, freshly built Rust extension or the full Rust
workspace checks. `make ci-precheck` stops at `cargo fmt` because this
machine has no `cargo`; the Python change does not modify Rust code.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this fixes the existing code-aware
compressor, which is not gated by the runtime rollout registry.
- Minimum rollout channel: Stable; no beta/canary/dev channel or unsafe
override is required.
- Stable/default behavior changed: only the CCR comment delimiter on
languages whose configured delimiter is not `#`.
- Kill switch / disable path: Set
`CodeCompressorConfig(enable_ccr=False)` to disable the CCR marker and
original-content storage.
- Unsafe override required: no.
- Qualification impact: final source now remains parseable for the
covered languages; CCR payload/hash behavior is preserved.
- Rollback path: Revert commit
`69afcfa7e3acac52b99e638af2331538b3dadea6`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] The change follows project style and uses the existing language
configuration.
- [x] Regression tests cover final output syntax and original-content
recovery.
- [x] Relevant existing tests pass locally.
- [x] `CHANGELOG.md` is unchanged; the Conventional Commit title
supplies the release-note entry.
- [x] No user-facing documentation change is needed for this
syntax-preservation fix.

## Additional Notes

Prepared with AI assistance (Codex), with executed local regression
tests and a Node syntax-check reproduction. The scope is restricted to
#4478 item 4.

An independent Codex review found no blocking defects, checked language
coverage and marker consumers, and independently ran the main compressor
test file (96 passed), changed-file Ruff and `git diff --check`. This is
automated review, not maintainer approval.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4583"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:37:17 -06:00
Mansurshakh JaparovandJerrettDavis 93279f3208 fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description

Closes #3697.

An unchanged SmartCrusher result now keeps the original bytes and
reports no structural compression. Formatting-only savings come from the
existing Rust JsonMinifier, not from pretending the JSON value changed.

The minifier preserves duplicate keys, order, escapes and number
literals. A mixed result cannot use one section's CCR marker to hide
unmarked loss in another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace
savings through JsonMinifier in Python and the eligible native live
zone.
- Keep all records and JSON literals; do not store originals or inject
CCR markers for lossless minification.
- Preserve mixed routing and reject uncovered lossy sections in both
tool-result formats. Explicit no-CCR behavior is unchanged.
- Replace incidental forwarding tests with native minification and
complete-record regressions; document the distinction.

## Testing

- [ ] Entire Python suite — not claimed.
- [x] Four consumer/native modules: `python -m pytest -q
tests/test_recursive_json.py tests/test_transforms_content_router.py
tests/test_transforms/test_smart_crusher_rust_parity.py
tests/test_transforms/test_smart_crusher_lossless_default.py`
- [x] Changed-file Ruff checks; normal commit hooks, including mypy and
commitlint.
- [x] Manual public-API runs, with and without the HTML extra.

### Test Output

Consumer/native modules:

```text
collected 136 items
============================= 136 passed in 36.43s =============================
```

Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native
build, Python checks, commitlint):

```text
======================== 180 passed, 1 warning in 8.35s ========================
✅ ci-precheck PASSED — safe to push.
To https://github.com/oxura/headroom.git
   64688217..0002af3e  HEAD -> fix/3697-unchanged-json-passthrough
```

The first push stopped on an inherited commit footer because
`origin/main` was stale. Fetching the actual upstream ref fixed the lint
range; the normal push then passed. No source merge, lint-rule change or
hook bypass.

Hosted checks on `0002af3e` now pass: all four Python shards, Windows
permissions, smoke tests, native wrappers, parity and Docker e2e.
[Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks).

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual
extension built from this checkout. No provider billing calls.
- Exact command / steps: the earlier localhost `/v1/compress` and
TransformPipeline runs exercised the original fix. For this follow-up,
call `ContentRouter.compress` on a data-only HTML page containing 60
complete records, with `smart_crusher_lossless_only=True`,
`smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run
it once without trafilatura, then with the locked 2.0.0 extra. The new
public-API test contains the full input/config. Separately call
`route_embedded_json` on a compact array followed by a padded array,
with Unicode and CRLF surroundings.
- Observed result: both HTML modes keep all 60 records and the
surrounding markup, with no unrecoverable section. The unchanged first
JSON span does not stop the second span's compaction; surrounding text
stays exact. Final Codecov is 96%; the two remaining lines are the
deferred HTML fallback return. The real no-HTML-extra API run executed
both, and local coverage also exercises the unchanged-span branch.
- Not tested: live provider acceptance, invoices or production cache hit
rates. No native eligibility or admission gates changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing JSON routing.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: eligible formatting savings return as
json_minifier; standalone SmartCrusher remains byte-exact on no-ops.
- Kill switch / disable path: existing routing configuration and
HEADROOM_LIVE_ZONE_DISABLE_ARMS.
- Unsafe override required: no.
- Qualification impact: no-op, formatting, structural and CCR paths
exercised separately.
- Rollback path: revert the minifier integration; no persistence
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

Addresses the [minifier
review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285).
The maintainer’s main merge is retained. The latest follow-up only
changes consumer tests.

AI-assisted; no subagents.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 21:32:42 -06:00
Andrew Barnes 75df01d627 fix(codec): reject densified blocks with mismatched row counts (#4271)
## Description

`expand_compacted()` now rejects blocks whose `[N]` header disagrees
with the number of CSV records. Previously, `[3]{id:int}\n1\n2\n`
returned two records as though the original array had been
reconstructed, leaving callers unable to distinguish truncation from
successful expansion.

The decoder returns its existing `None` failure result on mismatch.
Valid multiline cells, single-column missing values, empty arrays and
the tolerated trailing blank line retain their behavior.

Compatibility limit: existing trailing-empty-row normalization can
discard an extra empty CSV record. This patch does not distinguish that
record from a trailing blank-line artifact; it validates the count after
that normalization.

## Type of Change

- [x] Bug fix

## Changes Made

- Validate the parsed CSV record count after dictionary legends and
existing trailing-newline normalization.

## Testing

- [x] Focused unit tests pass (full suite limitation below)
- [x] Linting passes
- [x] Type checking passes on macOS (Windows limitation below)
- [x] New regression tests added
- [x] Manual testing performed

- Eight new regression cases failed before the fix and pass after it
(missing/extra rows, empty bodies, dictionary-encoded values, and
JSON-string wrappers).
- Codec and live-agent densification suites: **41 passed on Windows and
41 passed on macOS arm64**.
- Repository-wide `ruff check .`: passed (Ruff 0.17.0).
- Changed-file `ruff format --check`: passed.
- `mypy --follow-imports=silent --ignore-missing-imports
headroom/transforms/compaction_codec.py`: passed.
- Full `pytest -x` attempt stopped at collection: optional `fastapi` is
not installed in this environment. The full suite is not claimed green.
- Repository pre-commit hooks passed except full-project mypy on
Windows: seven existing POSIX-only `os` attribute errors in unchanged
`fileperms.py` and `proxy/interceptors/astgrep.py`. Scoped codec type
checking passed.
- Full-project mypy 1.20.2 on macOS: **no issues found in 572 source
files** (`uvx --from mypy==1.20.2 mypy --ignore-missing-imports
headroom`).

### Test Output

```text
python -m pytest tests/test_transforms/test_compaction_codec.py tests/test_densify_agent_mode.py -q
Windows / Python 3.13.12: 41 passed in 2.74s
macOS arm64 / Python 3.13.15: 41 passed in 2.99s
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.12, current Python source with the
published Headroom 0.41.0 native wheel. No provider API calls, ML model
downloads or mocks. `HEADROOM_TELEMETRY=off`, `HEADROOM_BEACON=0`,
`HEADROOM_OFFLINE=1`; `gpt-4o` used only for token counting.

- Exact command / steps: Run the Python snippet below from the checkout
after installing the native extension.

```python
import json
from headroom import densify, expand_compacted

records = [{"id": i, "name": f"item_{i}", "status": "active"} for i in range(100)]
result = densify(
    [{"role": "tool", "tool_call_id": "demo", "content": json.dumps(records)}],
    model="gpt-4o", protect_recent=0,
)
block = result.messages[0]["content"]
if block.startswith('"'):
    block = json.loads(block)
assert expand_compacted(block) == records
truncated = block.rstrip("\n").rsplit("\n", 1)[0] + "\n"
print(f"Full block: {len(records)} records recovered exactly")
print(f"After removing the last data row: {expand_compacted(truncated)!r}")
assert expand_compacted(truncated) is None
```

- Observed result: All 100 records recovered exactly; deleting the final
row returns `None`.

```text
Full block: 100 records recovered exactly
After removing the last data row: None
```

- Not tested: full optional-dependency suite, native-source rebuild,
live provider/proxy traffic. This change does not validate every
possible malformed CSV cell; its scope is the declared row count.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: not applicable.
- Stable/default behavior changed: malformed row-count blocks return
`None`.
- Kill switch / disable path: no new flag; revert this commit.
- Unsafe override required: no.
- Qualification impact: no dependency or wire-format change.
- Rollback path: revert this commit to restore prior decoder behavior.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (decoder
docstring)
- [x] My changes generate no new warnings in the checks run
- [x] I have added tests that prove my fix is effective
- [x] New and existing focused unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Screenshots (if applicable)

Not applicable; decoder behavior is shown above.

## Additional Notes

The pre-push hook's documented bypass was used because Windows has no
`make`, and the Mac validation environment has no Rust toolchain. The
full native `ci-precheck` is unverified. Windows pre-commit mypy was
skipped after the same pinned full-project check passed on macOS; all
other pre-commit checks passed.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4271"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
2026-11-03 19:28:06 -06:00
Yuchuan HeandJerrettDavis 312ffec8ea fix(router): preserve read protection for quiet sed options (#4521)
## Description

With `HEADROOM_PROTECT_READS=1`, `sed -n '1,60p' file.py` achieves
byte-exact read protection, but equivalent `-ne`, `-En`, `--quiet`, and
`--silent` forms currently do not. The classifier previously checked for
a literal `-n` token across the raw command string, which also caused it
to mistake `-n` inside an expression argument as a quiet flag.

This PR properly parses sed options before classifying whether a command
is a read. It handles grouped flags and long options, respects script
arguments and `--` delimiters, and excludes in-place edits (`-i`). GNU
and BSD `-l` forms remain recognized.

Addresses the quiet-option parsing portion of item 3 in #4478 (quoted
redirect parsing and other items in that issue remain separate). Note
that `sed -n1,80p` is invalid GNU sed syntax and remains unhandled.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Recognize quiet sed options using standard-library shell tokenization
and robust option parsing.
- Keep script arguments separate from options, and exclude in-place
edits from protected reads.
- Add command-classification unit tests and end-to-end
`ContentRouter.apply()` regression coverage for all five quiet forms.

## Testing

- [x] Unit tests pass (`pytest`): focused and related suites below
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): native Linux pre-commit
configuration
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Focused tests against unchanged main's router:
16 failed, 44 passed

# This patch:
60 passed

# Read detection, Responses read protection, read audit, maturation, lifecycle:
147 passed

# make ci-precheck-rust (Rust 1.95.0):
# cargo fmt --all -- --check; cargo clippy --workspace -- -D warnings;
# cargo test --workspace
1659 passed, 3 ignored

# make ci-precheck-python (including native extension rebuild):
177 passed, 4 skipped

# ruff check .
All checks passed!
# ruff format --check .
1834 files already formatted
```

The native extension rebuild and import verification pass cleanly. The
repository's unmodified pre-commit configuration passes on native Linux
(WSL Ubuntu, Python 3.14.4), including mypy, Ruff, formatting, conflict
checks, version checks, and commit-message hooks.

On Windows, the default mypy hook reports seven POSIX-only `os`
attribute errors in unchanged files (`fileperms.py` and
`proxy/interceptors/astgrep.py`), matching unmodified main (#4520).
Similarly, broader Windows test failures (18 failed, 5367 passed, 190
skipped, 2 errors) reproduce identically on main and stem from known
platform differences (symlink privileges, SIGHUP handling, and Windows
environment-variable size limits).

## Real Behavior Proof

- Environment: Windows x86_64, Python 3.13.9, GNU sed 4.9, current-main
Python sources with native `headroom._core` extension. ML and code
compression are disabled to directly isolate the read-protection route.
- Exact command / steps: Run the script below with main's router versus
this patch. It executes real GNU sed against a temporary Python source
file and passes each tool result through `ContentRouter.apply()`.
- Observed result: The standard `-n` control is protected on both
versions. On main, the other four forms miss `router:read_protected`.
With this patch, all five forms correctly trigger read protection and
preserve the 7,998-character output byte-for-byte.
- Not tested: Focuses on command parsing and routing behavior. BSD
option syntax is validated via parser unit tests; live proxy traffic and
model calls were not required.

Save as `repro_sed_quiet_reads.py`, then run `python
repro_sed_quiet_reads.py`:

```python
"""Run real GNU sed output through ContentRouter, without a model or provider."""
import json
import os
from pathlib import Path
import subprocess
import tempfile

os.environ["HEADROOM_OFFLINE"] = "1"
os.environ["HEADROOM_TELEMETRY"] = "off"
os.environ["HEADROOM_PROTECT_READS"] = "1"

from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
from headroom.tokenizers.estimator import EstimatingTokenCounter

sed = Path("C:/Program Files/Git/usr/bin/sed.exe")
source = "\n\n".join(
    f"def compute_{i}(value):\n"
    '    """Compute a checked result for a caller."""\n'
    "    result = value * 2\n"
    "    if result < 0:\n"
    '        raise ValueError("negative result")\n'
    "    return result\n"
    for i in range(45)
)
with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "sample.py"
    path.write_text(source, encoding="utf-8", newline="\n")
    for flags in [["-n"], ["-ne"], ["--quiet"], ["--silent"], ["-En"]]:
        output = subprocess.check_output(
            [str(sed), *flags, "1,400p", str(path)], text=True, encoding="utf-8"
        )
        command = f"sed {' '.join(flags)} '1,400p' {path.as_posix()}"
        router = ContentRouter(ContentRouterConfig(
            enable_code_aware=False, enable_kompress=False,
            protect_analysis_context=False, protect_recent_reads_fraction=0.0,
        ))
        messages = [
            {"role": "user", "content": "Read the module."},
            {"role": "assistant", "content": None, "tool_calls": [{
                "id": "read-1", "type": "function", "function": {
                    "name": "Bash", "arguments": json.dumps({"command": command})
                }
            }]},
            {"role": "tool", "tool_call_id": "read-1", "content": output},
        ]
        result = router.apply(messages, EstimatingTokenCounter(), protect_recent=0)
        actual = result.messages[2]["content"]
        print(json.dumps({
            "flags": flags, "input_chars": len(output), "output_chars": len(actual),
            "byte_exact": output == actual, "transforms": result.transforms_applied,
        }), flush=True)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: sed read classification; byte-exact
routing changes when `HEADROOM_PROTECT_READS` is enabled (default
remains unchanged).
- Kill switch / disable path: `HEADROOM_PROTECT_READS=0` disables read
protection.
- Unsafe override required: no.
- Qualification impact: equivalent quiet reads now retain their original
output; no savings or task-quality measurement is claimed.
- Rollback path: revert this patch; no persisted-state migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing focused unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

I implemented and verified this with Codex assistance for drafting and
local tests.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 17:10:38 -06:00
Kenneth WongandKenneth Wong 31f1304026 fix(privacy): turn off ONNX Runtime's Microsoft telemetry by default (#4602)
## Description

Since 1.31.0, ONNX Runtime's Linux and macOS builds include Microsoft's
1DS telemetry client
(`onnxruntime/core/platform/posix/telemetry_1ds.cc`, absent through
1.30.0). It is **on by default**. `OneDsTelemetry::Initialize` turns it
off only when a CI variable (`GITHUB_ACTIONS`, `TF_BUILD`, ...),
`ORT_RUNNING_UNIT_TESTS` or `ORT_DISABLE_TELEMETRY` is set. Otherwise
ORT:

- writes a device id under `~/Library/Application
Support/Microsoft/DeveloperTools/.onnxruntime/` on macOS, and sends a
SHA-256 of it with each event;
- queues events in an `onnxruntime.db` next to it and uploads them to
`https://mobile.events.data.microsoft.com/OneCollector/1.0`;
- sends a `SessionCreation` event for each session, with
`modelFileName`, `modelGraphName`, `modelProducerName`,
`modelGraphHash`, `modelWeightHash`, execution provider ids and
versions, and hardware vendor and device types.

Headroom requires `onnxruntime>=1.24.0` on Python 3.11+ and currently
resolves 1.31.1. The proxy loads ORT twice: the Python package
(Kompress, image router) and the dylib the Rust core loads through
`ORT_DYLIB_PATH` (Magika, embeddings). Each copy initializes the
uploader. In my runs the proxy opened a TLS connection to Microsoft
about 11 s after start, while the models loaded, and sent about 4 KB.
`HEADROOM_OFFLINE` doesn't touch it, and
`docs/content/docs/security-model.mdx` doesn't list this destination. CI
doesn't see it because GitHub Actions sets `GITHUB_ACTIONS`, which ORT
treats as CI. The same client is in the manylinux wheels (checked in
`onnxruntime-1.31.1-cp311-cp311-manylinux_2_28_aarch64.whl`), so Docker
deployments are affected too.

ORT reads `ORT_DISABLE_TELEMETRY` once per copy, when that copy
initializes, and then never creates the uploader or the device id. So
`import headroom` now sets it with
`os.environ.setdefault("ORT_DISABLE_TELEMETRY", "1")`, in the same place
as the existing `ensure_ort_dylib_pinned()` hook, before anything in the
package can load ORT. An explicit caller value wins, including `0` for
anyone who wants ORT's telemetry on.

Design choice: this is off always, not only under `HEADROOM_OFFLINE` (as
`pin_crewai_telemetry_off` does for CrewAI). The events describe the
user's models and hardware and go to a third party the user never chose.
Headroom's own beacon is opt-out and documented, but this upload was
neither documented nor tied to any Headroom setting. If maintainers
prefer offline-only, `_OfflineEnvPin("ORT_DISABLE_TELEMETRY", "1")` in
`headroom/offline.py` is a small change to this PR.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/_ort.py`: `disable_ort_telemetry_by_default()` sets
`ORT_DISABLE_TELEMETRY=1` unless the variable is already set.
- `headroom/__init__.py`: calls it at package import, next to
`ensure_ort_dylib_pinned()`.
- `crates/headroom-core/src/transforms/magika_detector.rs`:
`init_ort_from_path` commits the environment with
`.with_telemetry(ort_telemetry_opted_in(ORT_DISABLE_TELEMETRY))`. The
`ort` crate's default is `true`, and it calls `EnableTelemetryEvents`.
With this change, telemetry is off when the variable is unset or set to
a value ORT treats as disabled, and on for any other explicit value, so
`ORT_DISABLE_TELEMETRY=0` re-enables it for Rust-loaded models too. This
only stops later events, after the uploader has started, so it is a
backup for Rust-only callers; the environment variable is the real fix.
A unit test covers each case.
- `tests/test_transforms/test_ort_dylib.py`: the default is set; a
caller's value (`0`) is kept; a subprocess `import headroom` leaves
`ORT_DISABLE_TELEMETRY=1` with `onnxruntime` not yet imported.
- `docs/content/docs/security-model.mdx`: a row for
`mobile.events.data.microsoft.com` in the network egress table.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_transforms/test_ort_dylib.py -k importing          # without the headroom/__init__.py change
E   AssertionError: assert {'ort_loaded'...'value': None} == {'ort_loaded'... 'value': '1'}
1 failed, 12 deselected
$ pytest -q tests/test_transforms/test_ort_dylib.py tests/test_package_init_lazy.py tests/test_offline_egress_chokepoint.py \
    tests/test_transforms/test_detect_fallback_1123.py tests/test_router_external_dispatch.py
223 passed, 2 skipped in 57.30s
$ ruff check / ruff format --check on the changed .py files (ruff 0.16.8)
All checks passed!
$ mypy headroom/_ort.py
headroom/memory/adapters/embedders.py:389: error: Unsupported operand types for - ("None" and "int")   # same single error on main
$ cargo clippy --all-features --tests -D warnings / cargo fmt --check -p headroom-core
clean
$ cargo test -p headroom-core --all-features --lib ort_telemetry
1 passed
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), Python 3.11.17, onnxruntime 1.31.1
(CoreML, Azure and CPU providers), no CI variables set; `origin/main` at
`fc1ba891` and this branch, each in its own worktree with the same
`headroom-ai[all]` environment.
- Exact command / steps: (1) a bare script that creates one CPU
`InferenceSession` for `chopratejas/technique-router-onnx` and idles 60
s, with `ORT_DISABLE_TELEMETRY` unset, then set to `1`. (2)
`PYTHONPATH=<worktree> python -m headroom.cli proxy --port 8799
--no-telemetry` with `HEADROOM_UPDATE_CHECK=off` and
`ORT_DISABLE_TELEMETRY` unset, running until Kompress reports healthy.
Both record every TCP flow of the process with `nettop -m tcp -p <pid>
-L 0 -s 1`, and remote owners come from `whois`.
- Observed result: (1) unset: one TLS flow to a Microsoft Corporation
address (4.150.223.107:443, about 9.5 KB in and 4 KB out), with
`onnxruntime.db` touched; set to `1`: no TCP flows, and the database is
untouched. (2) `main`: 2 of 2 runs open a flow to Microsoft
(74.151.238.96, 40.74.98.197) about 11 s after start, with about 9.5 KB
in and 4 KB out; the collector rotates across Azure addresses. This
branch: no flow to Microsoft in 6 runs; only the Anthropic upstream,
plus litellm's GitHub pricing fetch in some runs.
- Not tested: Linux and Windows at runtime. The Linux wheel contains the
same client (`OneDsTelemetry` and the collector URL appear in
`libonnxruntime.so.1.31.1`), but I only measured macOS. Windows uses
ORT's separate ETW provider, which doesn't read this variable.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, process environment default
- Stable/default behavior changed: yes; ONNX Runtime no longer uploads
usage telemetry to Microsoft from Headroom processes unless
`ORT_DISABLE_TELEMETRY` is set to a false value
- Kill switch / disable path: `ORT_DISABLE_TELEMETRY=0` restores ORT's
own default
- Unsafe override required: no
- Qualification impact: none; inference paths, providers and models are
unchanged
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

A host that imports `onnxruntime` before `headroom` has already
initialized ORT, so it has to set the variable itself. This is the same
limitation `pin_litellm_cost_map_local` documents.

In one early run, the fixed proxy did open the Microsoft flow. I
couldn't reproduce it in six later runs, and its log reports a different
computed `__version__` (`0.43.0-dev` against `0.42.1-dev` for the same
tree), so the import state of that run is uncertain. Setting the
variable before the process starts was clean in every run, for both the
bare script and the proxy.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
2026-11-03 17:01:04 -06:00
3c1fcbdfde fix(compaction): reject negative dictionary indices and non-array legends in expand_compacted (#4470)
## Description

`expand_compacted()` promises `None` when a densified block cannot be
losslessly reversed, but value-factored (`__dict:K`) cells were decoded
with raw Python indexing:

- a wire index of `-1` silently decoded to the **last** legend value
(data corruption, no error);
- a non-array legend (`@tag=42`, `@tag="ab"`, `@tag={"a":1}`,
`@tag=null`) raised `TypeError`/`KeyError` or was indexed character by
character.

Fixes #4280

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/compaction_codec.py`: reject a legend whose JSON
payload is not a list; reject negative dictionary indices before
indexing. Both paths return `None`, as the docstring already promises.
- `tests/test_transforms/test_compaction_codec.py`: regression tests for
the negative index, four non-array legend shapes, and a guard that valid
first/last indices, a quoted literal `"-1"`, `null`, and missing cells
in a dictionary column still decode.

## Testing

- [x] Unit tests pass locally
- [x] New tests added for the fix

```
$ python -m pytest tests/test_transforms/test_compaction_codec.py tests/test_densify_agent_mode.py -q
37 passed in 1.58s

# with the codec fix reverted, new tests:
5 failed, 1 passed, 17 deselected

$ ruff check / ruff format --check (touched files)
All checks passed! / 2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, local build of headroom with
native `headroom._core`, branch rebased on upstream/main
- Exact command / steps: `expand_compacted('[1]{tag:string}
__dict:1\n@tag=["first","last"]\n-1\n')` and
`expand_compacted('[1]{tag:string} __dict:1\n@tag=42\n0\n')`, plus the
codec and densify-agent-mode test files above
- Observed result: before: `[{"tag": "last"}]` and `TypeError`; after:
`None` for both; full codec + agent densification suites pass (37 tests)
- Not tested: declared row-count mismatch validation (separate issue
#4271)

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, decoder correctness fix
- Stable/default behavior changed: only for malformed blocks, which now
return None instead of wrong data or an exception
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4470"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 23:34:33 -05:00
HetCreepandClaude Opus 5.5 4c3799fd81 fix(read-lifecycle): keep Reads fresh after a failed Edit or Write (#4461)
## Description

`ReadLifecycleManager` counted every `Edit`/`Write` tool_use as a change
to the file, so an Edit whose tool_result had `is_error: true` still
turned earlier Reads of that file STALE and replaced their content with
a marker, although the file never changed. This collects the tool_use
ids whose tool_result is an error and leaves them out of the edit
operations.

Fixes #4460

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `ReadLifecycleManager._failed_tool_use_ids` (Anthropic
`tool_result` blocks with `is_error` set) and skip those ids when
building the edit operations in `_build_file_operation_index`.
- Add two tests to `tests/test_transforms/test_read_lifecycle.py`: a
failed Edit leaves the Read fresh; a later successful Edit still makes
it stale.

## Testing

Rebased onto upstream/main `3fef15d3` (2026-10-24): the new tests pass
(25 passed); with only the source change reverted, 1 fails. The
real-behavior run below was done on `e995eb9f`.

Review round 1 (6aa14c9d, Devin findings): only a `<tool_use_error>`
(pre-execution validation) keeps earlier Reads fresh; any other error,
such as an interrupted call that may have partly run, still counts as an
edit; an error result whose `tool_use_id` is not a string is ignored
instead of raising. `tests/test_transforms/test_read_lifecycle.py`: 28
passed; on the previous head the two new tests fail, and on main the
fresh-read tests fail.

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests, source change reverted
E   AssertionError: assert 1 == 0
FAILED tests/test_transforms/test_read_lifecycle.py::TestAnthropicFormat::test_failed_edit_does_not_make_read_stale
1 failed, 24 passed

# with the fix
tests/test_transforms/test_read_lifecycle.py: 25 passed
tests/test_read_maturation.py tests/test_read_maturation_handler_nobust.py tests/test_transforms: 510 passed, 78 skipped, 1 failed
  (the 1 failure is test_kompress_compressor.py::TestLazyImports::test_unloadable_native_dep_degrades_instead_of_raising;
   it fails the same way with the source change reverted)

ruff check: All checks passed!
ruff format --check: 2 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu (WSL2, Linux 6.18), Python 3.12.15, upstream main
e995eb9f plus this change. Real proxy on 127.0.0.1:18980 started with
`HEADROOM_MODE=token` (the default cache mode keeps earlier turns in the
frozen prefix) and `ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18981`, a
local stub upstream that logs each tool_result it receives. The proxy
log banner shows `Anthropic: http://127.0.0.1:18981`, so no request left
the machine. `headroom/_core.abi3.so` was taken from the official
headroom-ai 0.41.0 wheel (untracked, not committed).
- Exact command / steps: start the stub (`stub.py 18981 stub.log`),
start the proxy (`HEADROOM_MODE=token
ANTHROPIC_TARGET_API_URL=http://127.0.0.1:18981 python -P -m
headroom.proxy.server --host 127.0.0.1 --port 18980 --no-cache
--no-rate-limit`), then run `client.py 18980`: one `POST /v1/messages`
with Read `/src/app.py` (6,980-char result), Edit `/src/app.py` whose
tool_result has `is_error: true`, and two more turns. Repeated with the
source change reverted (before), with it applied (after), and with a
successful Edit result (control).
- Observed result: before, the upstream received the earlier Read result
replaced by a stale marker (151 chars) after an Edit whose tool_result
had is_error=True; after, the Read result arrives unchanged (6980
chars); the control (Edit result without is_error) is still replaced.
Stub log:

  ```text
  # before (change reverted): upstream receives the Read result replaced
  STUB POST /v1/messages bytes=1494
STUB msg[2] tool_result toolu_r1 is_error=None len=151 head='[Read
content stale: /src/app.py was modified after this read — re-read the
file for current content. Retrieve'
STUB msg[4] tool_result toolu_e1 is_error=True len=69
head='<tool_use_error>String to replace not found in
file.</tool_use_error>'

  # after (this change): upstream receives the Read result unchanged
  STUB POST /v1/messages bytes=8206
STUB msg[2] tool_result toolu_r1 is_error=None len=6980 head='def
handler_0(request):\n return process(request, 0)\n\ndef
handler_1(request):\n return process(request, 1)'
STUB msg[4] tool_result toolu_e1 is_error=True len=69
head='<tool_use_error>String to replace not found in
file.</tool_use_error>'

  # control (this change, Edit result without is_error): still replaced
  STUB POST /v1/messages bytes=1460
STUB msg[2] tool_result toolu_r1 is_error=None len=151 head='[Read
content stale: /src/app.py was modified after this read — re-read the
file for current content. Retrieve'
STUB msg[4] tool_result toolu_e1 is_error=None len=51 head='The file
/src/app.py has been updated successfully.'
  ```

Scripts and raw outputs:
`/var/tmp/cb-pub/proof/fix-read-lifecycle-failed-edit/` (WSL):
`stub.py`, `client.py`, `run.sh`, `before.out`, `after.out`,
`control.out`.
- Not tested: a real Anthropic upstream (stub only); OpenAI-format
conversations (tool messages carry no `is_error`, so behavior there is
unchanged); streaming requests; cache mode; the Rust core is the 0.41.0
wheel's, not one built from this checkout; mypy; the full test suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for Anthropic conversations
containing an Edit/Write whose tool_result has `is_error`; earlier Reads
of that file are no longer replaced by a stale marker
- Kill switch / disable path: `headroom proxy --no-read-lifecycle`
- Unsafe override required: no
- Qualification impact: none known
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Not included: `MultiEdit`/`NotebookEdit` are still missing from
`_MUTATING_TOOL_NAMES` (`NotebookEdit` also uses `notebook_path`, which
the path extraction does not read); that is a separate change. Only the
listed test files were run, not the whole suite.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 23:32:24 -05:00
HetCreepandClaude Opus 5.5 cddd2fd60e fix(transforms): tolerate malformed assistant tool calls in ContentRouter.apply (#4459)
## Description

A tool call with `"function": null` (or a non-object), `arguments` that
decode to a list/string/number, or a `file_path` that is not a string
raised out of `ReadLifecycleManager`, `ReadMaturationManager` and
`ContentRouter._build_tool_name_map`. `TransformPipeline.apply`
re-raised and counted a circuit-breaker failure, so three such requests
in a row switched compression off for the cooldown (60 s default) for
every request on that pipeline. The three scanners now skip the
malformed call, as the surrounding `isinstance(tc, dict)` checks already
do for a non-dict `tool_calls` entry.

Fixes #4458

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/read_lifecycle.py` (`_build_tool_metadata`): skip
a call whose `function` is not a dict; only read `file_path` / `offset`
/ `limit` when the decoded `arguments` is a dict; treat a non-string
`file_path` as absent (it is later used as a dict key). Same non-string
`file_path` guard in the Anthropic `tool_use` branch.
- `headroom/transforms/read_maturation.py` (`_scan_activity`): same
`function` and decoded-`arguments` guards; non-dict `input` and
non-string `file_path` treated as absent in both branches.
- `headroom/transforms/content_router.py` (`_build_tool_name_map`): skip
a call whose `function` is not a dict.
- `tests/test_transforms/test_malformed_tool_calls.py`: 9 malformed
shapes (`function` null / string, `arguments` decoding to list / string
/ number, `file_path` list in OpenAI arguments and in `tool_use`, `path`
dict, `input` list) run through each of the three entry points (27
cases).

## Testing

Rebased onto upstream/main `3fef15d3` (2026-10-24): the new tests pass
(27 passed); with only the source change reverted, 25 fail. The
real-behavior run below was done on `e995eb9f`.

Review round 1 (42ddb602, Devin findings): results of calls whose
`function` is not an object are now passed through verbatim (before: an
18,690-char log result was compressed to 103 chars), and read maturation
skips calls whose file cannot be read from the arguments. The tests now
assert the tool result is unchanged and the maturation test runs eight
quiet turns: 27 passed; on the previous head 9 of them fail.
Neighbouring suites (134 files touching the router, read lifecycle or
maturation): the only failures are environment ones (memory_bridge
fixtures, offline-egress, kompress), and they fail on main too.

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

(`ruff check` and `ruff format --check` were run on the four changed
files only; `pytest` on the new file and the neighbouring
read-lifecycle, read-maturation and content-router test files, not the
whole suite.)

### Test Output

```text
# new test file with only the source change absent (tests added on top of upstream/main e995eb9f)
25 failed, 2 passed
  e.g. AttributeError: 'NoneType' object has no attribute 'get'
       TypeError: unhashable type: 'dict'   (file_ops[file_path].append in _build_file_operation_index)

# with the fix
tests/test_transforms/test_malformed_tool_calls.py ..................... [ 77%]
......                                                                   [100%]
27 passed in 1.52s

# neighbouring tests with the fix
tests/test_transforms/test_read_lifecycle.py tests/test_read_maturation.py
tests/test_read_maturation_handler_nobust.py tests/test_transforms/test_content_router.py
120 passed, 1 warning in 8.93s

ruff check: All checks passed!
ruff format --check: 4 files already formatted
```

## Real Behavior Proof

- Environment: Ubuntu on WSL2 (Linux 6.18, x86_64), Python 3.12.15,
upstream/main e995eb9f with and without this commit, Rust core
`headroom/_core.abi3.so` copied (untracked) from the official
headroom-ai 0.41.0 wheel. The real proxy `python -m
headroom.proxy.server --host 127.0.0.1 --port 18970 --no-cache
--no-rate-limit` with `OPENAI_TARGET_API_URL=http://127.0.0.1:18971`, a
local stub upstream that logs every request it receives (so the stub log
shows the proxy pointed at it).
- Exact command / steps: `client36.py` sends six `POST
/v1/chat/completions` in order: a well-formed control (assistant call
`list_users`, tool result of 400 JSON rows, 31,260 bytes), three
requests whose assistant `tool_calls[0].function` is `null` (same tool
result), then two well-formed requests with new tool-result content (the
proxy's compression cache would otherwise answer for repeated content).
Run once on the base commit and once with this commit. The stub records
the body size it receives.
- Observed result: before, the three malformed requests are each
forwarded unoptimized with `Optimization failed: AttributeError:
'NoneType' object has no attribute 'get'`, the third opens `Pipeline
circuit breaker OPEN after 3 consecutive failures; passing messages
through for 60s`, and the two well-formed requests after it reach the
upstream uncompressed (31,234 bytes). After, no request fails, the
breaker never opens, and every request is compressed (the malformed ones
too, since the bad call is simply ignored). Excerpts below.
- Not tested: a real OpenAI or Anthropic API (stub upstream only); the
Anthropic `/v1/messages` and the OpenAI Responses handlers over HTTP
(the `tool_use` and string-`file_path` shapes are covered by the unit
tests only); streaming requests; malformed `offset` / `limit` values (a
non-numeric `offset` is still not guarded); Windows or macOS; a Rust
core built from this checkout (the 0.41.0 wheel's was used); how many
callers share one pipeline in a real deployment.

Before (base e995eb9f), proxy log and stub:

```text
[hr_1792691400_000002] Optimization failed: AttributeError: 'NoneType' object has no attribute 'get' (request forwarded unoptimized)
[hr_1792691400_000003] Optimization failed: AttributeError: 'NoneType' object has no attribute 'get' (request forwarded unoptimized)
Pipeline circuit breaker OPEN after 3 consecutive failures; passing messages through for 60s
[hr_1792691400_000004] Optimization failed: AttributeError: 'NoneType' object has no attribute 'get' (request forwarded unoptimized)
STUB-UPSTREAM call #1 body_bytes=10090 tool_content_chars=9433    <- control, compressed
STUB-UPSTREAM call #2 body_bytes=31200 tool_content_chars=26180   <- malformed
STUB-UPSTREAM call #3 body_bytes=31200 tool_content_chars=26180
STUB-UPSTREAM call #4 body_bytes=31200 tool_content_chars=26180
STUB-UPSTREAM call #5 body_bytes=31234 tool_content_chars=26180   <- well-formed, NOT compressed
STUB-UPSTREAM call #6 body_bytes=31234 tool_content_chars=26180
```

After (this commit): no `Optimization failed`, no `circuit breaker`
line, six `Pipeline complete: ... saved 6786 ... 62.6% reduction` lines,
and:

```text
STUB-UPSTREAM call #1 body_bytes=10090 tool_content_chars=9433
STUB-UPSTREAM call #2 body_bytes=10056 tool_content_chars=9433
STUB-UPSTREAM call #3 body_bytes=10056 tool_content_chars=9433
STUB-UPSTREAM call #4 body_bytes=10056 tool_content_chars=9433
STUB-UPSTREAM call #5 body_bytes=10090 tool_content_chars=9433
STUB-UPSTREAM call #6 body_bytes=10090 tool_content_chars=9433
```

Scripts and raw outputs (WSL):
`/var/tmp/cb-pub/proof/fix-malformed-tool-calls-apply/` (`stub36.py`,
`client36.py`, `run36proof.sh`, `before.*`, `after.*`, `src.patch`).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, narrowly: requests with a
malformed tool call are now compressed instead of failing the router and
counting toward the circuit breaker; well-formed requests are unaffected
- Kill switch / disable path: none; no config involved
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Design choice to review: a malformed call is skipped (it contributes no
tool name, no file path), not rejected, so the request still goes
through compression. The alternative is validating tool calls at handler
ingress; I did not find such validation today and kept the fix local to
the three scanners that crashed. Documentation: no doc describes this
behavior.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 23:32:07 -05:00
e11afdbdf8 fix(smart-crusher): match audit-safe protected_patterns against unescaped text (#4294)
## Description

Audit-safe mode (`SmartCrusherConfig(audit_safe=True,
protected_patterns=[...])`, #1705) guarantees that rows matching a
protected regex survive compression. The regexes run against
`SmartCrusher._canon(row)`, which is `json.dumps(row, sort_keys=True,
default=str)`. `json.dumps` defaults to `ensure_ascii=True`, so every
non-ASCII character is escaped before the pattern sees it:

```python
_canon({"city": "Kraków"})   # '{"city": "Krak\\u00f3w"}'
re.search("Kraków", ...)     # None
```

A pattern that contains any non-ASCII character can never match:
accented names, CJK error text, currency symbols such as `€`.
`_scan_protected_rows` returns `[]`, nothing is spliced back or
verified, and the row is sampled out like any other. Nothing warns,
because from the crusher's point of view there was nothing to protect.
That's the exact failure audit-safe mode exists to prevent.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/smart_crusher.py`: `_canon()` serializes with
`ensure_ascii=False`, so patterns are matched against the real
characters. `_canon` is also the identity key in
`_splice_missing_protected` / verification, and both sides of that
comparison go through it, so identity stays consistent. Rows without
non-ASCII text serialize exactly as before.
- `tests/test_transforms/test_smart_crusher_audit_safe.py`:
`test_protected_patterns_match_non_ascii_text`, parametrized over
`Kraków`, `错误` and `Zürich`, builds a 600-row array where row 437's
`city` is the pattern. It checks that `_scan_protected_rows` finds that
row and that it survives a real `_smart_crush_content` lossy sample (600
→ ~16 rows).

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_transforms/`,
`tests/test_garbled_compression_fixes.py`,
`tests/test_smart_crusher.py`,
`tests/test_smart_crusher_toin_attachment.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test on `main` (fix reverted, test kept) fails at the scan:

```text
$ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q -k non_ascii
E   AssertionError: assert [] == [{'id': 437, ...', 'ms': 185}]
E     Right contains one more item: {'id': 437, 'city': 'Kraków', 'ms': 185}
```

With the scan assertion removed, so the crush outcome on `main` is
visible, all three rows are sampled out:

```text
E   AssertionError: assert {'id': 437, 'city': 'Kraków', 'ms': 185} in [{'id': 0, 'city': 'Berlin', 'ms': 0}, {'id': 1, ...}, ...]
E   AssertionError: assert {'id': 437, 'city': '错误', 'ms': 185} in [...]
E   AssertionError: assert {'id': 437, 'city': 'Zürich', 'ms': 185} in [...]
======================= 3 failed, 11 deselected in 9.52s =======================
```

With the fix:

```text
$ python -m pytest tests/test_transforms/test_smart_crusher_audit_safe.py -q
============================= 14 passed in 28.95s ==============================

$ python -m pytest tests/test_transforms tests/test_garbled_compression_fixes.py tests/test_smart_crusher.py tests/test_smart_crusher_toin_attachment.py -q
================== 519 passed, 78 skipped in 67.42s (0:01:07) ==================

$ ruff check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py
All checks passed!
$ ruff format --check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_audit_safe.py
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel),
`HEADROOM_OFFLINE=1`
- Exact command / steps: ran the full `SmartCrusher.apply(messages,
tokenizer)` transform (the path
`test_apply_preserves_protected_rows_in_tool_message` uses for the
proxy) with `SmartCrusherConfig(audit_safe=True,
protected_patterns=["Kraków"], min_tokens_to_crush=10)` on an OpenAI
tool message holding a 600-row JSON array (UTF-8, `ensure_ascii=False`,
as tool output arrives) in which one row's `city` is `Kraków`. I parsed
the compressed tool content and checked for that row, on `main` and on
this branch.
- Observed result: on `main` the protected row is dropped: 16 of 600
rows kept, `Kraków` row kept: False. On this branch it is kept: 17 of
600 rows, the 16 sampled plus the spliced protected row.

  ```text
  == main
  rows kept: 16 of 600 | Kraków row kept: False
  == this branch
  rows kept: 17 of 600 | Kraków row kept: True
  ```
- Not tested: the proxy end to end with an audit-safe config; patterns
that deliberately match the escaped form (a regex like `Krak\\u00f3w`
with an escaped backslash). I don't know of anyone relying on that, and
such a pattern stops matching after this change. A regex escape like
`Krak\u00f3w` means the character `ó` to `re`, so it never matched on
`main` and now does.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for audit-safe users whose
`protected_patterns` or protected rows contain non-ASCII text: those
rows are now protected as documented. ASCII-only rows serialize
identically, so existing audit-safe behaviour is unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Docs unchecked: the audit-safe docs don't mention ASCII. This makes
non-ASCII patterns behave the way the docs already imply.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:37:46 -07:00
Mansurshakh JaparovandJerrettDavis 9762a14a40 fix(smart-crusher): preserve wrapped retrieval results in apply (#4149)
## Current-head security-gate integration

Head `17ac399ac38930a265d8e5108c703110cd3e2ebb` incorporates upstream
main `49751ec4` (#4178). The upstream change is Cargo.lock only;
contributor logic and behavioral regressions are unchanged. This section
supersedes older current-head/check-status references below; earlier
runtime evidence is retained as historical implementation qualification.

Normal commit/pre-push hooks passed without bypass: Rust formatting,
warnings-as-errors Clippy, complete workspace tests, native editable
rebuild/import verification, **181 critical Python tests**, and
branch-range commitlint. Fresh local `cargo audit --file Cargo.lock`
also passed (546 dependencies, no findings). Hosted checks for this new
head must complete independently; local results are not a claim that all
hosted checks are green.

## Description

Closes #4147. Related: #3563, #3837 and the already merged router fix
#3915.

Close the remaining direct Python `SmartCrusher.apply()` parity gap:
resolve originating wrapper calls with the existing shared
`unwrap_tool_call_name` parser before applying the CCR retrieval
exemption. This is not a duplicate proxy/router fix, a blanket wrapper
exclusion, or a new content-based recovery heuristic.

## Type of Change

- [x] Bug fix (non-breaking correction)
- [x] Documentation update

## Changes Made

- Resolve OpenAI originating function arguments and Anthropic tool-use
input to their effective tool names in the existing call-ID index.
- Preserve null-function handling, missing-ID/name behavior and the
shared parser's conservative ambiguity/malformed-input rules.
- Add four real-native regression cases: OpenAI/Anthropic × Hermes
`tool_call`/`functions.exec`, each checking byte-exact retrieval output
and a genuinely compressed ordinary wrapper-output control.
- Remove old tautological compression assertions and retain
named-retrieval consumer coverage.
- Document direct `apply()` recovery identity, the need for originating
call history, and the context-free `crush()` limit. Native compression,
thresholds, CCR storage and the parser itself are unchanged.

## Testing

- [x] Real native regression and shared wrapper safety suites pass
- [x] Ruff passes on the two changed Python files
- [x] Scoped mypy passes on the changed source file
- [x] Actual rebuilt native API and real tokenizer exercised before and
after

### Test Output

```text
Before: 4 wrapped-recovery regressions failed at byte preservation.
After: 65 passed, 1 warning in 126.20s
Ruff: All checks passed!
mypy: Success: no issues found in 1 source file
```

The warning is the existing Anthropic tokenizer approximation in an
unchanged Hermes test. The scoped suite comprises the direct retrieval
exemption tests plus Hermes and orchestrator-wrapper safety coverage;
this is not a full-Python-suite claim.

Normal commit/push hooks completed without bypass: Rust fmt, workspace
Clippy
with `-D warnings`, workspace Rust tests, native extension
build/install/import
verification, critical Python suite (**177 passed, 4 skipped, 1 warning
in
25.48s**) and commitlint. The ordinary push succeeded with the genuine
installed
ORT library explicitly resolved. No native backend or gate threshold
changed.

### Review follow-up verification

The non-string-name caller correction is in
`40fa055854b1c4b60da84f78d534976bc3fd40fb`; the reviewed thread is
resolved. Verification against that exact published Python source: **83
passed, 1 existing Anthropic approximation warning in 16.44s**, across
direct retrieval, Hermes and orchestrator safety suites. Changed-file
Ruff and scoped mypy passed.

A separate real native API smoke reused the genuine unchanged Rust
extension and real OpenAI tokenizer: all four wire/wrapper cases
returned exactly 27,490 recovery bytes, and all ordinary controls still
compressed. An OpenAI `name=["Bash"]` returned its original `"ok"`
result. Anthropic malformed names are tested at the identity-index
boundary: the bundled tokenizer rejects such malformed tool-use names
earlier, and this PR does not change or claim that tokenizer behavior.
No mocked native backend, live provider request, or live retrieval-store
claim.

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.14, latest-main base
`976aa714ef3e19174dc0cb586cee7b20d064e4ef`, freshly built
`headroom/_core.abi3.so`, real OpenAI `gpt-4o` tokenizer. Resolve the
installed genuine `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`; no
native fallback, mocked compressor or provider request.
- Exact command / steps: run `python -m pytest -q
tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py
tests/test_hermes_tool_call_unwrap.py
tests/test_orchestrator_wrapper_ccr_retrieve.py`, Ruff on the two
changed files and mypy on `headroom/transforms/smart_crusher.py`.
Separately call the production native-backed `SmartCrusher.apply()` with
a 27,490-byte synthetic recovery result and an equally sized ordinary
result; originating wrapper arguments invoke retrieval only for the
recovery call. Exercise both wire formats and both wrappers. The
isolated reproduction uses the existing test configuration
`min_tokens_to_crush=0` unchanged before/after; production/native
thresholds are not changed.
- Observed result: before, all four recovery payloads became CCR-marked
505/509-byte outputs. After, all four returned exactly 27,490 bytes,
matching the original byte for byte, with no new recovery CCR marker.
Every ordinary wrapper result still compressed. Copied runtime results
are below.
- Not tested: a live provider session, proxy HTTP route, real client
integration, or a complete retrieval loop. This is a real direct-library
reproduction using synthetic returned recovery bytes, not a claim that a
synthetic hash was retrieved from a live store.

| Wire | Wrapper | Recovered bytes before | Recovered bytes after |
Exact after | Ordinary output compressed after |
|---|---|---:|---:|---|---|
| OpenAI | `tool_call` | 505 | 27490 | Yes | Yes |
| OpenAI | `functions.exec` | 505 | 27490 | Yes | Yes |
| Anthropic | `tool_call` | 509 | 27490 | Yes | Yes |
| Anthropic | `functions.exec` | 509 | 27490 | Yes | Yes |

## Runtime Rollout Safety

- Rollout-managed feature(s): None; existing originating-call identity
and CCR exemption only.
- Minimum rollout channel: Stable; existing rollout controls unchanged.
- Stable/default behavior changed: Direct `apply()` no longer
re-compresses recognized wrapped recovery results. Unrelated wrapper
output still compresses; malformed/ambiguous wrappers retain normal
shared-parser behavior.
- Kill switch / disable path: No new switch; revert this identity-map
correction to restore prior direct-transform behavior.
- Unsafe override required: None; runtime qualification used only
synthetic local messages.
- Qualification impact: Both wire formats, both wrapper families, real
native output/control behavior and shared parser/router safety coverage
qualified. Live-client/provider/proxy limits remain explicit above.
- Rollback path: Revert the code/tests/docs to base
`976aa714ef3e19174dc0cb586cee7b20d064e4ef`; no data/schema migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Shared wrapper parsing reused, no second identity convention
- [x] Documentation updated
- [x] Native and existing wrapper safety regressions pass
- [x] No new source warnings, dependencies or native policy changes
- [x] `CHANGELOG.md` untouched; Conventional Commit PR title used

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-13 16:37:18 -07:00
Tejas ChopraandClaude Opus 5.5 e4a1ec19be refactor(transforms): deprecate unused compress_with_stats, extract_batch and align_for_cache (#4098)
## Description

`DiffCompressor.compress_with_stats`, `HTMLExtractor.extract_batch` and
`cache_aligner.align_for_cache` are public. The first two are methods on
classes exported from `headroom.transforms`, and the third's docstring
called it a stable public API. None is called anywhere in Headroom, the
plugins or the consumer repos; only tests call them. Each now emits a
`DeprecationWarning` that names the replacement.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `DiffCompressor.compress_with_stats` warns and points to
`DiffCompressor.compress`. Unlike `compress()`, it never wrote its CCR
entry to the compression store, so a retrieval marker in its output
could dangle. The docstring now says so.
- `HTMLExtractor.extract_batch` warns and points to calling
`HTMLExtractor.extract` per page.
- `align_for_cache` warns and points to
`CacheAligner(config).apply(messages, tokenizer)` and the
`stable_prefix_hash:` marker.
- The existing tests now assert the warnings. A new test checks that
`align_for_cache` warns at the caller and still returns the messages and
the hash.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_cache_aligner_detector_only.py tests/test_cache_aligner_prefix_stability.py tests/test_transforms/test_html_extractor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms/test_diff_compressor.py
118 passed in 3.87s
$ ruff check / format: All checks passed;  mypy: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'align_for_cache(...);
DiffCompressor().compress_with_stats(...)'`.
- Observed result: The proxy serves both requests (200) with the same
savings as main. Each deprecated call emits one DeprecationWarning
attributed to the caller and still returns its result. Output below.
- Not tested: Third-party callers; none is known.

```text
branch: cleanup/transforms-deprecate-unused-methods @ a3e57303
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51205.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -W always -c "align_for_cache([...]); DiffCompressor().compress_with_stats(...)"
<string>:4: DeprecationWarning: align_for_cache is deprecated and will be removed in a future release; use CacheAligner(config).apply(messages, tokenizer) and read the stable_prefix_hash: marker from markers_inserted.
<string>:5: DeprecationWarning: DiffCompressor.compress_with_stats is deprecated and will be removed in a future release; use DiffCompressor.compress instead.
ok
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only a DeprecationWarning when the
deprecated methods are called.
- Kill switch / disable path: Not applicable; the warning is a standard
DeprecationWarning and can be filtered with `warnings`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

Deprecated (not removed): `DiffCompressor.compress_with_stats`,
`HTMLExtractor.extract_batch`, `cache_aligner.align_for_cache`. The
warnings fire on every call, which is standard. Python's default warning
filters show each one once per call site.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:03:03 -07:00
Suliman AbdulrazzaqandJerrettDavis 3db1a94443 fix(smart-crusher): keep string cells in json columns from decoding as other types (#4053)
## Description

The SmartCrusher lossless table (`csv-schema`) types a column `json`
when its cells mix types, for example `[26]{id:int,v:json?}`.
`_decode_cell` in `headroom/transforms/compaction_codec.py` reads every
cell of a `json` column with `json.loads` and only keeps the raw text
when that fails. The Rust encoder did not know the column type when it
rendered a string cell, so strings that spell a JSON value came back as
that value:

- the string `null` was written `"null"` and decoded as `None`
- the string `true` was written `true` (byte-identical to the bool) and
decoded as `True`
- the string `1` was written `1` and decoded as the int `1`
- `NaN` decoded as a float nan, and the string `"q"` (a string that
carries its own quotes) decoded as `q`

A string that parses as a JSON array or object (`[1]`, `{"a":1}`) never
reaches the formatter as a string: the classifier already un-escapes it
(`CellClass::StringifiedJson`), which is existing behavior and is not
changed here.

This changes the encoder only. In a column whose type tag is `json`, a
string cell is now written as a JSON string literal (then CSV-quoted)
when `json.loads` could parse it into something else. The decoder
already returns the string for that form, so it is not touched.
Everything else renders as before.

"Could parse into something else" is a conservative superset: after
trimming JSON white space (space, tab, newline, carriage return), the
text starts with `[`, `{`, `"`, `-` or an ASCII digit, or is exactly
`true`, `false`, `null`, `NaN` or `Infinity`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

-
`crates/headroom-core/src/transforms/smart_crusher/compaction/formatter.rs`:
`write_table` works out once which columns have type tag `json` and
passes that flag to `format_cell`. For a `Value::String` cell in such a
column, `is_ambiguous_json_text` decides whether it is rendered as
`csv_quote(serde_json::to_string(s))`. Non-string cells, the
`string`/`int`/`float`/`bool` columns, the JSON formatter and the
Markdown-KV formatter are unchanged. Bucketed tables go through
`write_table` too, so they get the same treatment.
- Rust tests in the same file:
`csv_formatter_json_column_quotes_strings_that_parse_as_json` (13
strings, exact wire output, with a `string` column as control),
`csv_formatter_json_column_leaves_other_strings_alone`, and
`csv_formatter_json_column_tells_strings_from_values`, which goes
through `compact` + `CsvSchemaFormatter` so the column type really comes
out as `json`, and checks that the string `"true"` and the bool `true`
no longer share a wire form.
- `tests/test_transforms/test_compaction_codec.py`: one test that
decodes a hand-written block using the new wire form next to the bare
forms for real values. It documents the contract the Rust change relies
on and passes with or without this change, since the decoder is not
changed.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` on the touched test file; `cargo fmt`
and `cargo clippy -D warnings` in CI)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Rust, GitHub Actions run on our fork (toolchain 1.95.0)
https://github.com/SulimanAbdulrazzaq/headroom/actions/runs/37829441316
  cargo fmt --all -- --check                                 ok
  cargo clippy -p headroom-core --all-targets -- -D warnings ok
  cargo test -p headroom-core smart_crusher                  345 passed; 0 failed
    (342 on main in run 37827119554; the 3 new tests are the difference)
  csv_formatter_json_column_leaves_other_strings_alone ... ok
  csv_formatter_json_column_quotes_strings_that_parse_as_json ... ok
  csv_formatter_json_column_tells_strings_from_values ... ok

# Python, 3.12
$ pytest tests/test_transforms/test_compaction_codec.py -q
17 passed   (16 existing + 1 new)
$ ruff check tests/test_transforms/test_compaction_codec.py          All checks passed!
$ ruff format --check tests/test_transforms/test_compaction_codec.py 1 file already formatted
```

## Real Behavior Proof

- Environment: Rust core built and tested on `ubuntu-latest` in GitHub
Actions with Rust 1.95.0 (the repository pin). Python 3.12.13 on Windows
11 for the decoder test and the reproducer.
- Exact command / steps: a script crushes 26 records `{"id": i, "v":
...}` whose `v` cycles through the strings `null`, `true`, `1`, `NaN`
and other values, decodes the result with `expand_compacted`, and
compares each row with its input. The encoded header is
`[26]{id:int,v:json?}`. In CI: `cargo test -p headroom-core
smart_crusher`.
- Observed result: before the change (the Rust core of the installed
`headroom-ai` 0.40.0 wheel), 8 of 26 rows differ, for example `'null'
(str) -> None`, `'true' (str) -> True`, `'1' (str) -> 1`, `'NaN' (str)
-> nan`. After the change the new Rust test shows the string `null` in a
`json` column is written `"""null"""` and the string `hello` is still
written `hello`; in the real-path test (through `compact` and
`CsvSchemaFormatter`) the rows are `1,"""null"""`, `2,null`,
`3,"""true"""`, `4,true`, `5,"""1"""`, `6,1`, `7,hello`, so each string
and each value has its own wire form. The Python test shows the decoder
returns `'null'`, `None`, `'true'`, `True`, `'1'`, `1`, `'NaN'` for such
a block.
- Not tested: the 26-row Python reproducer against a freshly built core
(the installed wheel on my machine is older and cannot see the Rust
change), and the Python wheel end to end. The Python decoder was not
changed.

### Maintainer native qualification (2026-10-08)

- Environment: Windows 11, Rust/MSVC, Python 3.12.13. This head contains
current `main`; contributor formatter logic was not modified.
- Exact commands: `cargo test -p headroom-core --lib
transforms::smart_crusher::compaction --locked`; `cargo build -p
headroom-py --features extension-module --locked`; then the PR's
`tests/test_transforms/test_compaction_codec.py` suite using that newly
built extension. A standalone native SmartCrusher/Python decoder probe
also encoded and decoded 100 records containing mixed
string/null/bool/numeric cells.
- Observed: **73 Rust compaction tests and 17 Python codec tests
passed**. The 100-record probe preserved every value and Python type,
including literal strings `null`, `true`, `false`, `1`, `NaN`,
`Infinity`, `-Infinity`, whitespace, quotes, empty strings, and embedded
newlines. No native encoder/decoder boundary was mocked.
- Scope: JSON-container-looking strings (`[1]`, `{"a":1}`) retain the
classifier's pre-existing `StringifiedJson` behavior. They are
explicitly outside this formatter repair; the author clarified that
limitation in the test. This PR fixes mixed-column primitive-literal
strings that actually reach the formatter, and does not claim a new
container-string round-trip contract.
- Hosted CI is tracked by the maintainer at this exact head; approval
remains pending until all applicable current checks pass.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: the lossless table text changes only
for string cells that look like JSON literals inside `json`-typed
columns. All other cells and all other column types render as before.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit. Older decoders read the new form
correctly, because it is a plain quoted JSON string, so nothing else
needs to change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Rust tests ran in CI on a fork, not locally, because the machine used
for development has no linker for the Rust toolchain. `mypy` and the
full Python suite were not run, only the touched test file. No other
test or fixture in the repository asserts the wire form of a string in a
`json` column (searched `crates`, `tests`, `docs` and `headroom` for
`json?` and `:json`).

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-08 22:40:54 -05:00
f37980f7f7 fix(router): bound ML compression per request, not just per block (#3721)
## Description

> Revised 2026-10-05 after review: the first ML call of a request is now
capped by the remaining ML budget through Kompress's existing
chunk-boundary time budget (no seed estimate), and budget-refused blocks
pass through unchanged so their bytes stay stable on later turns. Third
review: the same remainder now caps `RemoteKompressCompressor` too, over
the whole HTTP request, and the remote call is declined when 50 ms or
less is left.

Closes #3711.

Thanks to @gaaaork for the report — and for the follow-up correction,
which is what made this findable. The corrected finding (it *is* the
Kompress stage; `kompress` alone reproduces it) was right, and this PR
answers the open question at the end of that comment.

### The question was: why did the size gate never fire?

> `headroom_kompress_size_gate_total{outcome="within"} 2584` — only
`within`. Either the gate is evaluated per chunk (each chunk under the
ceiling, while their sum takes 70s), or it is not applied on this path.

**It is the first.** The gate lives inside `_try_ml_compressor`
(`content_router.py:4152`), and `_compress_mixed` calls that function
**once per section** in a serial loop (`content_router.py:2815-2841`).
So the gate only ever sees one section. It bounds **per-call size**;
nothing bounded the **request**.

Reproduced with the reported shape — a `tool_result` of prose blocks
separated by small JSON objects — instrumented through the real router:

```
strategy                 : router:tool_result:mixed
_try_ml_compressor calls : 8
size-gate outcomes       : {'within': 8}     <- no 'exceeded', exactly as reported
```

Eight separate model entries, eight independent gate decisions, no
aggregate accounting. The 30s `compression_first_stage` budget is
per-request while the only guard was per-section — a granularity
mismatch. And there was no cumulative deadline either: the one existing
helper (`_compression_deadline_seconds`) is used at a single site and
only when `len(pending_tasks) == 1`, so the section loop ran unbounded.

One note on the payload shape, since it cost me a false start: plain
user text never reaches the ML stage at all
(`router:protected:user_message`). The content has to arrive as a
`tool_result` to reproduce — which is exactly what was reported.

### Why not just gate on the whole request

That was the suggestion in the issue, and on its own it is not enough.
It would fix the 1.4MB case while leaving a request of 100 sections at
40k tokens each equally able to hang — every section still passes any
size ceiling. The guard needs to match the budget's *granularity*, not
just its scope.

So this adds a **request-scoped wall-clock ceiling**. Time is also the
honest unit here: the estimator is already known to misjudge dense
payloads (see the `#1171` comment above the gate), and the symptom being
protected against is wall-clock, not tokens.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_ml_stage_deadline_seconds()` — `HEADROOM_ML_STAGE_DEADLINE_SECONDS`,
default **15s**, deliberately under the 30s
`HEADROOM_COMPRESSION_TIMEOUT_SECONDS` so we degrade to a *partial*
compression **before** the executor abandons a non-preemptible worker.
Losing some savings beats losing the request and then quarantining the
stage for everything behind it. `0` restores the previous unbounded
behaviour.
- Enabled by `apply()` — the one place that owns a whole request — and
carried on `_PerRequestRuntimeState`, so the existing `copy_context()`
fan-out (documented under #3486/#3556) propagates it to worker threads
for free. The budget counts only time spent **inside** the ML compressor
(cumulative, charged in a `finally`), and is clamped to half of
`HEADROOM_COMPRESSION_TIMEOUT_SECONDS`.
- Admission: a block is refused when its projected cost (this request's
worst measured seconds-per-token) exceeds what is left. Every admitted
call — **including the first, which has no measurement** — is also
**capped**: the remainder goes to Kompress as
`_time_budget_cap_seconds`, lowering its existing per-call time budget,
which Kompress already checks at every chunk boundary. So no call can
run more than one chunk past the budget, and parallel workers each end
by their admission time plus the remainder.
- Remote Kompress (`HEADROOM_KOMPRESS_ENDPOINT`) gets the same cap.
httpx timeouts are per phase, so the cap is applied to the request as a
whole: the POST runs on a small worker pool (16) and is awaited with one
overall deadline. Admission is bounded: a permit per worker is taken
before submit, waiting at most until the caller's deadline, so a
saturated call fails open unchanged and its payload is never enqueued.
The worker holds its permit until it actually finishes, and it finishes
by an absolute deadline: per-phase timeouts are capped to the time left
and the body is streamed and cut off at the deadline, so a slow-drip
response cannot hold a worker. At or below 50 ms left, no request is
sent and the block passes through unchanged. With no request budget
(direct callers) the call and its 20s timeout are exactly as before.
- Blocks refused by the budget (or cut off by the cap) **pass through
unchanged**. Not the size gate's lossy fallback: the refusal depends on
the request, not the content, and the router caches that verdict, so raw
bytes are the only ones that stay stable on later turns (prompt-cache
safety).
- Recorded as `headroom_kompress_size_gate_total{outcome="deadline"}`,
so this is visible in the metric you were already reading, next to
`within`/`exceeded`.
- Direct `compress()` callers (tests, `/v1/compress`) stay unarmed and
keep the old behaviour.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed


The original four tests, using a stub in place of Kompress so they
assert the *budget* rather than ONNX latency (the real model is not
available in CI, and the bug is about how many times a slow stage is
entered):

- `test_size_gate_alone_cannot_bound_a_request` — pins the precondition:
many `within`, zero `exceeded`, repeated model entry. Passes on `main`;
it *is* the bug.
- `test_ml_deadline_stops_further_ml_work_once_the_budget_is_spent` —
fails on `main`. Measures the saving by running the same payload with
the ceiling disabled and asserting strictly more model calls.
- `test_deadline_zero_restores_previous_unbounded_behaviour`
- `test_direct_compress_callers_stay_unarmed` — fails on `main`.

Verified fail-before/pass-after by reverting only `content_router.py`
and keeping the tests.

Regression sweep `-k "router or kompress or mixed or compress"`: **2555
passed**, 10 failed — and those same 10 fail identically on
`origin/main` with this file reverted, so **zero new failures**. Ruff
0.16.7 (the CI pin) clean.

Added after review (each fails with the source reverted and the tests
kept, passes with it):

- `test_non_ml_prework_does_not_consume_the_ml_budget`,
`test_a_slow_call_cannot_be_admitted_against_a_budget_it_will_overrun`,
`test_budget_is_clamped_below_the_timeout_that_quarantines_the_stage`,
`test_a_failing_ml_call_still_draws_down_the_budget` (first review).
- `test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against` —
first block's worst case is 3s under a 1s executor timeout (0.5s ML
budget); asserts the first call is handed a cap ≤ 0.5s and total ML time
stays within budget + one chunk.
- `test_blocks_refused_by_the_budget_pass_through_unchanged` — refused
block is byte-identical even with TextCrusher available.
- `test_time_budget_cap_stops_a_call_and_returns_it_unchanged`,
`test_time_budget_cap_bounds_the_batched_path_too` (real
`KompressCompressor`, fake model/clock) — the cap stops at the first
chunk boundary past it and returns the block unchanged, on both the
single and batched paths.
- `test_cold_remote_call_cannot_overrun_a_tiny_ml_budget` (third review,
mirrors the reviewer's probe): real `RemoteKompressCompressor` over
`httpx.MockTransport` with a 100 ms endpoint, 10 ms ML budget, 40 ms
outer timeout, via public `apply()`. Asserts ML time < 40 ms, no HTTP
request sent, prose blocks byte-identical.
- `test_cold_remote_call_is_bounded_as_a_whole_by_the_ml_budget`: 200 ms
budget, endpoint that hangs for 5 s. Asserts the first request's
per-phase timeouts are lowered to the budget, total ML time ≤ budget +
0.1 s, blocks unchanged.
- `test_remote_kompress_keeps_its_default_timeout_without_a_budget`:
uncapped call keeps the 20s timeouts and does not hop threads.
-
`test_saturated_remote_calls_fail_open_without_queueing_their_payloads`
(fourth review, the reviewer's repro): 16 workers held by requests whose
200 ms callers timed out, then 5 calls with 60 ms caps. Each fails open
unchanged within its cap, the executor queue stays empty, none reaches
the endpoint, and capacity returns once the held requests end.
- `test_slow_drip_response_is_cut_off_at_an_absolute_deadline`: a body
dripping a chunk every 50 ms for 2 s under a 200 ms cap; the worker
stops at the deadline and returns its permit well before the drip ends.
- `test_permit_is_held_until_the_worker_finishes_not_the_caller`.

### Test Output

```text
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest -q \
    tests/test_content_router*.py tests/test_transforms/test_content_router*.py \
    tests/test_transforms/test_kompress*.py tests/test_kompress*.py tests/test_force_kompress_all.py \
    tests/test_remote_kompress_dropin.py tests/test_transforms_content_router.py tests/test_router_*.py \
    tests/test_onnx_router_threads.py tests/test_proxy_disable_kompress.py tests/test_proxy_per_provider_kompress.py
486 passed, 2 skipped, 1 warning in 19.42s

# with only the source hunks reverted:
FAILED tests/test_content_router_ml_stage_deadline.py::test_first_ml_call_is_capped_by_the_budget_it_was_admitted_against
FAILED tests/test_content_router_ml_stage_deadline.py::test_blocks_refused_by_the_budget_pass_through_unchanged
FAILED tests/test_transforms/test_kompress_deadline.py::test_time_budget_cap_stops_a_call_and_returns_it_unchanged
FAILED tests/test_transforms/test_kompress_deadline.py::test_time_budget_cap_bounds_the_batched_path_too
```

## Real Behavior Proof

- Environment: local macOS, Python 3.14, branch merged with current
`main`.
- Exact command / steps: the pytest invocation above; the reporter's
payload shape (prose blocks separated by small JSON objects inside a
`tool_result`) driven through the real `ContentRouter.apply()` with a
stub in place of the ONNX model.
- Observed result: ML time stays within the budget (plus at most one
chunk) even when the first block's worst case exceeds the executor
timeout; refused blocks are byte-identical.
- Not tested: real Kompress ONNX latency on the reporter's hardware
(model not downloaded locally); a real remote Kompress endpoint (covered
network-free via `httpx.MockTransport`).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: yes — a request now spends at most
15s (or half of `HEADROOM_COMPRESSION_TIMEOUT_SECONDS`, if lower) inside
ML; blocks past that are sent uncompressed.
- Kill switch / disable path: `HEADROOM_ML_STAGE_DEADLINE_SECONDS=0`
restores the previous unbounded behaviour.
- Unsafe override required: no.
- Qualification impact: none expected; only requests whose ML stage
exceeds the budget see less compression.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

### What this does not claim to fix

The ~70s figure is still the reporter's measurement, not mine — I could
not reproduce it locally because the Kompress model is not downloaded on
this machine, so I discarded my own latency numbers as meaningless. What
this PR fixes is the *consequence*: a request that spends too long in ML
now degrades to partial compression instead of blowing the budget and
quarantining the stage.

@gaaaork — you offered to test a patch. If you can run this with
`HEADROOM_ML_STAGE_DEADLINE_SECONDS` at the default 15, the stage should
return in ~15s with partial compression and no
`CompressionQuarantinedError`, and `headroom_kompress_size_gate_total`
should start showing `deadline`. The separate question of why one
40k-token section takes seconds on your hardware is still worth chasing
— this bounds the blast radius, it does not make the model faster.

Documentation: no docs page covers `HEADROOM_ML_STAGE_DEADLINE_SECONDS`
yet; mypy not run locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3721"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-06 08:39:09 -05:00
84f0e84dcc feat(compress): live-agent densify mode, lossless and cache-safe (#1402)
## Description

Revised 2026-10-05 after review: the codec now decodes main's native
null encoding (bare `null` = null, empty cell = missing key) with no
formatter change, and agent-mode `tokens_after` is recounted with
`count_messages` after factoring and reverts.

Adds an opt-in agent densification mode that only shrinks content,
verifies reversibility message by message, and reverts any output that
cannot be proven lossless. It also adds a pure-Python inverse decoder
and reversible low-cardinality value factoring for the existing
`csv-schema` format.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `compress(mode="agent")` and `densify()` with verified-lossless
fallback.
- Add public `expand_compacted()` and `is_compacted()` helpers.
- Decode `csv-schema` cells exactly as main's native formatter (#3665)
encodes them: empty cell = missing key, bare `null` = null, quoted `""`
/ `"null"` = strings. This PR no longer changes the Rust formatter.
- Recount `tokens_after` with the pipeline tokenizer's `count_messages`
after agent-mode value factoring and lossless reverts.
- New `CompressConfig` / `CompressResult` fields sit after
`diagnostics`, so existing positional construction is unchanged.
- Add reversible value factoring only when it reduces output size.
- Preserve current `main` dependency placement, security pins, CCR
gating, and Windows ONNX initialization.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
tests/test_transforms/test_compaction_codec.py + tests/test_densify_agent_mode.py: 30 passed (native extension built from this branch)
tests/test_compress_*.py + tests/test_transforms/ + agent-mode suite: 577 passed, 67 skipped
cargo test -p headroom-core --lib compaction: 73 passed
ruff check / ruff format --check on touched files: passed
```

## Real Behavior Proof

- Environment: macOS, locked `uv` environment, native extension built
from this branch, `HEADROOM_*` env scrubbed
- Exact command / steps: public `headroom.densify()` on a tool result of
40 records with three repeated long paths, some `null` paths and some
literal `"null"` strings
- Observed result: `factored: True | reverted: 0 | lossless: True`;
`tokens_before: 1269 | tokens_after: 407 |
count_messages(result.messages): 407`; `expand_compacted(content) ==
records: True`
- Not tested: live multi-turn prompt-cache hit rate and proxy wiring;
agent mode remains an opt-in library API

## Runtime Rollout Safety

- Rollout-managed feature(s): none; `mode="agent"` / `densify()` is an
opt-in library API
- Minimum rollout channel: not applicable (library opt-in, not wired
into the proxy)
- Stable/default behavior changed: no; default `compress()` and native
formatter output are unchanged
- Kill switch / disable path: do not pass `mode="agent"` / do not call
`densify()`
- Unsafe override required: no
- Qualification impact: none for the proxy path
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable: this PR does not change a user interface.

## Additional Notes

The branch was rebuilt on current `main`. The older
dependency-relocation portion was intentionally omitted: current `main`
keeps those dependencies available and includes a security exclusion for
the compromised `ast-grep-cli` release. The current CCR gate already
supersedes the PR's older gate implementation and was retained.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/1402"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 08:28:46 -05:00
ffc359976b fix: protect file reads in chained shell commands (#2668)
## Description

`_is_read_command` decides whether a shell command's output is file
content the agent will patch from. Output classified as a read is passed
verbatim; everything else is eligible for lossy Kompress compression.

It inspected only the **first program**, after peeling `cd`/`sudo`/`rtk`
wrappers. Agents routinely batch a read behind other work, so every
chained read fell through to the compression path despite
`HEADROOM_PROTECT_READS=1`:

```
wc -l a.py b.py && sed -n '1,60p' c.py     # parsed as `wc`   -> not a read
grep -n X f.py | head -40; head -80 g.py   # parsed as `grep` -> not a read
ls -la || cat f.py                         # parsed as `ls`   -> not a read
```

Kompress operates on words and rejoins the survivors, so file content
arrives with function words missing. That is precisely the failure read
protection exists to prevent: the agent either re-reads to recover
detail (turn inflation) or builds an exact-match `old_string` from text
that no longer matches the file on disk.

Reported by a Headroom desktop user who saw source content garbled while
the `Read` tool path stayed byte-exact — which is what pointed at the
shell-command classifier rather than the tool exclusion list.

No linked issue; reported directly.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_is_read_command` now scans every `;` / `&&` / `||` segment and
protects if **any** segment is a read, instead of matching only the
first program.
- Extracted the existing single-command logic into `_segment_is_read`,
unchanged in behavior for a non-chained command.
- Added `_command_segments`, which splits on `;` / `&&` / `||` and
reduces each segment to its first pipeline stage.
- Pipelines are deliberately **not** split: a downstream stage consumes
the previous stage's output rather than a file, so `grep … | head -40`
stays compressible while `cat f.py | head -40` is still caught via its
first stage.
- The whole-string write/heredoc guard is unchanged, so a heredoc body
containing `;` cannot be split into a bogus read segment.
- Two latent false positives close as a consequence of per-segment
evaluation: a `-n` flag in a sibling segment no longer qualifies a bare
`sed`, and the lockfile carve-out now applies to the segment naming the
lockfile rather than the whole command.
- New test file `tests/test_transforms/test_read_command_detection.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — see Additional Notes
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_transforms/ \
    tests/test_observed_wire_shapes.py tests/test_content_router_exclude_tools.py -q
======================= 408 passed, 62 skipped in 13.46s =======================

$ uv run --frozen --extra dev pytest tests/test_transforms/test_read_command_detection.py -q \
    --cov=headroom.transforms.content_router --cov-report=term-missing --cov-branch
30 passed in 0.45s
# no uncovered lines or partial branches remain in the changed range (614-654)

$ uvx ruff check headroom tests --output-format concise
All checks passed!

$ uvx ruff format --check headroom/transforms/content_router.py \
    tests/test_transforms/test_read_command_detection.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom
Success: no issues found in 512 source files
```

## Real Behavior Proof

- Environment: macOS (Darwin 24.6.0), Python 3.10.18, worktree branched
from `upstream/main` @ `28aa53dc`; plus a live Headroom desktop proxy
running `headroom-ai` 0.32.1 with Claude Code pointed at
`127.0.0.1:6767`.
- Exact command / steps: (1) pre-fix on the live proxy, ran `grep -n
"Read" transforms/compression_policy.py | head -40; echo "=== POLICY
HEAD ==="; head -80 transforms/compression_policy.py` through Claude
Code's Bash tool; (2) ran `ContentRouter().apply(...)` with
`HEADROOM_PROTECT_READS=1` over a `role: tool` bash observation produced
by `wc -l app/services/inventory.py && sed -n '1,200p'
app/services/inventory.py` (11,007 chars of plain Python source, aged
behind 20 follow-up turns so the positional "recent code" guard no
longer covers it), against both the pre-fix and post-fix
`content_router.py`.
- Observed result: step 1 returned the file's own docstring with
function words dropped and a retrieval marker appended; step 2 went from
no protection transform at all (pre-fix — the observation enters the
compression path) to `router:read_protected` (post-fix). Verbatim output
below.
- Not tested: byte-level Kompress corruption inside the repo venv (the
ML model is not downloaded there — `Kompress model not ready; requests
will not be compressed` — so step 2 shows the routing change and step 1
shows the corruption); `magika` unavailable in that venv (`no pip
onnxruntime native library was found`) so detection fell through to the
regex tier during step 2, which affects which compressor a released read
routes to but not whether a read is protected; Windows shell command
strings; fleet-scale traffic replay, so the savings impact of
newly-protected reads is unquantified.

### Proof detail

Step 1 — marker appended to the returned file content:

```
[478 items compressed to 305. Retrieve more: hash=5d959a193f8641579c741ad5]
```

On disk:

> Hand-mirrored port **of**
`headroom_core::compression_policy::CompressionPolicy` (Rust). **The**
Rust crate **is the** source **of** truth; **this** module exists **so
the** Python proxy's `TransformPipeline` (which still runs
`CacheAligner` **and other** detector-only transforms) **can** read
**the** same per-mode flags **the** Rust dispatcher does.

As received by the model:

> Hand-mirrored port
`headroom_core::compression_policy::CompressionPolicy` (Rust). Rust
crate source truth; module exists Python proxy's `TransformPipeline`
(which still runs `CacheAligner` detector-only transforms) read same
per-mode flags Rust dispatcher does.

Step 2 — routing branch taken by the tool observation:

```text
##### BEFORE (upstream/main content_router.py) #####
command:               wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars:           11007
file content verbatim: True
transforms:            ['router:protected:user_message', ... x10]      <- no read protection

##### AFTER (this PR) #####
command:               wc -l app/services/inventory.py && sed -n '1,200p' app/services/inventory.py
input chars:           11007
file content verbatim: True
transforms:            ['router:read_protected', 'router:protected:user_message', ... x10]
```

Before the fix the observation carries no protection transform at all
and enters the compression path. After the fix it is
`router:read_protected`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs describe this internal classifier; its docstrings are
updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — no UI surface.

## Additional Notes

- `ruff check .` reports 4 errors, all pre-existing on `upstream/main`
and all in `plugins/headroom-oauth2/` (`RUF022`, `BLE001` x2, `S110`).
Untouched by this PR, so the box is left unchecked rather than claimed.
`ruff check headroom tests` is clean.
- Trade-off: the write/heredoc guard stays whole-string rather than
per-segment. That keeps `cat a.py && echo x > b` unprotected, which is
the pre-existing behavior — chosen so a heredoc body containing `;` can
never be split into a segment that looks like a read. Narrowing it
further is a separate change.
- Follow-up for the reporter's second issue (not in scope here): passive
traffic learning has no off switch, and the desktop starts the proxy
with `--learn` unconditionally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 16:29:01 -05:00
Radhakrishnan Pachyappan fef99cc5b4 feat(compress): add per-message compression diagnostics (#3058)
## Description

Adds `CompressConfig.diagnostics` (bool, default `False`) and a
companion `HEADROOM_DIAGNOSTICS=1` environment variable that enable
per-message decision logging on every call to `compress()`.

When active, `CompressResult.diagnostics` is a list of `MessageDecision`
objects — one per input message — each describing:

| Field | Description |
|---|---|
| `message_index` | Zero-based position in the original messages list |
| `role` | `"user"` / `"assistant"` / `"tool"` / `"system"` / … |
| `tokens_before` | Token count before the ContentRouter acted on this
message |
| `tokens_after` | Token count after (equals `tokens_before` for
protected/passthrough) |
| `action` | What was done: `"compressed:kompress:0.72"`,
`"protected:user_message"`, `"passthrough:frozen"`,
`"passthrough:small"`, etc. |

Closes #2855.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/config.py`** — add `MessageDecision` dataclass; add
`message_decisions: list[MessageDecision]` field to `TransformResult`
- **`headroom/compress.py`** — add `CompressConfig.diagnostics`; add
`CompressResult.diagnostics`; derive `collect_diagnostics` from config +
env var; pass to pipeline; collect from `TransformResult`
- **`headroom/transforms/pipeline.py`** — accumulate `message_decisions`
from each transform result
- **`headroom/transforms/content_router.py`** — instrument every
decision branch in `apply()` with `_diag[i] = "action:..."` strings when
`collect_diagnostics=True`; build `MessageDecision` list after Pass 3
(zero overhead when flag is off)
- **`headroom/__init__.py`** — export `MessageDecision` in `__all__` and
lazy-import map
- **`tests/test_compress_diagnostics.py`** — 14 tests covering the full
`compress() <-> pipeline` contract via a mock pipeline

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_compress_diagnostics.py -v
========================= test session starts ==========================
collected 14 items

tests/test_compress_diagnostics.py::test_message_decision_dataclass PASSED
tests/test_compress_diagnostics.py::test_message_decision_importable_from_headroom PASSED
tests/test_compress_diagnostics.py::test_transform_result_has_message_decisions_field PASSED
tests/test_compress_diagnostics.py::test_transform_result_message_decisions_populated PASSED
tests/test_compress_diagnostics.py::test_compress_config_diagnostics_defaults_false PASSED
tests/test_compress_diagnostics.py::test_compress_config_diagnostics_can_be_set PASSED
tests/test_compress_diagnostics.py::test_compress_result_diagnostics_none_by_default PASSED
tests/test_compress_diagnostics.py::test_compress_passes_collect_diagnostics_when_config_true PASSED
tests/test_compress_diagnostics.py::test_compress_result_diagnostics_list_when_config_true PASSED
tests/test_compress_diagnostics.py::test_compress_result_diagnostics_contains_decisions PASSED
tests/test_compress_diagnostics.py::test_env_var_enables_diagnostics PASSED
tests/test_compress_diagnostics.py::test_env_var_absent_keeps_diagnostics_none PASSED
tests/test_compress_diagnostics.py::test_env_var_overrides_config_false PASSED
tests/test_compress_diagnostics.py::test_no_diagnostics_when_optimize_false PASSED
========================== 14 passed in 0.42s ==========================

$ ruff check .
All checks passed!
```

## Real Behavior Proof

- Environment: Python 3.12, headroom dev install
- Exact command / steps: `python -m pytest
tests/test_compress_diagnostics.py -v`
- Observed result: 14 tests pass; diagnostics list is populated when
`CompressConfig(diagnostics=True)` or `HEADROOM_DIAGNOSTICS=1`
- Not tested: ContentRouter integration tests (require compiled
`headroom._core` extension — run in CI)

## Runtime Rollout Safety

- Rollout-managed feature(s): None — diagnostics are opt-in via
`CompressConfig.diagnostics=True` or `HEADROOM_DIAGNOSTICS=1`
- Minimum rollout channel: N/A (additive, opt-in only)
- Stable/default behavior changed: No — `CompressConfig.diagnostics`
defaults to `False`; `CompressResult.diagnostics` is `None` unless
explicitly requested
- Kill switch / disable path: Simply omit `diagnostics=True` / unset
`HEADROOM_DIAGNOSTICS`
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this PR; no schema or data migration involved

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — no UI changes.

## Additional Notes

The `collect_diagnostics` flag incurs zero overhead in the hot path when
not set: the `_diag` dict in `ContentRouter.apply()` is never allocated
and no `MessageDecision` objects are constructed.

Signed-off-by: Radha Krishnan P <gingeekrishna@gmail.com>

---------

Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech>
Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com>
2026-10-01 00:04:47 -05:00
8f3d677354 fix(kompress): degrade when a native dep is installed but unloadable (#3133)
## Description

`_is_pytorch_available()` and `_is_onnx_available()` catch only
`ImportError`. An
installed-but-unloadable native dependency raises `OSError` instead: on
Windows
without the MSVC redistributable, `import torch` fails with `WinError
126` loading
`c10.dll`.

So a probe whose contract is to return a bool raises instead, and
`is_kompress_available()` raises with it.
`content_router._get_kompress()` guards its
call with `except ImportError` too, so the `OSError` propagates past
both guards into
the request path rather than degrading to "kompress unavailable".

Because the compressor is only cached on success, this repeats on every
request, and
Python drops failed modules from `sys.modules`, so each one pays a full
`torch` import
that ends in the same error.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_is_pytorch_available()` / `_is_onnx_available()` now catch
`(ImportError, OSError)`, so an installed-but-unloadable native
dependency degrades to "unavailable" instead of raising out of a
bool-returning probe.
- Both probes are `lru_cache`d. The answer cannot change within a
process, and nothing else memoises a *failing* import, so without this
the full failing `torch` import is retried on every request.
- Added `test_unloadable_native_dep_degrades_instead_of_raising`, which
simulates the Windows condition (onnxruntime absent so the `or` cannot
short-circuit, torch raising `OSError`) and asserts both the graceful
`False` and the single import attempt.
- `test_module_import_without_torch` now calls `cache_clear()` around
its assertion, which memoising makes necessary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --extra dev pytest tests/test_transforms/ -q
======================= 384 passed, 67 skipped in 18.13s =======================

$ uvx ruff check .
All checks passed!

$ uv run --extra dev mypy headroom/transforms/kompress_compressor.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Reproduced from a Windows Server 2022 (10.0.20348) box
where all three of vcruntime140.dll, vcruntime140_1.dll and msvcp140.dll
are absent from System32, so the installed torch cannot load its DLLs.
The behaviour was then isolated on macOS 26.5 / CPython 3.10.18 by
simulating that condition, since the failure is an exception type and
not platform-specific logic.
- Exact command / steps: On the Windows box, the packaged proxy logged
`OSError: [WinError 126] ... Error loading "...\torch\lib\c10.dll"`
escaping `_is_pytorch_available` at kompress_compressor.py:489 via
`_load_kompress` at :1002. To isolate, patch `builtins.__import__` so
`onnxruntime`/`optimum` raise `ImportError` (absent) and `torch` raises
`OSError` (installed, unloadable), then call `is_kompress_available()`
directly and through the same `try/except ImportError` guard
`content_router._get_kompress()` uses, counting torch import attempts
across 3 requests.
- Observed result: Before the change, `is_kompress_available()` raised
`OSError` rather than returning a bool, the error escaped the caller's
`except ImportError` on 3/3 requests, and torch was imported 4 times.
After the change, it returns `False`, 0/3 requests raise, and torch is
imported once. This is covered by the new test.
- Not tested: The fixed build has not been run end-to-end on the Windows
box, because installing the MSVC redistributable there would remove the
very condition under test. Whether kompress then loads successfully on
Windows once the redistributable is present is out of scope for this PR,
which only makes its absence degrade correctly.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. This is an availability probe on an
existing optional dependency, not a gated feature.
- Minimum rollout channel: N/A
- Stable/default behavior changed: No behaviour change where the
dependencies import cleanly. The probe returns the same value; only the
installed-but-unloadable case changes, from raising to returning False.
- Kill switch / disable path: N/A. Existing kompress controls
(HEADROOM_KOMPRESS_ENDPOINT, the router's own gates) are untouched.
- Unsafe override required: No
- Qualification impact: None expected. The one existing test that
changed did so because memoising the probe makes an explicit
cache_clear() necessary; its assertion is unchanged.
- Rollback path: Revert the commit. There is no persisted state or
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines


🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-01 00:03:11 -05:00
lorenzozane c7ebaedd74 fix(proxy): preserve OpenCode Headroom retrieve results (#3782)
## Description

When OpenCode reports the retrieve tool as `headroom_headroom_retrieve`,
its result could miss the existing retrieve exclusion and be compressed
again. This adds that exact name to the shared alias mapping so the
existing ContentRouter pass-through guard can preserve the retrieved
content.

Fixes #3780

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Map `headroom_headroom_retrieve` to the existing `headroom_retrieve`
exclusion.
- Add alias coverage and a ContentRouter regression test for the
OpenCode name.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
python -m pytest -q tests/test_tool_name_aliases.py: 5 passed
Changed-file Ruff checks, format check, compileall, and git diff --check: passed
python -m pytest -q tests/test_transforms/test_content_router_ccr_retrieve_exemption.py: collection blocked by ModuleNotFoundError: headroom._core
Full pytest, repository-wide Ruff, mypy, and live OpenCode/proxy verification were not run.
```

## Real Behavior Proof

- Environment: local submitter checkout; the native `headroom._core`
extension is unavailable.
- Exact command / steps: `python -m pytest -q
tests/test_tool_name_aliases.py` after the alias change.
- Observed result: 5 tests passed, including the exact OpenCode tool
name and negative alias checks.
- Not tested: the ContentRouter regression could not be collected
locally; the issue-reported cache/token OpenCode runtime was not
independently reproduced after this change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: the matching OpenCode retrieve name
now follows the existing exclusion behavior.
- Kill switch / disable path: None.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The focused alias test passes, but the end-to-end ContentRouter test and
live runtime behavior remain unverified in this environment.
2026-09-24 22:26:30 -05:00
Tejas ChopraandClaude Opus 5 21d166b77a test: stop three sources of cross-test pollution on main (#3774)
## What

Three tests fail on `main` in a full-suite run and pass when their file
is run alone. Each is state one test leaves behind that a later test
reads. This PR isolates all three. **No production code changes.**

| Failing test | Left behind by | Fix |
|---|---|---|
| `/readyz` readiness tests reporting kompress `ready: true` |
`test_kompress_failsafe.py::test_canary_disabled_via_env` calling
`compressor.preload()`, which writes the model into the module-level
`_kompress_cache` | autouse fixture snapshots/restores `_kompress_cache`
|
| `test_runtime_log_refuses_a_symlinked_path` |
`tests/gateway/test_compress_turn_seam.py` leaking an
`_OwnerOnlyRotatingFileHandler` on the `headroom` logger | autouse
fixture detaches + closes any handler named `_PROXY_LOG_HANDLER_NAME` |
| 2 × `test_dense_line_elider.py` | the optional `html` extra
(`trafilatura`) being installed, so `HTMLExtractor` wins routing instead
of the elider | both tests pin `enable_html_extractor=False` |

The third is not pollution between tests but pollution from the
environment: CI does not install the `html` extra, so CI has never seen
these two fail. Anyone with `pip install headroom[html]` in their venv
does.

## How to verify

Before:

```
pytest tests/test_kompress_failsafe.py tests/test_proxy_health.py   # health tests fail
pytest tests/gateway/test_compress_turn_seam.py tests/test_install/test_runtime.py   # symlink test fails
pytest tests/test_transforms/test_dense_line_elider.py   # fails iff trafilatura installed
```

After: all pass, and each file still passes alone.

Slices run locally (venv `bin` on PATH):
- `-k "kompress or health or elider or breakpoint or transform"` → 1040
passed, 17 skipped
- `-k "install or savings or stats or memory or learn or router"` → 2960
passed, 59 skipped

## Behaviour change

None for users. For test authors: the kompress model cache and the proxy
log handler no longer survive a test, so a test that *relied* on a
previous test having preloaded the model would now have to preload it
itself. No test in the suite does.

## Two defects found while debugging this, deliberately not fixed here

1. **`HTMLExtractor` emits lossy output with no CCR retrieval marker.**
A 3232-char HTML tool result is reduced to `"hi there"` —
`<script>`/`<style>` stripped — and booked as 99.8% savings with no
marker, so the original cannot be retrieved. Present since 9263b420.
Filed separately; this is a data-loss bug, not a test bug.
2. **Tests write to the developer's real `~/.headroom/logs/`.** Fixture
(2) contains the blast radius but does not redirect the writes.

Keeping both out of this PR so it stays reviewable as a pure
test-isolation change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 16:13:24 -05:00
Raphael Malikian c1fc84ade6 fix(smart-crusher): disclose dropped scalar-array items with a retrievable ccr sentinel (#3692)
## Description

On default config, `SmartCrusher` silently truncated long
string/number/mixed arrays down to `max_items_after_crush` and emitted
no disclosure at all — valid JSON with no marker, no sentinel, and
untouched sibling count fields. This contradicts the repo's own "Safety
first: never drop user/assistant content" principle.

The dict-array path already disclosed drops via a `{"_ccr_dropped":
...}` sentinel; scalar arrays had no equivalent. This PR gives them one:
when the `StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, it appends `"… N more items <<ccr:HASH
N_items_offloaded>>"` and stores the full original array in the CCR
store (same SHA256-12 hashing the Python bridge and retrieval use), so
the marker round-trips through `headroom_retrieve`. No-op when nothing
is dropped or the marker gate is off.

Closes #3650

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

| File | What changed |
|---|---|
| `crates/headroom-core/src/transforms/smart_crusher/crusher.rs` | New
`append_scalar_drop_sentinel()`: when
`StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, append the sentinel and store the full
original array in the CCR store. No-op when nothing is dropped or the
marker gate is off. |
| `headroom/transforms/smart_crusher.py` | `is_ccr_sentinel()` now
recognizes the string sentinel shape (previously only the dict shape),
so downstream code can filter it like the dict sentinel; docstring
updates. |
| `tests/test_transforms/test_smart_crusher_issue_3650.py` | New
regression tests: sentinel is emitted with the right drop count, carries
a retrievable CCR marker whose hash resolves to the full original array,
and is suppressed when the marker gate is off. |
| `crates/.../smart_crusher/crusher.rs` (tests module) | Rust regression
tests mirroring the above (sentinel emission, marker retrieval of the
full 120-item array, gate-off suppression). |
| `tests/parity/fixtures/smart_crusher/*.json` (3 files) | Recorded
outputs updated — strategy strings now reflect the trailing sentinel. |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
cargo test --lib transforms::smart_crusher  → 337 passed, 0 failed
  (includes the new #3650 Rust regression tests)

pytest tests/test_transforms/test_smart_crusher_issue_3650.py \
       tests/test_transforms/test_smart_crusher_bugs.py \
       tests/test_transforms/test_smart_crusher_ccr_roundtrip.py \
       tests/test_transforms/test_smart_crusher_rust_parity.py \
       tests/test_transforms/test_content_router.py  → 106 passed
  (includes 18 Rust-parity fixture tests)

ruff check and ruff format --check on the touched Python files → clean
```

## Real Behavior Proof

- Environment: Linux container, Python 3.12, editable install from
`main` with the maturin-built native `_core` module (no ONNX model
involved — this path is pure Rust crusher).
- Exact command / steps: the repro snippet from #3650, verbatim
(`ContentRouter(ContentRouterConfig())`, 120 slugs + `"total": 120`).
- Observed result: **before** → 16 of 120 kept, `"&lt;&lt;ccr:" in out`
False; **after** → 16 kept + `'… 104 more items &lt;&lt;ccr:4f03caae30bb
104_items_offloaded&gt;&gt;'`, marker present, and the marker's hash
retrieves all 120 originals from the CCR store. The `total` sibling
field is left untouched — the truncation is now disclosed in-band next
to it rather than silently contradicting it.
- Not tested: the ONNX/Kompress model path from #3673 (a separate issue;
this fix is model-independent), Windows/macOS install flows, and the
`lossless_only=True` path (unchanged: nothing dropped there, so no
sentinel is emitted).

## Runtime Rollout Safety

- Rollout-managed feature(s): CCR marker gate (`enable_ccr_marker`)
- Minimum rollout channel: n/a (open-source library; applies to default
config where the marker gate is on)
- Stable/default behavior changed: Yes — truncated scalar arrays now
carry a trailing sentinel string instead of being silently shortened;
sibling count fields untouched
- Kill switch / disable path: set `enable_ccr_marker=False` — no
sentinel emitted, gate-off path unchanged
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — library behavior; observed output captured under Real Behavior
Proof.

## Additional Notes

Raphael Malikian — rtmalikian@gmail.com

If you'd like similar help with your own project or codebase, I'm open
to collaborate — feel free to reach out.

*This PR was prepared with AI assistance and reviewed by the author
before submission.*

---------

Signed-off-by: Raphael Malikian <rtmalikian@gmail.com>
2026-09-24 11:19:23 -05:00
6feb1fb22a fix(content_detector): detect CMTrace (SCCM/Intune) logs as BUILD_OUTPUT (#3712)
## Description

CMTrace-format logs — what SCCM, MDT and Intune Win32 app/script
deployments
write on Windows — are never detected as `BUILD_OUTPUT`. Two independent
causes,
and the second shadows a fix to the first, so both are fixed together.

Measured on a corpus of 253 real Intune/SCCM deployment logs,
`detect_content_type`
returned `BUILD_OUTPUT` for **zero** of them.

Closes #<!-- issue number if you filed one; otherwise delete this line
-->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- **Both implementations** — added `^<!\[LOG\[` to `_LOG_PATTERNS` /
  `LOG_PATTERNS`. CMTrace puts its timestamp in an attribute *after* the
  message, so the anchored `^\s*\d{4}-\d{2}-\d{2}`,
`^\s*\[\d{2}:\d{2}:\d{2}\]` and `^={3,}` patterns cannot fire, and a
record
need not contain ERROR/WARN/INFO. `_try_detect_log` returned `None` at
its
  `pattern_matches == 0` guard.
- **Rust only** — ported Python's `_prefix_looks_like_path` as
`prefix_looks_like_path` and applied it in `try_detect_search`. A
CMTrace
  divider contains no whitespace, so `SEARCH_RESULT_PATTERN` runs from
`<![LOG[` through `]LOG]!><time="12`, then `:`, then `00`, then `:` — a
clean
  `file:line:` match. Python rejects it via `_is_search_result_line`'s
path-shape guard (no `<`, `>`, `=`); Rust applied the regex bare. Since
search is dispatched before logs, Rust would otherwise shadow the fix
above.
  Porting the guard, rather than tightening the Rust regex, keeps
`SEARCH_RESULT_PATTERN` byte-identical across the two and converges the
  logic on Python's.
- Tests on both sides: CMTrace multi-line → `BUILD_OUTPUT`, CMTrace
single-line
→ `BUILD_OUTPUT`, the divider is rejected as a search line, plus
regression
  guards that ordinary and extensionless grep output still detect as
`SEARCH_RESULTS`. The Rust suite additionally pins
`prefix_looks_like_path`
  against Python's rules directly.

### Note on the parity harness

The Rust/Python divergence above was **not** caught by
`parity-run --only content_detector`, which reports `matched=21
diffed=0` both
before and after this change. None of the 21 recorded fixtures exercise
a
`<`-prefixed `file:line:` shape, so the drift was invisible to it. Worth
knowing
if the harness is being relied on as the parity guarantee.

### Two further Rust divergences, reported not fixed

Found while porting the guard; left alone to keep this PR to one logical
change.
Happy to open a separate issue or PR:

- Rust has no equivalent of `_GREP_COLON_DASH_PATTERN` or
`_is_grep_context_line` — the `grep -A/-B/-C` context-line handling
added in
  #3580.
- Rust returns on `matching_lines == 0`; Python uses an absolute floor
of
`matching_lines < 2`, added deliberately because the search compressor
drops
  non-matching lines, so a single coincidental match is data loss.

### Why a second, single-line test

Real Intune logs are emitted as one continuous stream with **no line
separators** — a 133 KB production sample held 571 records and 2 line
feeds.
That form never reaches `_try_detect_search` (it exits at
`matching_lines < 2`),
so it fails on Bug 1 alone and lands in `PLAIN_TEXT`. The multi-line
form is
what exercises Bug 2. Both shapes occur, so both are tested.

## Testing

- [x] Unit tests pass (`pytest`) — 12,680 passed; see note on
pre-existing Windows failures below
- [x] Linting passes (`ruff check .`) — `All checks passed!`
- [x] Type checking passes (`mypy headroom`) — 1 pre-existing error in
an unrelated file, none in the files this PR touches
- [x] New tests added for new functionality — 5 Python, 5 Rust
- [ ] Manual testing performed

### Test Output

Python, on 3.13.15 / pytest 9.0.3 — **5 passed:**

```text
tests/test_transforms/test_content_detector_cmtrace.py::test_multiline_cmtrace_detected_as_build_output PASSED   [ 20%]
tests/test_transforms/test_content_detector_cmtrace.py::test_single_line_cmtrace_detected_as_build_output PASSED [ 40%]
tests/test_transforms/test_content_detector_cmtrace.py::test_cmtrace_divider_is_not_treated_as_a_search_result_line PASSED [ 60%]
tests/test_transforms/test_content_detector_cmtrace.py::test_grep_output_still_detected_as_search_results PASSED [ 80%]
tests/test_transforms/test_content_detector_cmtrace.py::test_extensionless_grep_paths_still_detected PASSED      [100%]

================================================== 5 passed in 0.71s ==================================================
```

Rust, `cargo test -p headroom-core content_detector` — **27 passed** (21
pre-existing + 6 new). The 21 pre-existing `content_detector` tests all
still
pass, including `search_results_detected` and `build_output_detected`,
which is
the direct evidence that adding the path-shape guard did not disturb
existing
detection:

```text
running 27 tests
test transforms::content_detector::tests::cmtrace_divider_is_rejected_by_the_path_shape_guard ... ok
test transforms::content_detector::tests::path_shape_guard_matches_python_rules ... ok
test transforms::content_detector::tests::multiline_cmtrace_detects_as_build_output ... ok
test transforms::content_detector::tests::single_line_cmtrace_detects_as_build_output ... ok
test transforms::content_detector::tests::grep_output_still_detects_as_search_results ... ok
test transforms::content_detector::tests::extensionless_grep_paths_still_detected ... ok
test transforms::content_detector::tests::search_results_detected ... ok
test transforms::content_detector::tests::build_output_detected ... ok
[...19 further pre-existing content_detector tests, all ok...]

test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 915 filtered out; finished in 0.07s
```

Rust↔Python parity harness over all 21 recorded `content_detector`
fixtures —
**already run, zero divergence**:

```text
$ cargo run -p headroom-parity --bin parity-run -- run --only content_detector
[content_detector] total=21 matched=21 skipped=0 diffed=0
```

Full Python suite — **already run**:

```text
45 failed, 12680 passed, 603 skipped, 6141 warnings, 81 errors in 2743.50s (0:45:43)
```

**The 126 failures/errors are pre-existing and platform-specific, none
related
to this change.** Evidence:

- They span 26 distinct test files. **Not one of them references
`content_detector` or `detect_content_type`** (checked by grep across
every
  failing file).
- 95 of 126 are `PermissionError: [WinError 32] The process cannot
access the
file because it is being used by another process` — Windows SQLite file
  locking, concentrated in `test_sqlite_graph_store.py` (37),
  `test_sqlite_vector_index.py` (18) and `test_memory/` (36).
- The remainder are Windows-platform assumptions: `module 'signal' has
no
attribute 'SIGHUP'`, `WindowsPath` vs POSIX path comparisons, CRLF-vs-LF
in
  `test_owned_asset_encoding.py`, and two missing external binaries
  (`ast-grep`, `difftastic`).

Per CONTRIBUTING ("Test/CI-only PR chasing a known `main` failure —
Don't.
We're tracking it"), these are left alone. Happy to confirm against a
clean
`main` run on request.

Lint — **already run, clean**:

```text
$ py -m uv run ruff check .
All checks passed!
```

Type check — **already run**:

```text
$ py -m uv run mypy headroom
headroom\integrations\langchain\chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 538 source files)
```

**That error is pre-existing and unrelated.**
`headroom/integrations/langchain/chat_model.py` is not touched by this
PR and contains no reference to
`content_detector` or `detect_content_type`. The two files this PR
changes
produce no mypy errors.

## Real Behavior Proof

- Environment: Windows 11 Enterprise 10.0.26200; Python 3.13.15 with
dependencies resolved from `uv.lock` (`py -m uv sync --extra dev`, uv
0.12.17); cargo/rustc 1.95.0; branch `fix/cmtrace-content-detection`
based on `94206e26` (release 0.38.0). Full detail under *Environment*
below.
- Exact command / steps: `detect_content_type` was run over a 40-record
synthetic CMTrace log three times — with the patch present, with `git
checkout HEAD~1 -- headroom/transforms/content_detector.py` reverting
only the patched file, and with it restored — each run printing whether
the fix was present in the module it actually imported. Full command
list under *Exact command / steps* below.
- Observed result: on identical input, `ContentType.BUILD_OUTPUT` with
the fix present and `ContentType.PLAIN_TEXT` with it reverted, toggled
in both directions and back; classification tracks the file state
exactly. Transcript under *Observed result* below.
- Not tested: end-to-end through the CLI or proxy (detection was
exercised directly, `Manual testing performed` is unchecked above); the
Rust half has unit and parity coverage but no before/after toggle of a
built binary; UTF-16 or BOM-prefixed CMTrace input; non-Windows CMTrace
producers; the 253-log corpus was measured before the patch and not
re-measured after. Detail under *Not tested* below.

### Environment

- Windows 11 Enterprise 10.0.26200
- Python 3.13.15, from `C:\Program Files\Python313`
- uv 0.12.17 (x86_64-pc-windows-msvc), installed via `py -m pip install
uv`
- Dependencies from `uv.lock` via `py -m uv sync --extra dev`
- cargo 1.95.0 / rustc 1.95.0
- Branch `fix/cmtrace-content-detection`, based on `94206e26` (release
0.38.0)

### Exact command / steps

```bash
# environment
py -m pip install uv
py -m uv sync --extra dev

# confirm the suite imports the working tree, not a built copy
py -m uv run python -c "import headroom.transforms.content_detector as m; print(m.__file__)"
#   -> <repo>\headroom\transforms\content_detector.py

# tests
py -m uv run pytest tests/test_transforms/test_content_detector_cmtrace.py -v
py -m uv run pytest
py -m uv run ruff check .
py -m uv run mypy headroom
cargo test -p headroom-core content_detector
cargo run -p headroom-parity --bin parity-run -- run --only content_detector

# before/after on the same input, reverting only the patched file
py -m uv run python -c "<detect command, below>"
git checkout HEAD~1 -- headroom/transforms/content_detector.py
py -m uv run python -c "<same command>"
git checkout HEAD -- headroom/transforms/content_detector.py
py -m uv run python -c "<same command>"
```

The `git checkout` toggle only proves anything if the import resolves to
the
working tree rather than an installed copy, which is why the detect
command
below prints the provenance of the module it imported alongside the
result.

### Observed result

Same input, same command, toggling only the patched file, on a synthetic
40-record CMTrace log containing no real data:

```text
$ py -m uv run python -c "import headroom.transforms.content_detector as m; src=open(m.__file__, encoding='utf-8').read(); print('fix_present =', 'CMTrace record opener' in src); print('result      =', m.detect_content_type(open(r'...\cmtrace-sample.log').read()).content_type)"
fix_present = True
result      = ContentType.BUILD_OUTPUT

$ git checkout HEAD~1 -- headroom/transforms/content_detector.py
$ py -m uv run python -c "...same command..."
fix_present = False
result      = ContentType.PLAIN_TEXT

$ git checkout HEAD -- headroom/transforms/content_detector.py
$ py -m uv run python -c "...same command..."
fix_present = True
result      = ContentType.BUILD_OUTPUT
```

Toggled in both directions and back, and the classification tracks the
file
state exactly.

Note the "before" value is `PLAIN_TEXT`, not `SEARCH_RESULTS`: Python's
existing
path-shape guard already keeps this content off the search path, so it
was
falling through to prose rather than being misrouted. That is the whole
of the
Python-side bug — the log detector simply never fired. The Rust port,
lacking
that guard, *would* have returned `SEARCH_RESULTS` here, which is why
this PR
ports it.

### Supporting analysis, gathered before the patch

Impact analysis against all 21 recorded parity fixtures in
`tests/parity/fixtures/content_detector/`. Each fixture input was
extracted and
run against the search pattern and against the new CMTrace pattern. **No
fixture
changes classification** — the only `search` fixture matches 4 of 4
lines either
way:

```
$ printf '%b\n' "$(...79e6b68a05a1c639.json input...)"
src/main.py:42:def process():
src/util.py:13:    return None
lib/x.py:7:class X:
tests/test_a.py:3:    assert True
matches: 4 of 4
```

Why the bare `file:line:` shape fires on this format at all — on the
same
synthetic log (40 records, 20 dividers):

```
$ grep -cP '^[^\s:]+:\d+:' cmtrace-sample.log
20
$ grep -oP '^[^\s:]+:\d+:' cmtrace-sample.log | head -1
<![LOG[===========================================================================]LOG]!><time="10:00:
```

20 of 40 lines matching gives ratio 0.50 and confidence `0.4 + 0.50*0.6
= 0.70`,
over the 0.6 cutoff. Python discards those matches at the path-shape
guard, so
it lands in `PLAIN_TEXT`; Rust, which had no such guard, is where that
0.70
would have been returned as `SEARCH_RESULTS` — the shadowing this PR
removes.

None of `_LOG_PATTERNS`' existing signals match that same file, each
checked
individually:

```
^\s*\d{4}-\d{2}-\d{2}                     0 matches
^\s*\[\d{2}:\d{2}:\d{2}\]                 0 matches
^={3,}|^-{3,}                             0 matches
\b(ERROR|FAIL|FAILED|FATAL|CRITICAL)\b    0 matches
\b(WARN|WARNING)\b                        0 matches
\b(INFO|DEBUG|TRACE)\b                    0 matches
```

**Downstream real-world measurement:** a tool built on `headroom-core`
processed
253 real Intune/SCCM deployment logs. `LogCompressor` was dispatched **0
times**
across the whole corpus; every log went to the prose path.

### Not tested

- End-to-end through the CLI or proxy. Detection was exercised directly
via
`detect_content_type`; `Manual testing performed` is left unchecked
above.
- A before/after toggle of a built Rust binary. The Rust half is covered
by its
  6 new unit tests and by `parity-run --only content_detector`.
- UTF-16 or BOM-prefixed CMTrace input — out of scope here, detection
operates
  on already-decoded text.
- Non-Windows CMTrace producers.
- Re-measurement of the 253-log corpus after the patch; that figure is a
  pre-patch measurement.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — pure detection logic in
`content_detector`, behind no flag, channel or rollout gate.
- Minimum rollout channel: N/A — not rollout-managed; it ships with the
ordinary release.
- Stable/default behavior changed: yes, narrowly — content beginning
`<![LOG[` now classifies as `BUILD_OUTPUT` instead of `PLAIN_TEXT`
(Python) or `SEARCH_RESULTS` (Rust), and in Rust a `file:line:` prefix
containing `<`, `>` or `=` no longer counts as a search line, which is
what Python has always done. All 21 recorded parity fixtures are
unchanged: `matched=21 diffed=0`.
- Kill switch / disable path: none — there is no runtime toggle;
disabling means reverting the commit.
- Unsafe override required: no.
- Qualification impact: `BUILD_OUTPUT` routes to `LogCompressor` rather
than the prose path, so measured compression ratio on CMTrace input
changes — see Additional Notes, which quantifies it and explains why it
is still the correct classification.
- Rollback path: single-commit revert; no migration, persisted state or
cache format is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A, no
user-facing docs cover detector internals
- [x] My changes generate no new warnings — `ruff check .` clean; `cargo
test` compiled without warnings on the touched crate
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally — 5 Python + 27 Rust;
parity `matched=21 diffed=0`
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

**One thing worth flagging to maintainers, because it changed our own
conclusion.** Correcting detection is necessary but, for this format,
not
sufficient.

We first patched this downstream by routing CMTrace to `BUILD_OUTPUT`.
On
measuring it, we reverted that change: `LogCompressor` returns this
content
**unchanged**. Real CMTrace logs have no line separators, so a
line-oriented
compressor sees one enormous line and has nothing to deduplicate.

| Route | Time | 266 KB CMTrace sample |
|---|---|---|
| `BUILD_OUTPUT` → `LogCompressor` | 0.10 s | 133 KB → 133 KB (**0%**) |
| prose/embedding path | 17.9 s | 266 KB → 9.5 KB (**96%**) |

4 of 5 sampled files saved exactly zero bytes on the log path; across
152 real
CMTrace files the prose path averages 71%.

This PR fixes detection only, because that is the actual bug and it is
correct
on its own terms — CMTrace *is* build output. But `LogCompressor` would
need
record-boundary awareness (splitting on `]LOG]!>` rather than `\n`)
before these
files genuinely benefit. Happy to open a separate issue or PR for that
if
maintainers want it; flagging it here so this change doesn't land and
look like
a regression on compression ratio.

**Not a performance report.** On our corpus these files average 2.7 s;
the slow
files are simply the large ones, regardless of format.

Co-authored-by: PRokicki <prokicki@hemmersbach.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-24 11:12:26 -05:00
Xmon DaiandJD Davis d971f7c398 fix(transforms): stop routing timestamped log rows as grep matches (#3748)
## Description

`_is_search_result_line` now rejects any line that `_TIMESTAMP_ROW_RE`
(from `lossless_compaction.py`, #3419) recognizes as a timestamped log
row, before the grep-shape patterns are tried.

The colon branch accepts `^[^\s:]+:\d+:` whenever the prefix passes
`_prefix_looks_like_path`, and that guard only rules out `<`, `>` and
`=`. A bare ISO date-hour prefix (`2026-09-23T10`) passes, so every row
of a timestamped log counts as a grep match at ratio 1.0 and the payload
routes to `SEARCH_RESULTS`.

Since #3419 the lossless fold skips timestamp rows, so the payload falls
through to the lossy `SearchCompressor`: 2,000 rows collapse to 5, the
note misreads them as "matches in `2026-09-23T10`", and the minute is
printed back as an integer (`10:00:00Z` becomes `10:0:00Z`, a time that
does not exist in the input).

Reusing the lossless module's regex keeps the two guards in lockstep: a
row the fold refuses to treat as grep output can no longer be classified
as grep output here either. The context-line branch
(`_is_grep_context_line`) already kept dates out via its `/`-or-`.`
requirement; this closes the same hole in the colon branch.

Closes #3736

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/content_detection.py`: `_is_search_result_line`
checks `_TIMESTAMP_ROW_RE.match(line)` (imported from
`lossless_compaction`) before the colon and context branches, so
timestamped log rows are never counted as grep matches.
- `tests/test_transforms/test_content_detector_timestamp_search.py`: 6
new regression tests (2 pinning the bug, 4 controls for real grep
output, date-named files like `logs/2026-09-23.log:42:`, and
extensionless `Makefile:12:`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Red-green on base 26a2c493 (clean worktree): the two bug-pinning tests fail
(_is_search_result_line returns True; _try_detect_search returns
SEARCH_RESULTS at confidence 1.0); after the fix all 6 pass.

Adjacent suites on this branch:
  tests/test_transforms/                     427 passed, 2 failed
    (test_dense_line_elider, test_smart_crusher_bugs reproduce identically
     on a clean base worktree: pre-existing)
  test_transforms_content_detection.py +
  test_transforms_content_router.py           95 passed
  test_lossless_timestamp_guard.py,
  test_search_compressor*.py,
  test_bash_search_lossless_fold.py           93 passed

ruff check / ruff format --check / mypy: clean on touched files.
```

## Real Behavior Proof

- Environment: WSL2, Python 3.13.15, editable install of this branch
(pure-Python detector; native ONNX path unavailable in this env, same
fallback the 0.38.0 reporter's router label shows).
- Exact command / steps: the 2,000-row ISO-8601 log from the issue,
driven through `ContentRouter.compress` (the same entry
`headroom_compress` uses).
- Observed result:

Base `26a2c493`, before the fix:

```
detect: ContentType.SEARCH_RESULTS 1.0
strategy: CompressionStrategy.SEARCH
2026-09-23T10:0:00Z INFO worker[0] processed batch 0 in 0ms
2026-09-23T10:0:01Z INFO worker[1] processed batch 1 in 13ms
2026-09-23T10:0:02Z INFO worker[2] processed batch 2 in 26ms
2026-09-23T10:0:03Z INFO worker[3] processed batch 3 in 39ms
2026-09-23T10:33:19Z INFO worker[7] processed batch 1999
```

That reproduces both reported defects: 5 of 2,000 rows survive, and the
minutes are mangled to integers (`10:0:00Z`).

This branch, after the fix:

```
detect: ContentType.BUILD_OUTPUT 0.8
strategy: CompressionStrategy.LOG
[2000 lines omitted: 2000 INFO]
[2000 lines compressed to 0. Retrieve more: hash=4803fbc9753df57f2e208836]
```

The payload now routes as a log, as the issue suggested. No fabricated
timestamps appear, and the original stays retrievable through the hash,
same as the lossless path.

- Not tested: the Rust detector in `crates/headroom-core` still lacks
the prefix guard (its `try_detect_search` runs `SEARCH_RESULT_PATTERN`
alone). I left it untouched because open PR #3712 is already porting
`_prefix_looks_like_path` into that file, and adding a parallel change
there would conflict. The Python guard runs on every native fallback
path, and `_detect_content` consults `_try_detect_search` for the
HTML-misroute override as well. The reporter's other open question
(whether `_core` applies the same rule) is answered: it does not, until
#3712-style parity work lands.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this is a pure-Python
content-detection guard.
- Minimum rollout channel: N/A (no feature flag involved).
- Stable/default behavior changed: only for payloads whose lines match
`_TIMESTAMP_ROW_RE`; those stop being classified as `SEARCH_RESULTS` and
route through the log/build strategies instead. Real grep output is
unaffected (controls above).
- Kill switch / disable path: revert the single guard call in
`_is_search_result_line`.
- Unsafe override required: no.
- Qualification impact: none; no qualification suites touched.
- Rollback path: single-commit revert, no state or schema changes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The Rust-side parity gap is intentionally out of scope to avoid
conflicting with #3712.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-24 10:48:05 -05:00
gglucass 7f2766ca4b fix(kompress): bound the inference deadline to the request, not each block (#3693)
## Description

`HEADROOM_COMPRESSION_DEADLINE_MS` (default 20s) is a budget on kompress
inference, checked at chunk boundaries against a clock that `compress()`
starts itself when the caller passes no `_deadline_started_at`. Nothing
outside `kompress_compressor.py` ever passes one, so the budget bounds a
**block**, not a request: a request with a dozen compressible blocks
gets a dozen full deadlines.

That is how one request runs past the pipeline's own
`COMPRESSION_TIMEOUT_SECONDS` (default 30s). The overrunning worker
cannot be preempted, so it becomes timeout debt and opens the
quarantine, and every request behind it forwards with NO compression
until that worker exits or the 60s cap lapses. Same cascade as #2360,
reached on the Anthropic path by ordinary large agentic requests rather
than by a Codex WS 5s deadline.

`apply()` now stamps one `time.perf_counter()` origin into the existing
per-request `_PerRequestRuntimeState` (#3486) and `_try_ml_compressor`
hands it to every kompress call for that request, so a big request
degrades to partial compression inside its budget instead of overrunning
and starving the requests queued behind it.

Related: #3653 (the thread/fd leak each quarantine event also causes)
and #2451 (the concurrency-1 execution slot that makes blocks
serialize).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `_PerRequestRuntimeState` carries `kompress_deadline_started_at`,
stamped once per `apply()` alongside the other per-request fields.
- `_try_ml_compressor` forwards that origin as `_deadline_started_at` on
every kompress call.
- `KompressCompressor.shares_request_deadline = True` gates the kwarg;
`RemoteKompressCompressor.compress()` has no such parameter, so it is
not handed one.
- Three tests in `tests/test_transforms/test_kompress_deadline.py`.

## Testing

- [x] Unit tests pass (`pytest`)

### Test Output


```text
$ uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_router_registry_smartcrusher.py -q
404 passed, 67 skipped in 32.37s

$ uv run --frozen --extra dev pytest tests/test_transforms/test_kompress_deadline.py -q
6 passed in 2.00s

# Against the unfixed source, the two behavioural tests fail:
FAILED test_every_block_of_one_request_draws_down_the_same_deadline
FAILED test_a_second_request_gets_a_fresh_deadline
2 failed, 4 passed
```


## Real Behavior Proof

- Environment: macOS 24.6 arm64, Python 3.12, headroom-ai 0.37.0
installed wheel, ONNX kompress (`chopratejas/kompress-v2-base`), backend
anthropic, one heavy Claude Code user over 2026-09-17..21.
- Exact command / steps: Measured the installed runtime directly. Timed
`KompressCompressor.compress()` on a real 20 KB block pulled from
`/transformations/feed` (2622 words, 8 chunks), four calls. Parsed 4100
`PERF` lines and every `CompressionQuarantinedError` request id out of
`~/.headroom/logs/proxy.log*`, bucketed by `tok_before`, and priced the
quarantined requests at the savings rate achieved by non-quarantined
requests in the same size bucket.
- Observed result: Inference costs ~1.6 ms/word: the 20 KB block takes
4.2s per call, reproducible (4548/4162/4392/4237 ms). `opt_ms` by size:
0-50k tokens n=376 avg 113 ms; 200-250k tokens n=101 avg 5192 ms, max
67247 ms. One day carried 266 "Kompress slow compress" markers totalling
682s of inference, max single inference 21.9s. 250 requests were refused
compression by the quarantine, costing ~3.56M tokens, 8.7% of that day's
achievable savings (per-day: 2.2%, 19.4%, 3.7%, 8.7%). With the deadline
shared, the same request stops kompressing at the budget and returns
instead of overrunning the 30s pipeline timeout.
- Not tested: The fleet-wide prevalence of the quarantine cascade, which
no counter reports today; and the CUDA/MPS batched path, which shares
one deadline inside `compress_batch` already and was not re-measured on
this change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A request whose kompress
inference exceeds `HEADROOM_COMPRESSION_DEADLINE_MS` in total now stops
compressing further blocks, where before each block restarted the
budget. That is a reduction in per-request compression for requests that
were previously overrunning the pipeline timeout, and an increase for
every request that was being quarantined behind them.
- Kill switch / disable path: Set `HEADROOM_COMPRESSION_DEADLINE_MS`
high (for example `600000`) to restore effectively unbounded per-request
inference.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit; the deadline returns to being
per-block.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-09-23 23:43:08 -05:00
gglucassandClaude Fable 5.1 9263b42099 feat(router): elide dense machine-generated lines no compressor can shrink (#3685)
## Description

Tool outputs that dump a fetched web page, a bundled asset or an encoded
blob (minified JS/CSS, base64, RSC payloads) arrive as a few very long
lines with almost no whitespace. No structural compressor can do
anything with them: `code_aware` is disabled, `log`/`text` return them
unchanged, and the HTML extractor returns either the whole thing or `""`
(which the unit layer then rejects as an empty block and keeps all of it
verbatim). The block rides through the router at ratio 1.0. On a Codex
web-research session these were the four largest tool outputs (75k
tokens) and the whole session compressed at 1.3% of new input.

This adds a last-resort pass in `_apply_strategy_to_content`: after the
strategy chain (and on the passthrough path), every line that is >= 300
chars with < 6% spaces keeps its head and tail and the middle becomes a
marker. Prose, code, logs, indented JSON, CSV and markdown all have far
more spaces and are never touched; JSON-shaped lines are left to
SmartCrusher. The pre-elision block goes into the CCR store and a
`Retrieve original: hash=` marker is appended, so the messages path's
marker-less lossy gate (#1307) keeps the result and the agent can
recover exact bytes. Runs only on strategies whose result is still plain
text (`html`, `log`, `text`, `code_aware`, `search`, `passthrough`),
never after SmartCrusher/tabular/config/diff/Kompress, and never in
lossless mode. New config flag `enable_dense_line_elision`, default on.

Also: an HTML extraction that comes back `""` on non-empty input is now
treated as "nothing extracted" (fall through) rather than "compressed to
zero tokens".

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Performance improvement

## Changes Made

- `headroom/transforms/dense_line_elider.py`: `is_dense_line` /
`elide_dense_lines`, byte-identical no-op when nothing is dense.
- `headroom/transforms/content_router.py`: `enable_dense_line_elision`
config, `_DENSE_ELIDE_AFTER` strategy set, `_elide_dense` (CCR store +
marker), hook after the strategy chain and on the passthrough path,
empty-HTML-extraction fallthrough.
- `tests/test_transforms/test_dense_line_elider.py`: 8 tests
(prose/code/JSON untouched, elision shape, router hook on/off, marker
retrievable from the CCR store, survives the messages-path lossy gate,
lossless mode never elides, search output keeps line structure).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --frozen --extra dev pytest tests/test_transforms/test_dense_line_elider.py -q -> 8 passed
uv run --frozen --extra dev pytest tests/test_transforms/ tests/test_no_ccr_lossy.py tests/test_transforms_content_router.py tests/test_openai_responses_compression_units.py tests/test_code_aware_regressions.py tests/test_router_registry_dispatch.py tests/test_router_external_dispatch.py tests/test_compression_observability.py tests/test_openai_responses_context_compaction.py tests/test_agent_savings.py tests/test_openai_responses_t3_replay_regression.py -q -> 588 passed, 72 skipped
uv run --frozen --extra dev pytest tests/ --ignore=tests/e2e -q -> 9 failed, 12619 passed, 597 skipped (see note below)
uvx ruff check headroom/transforms tests/test_transforms/test_dense_line_elider.py  -> All checks passed!
uv run --frozen --extra dev mypy headroom/transforms/content_router.py headroom/transforms/dense_line_elider.py -> Success: no issues found in 2 source files
```

The 9 full-suite failures are unrelated to this change:
`tests/test_provider_grok.py` (2) and `tests/test_cli/test_wrap_grok.py`
fail because this shell exports `GROK_CLI_CHAT_PROXY_BASE_URL` (all pass
with it unset); the other 7 (`test_proxy_health.py` readyz/kompress x4,
`test_read_maturation_handler_nobust.py` x2,
`test_providers/test_deepseek.py` x1) are order-dependent, pass when run
in isolation, and fail identically with `upstream/main`'s
`content_router.py` swapped in.

## Real Behavior Proof

- Environment: this branch, `HeadroomProxy(ProxyConfig(stateless=True,
mode="token"))`, `HEADROOM_DISABLE_KOMPRESS=1`, two real
`~/.codex/sessions/*.jsonl` rollouts replayed through
`_compress_openai_responses_payload` one request per tool-output item
(`input=items[:i+1]`), novel `tokens_saved` (this pass minus previous
pass) over tiktoken of the items added since the previous request. Same
script with `elide_dense_lines` monkeypatched to a no-op for the "off"
arm.
- Exact command / steps: `HEADROOM_ELIDE_TEST=off|on uv run --frozen
--extra dev python replay_onoff.py <rollout.jsonl> <n_outputs> <model>`
- Observed result: see the block below (web-research session 1.3% ->
15.3% of new input, coding session unchanged at 34.6%).

```text
2026-09-09 gpt-5.6-sol elision off: novel=4377  new_input=344023 rate=1.3%
2026-09-09 gpt-5.6-sol elision on:  novel=52737 new_input=344023 rate=15.3%
2026-09-07 gpt-6-astra elision off: novel=51520 new_input=148924 rate=34.6%
2026-09-07 gpt-6-astra elision on:  novel=51520 new_input=148924 rate=34.6%
```

The web-research session (curl'd pages, truncated minified bundles) goes
1.3% -> 15.3%; the coding session (code-mode JSON chunk outputs, no
dense lines) is byte-for-byte unchanged. Across 14 local Codex sessions,
dense lines are 13.5% of all tool-output characters (0% to 54% per
session), so the fleet effect will be uneven and largest for
research-heavy users. What remains un-elided on the research session is
inside `embedded_json` spans (SSE `data: {...}` lines) and is excluded
on purpose: eliding inside those would corrupt the JSON.

- Not tested: live proxy traffic with a real Codex or Claude Code client
(replay only); Kompress enabled alongside elision (replays ran with
`HEADROOM_DISABLE_KOMPRESS=1`, matching the desktop's production
config); a CCR `/v1/retrieve` round trip over HTTP (the
store-and-retrieve check is in-process in the tests).

Desktop-side: measured 2026-09-20 on the `user_daily_savings` rollup,
Codex-dominant users sit at a median 6.5% strict input rate (n=341
user-days) vs 25.7% for Claude Code (n=478) at identical cache coverage;
this is the largest single lever found for that gap.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes.
`ContentRouterConfig.enable_dense_line_elision` defaults to on, so
blocks with dense machine-generated lines that previously passed through
at ratio 1.0 are now elided with a CCR retrieval marker. Blocks with no
such lines are byte-identical to before.
- Kill switch / disable path:
`ContentRouterConfig(enable_dense_line_elision=False)`; `lossless=True`
also disables it.
- Unsafe override required: No.
- Qualification impact: None; the elision is a deterministic post-pass
and does not touch routing decisions, strategy selection or the
frozen-prefix logic.
- Rollback path: Revert this PR, or set
`enable_dense_line_elision=False`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation: no docs page lists `ContentRouterConfig` flags today, so
there is nothing to update; happy to add one if you want the flag
documented.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 09:38:51 -07:00
Suliman AbdulrazzaqandTejas Chopra 58c7f72106 fix(smart-crusher): keep non-dict arrays and object keys whole under lossless_only (#3628)
## Description

`lossless_only` is documented as producing marker-free, byte-recoverable
output where rows are never dropped. Only the dict-array row-drop path
(`crush_array_with_source`) checked it, though. The string, number and
mixed-array crushers and the object-key crusher kept sampling in strict
mode, and they drop items without emitting any CCR marker. So `{"slugs":
[53 strings]}` came back with 15 slugs whether `lossless_only` was on or
off, and nothing in the output showed that items were removed.

Each of those four crushers now returns its input unchanged under
`lossless_only`, as `crush_array_with_source` already does for dict
arrays. Behavior with `lossless_only` off is unchanged.

Closes #3625

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crushers.rs`: `crush_string_array`, `crush_number_array` and
`crush_object` return their input under `lossless_only`, with strategies
`string:lossless_only`, `number:lossless_only` and
`object:lossless_only`. Each check comes after the crusher's existing
passthrough checks.
- `crusher.rs`: `crush_mixed_array` returns its items under
`lossless_only` (`mixed:lossless_only`).
- `config.rs`: the `lossless_only` doc comment now names these crushers.
- Tests:
  - three unit tests in `crushers.rs`;
- two in `crusher.rs`, one of them end to end through `crush()` using
the #3625 payload;
  - one Python test through the public `SmartCrusher` wrapper.

Each Rust test first asserts that the default config really drops items
on the same input, so the strict assertion cannot pass vacuously.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Before the fix (origin/main 2c56a1b3 plus these tests only)
$ cargo test -p headroom-core --lib lossless_only
test result: FAILED. 3 passed; 5 failed; 0 ignored; 0 measured; 922 filtered out
$ pytest tests/test_transforms/test_smart_crusher_bugs.py -q
1 failed, 6 passed

# With the fix
$ cargo fmt --all -- --check                                  # exit 0
$ cargo clippy -p headroom-core --all-targets -- -D warnings   # exit 0
$ cargo test -p headroom-core --lib lossless_only
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 922 filtered out
$ cargo test -p headroom-core --lib smart_crusher
test result: ok. 334 passed; 0 failed; 0 ignored; 0 measured; 596 filtered out
$ cargo test -p headroom-core
test result: ok. 929 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out   (lib; every integration test binary also ok)
$ cargo run -p headroom-parity -- run --fixtures tests/parity/fixtures
[smart_crusher   ] total=17 matched=17 skipped=0 diffed=0
$ pytest (the file list from make ci-precheck-python) -q
176 passed, 4 skipped, 1 warning
$ pytest tests/test_lossless_mode.py tests/test_lossless_then_lossy.py tests/test_agent_savings.py tests/test_transforms_config_compressor.py -q
121 passed
$ ruff check .              # All checks passed!
$ ruff format --check .     # 1605 files already formatted
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.14.2, Rust 1.95.0, headroom-ai
0.37.0 editable install built with `uv sync --frozen --extra dev`, which
builds the Rust core via maturin.
- Exact command / steps: ran the #3625 reproduction script unchanged,
then the same payload through the `SmartCrusher` wrapper and through
`ContentRouter`, first on `origin/main` and then on this branch after
`uv sync --frozen --extra dev --reinstall-package headroom-ai`. In full:
  1. Ran the reproduction script from #3625 unchanged.
2. Sent the payload below through
`SmartCrusher(config=SmartCrusherConfig(lossless_only=True)).crush(json.dumps(doc))`.
3. Sent it through
`ContentRouter(ContentRouterConfig(smart_crusher_lossless_only=True)).compress(json.dumps(doc),
context="")`, which is the config `HEADROOM_LOSSLESS_ONLY=1` sets.

  ```python
  doc = {
      "slugs": [f"r{i}" for i in range(53)],
      "sizes": list(range(1, 41)),
      "mixed": [f"entry-{i}" if i % 2 == 0 else i for i in range(40)],
"meta": {f"k{i:02d}": f"long description for entry {i}, above the
small-value floor" for i in range(40)},
  }
  ```
- Observed result: on `origin/main`, strict mode returned 15 of 53 slugs
and no CCR marker; on this branch it returns all 53 and the output
decodes back to the input. Raw output:
  ```text
  # origin/main
{'lossless_only': True} -> slugs 15/53 ccr_marker=False
{'lossless_only': False} -> slugs 15/53 ccr_marker=False
{'lossless_only': True, 'enable_ccr_marker': True} -> slugs 15/53
ccr_marker=False
{'lossless_only': True, 'enable_ccr_marker': False} -> slugs 15/53
ccr_marker=False
SmartCrusher lossless_only=True: slugs 15/53, sizes 15/40, mixed 21/40,
meta keys 15/40 | equal to input: False | marker: False
ContentRouter(smart_crusher_lossless_only=True): slugs 15/53, sizes
15/40, mixed 21/40, meta keys 15/40 | equal to input: False | marker:
False

  # this branch
{'lossless_only': True} -> slugs 53/53 ccr_marker=False
{'lossless_only': False} -> slugs 15/53 ccr_marker=False
{'lossless_only': True, 'enable_ccr_marker': True} -> slugs 53/53
ccr_marker=False
{'lossless_only': True, 'enable_ccr_marker': False} -> slugs 53/53
ccr_marker=False
SmartCrusher lossless_only=True: slugs 53/53, sizes 40/40, mixed 40/40,
meta keys 40/40 | equal to input: True | marker: False
ContentRouter(smart_crusher_lossless_only=True): slugs 53/53, sizes
40/40, mixed 40/40, meta keys 40/40 | equal to input: True | marker:
False
  ```
`ContentRouter(ContentRouterConfig(lossless=True))`, which is what
`--lossless` builds, returned the input intact both before and after,
because that mode stops at the byte-exact fold.
- Not tested: a running `headroom proxy` process (the router ran
in-process with the same config), macOS and Windows, the full `pytest`
suite, and `cargo test` / `cargo clippy` for the whole workspace. Only
`headroom-core` changed, and the parity harness was run.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: no. Only `lossless_only=True` (for
example `HEADROOM_LOSSLESS_ONLY=1`) changes: string, number and mixed
arrays and wide objects are no longer truncated.
- Kill switch / disable path: leave `lossless_only` off, or unset
`HEADROOM_LOSSLESS_ONLY`.
- Unsafe override required: no.
- Qualification impact: strict lossless mode compresses these JSON
shapes less, because they now pass through intact.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- The unit-test and "existing tests" boxes are unchecked because only
the file sets above ran, not the full suite. mypy was not run because no
module under `headroom/` changed. No user docs mention `lossless_only`,
so only the Rust doc comment was updated.
- With `lossless_only` off, behavior is unchanged: these crushers still
drop items without a CCR marker. Emitting a marker there is the other
option #3625 raises, and it needs a separate decision.
- Under `lossless_only`, dict arrays can still be rendered as the
lossless CSV-plus-schema table string when compaction wins. That is the
existing documented compaction, not a drop.
- This doesn't overlap with #3401. The two PRs edit the shared test file
far apart, and the new test does not use `max_items_after_crush`.

Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-09-17 16:42:37 -07:00
7bd4dbaf8f fix(kompress): append the CCR marker whenever the saving pays for it (#3484)
## Description

Kompress previously added a CCR retrieval marker only below a fixed
compression ratio. Smaller reductions could therefore produce lossy
output without a retrieval marker, which the router discarded.

The single, batch, and remote paths now compare the complete original
payload with the complete candidate plus retrieval marker. A marked
candidate ships only when it is smaller under the same `cl100k_base`
estimate, and its result accounting uses those measured counts.
Otherwise the original passes through. CCR-disabled behavior is
unchanged.

This is a fixed tokenizer estimate, not an exact billing count for every
OpenAI model or other provider. If the encoder is unavailable, the
fallback compares character counts rather than claiming a provider-token
bound.

## Type of Change

- [x] Bug fix

## Testing

Latest local validation: 81 tests passed across marker, remote,
deadline, compressor, size-gate, and CCR policy/resolution suites. Ruff
check and format check pass for the five changed Python files.

## Real Behavior Proof

- Environment: macOS, Python 3.13, deterministic compressor stubs with
real source-derived CCR hashes and `cl100k_base` encoding.
- Exact command / steps: Run the marker/remote/deadline suites and
related compressor/CCR suites. Re-run the single/batch boundary probe
using `" ".join(["alpha"] * 36 + ["bureaucratic"] + ["alpha"] * 63)`
with the supplied drop-first-36 model stub.
- Observed result: The formerly expanding candidate (100 original
tokens, 103 with marker) now passes through unchanged, reporting 100 to
100 with no retrieval key. Tests also verify that candidates with net
savings ship with marker-inclusive accounting, and partial deadline
completion preserves the unprocessed tail.
- Not tested: exact billing equivalence for other model tokenizers, live
model/provider traffic, or long-term retrieval-rate effects.
Current-head CI is pending.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: CCR-enabled Kompress uses
full-payload net savings rather than a fixed ratio threshold for marker
admission. Marker-inclusive successful result counts use the same
tokenizer estimate as the gate.
- Kill switch / disable path: `KompressConfig(enable_ccr=False)`
disables CCR markers as before.
- Unsafe override required: No.
- Qualification impact: Compression accounting includes the marker on
admitted CCR results; provider billing remains an estimate.
- Rollback path: Revert this PR.

## Review Readiness

- [x] Prior marker-accounting findings resolved.
- [x] Focused and related local checks pass.
- [ ] Current-head CI complete.
- Final human review is required.

## Checklist

- [x] Single, batch, remote, and deadline regressions covered.
- [x] No manual CHANGELOG edit.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
2026-09-09 11:31:10 -05:00
Tejas ChopraandClaude Fable 5 8884d87378 fix(transforms): stop compression garbling mixed subagent output (#3286)
## The report

A user's model called compressed subagent output "too garbled to use"
and burned CCR retrievals to reconstruct it — **not** because it needed
more context. One retrieval returned nothing but the Claude Code harness
sanitizer banner, reported as `original_item_count: 33,
compressed_item_count: 25`.

Root-cause chain (verified by reproduction): the harness prepends a
bracket-delimited banner (`[harness: ... you.]` — exactly 33
whitespace-delimited words) and neutralizes `<` → `<\`. Headroom's
mixed-content splitter typed the banner as JSON (bracket balance, no
validation) → SmartCrusher couldn't parse it → the fallback chain fed it
to lossy Kompress → Kompress word-dropped the banner 33→25 and stored it
behind a retrieval hash. Meanwhile tabular sections rendered as
quote-wrapped JSON-string blobs with `\n` as two-character escapes, and
`ensure_ascii=True` boundaries turned the output's unicode (`→ └ ✓`)
into `\uXXXX` soup. The model reasonably concluded the output was
garbled.

## Fixes

1. **`split_into_sections` validates JSON before typing a block
`JSON_ARRAY`** — same validation its own mixed-content gate
(`_has_valid_json_block_with_text`) has always used. Tag-protection
placeholders, which self-isolated only by accident of that bug
(`{{HEADROOM_TAG_N}}` bracket-balances), are now isolated explicitly via
a new `isolate=` parameter fed by the router; contiguous prose fragments
re-coalesce so the `\n\n` reassembly stops doubling newlines in
uncompressed prose.
2. **Kompress gets a real floor: `min_input_words = 64`**
(config-tunable, clamped at the historical 10), applied on the
in-process, batch, apply, and remote paths. Below it, lossy
word-dropping is a net loss — the retrieval marker alone is ~20 words —
and short blocks are disproportionately instruction-like.
3. **The mixed path unwraps SmartCrusher's whole-array CSV render** when
it comes back as a bare JSON string, splicing raw readable lines into
the text instead of a quoted escape blob.
4. **`ensure_ascii=False` at model-visible boundaries**: MCP
retrieve/stats responses and the audit-safe splice reserialization
(which now also matches serde_json's non-escaping behavior).
5. **Kompress honesty**: the marker says `N words compressed to M`
(shared `ccr_retrieval_marker` helper, unit-tested), and
`store_kompress_in_ccr` no longer writes word counts into the store's
*item count* fields — token counts already carry the size story.

The upstream trigger (the harness's `<` → `<\` neutralization corrupting
JSON semantics) is not Headroom's to fix, but with #1 and #2 the banner
now passes through byte-intact and nothing lossy touches it.

## Testing

- New `tests/test_garbled_compression_fixes.py` (12 tests) pins every
fix, including an end-to-end router pass over a reconstructed
harness-sanitized fixture asserting the banner survives byte-identical
and no `\uXXXX` appears.
- Existing small-fixture kompress/router tests updated to set
`min_input_words=10` explicitly (they test other mechanics; fixtures sit
under the new production floor by design).
- Affected sweep (`-k "compress or ccr or crusher or router or mixed or
kompress or hermes"`, ~2.9k tests): green apart from order-dependent
flakes that shift identity between runs (deepseek tokenizer `AutoConfig`
import, hermes/proxy-ccr) — each passes standalone and in direct
combination with the new tests; the full CI shards are the authoritative
check.
- ruff 0.16.3 `check` + `format --check` clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-27 13:22:26 +05:30
JD Davis a708c0571e fix(ci): prevent native detector from hanging test shards (#2996)
## Description

CI shard 4 was not merely slow: after thousands of fast tests it parked
indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The
router watchdogged only the first native call and then permanently
trusted direct calls via `_detect_native_verified`. Earlier suite
activity can change ORT/native state after that first success, making a
later call deadlock until GitHub cancels the job.

This keeps every native call bounded by the existing watchdog, activates
the process-wide pure-Python circuit breaker after a timeout, restores
the test-job ceiling to 30 minutes, and removes a separate wall-clock
scheduler assertion that generated false shard-1 failures despite the
structural regression guards passing.

No issue is auto-closed by this infrastructure repair.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed the unsafe process-lifetime `_detect_native_verified` fast
path.
- Kept every native detection call behind the existing bounded watchdog.
- Preserved the process-wide fallback circuit breaker so only the first
wedged call consumes the watchdog budget.
- Added a success-then-hang regression test.
- Isolated native circuit-breaker state in fallback exception tests.
- Restored the CI test timeout from the temporary 90-minute diagnostic
ceiling to 30 minutes.
- Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with
its meaningful absolute regression ceiling while retaining source-level
guards against the removed semaphore and nested executor.
- Corrected import order and formatting defects inherited from current
main so the synthetic merge commit passes repository-wide lint.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Exact local shard-4 command with coverage:
2723 passed, 172 skipped, 8661 deselected in 108.40s

Focused detector/router suite:
62 passed

Codex scheduler suite:
3 passed, 1 skipped

ruff check .
All checks passed!

ruff format --check .
1411 files already formatted

mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```

Exact-head GitHub CI on `28f284c7a156d5be2fac2d21ba904a19d3389e6d` is
entirely green. Test jobs 1–4, test-extras, test-agno, build, wheel,
lint, CodeQL, dependency audit, secret scan, smoke, governance, and
conflict checks all passed. Remaining skips are path-filtered jobs not
applicable to this diff.

## Real Behavior Proof

- Environment: macOS arm64/Python 3.13 locally; GitHub-hosted
Ubuntu/Python 3.12 using the production CI workflow and prebuilt wheel.
- Exact command / steps: reproduced `pytest tests scripts/tests --splits
4 --group 4 ...` hanging in native detection; sampled the parked
process; reran with `pytest-timeout` to locate `_rust_detect`; applied
the correction; reran the exact shard locally and all four CI shards
remotely.
- Observed result: local shard 4 completed in 1:48. GitHub shard 4's
pytest step completed in 5:45 and its full job in 8:06 under the
restored 30-minute ceiling. All four shards passed on the same head.
- Not tested: deliberately wedging a real production ORT runtime outside
the deterministic mocked regression; the watchdog behavior is covered
with a native-call fake that succeeds once and then never returns.

## Runtime Rollout Safety

- Rollout-managed feature(s): native content detection watchdog and
fallback only.
- Minimum rollout channel: normal patch release; no staged feature flag
required.
- Stable/default behavior changed: every native detection call remains
watchdog-bounded instead of only the first successful call.
- Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` bypasses
native detection; `HEADROOM_DETECT_TIMEOUT_SECS` controls the watchdog
budget.
- Unsafe override required: none.
- Qualification impact: full Python CI matrix must remain green; exact
shard-4 completion is the primary qualification evidence.
- Rollback path: human revert of this PR if bounded calls cause an
unexpected regression; setting the Python backend provides an immediate
operational fallback without code rollback.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — inline
lifecycle documentation and PR operational notes; no user-facing docs
change is needed
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; no UI change.

## Additional Notes

Human review only. No merge or auto-merge action has been configured.
The branch includes current main and preserves the MCP SDK compatibility
cap `mcp>=1.28.1,<2.0.0`.
2026-08-13 20:47:55 -05:00
JD Davis a3fe5cb65b fix(onnx): enforce Rust API-24 runtime compatibility (#2979)
## Description

Rust fastembed enables ORT C API 24, but the Python dependency allowed
ONNX Runtime 1.23.2. Entering ort's initializer with that library
deadlocks permanently instead of returning an error. Align dependency
resolution where compatible wheels exist and preflight native detection
where they do not.

Closes #2960

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Require ONNX Runtime 1.24+ for Python 3.11+ in the proxy and voice
extras.
- Keep the available pre-1.24 runtime on Python 3.10 for Python ONNX
consumers.
- Refuse to auto-pin an incompatible runtime into the Rust extension.
- Bypass native detection immediately when API 24 is unavailable,
preserving Python fallback without a five-second watchdog delay or stuck
native thread.
- Add dependency, pinning, override, and router regression coverage.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest -q tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py tests/test_onnx_runtime.py tests/test_transforms/test_content_router.py
88 passed in 9.31s

$ uv run ruff check headroom/_ort.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS arm64; Python 3.13.14 and uv-managed Python
3.10.20.
- Exact command / steps: run the issue's direct
`headroom._core.detect_content_type` call in a subprocess with a
12-second timeout on Python 3.13; run `_detect_content` on Python 3.10
after resolving the proxy extra.
- Observed result: Python 3.13 resolves ORT 1.26.0 and native detection
returns `json_array`; Python 3.10 resolves ORT 1.23.2, leaves
`ORT_DYLIB_PATH` unset, reports compatibility false, and immediately
returns the Python `json_array` fallback.
- Not tested: Linux-specific shared-object execution locally; CI's
existing Linux Rust job already preflights ORT 1.24+ and exercises
native tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): Native Rust content detection.
- Minimum rollout channel: Stable/default; this is a deadlock prevention
guard.
- Stable/default behavior changed: Python 3.11+ installs a compatible
ORT; Python 3.10 skips incompatible native detection.
- Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` remains
available; an explicit `ORT_DYLIB_PATH` remains an operator override.
- Unsafe override required: No.
- Qualification impact: Native detection stays enabled only with
API-24-compatible ORT.
- Rollback path: Revert this PR, which restores the old watchdog-only
degradation.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The large lockfile diff is dependency resolution: Python 3.10 keeps ORT
1.23.2 while 3.11+ resolves 1.26.0. The functional Python change is
intentionally small and keeps explicit `ORT_DYLIB_PATH` overrides
working.
2026-08-13 15:05:41 -05:00
Abhay SinghandJerrettDavis 3bb02f8f75 fix(transforms/smart_crusher): don't crash on a tool call with a null function (#2232)
## Description

A tool call whose `function` field is explicitly `null` crashes
SmartCrusher's per-request context extraction.

`_extract_context_from_messages` (called at the top of `apply()`) walks
recent assistant tool calls:

```python
for tc in msg.get("tool_calls", []):
    if isinstance(tc, dict):
        func = tc.get("function", {})
        args = func.get("arguments", "")
```

`dict.get("function", {})` only substitutes `{}` when the key is
**missing**. When the key is present but `null` — `{"id": "1", "type":
"function", "function": null}`, which clients emit for a partial or
streamed tool call — `func` is `None`, and `None.get("arguments")`
raises `AttributeError`. That propagates out of
`_extract_context_from_messages` and crashes `apply()` for the entire
request, so the request either errors or has to fail open to
uncompressed with a logged traceback.

The sibling `_build_tool_name_index` in the same file already guards
this exact shape with `(tc.get("function") or {})` — this call site just
wasn't updated to match.

## Fix

Use the same null-safe form:

```python
func = tc.get("function") or {}
```

`None` (and any other falsy value) now collapses to `{}`, the null tool
call contributes no context, and extraction continues to the next call.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/smart_crusher.py`: `tc.get("function", {})` →
`tc.get("function") or {}` in `_extract_context_from_messages`.
- `tests/test_transforms/test_smart_crusher_bugs.py`: new test asserting
a `{"function": null}` tool call doesn't crash extraction and later
calls are still read.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/transforms/smart_crusher.py tests/test_transforms/test_smart_crusher_bugs.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/transforms/smart_crusher.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the extraction loop with a dependency-free script and left
the full pytest to CI.
- Exact command / steps: ran an assistant message with tool calls
`[{"function": null}, {"function": {"arguments": "keep-me"}}]` through
the OLD `get("function", {})` loop and the NEW `get("function") or {}`
loop.
- Observed result: OLD raises `AttributeError` on the null function; NEW
skips it and returns `"keep-me"` from the following call.
- Not tested: a live proxy request carrying a null-function tool call;
full local `pytest` deferred to CI (OOM).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new test uses the
existing `_make_crusher` helper in
`tests/test_transforms/test_smart_crusher_bugs.py`, so it runs under the
normal CI pytest job; behaviour is additionally verified by the
standalone proof above.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-11 23:39:37 -05:00
gglucass d7bc1e275f fix(content-router): protect custom-tag blocks before mixed-content section split
Protect custom-tag blocks during mixed-content routing.
2026-08-11 18:16:00 -07:00
michaeltarletonandMichael Tarleton 677e09735a fix(transforms): stop ContentRouter recompressing headroom_retrieve results (#2654)
## Description

`ContentRouter` (the transform actually registered in the default/proxy
compression
pipeline -- see `transforms/pipeline.py`) recompresses the output of its
own
`headroom_retrieve` tool. That tool's entire contract is returning
already-retrieved,
original content verbatim; recompressing it produces a new
`<<ccr:hash>>` marker the
caller can never redeem -- an unresolvable retrieval loop.

`SmartCrusher` already has a guard against this exact failure mode
(#1077), but only
on its `apply()` entry point. `ContentRouter` calls the lower-level
`SmartCrusher.crush()` directly, bypassing that guard entirely, since
`crush()` takes
a raw content string with no tool identity at all.

Closes #1077 (reopens the same failure mode ContentRouter's own call
path, which #1077's
original fix did not cover).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `transforms/content_router.py`: adds an unconditional guard to all
three of the
places `ContentRouter` can hand a `headroom_retrieve` result to
compression:
the OpenAI-shape `role:"tool"`/legacy `role:"function"` string-content
loop, the
Anthropic-shape `tool_result` block loop, and a third, distinct shape --
top-level `{"type": "text"}` blocks under a `role:"tool"`/`"function"`
message
that never go through a `tool_result` wrapper (a real, already-tested
wire shape
in this codebase; see
`test_tool_role_text_blocks_compressed_by_default`). All
three use `is_tool_excluded()` (not a bare comparison) because
MCP-served tools
appear here under their qualified form, e.g.
`mcp__headroom__headroom_retrieve`.
Legacy `role:"function"` messages carry no call id in that shape, so the
tool
name is read directly off the message's `name` field instead of through
the
  id-keyed `tool_name_map`.
- Hoisted the per-iteration `is_tool_excluded(...,
("headroom_retrieve",))` calls
into a single precomputed `ccr_retrieve_tool_ids` set, computed once
alongside
the existing `excluded_tool_ids` set, rather than recomputing aliases on
every
  message/block.
- `config.py`: adds `"headroom_retrieve"` to `DEFAULT_EXCLUDE_TOOLS` and
`DEFAULT_VERBATIM_EXCLUDE_TOOLS` -- this also covers a third path
(cross-turn
message dedup, `_cross_turn_dedup_messages`) that consults the same
frozensets
and has no dedicated guard of its own. Also hardens
`_tool_name_aliases()`
against a non-string tool name (pre-existing fragility, not introduced
by this
PR, but shares the same call path) by returning no aliases instead of
crashing
  on `.lower()`.
- Documentation: updated `ContentRouterConfig.exclude_tools`'s field
comment (was
stale -- didn't mention this override is unconditional even when a
caller
  explicitly empties `exclude_tools`), and added a comment on
  `DEFAULT_VERBATIM_EXCLUDE_TOOLS` noting all three real consumers.
- Kept `"headroom_retrieve"` as a literal string (matching every other
entry in
those frozensets) rather than importing the existing `CCR_TOOL_NAME`
constant
from `ccr.tool_injection` into `content_router.py` -- that module is
imported
eagerly by `pipeline.py` (unlike `smart_crusher.py`, which imports the
same
constant lazily), so pulling in `headroom.ccr` there would add a new
eager-import
  edge to a hot module for a one-line DRY win. Happy to change this if a
  maintainer prefers the constant.

**Known, accepted tradeoff:** `is_tool_excluded()`'s alias matching
strips any
`mcp__<server>__` prefix before comparing, so a third-party MCP server
exposing a
tool literally named `headroom_retrieve` would also match. Narrowing
this to
headroom's own server specifically would need a bespoke check
inconsistent with
how every other excluded-tool entry is matched in this codebase; given
how specific
the name is, the collision risk is accepted rather than special-cased.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_transforms/ tests/test_transforms_content_router.py -q
1 failed, 420 passed, 62 skipped in 12.50s
FAILED tests/test_transforms/test_kompress_compressor.py::...test_onnx_session_options_read_thread_caps
  (pre-existing, unrelated to this diff -- confirmed via `git stash` that it fails
  identically against unmodified upstream/main; an ONNX thread-cap assertion, not
  a compression-routing test)

$ uv run ruff check headroom/config.py headroom/transforms/content_router.py \
    tests/test_transforms/test_content_router_ccr_retrieve_exemption.py \
    tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py
All checks passed!

$ uv run ruff format --check <same files>
5 files already formatted

$ uv run mypy headroom/config.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```

- `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`:
10 tests --
MCP-qualified name (Anthropic + OpenAI shape), bare name,
unconditional-even-with-
`exclude_tools=frozenset()`, negative control (normal tools still
compressed,
asserted via the absence of the `router:excluded:ccr_retrieve` marker),
the
top-level-text-block shape, legacy `role:"function"`, litellm list-form
content
nested in a `tool_result` block, mixed retrieve+normal blocks in one
turn, and a
content well below the compression floor (proving the guard is
size-independent).
- `tests/test_transforms/test_content_router.py`:
`test_anthropic_mcp_bare_tool_alias_exclude_tools`
(#1822) updated to assert the new, stronger byte-verbatim guarantee for
`headroom_retrieve` specifically;
`test_anthropic_mcp_bare_tool_alias_exclude_tools_generic`
added to keep the original #1822 general-mechanism coverage (bare-alias
matching
  for an arbitrary, non-exempt tool).
- `tests/test_transforms_content_router.py`: updated 10 pre-existing
`_process_content_blocks()` unit tests for the new
`ccr_retrieve_tool_ids`
parameter (all pass empty sets -- none of those tests involve
`headroom_retrieve`).
- Verified the local installed package copy (a separate, drifted
internal version)
with a standalone repro script exercising the two new shapes directly
against
`ContentRouter.apply()` -- both correctly report
`router:excluded:ccr_retrieve`.

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.7, `uv sync --extra dev` on this
branch.
- Exact command / steps: standalone repro building an assistant
`tool_use` for
`mcp__headroom__headroom_retrieve` paired with a large-JSON
`tool_result`,
through `ContentRouter().apply()`; repeated for the top-level-text-block
and
  legacy-`function`-role shapes.
- Observed result: unpatched (Anthropic `tool_result` shape, `git stash`
to
`upstream/main`), the retrieve output was rewritten 3680 -> 1861 bytes
(mangled
into a compact tabular form); patched (this branch), it is forwarded
3680 -> 3680
bytes byte-identical, no `<<ccr:` marker present. The two additional
shapes fixed
in this PR's second commit -- top-level text block under `role:"tool"`,
and
legacy OpenAI `role:"function"` -- both report `excluded=True`
(protected)
against this branch, where they reported `excluded=False` (recompressed)
before
  the second commit.
- Not tested: the actual `headroom mcp serve` + `headroom wrap` proxy
end-to-end
  over a live Anthropic API call (would need API credentials); the
OpenAI-chat-completions `CompressionUnit` path (out of scope, see #1176
below);
  the opt-in `ToolResultInterceptorTransform` path.

## Relationship to other issues/PRs

- Issue #1077 (closed) is this exact bug; PR #1323 fixed it only for
`SmartCrusher.apply()`'s own call path (the "legacy" pipeline path, per
`smart_crusher.py`'s own comment), not `ContentRouter`, which is what
the
  default/proxy pipeline actually uses.
- Open PR #1176 addresses an adjacent, non-overlapping gap: the
`CompressionUnit`-based OpenAI chat-completions path
(`router.compress()` calls
in `transforms/compression_units.py`/`compression_batches.py`), which
has no
tool-identity context at all and needs its own capture/restore
mechanism. This
  PR does not touch that path.
- Filed #2656 as a follow-up: code review on this PR found the same bug
class
still reachable through `SmartCrusher.apply()`'s own bare-name guard
(not
alias-aware, so it misses the MCP-qualified form) and through two
unguarded
direct `.crush()` calls in the LangGraph and Strands integrations. Both
are
pre-existing, narrower/separate call paths from `ContentRouter`'s
primary proxy
pipeline, so tracking them separately keeps this PR reviewable as one
logical
  change.
- Also not covered by this PR (flagging rather than silently omitting):
`proxy/system_compaction.py`'s `router.compress(text, context="")` call,
and the
opt-in `ToolResultInterceptorTransform` (`HEADROOM_INTERCEPT_ENABLED=1`)
--
  neither was checked for CCR-awareness.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` -- it is generated by
release-please from
      my Conventional Commit PR title (a CI guard enforces this)

## Screenshots (if applicable)

N/A -- this is a backend compression-routing fix with no UI surface.

## Additional Notes

This PR is two commits: the first commit added the initial two-loop
guard; a
second commit followed after code review found the guard was incomplete
for two
additional wire shapes (top-level text blocks, legacy `role:"function"`)
and
added the missing test coverage plus a few cleanup items (deduplicated
guard
logic, comment accuracy, a pre-existing non-string-tool-name fragility).
See
`Changes Made` above for the full list. Filed #2656 for the remaining
out-of-scope gaps found during that same review.

---------

Co-authored-by: Michael Tarleton <mtarleton@istation.com>
2026-08-03 20:17:06 -07:00
Tejas Chopra 3e348f327f fix(ccr): stop persisting retrieval markers as original content (#2694) (#2703)
## Description

CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at
all — where the original bytes belonged, so `headroom_retrieve(hash)`
answered with the very placeholder the caller was trying to resolve. For
a base64/credential field that is permanent, silent data loss: the inner
marker's hash is the only handle on the real payload, and it disappears
from anywhere the model can see.

Four sites, one root cause — **a compressed intermediate (or nothing)
was stored in place of the source**, the same defect class as #1209 (tag
placeholders persisted as originals).

Closes #2694

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`compaction/walker.rs`** — `walk_array` compacted through the
store-LESS `compact()`. Opaque cells inside a compacted table got a
marker whose payload was **never written**, so retrieval 404'd forever.
Now uses `compact_with_store` so the emitted hash resolves.
- **`compaction/classifier.rs`** — nothing stopped an already-marked
string from being offloaded a second time, which stashed the MARKER as
the new entry's "original". Marker-bearing text is our own output, not
source content, so it is never classified opaque. One guard at the choke
point both the walker and the table compactor share.
- **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop
marker hashed and stored rows whose leaves were **already** rewritten
(prose compressed, blobs marker-substituted), so retrieving dropped rows
returned compressed output. Now hashes and stashes the pre-processing
array via `crush_array_with_source`.
- **`content_router.py`** — compression pinning matched only `Retrieve
more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so
opaque-blob output was readmitted to the compressor on a later turn —
the path that feeds the corruption above. Consolidated into
`_is_already_compressed()` and applied at all three pinning sites.
- **`cache/compression_store.py`** — store-level guard: refuse to
persist a *bare* marker as `original_content` and log at ERROR, so a
future producer regression surfaces loudly instead of silently
converting "retrievable" into "gone". Deliberately narrow — originals
may legally *contain* markers (nested offloads); only a bare marker is
rejected.
- **Regression tests** —
`test_nested_table_markers_resolve_to_source_bytes` (asserts payloads
are verbatim-retrievable, not merely that a marker was emitted) and
`test_already_marked_content_is_not_re_offloaded`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo build -p headroom-core
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s

$ cargo test -p headroom-core --lib smart_crusher
test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s

$ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q
16 passed in 0.68s

$ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q
50 passed in 12.52s

$ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q
100 passed in 13.14s

$ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \
      tests/test_transforms/test_smart_crusher_ccr_roundtrip.py
All checks passed!

$ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai
0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt
via `maturin develop --release`.
- **Exact command / steps:** compact a nested document — 5 rows whose
`detail` field is a stringified sub-array of 6 base64 blobs (1600 B
each) — then, for every `<<ccr:HASH>>` marker in the output, call
`ccr_get(HASH)` and check the payload is the verbatim source rather than
a marker.

```python
inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)]
doc   = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]}
out   = SmartCrusher().compact_document_json(json.dumps(doc))
for h in re.findall(r"<<ccr:([0-9a-f]+)", out):
    payload = crusher.ccr_get(h)          # must be real bytes, not a marker
```

- **Observed result — BEFORE (on `main`):** all six payloads collapsed
into a single dead marker. The rendered sub-table was re-classified
opaque (`html`, because `<<` reads as a tag), offloaded again, and its
payload never stored — so the six inner hashes were erased from the
visible text *and* the outer hash resolved to nothing.

```text
{"rows":"[5]{detail:string,id:int,note:string}
         <<ccr:3fb1d44933da,html,289B>>,0,x
         <<ccr:3fb1d44933da,html,289B>>,1,x ..."}

3fb1d44933da -> RUST MISS          # unrecoverable — 6 × 1600 B gone
```

- **Observed result — AFTER (this branch):** the sub-table stays inline,
each blob keeps its own marker, and every marker resolves to verbatim
source.

```text
{"rows":"[5]{detail:string,id:int,note:string}
         \"[6]{k:string,tok:string,v:int}
         key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."}

  6ad5846997f4: resolves, len=1600, is-verbatim-source=True
  78a0bd9364a7: resolves, len=1600, is-verbatim-source=True
  955b1fed2ef7: resolves, len=1600, is-verbatim-source=True
  a0cef69da7f0: resolves, len=1600, is-verbatim-source=True
  dfcde5e940c0: resolves, len=1600, is-verbatim-source=True
  e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True

RESULT: PASS — every marker resolves to real source bytes
```

## Notes for reviewers

- The issue also reports **function words dropped from retained prose**
(`is`, `a`, `the`) and **interleaved log output corrupting `headroom
doctor`'s table borders**. Those are separate defects on different paths
(extractive prose compression and log-handler buffering respectively)
and are **not** addressed here — this PR is scoped to the CCR
store/retrieve corruption. They should be tracked separately; the prose
one overlaps #2586.
- The `crusher.rs` prose-hook fix is on the Rust pipeline
(`json_offload`) rather than the Python proxy path, but it is the same
store-the-intermediate bug and was cheap to close while in the file.
2026-08-02 13:10:41 -07:00
Raúl 1a2688b57f test(kompress): close patch-coverage gaps from #2716 (#2721)
## Description

Codecov flagged 9 uncovered lines on #2716 after it merged:
`hf_entry_known_absent`'s own body
in `headroom/onnx_runtime.py` was only ever exercised indirectly (every
existing test in
`tests/test_transforms/test_kompress_compressor.py` monkeypatched it
away rather than calling the
real implementation), and `_load_pytorch_weights` /
`_load_kompress_pytorch` in
`headroom/transforms/kompress_compressor.py` had three untested
branches: the double cache-miss
under `allow_download=False` (merged.pt confirmed absent AND the plain
fallback also not cached),
a genuine non-404 download failure propagating instead of silently
falling back, and the
already-cached fast path in `_load_kompress_pytorch`.

## Type of Change

- [ ] Bug fix
- [ ] New feature
- [x] Test coverage improvement, no production code change

## Changes Made

- `tests/test_onnx_runtime.py`: added `_write_fake_hf_cache` (builds a
minimal on-disk HF hub
cache layout, including the `.no_exist/<hash>/<filename>` marker
huggingface_hub writes after a
real 404) and three direct tests of `hf_entry_known_absent` against the
real
  `huggingface_hub.try_to_load_from_cache`, not a mock of it.
- `tests/test_transforms/test_kompress_compressor.py`: added
  `test_cache_only_raises_when_confirmed_absent_but_plain_also_missing`,
`test_genuine_download_failure_propagates_instead_of_falling_back`, and
a new
`TestLoadKompressPytorchCaching` class covering the already-cached fast
path.

## Testing

```text
$ .venv/bin/python3 -m pytest tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py -q
51 passed

$ .venv/bin/python3 -m pytest tests/ -k "kompress or onnx_runtime" -q --cov=headroom.transforms.kompress_compressor --cov=headroom.onnx_runtime --cov-report=term-missing
# before: onnx_runtime.py Missing includes 132-136 (hf_entry_known_absent's entire body);
#         kompress_compressor.py Missing includes 805-806, 818, 836
# after:  none of those lines appear in Missing anymore
191 passed, 7 skipped

$ .venv/bin/python3 -m ruff format --check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py
2 files already formatted
$ .venv/bin/python3 -m ruff check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py
All checks passed!
```

## Review Readiness

- Test-only, additive diff (113 insertions, 0 deletions, 0 lines touched
outside the two test
  files). No behavior change possible.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit `CHANGELOG.md`

## Additional Notes

Not closing: the remaining branch-partial on the `device == "auto"`
cuda/mps/cpu selection in
`_load_kompress_pytorch` (would need mocking `torch.cuda.is_available()`
/
`torch.backends.mps.is_available()` for marginal benefit); left as-is.
2026-08-02 10:45:44 -07:00
Raúl 46da91b2f1 fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors (#2716)
# PR draft: fix(kompress): load merged.pt for the v2 checkpoint instead
of the unmerged PEFT safetensors

Branch: `rnoz/fix-kompress-merged-checkpoint` (off `upstream/main`). Two
commits.
Issue: https://github.com/headroomlabs-ai/headroom/issues/2714 (filed,
open).

---

## Description

`_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py`
downloaded `model.safetensors` from the default model repo
`chopratejas/kompress-v2-base` and loaded it with `strict=False`,
discarding the missing/unexpected key report. That file is the unmerged
PEFT checkpoint (encoder keys prefixed `encoder.base_model.model...`),
which never matches `HeadroomCompressorModel`'s plain `encoder.*` keys.
The LoRA-adapted encoder weights were silently dropped while
`token_head`/`span_conv` happened to match and loaded fine, so the model
ran with a stock, non-adapted `answerdotai/ModernBERT-base` encoder
feeding correctly trained decision heads, with no error and a healthy
status reported everywhere.

`scripts/export_kompress_v2_onnx.py` already documents this exact
mismatch and loads the correct `merged.pt` sub-state-dicts for its own
export path. This PR mirrors that same loading logic into the runtime
PyTorch loader, with a fallback to the plain `model.safetensors` format
for repos that never shipped a `merged.pt` (verified against the v1
`chopratejas/kompress-base` repo via the public HF API, which has no
`merged.pt`). Both paths now check the missing/unexpected key report and
raise instead of silently proceeding on a mismatch.

A second commit fixes a gap an adversarial review caught in the first:
the cache-only (`allow_download=False`, startup preload) path could not
tell "this repo genuinely has no merged.pt" apart from "merged.pt exists
but is not downloaded yet", so it would have fallen back to a stale
`model.safetensors` left over from before this fix on exactly the
upgrade path this PR is meant to close. It now uses `huggingface_hub`'s
own `.no_exist` cache marker (via a new `hf_entry_known_absent()` helper
in `headroom/onnx_runtime.py`) to make that distinction without a
network call, and only falls back when absence is confirmed; otherwise
it raises `KompressModelNotCached` so the caller defers instead of
guessing.

Closes #2714

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/kompress_compressor.py`: added
`_load_merged_state_dict`, `_load_plain_state_dict`, and
`_load_pytorch_weights`, replacing the inline `model.safetensors`
download + `load_state_dict(strict=False)` call in
`_load_kompress_pytorch`. `merged.pt` is tried first; the plain format
is only used when its absence is confirmed.
- `headroom/onnx_runtime.py`: added `hf_entry_known_absent()`, a thin
wrapper around `huggingface_hub.try_to_load_from_cache()` that reads the
on-disk `.no_exist` marker HF writes after a real 404, so cache-only
code can distinguish "confirmed absent" from "never checked" without
hitting the network.
- `tests/test_transforms/test_kompress_compressor.py`: added
`TestPytorchWeightLoading` (8 tests) covering the merged-checkpoint
happy path, missing-section and key-mismatch failures, the plain-format
fallback for repos without `merged.pt`, and the cache-only ambiguity fix
(confirmed-absent vs unconfirmed).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] Manual testing performed (real download against the live
`chopratejas/kompress-v2-base` repo, not just mocks)

### Test Output

```text
$ .venv/bin/python3 -m pytest tests/ -k kompress -q
178 passed, 7 skipped in 22.39s

$ .venv/bin/python3 -m ruff check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py
All checks passed!

$ .venv/bin/python3 -m ruff format --check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py
3 files already formatted

$ .venv/bin/python3 -m mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

Ran the actual fixed loader against the live
`chopratejas/kompress-v2-base` HF repo (not a mock), before and after
each fix:

```text
# BEFORE (parsed the real cached model.safetensors header by hand, no safetensors lib needed):
total tensors: 316
  encoder.base_model.model.embeddings.norm.weight
  encoder.base_model.model.embeddings.tok_embeddings.weight
  ...(all 310 encoder tensors share this prefix)...
  span_conv.0.bias / span_conv.0.weight / span_conv.2.bias / span_conv.2.weight
  token_head.bias / token_head.weight
exact prefix match count with plain "encoder.<rest>": 0

# This confirms the pre-fix code's model.load_state_dict(state_dict, strict=False)
# silently dropped every encoder weight (0 keys match HeadroomCompressorModel.encoder),
# while token_head/span_conv happened to match and loaded.

# AFTER (commit 1, real merged.pt download + load):
$ .venv/bin/python3 -c "
import headroom.transforms.kompress_compressor as kmod
model = kmod._get_model_class()()
kmod._load_pytorch_weights(model, 'chopratejas/kompress-v2-base', allow_download=True)
print('SUCCESS: 0 missing/unexpected keys across all three sections')
"
SUCCESS: 0 missing/unexpected keys across all three sections

# AFTER (full pipeline, real end-to-end compression through the public API):
$ .venv/bin/python3 -c "
import headroom.transforms.kompress_compressor as kmod
compressor = kmod.KompressCompressor()
result = compressor.compress(sample_traceback_plus_boilerplate_text)
print(result.original_tokens, result.compressed_tokens, result.tokens_saved)
print('ValueError: bad input' in result.compressed)
"
497 454 43
True   # must-keep line (the actual error) survived compression

# AFTER (commit 2, cache-only ambiguity): unit tests
# test_cache_only_defers_instead_of_using_stale_plain_checkpoint: PASSED
# test_cache_only_uses_plain_checkpoint_when_merged_pt_confirmed_absent: PASSED
```

## Review Readiness

- [x] I have performed a self-review
- [x] An independent adversarial review pass was run on both commits
before this PR was opened; its one finding (the cache-only ambiguity) is
fixed in commit 2, verified with new regression tests
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal loader behavior, no public API or config surface changed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective (regression
tests for both the original silent-drop bug and the cache-only ambiguity
found in review)
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit `CHANGELOG.md`

## Additional Notes

- Both `merged.pt` and the plain `model.safetensors` fallback now raise
loudly on any state-dict mismatch instead of proceeding with
partially-loaded weights, closing the general silent-failure class this
bug belonged to, not just this one instance of it.
- No other call sites of `_load_kompress_pytorch` or its removed inline
code exist; its public signature is unchanged.
2026-08-02 08:10:26 -07:00
6d5516dcb8 feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description

Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already
*detected* as code (Magika labels in `headroom/compression/detector.py`
include `php`, and the Rust `magika_detector.rs` lists it too) but there
was no PHP `LangConfig`, so PHP content silently passed through
uncompressed. This wires PHP through the tree-sitter compression path
following the C# pattern (the most recently added, fully functional
language — deliberately not the quarantined Perl path).

A secondary detection bug is fixed along the way: PHP's `$variables`
match Perl's prefilter regex, and the existing Perl-dominance guard in
`detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php`
open tag — which no Perl source contains — now drops Perl from the
candidate set before that guard runs.

## Type of Change

- [ ] Bug fix
- [x] New feature
- [ ] Documentation update
- [ ] Refactor
- [ ] Other

## Changes Made

- `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` +
`phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the
actual tree-sitter-php grammar (node names verified by parsing samples):
`namespace_use_declaration` imports,
`function_definition`/`method_declaration` functions,
`class_declaration`/`interface_declaration`/`trait_declaration` classes,
`enum_declaration` types, `declaration_list` class bodies,
`compound_statement` function bodies. `namespace_definition` maps to
`package_node` so statement-scoped `namespace App;` hoists ahead of the
`use` imports (required PHP ordering); the rare block-scoped `namespace
A { }` form takes the same path and is preserved verbatim — valid
output, just no compression inside the block. PHP prefilter regexes
added; supported-languages error message updated; `<?php`-tag Perl
disambiguation in `detect_language`.
- `headroom/transforms/content_detector.py`: `php` entry in
`_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the
code-aware route.
- `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport`
mirroring `TestCSharpSupport` — signatures preserved / bodies elided,
`<?php` → `namespace` → `use` → declarations ordering, auto-detection
despite the Perl sigil overlap, alias coercion, malformed passthrough.
- `tests/test_code_compressor_language_alias.py`: `php` in the canonical
list, `phtml` in the alias table.
- `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2
supported-languages row.

No new dependency: `tree-sitter-language-pack` (the existing `[code]`
extra) already ships the PHP grammar. No Rust changes needed.

## Testing

- [x] New unit tests added and passing
- [x] Full affected test suites pass locally

**Test Output**

```
$ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q
============================= 120 passed in 7.81s =============================

$ python -m pytest tests/test_transforms/ -q
3 failed, 443 passed   # the 3 failures (kompress ONNX thread caps, kompress size gate,
                       # text_crusher unicode parity) reproduce identically on a clean
                       # upstream/main checkout in this environment — pre-existing local
                       # ONNX runtime quirks, unrelated to this change

$ ruff check . (0.15.17, CI-pinned) → All checks passed!  |  ruff format --check → clean
$ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, tree-sitter +
tree-sitter-language-pack (<1.0) installed, branch
`feat/201-php-code-compression` off `upstream/main`.
- Exact command / steps: parsed PHP samples (namespaced class w/
methods, block-scoped namespace, mixed HTML+PHP) with
`tree_sitter_language_pack.get_parser('php')` to verify every node name
used in the config; then ran `CodeAwareCompressor().compress(php_code,
language="php")` and `compress(php_code)` (auto-detection) on a 48-line
realistic service class.
- Observed result: explicit and auto-detected paths both return
`language=CodeLanguage.PHP`, `compression_ratio=0.64`,
`syntax_valid=True`; method bodies elided to `// [N lines omitted]`
while `<?php`, `namespace`, `use` lines, class header, and all
signatures are preserved verbatim in the original order. Before the
detection fix, auto-detection returned `UNKNOWN` (Perl prefilter
dominance) — reproduced and then verified fixed.
- Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]`
on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic
mixed case); these fall back to verbatim preservation via the
uncaptured-node pass or malformed-passthrough, both of which are covered
by tests for the simple cases.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-31 15:54:13 -07:00
ParideboyandClaude Opus 4.8 c811007f81 fix(kompress): match all ONNX backends with startswith, not exact "onnx" (#2448)
## Description

With `HEADROOM_KOMPRESS_BACKEND=onnx_coreml`, every Kompress compression
call and the startup canary crash with `'_OnnxModel' object has no
attribute 'parameters'`, so Kompress silently degrades to passthrough
and `/health` reports `kompress: unhealthy, backend: null`.

Root cause: `headroom/transforms/kompress_compressor.py` gated the
ONNX-vs-PyTorch branch with an exact string match `backend == "onnx"`.
But `_load_kompress_onnx` returns `onnx_coreml` (CoreML) or `onnx_cpu` —
never the bare string `onnx`. So under `onnx_coreml` the code built
PyTorch tensors and dispatched to a device via
`next(model.parameters())`, which the `_OnnxModel` wrapper doesn't
implement. This is the accelerated backend Apple Silicon users reach
for, so the fast path is exactly the broken one.

Fixes #2442

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Change the four exact-match `backend == "onnx"` sites in
`headroom/transforms/kompress_compressor.py` to
`backend.startswith("onnx")`, matching the convention already used by
`_model_device_type`: `_timed_canary`, `compress`, `compress_batch`, and
the batch-parallelism guard in `_should_use_sequential_fallback`.
- Update the guard comment ("ONNX CPU provider" → "ONNX EPs") since it
now covers all ONNX execution providers.
- Add regression tests exercising `_timed_canary` on `onnx_coreml` (must
take the numpy path and never touch `.parameters()`) with a negative
control proving the PyTorch branch still dispatches to a device.
- Leave `CHANGELOG.md` untouched — release-please generates it from
conventional commits.
- Out of scope: the secondary `/health` under-reporting the issue flags
as informational (deferred-preload warmup object never flips to
`loaded`).

## Testing

- [x] Unit tests pass (`python -m pytest
tests/test_transforms/test_kompress_compressor.py::TestOnnxBackendPrefixGating
-q`)
- [x] Linting passes (`ruff check`, `ruff format --check` on the two
changed files)
- [x] Type checking passes (`mypy
headroom/transforms/kompress_compressor.py --ignore-missing-imports`)
- [x] New tests added for new functionality

### Test Output

```text
$ python -m pytest tests/test_transforms/test_kompress_compressor.py::TestOnnxBackendPrefixGating -q
collected 2 items
tests\test_transforms\test_kompress_compressor.py ..                     [100%]
2 passed in 2.20s

$ ruff check headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, local dev checkout on a branch
off upstream/main (no Apple Silicon / CoreML hardware available)
- Exact command / steps: Ran the new `TestOnnxBackendPrefixGating`
regression; then temporarily reverted one site back to `backend ==
"onnx"` and re-ran to confirm the test discriminates.
- Observed result: With the fix, `_timed_canary(model, tokenizer,
"onnx_coreml")` returns a float and never touches `.parameters()`.
Reverting one site makes the onnx_coreml test fail (it takes the `pt`
tensor path and hits the paramless model), proving the test catches the
exact bug. The issue reporter separately verified the fix on real Apple
Silicon hardware (onnxruntime 1.27.0, CoreMLExecutionProvider): zero
occurrences of the error afterward and compression completing on the
CoreML session.
- Not tested: End-to-end run on real CoreML hardware from this
environment — reproduced via the unit-level device-dispatch seam
instead; hardware confirmation is in the issue.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 17:29:08 -07:00
Abhay SinghandJerrettDavis b75999017f fix(transforms/kompress-remote): keep compress fail-open on malformed 200 (#2320)
## Description

`RemoteKompressCompressor` (the opt-in `HEADROOM_KOMPRESS_ENDPOINT`
remote compression client) documents a fail-open contract in its own
docstring:

> Fails OPEN: any network/HTTP error returns the content verbatim so a
flaky endpoint degrades compression rather than breaking the proxy.

But only the network call and the `compressed` field check actually run
inside the fail-open guard. The metadata coercions run **after** the
`except`, outside it:

```python
try:
    resp = self._client.post(...)
    resp.raise_for_status()
    data = resp.json()
    compressed = data["compressed"]
    if not isinstance(compressed, str):
        raise TypeError("...")
except Exception as e:  # fail OPEN
    logger.warning("Remote Kompress failed (%s); passing through", e)
    return self._passthrough(content, n_words)

result = KompressResult(
    compressed=compressed,
    original=content,
    original_tokens=int(data.get("original_tokens", n_words)),
    compressed_tokens=int(data.get("compressed_tokens", len(compressed.split()))),
    compression_ratio=float(data.get("compression_ratio", 1.0)),   # <-- outside the guard
    model_used=str(data.get("model_used", self.config.model_id)),
)
```

So a hosted `/compress` endpoint that returns a 200 with a valid
`compressed` string but a malformed metadata field escapes the guard and
raises out of `compress`, breaking the proxy request instead of passing
through. The most realistic trigger is an explicit JSON `null`:
`data.get("compression_ratio", 1.0)` returns `None` for a **present**
key (the default only applies to a missing key), and `float(None)`
raises `TypeError`. A non-numeric string like `"original_tokens":
"lots"` raises `ValueError` the same way. Since the whole point of the
flag is to support arbitrary self-hosted endpoints, a slightly-off but
well-meaning endpoint (sending `null` for a field it could not compute)
takes down the request path this class exists to protect.

## Fix

Move the response parsing (the `KompressResult` construction with its
`int`/`float`/`str` coercions) inside the fail-open `try`, so any
malformed field degrades to verbatim passthrough like every other
bad-response case:

```python
try:
    ...
    compressed = data["compressed"]
    if not isinstance(compressed, str):
        raise TypeError("...")
    result = KompressResult(
        compressed=compressed,
        original=content,
        original_tokens=int(data.get("original_tokens", n_words)),
        compressed_tokens=int(data.get("compressed_tokens", len(compressed.split()))),
        compression_ratio=float(data.get("compression_ratio", 1.0)),
        model_used=str(data.get("model_used", self.config.model_id)),
    )
except Exception as e:  # fail OPEN
    logger.warning("Remote Kompress failed (%s); passing through", e)
    return self._passthrough(content, n_words)
```

No behavior change on a well-formed response; only the malformed-200
path changes (raise to passthrough).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/kompress_remote.py`: move the `KompressResult`
construction and its field coercions inside the fail-open `try`.
- `tests/test_transforms/test_kompress_remote.py`: add
`test_remote_kompress_null_numeric_field_fails_open` (explicit JSON
`null`) and `test_remote_kompress_non_numeric_field_fails_open`
(non-numeric string), both asserting verbatim passthrough.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/transforms/kompress_remote.py tests/test_transforms/test_kompress_remote.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/transforms/kompress_remote.py tests/test_transforms/test_kompress_remote.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/transforms/kompress_remote.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the control flow with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (coercions outside the `try`)
and NEW (inside the `try`) parsing against a 200 body `{"compressed":
"short result", "compression_ratio": null}` and against a well-formed
body.
- Observed result: OLD raised `TypeError` on the null field (proxy
request breaks); NEW returned passthrough; a well-formed body still
compressed under NEW. The added tests assert both malformed cases
(`null` and non-numeric string) return the original content with
`compression_ratio == 1.0`.
- Not tested: a live remote Kompress endpoint; the added tests drive
`RemoteKompressCompressor` through an `httpx.MockTransport`, matching
the existing test harness in this file.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added tests reuse the
existing `httpx.MockTransport` harness in `test_kompress_remote.py` and
run under the normal CI pytest job, and the behavior is corroborated by
the standalone proof above.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-17 12:11:41 -07:00