Commit Graph
257 Commits
Author SHA1 Message Date
zhengqiuyangandJD Davis 303f521812 fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description

A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.

This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.

Closes #4874

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
  string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored

$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored

# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
  `fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib

compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds

`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
  it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
  it the string renders as the JSON string literal (`"""[]"""`), which
  `expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
  included for CI, where the wheel is built normally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:50:47 -06:00
3aafdf2dcc fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description

When the code compressor declined a block of source code, the router
passed the block to Kompress. Kompress is a prose model. It drops
low-salience words, and in code those words are keywords such as `for`,
`in`, `if` and `import`. The result no longer parses. #4214 shows a
fenced Python script that comes back with `for` and `in` removed.

This PR stops that in the Python router and in the Rust live-zone
dispatcher. When the code compressor declines, the code is forwarded
unchanged.

Closes #4214

Credit: the Python change is oxura's work from
https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018
(277ca1f). Their commit is the first commit here, rebased onto `main`,
and oxura is its author. The test, docs and Rust commits were added on
top. oxura asked us to open this PR from our fork
([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)),
and said they have not run this branch, so the model and native results
below are ours, not theirs.

## Behaviour change to review

With `enable_code_aware=False` (the default), source code now stays
unchanged. Before this PR, it was sent to Kompress and lost keywords.
With `enable_code_aware=True`, code that the code compressor declines
also stays unchanged. Before, Kompress ran on it.
A code block the code compressor does compress is unaffected. Prose,
logs, JSON and tables are unaffected.

Side effect: a code block that Kompress used to shrink (and break) now
keeps its full size. This is the intended trade, but it means fewer
tokens saved on code when `enable_code_aware=False`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
- [ ] New feature
- [ ] Breaking change (the default-config behaviour change above is
intended, and it only stops damage to code)

## Changes Made

Five commits:

- `596764c3` `fix(router): keep source code out of prose fallbacks`.
Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in
`headroom/transforms/content_router.py`. This is 277ca1f rebased onto
`main`, and it differs from 277ca1f in two ways only: the two test-file
conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because
`AGENTS.md` says not to add content under `wiki/`.
- `7705a4e6` `test(router): repin the code-to-Kompress tests and add the
#4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by:
oxura` trailer. Five tests asserted the old fallback. They are rewritten
to assert the new behaviour (code unchanged, Kompress never sees it)
instead of being deleted. Adds a regression test for the script from
#4214, with `enable_code_aware` both off and on. Updates
`docs/content/docs/code-compression.mdx` and
`docs/content/docs/how-compression-works.mdx`, which said that detected
code goes to Kompress.
- `6631668a` `fix(live-zone): leave source code unchanged when the code
compressor declines`. Author: Tao Hansen. Same fix in
`crates/headroom-core/src/transforms/live_zone.rs`: a declined
`SourceCode` block is now a NoOp, not a Kompress input. Tests:
`tests/live_zone_source_code_decline.rs` (single test, needs no model),
and new cases in `tests/live_zone_dispatch.rs`.
- `d8776283` `fix(router): turn off the code compressor's internal
Kompress fallback`. Author: Tao Hansen. Follows review: the router's
code compressor was still built with
`fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown
languages, Perl and AST failures could build a `KompressCompressor`
inside `CodeAwareCompressor`. The router now always sets
`fallback_to_kompress=False`. The #4719 test that pinned `True`
(`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as
`test_enabled_kompress_still_turns_off_code_aware_fallback`, and the
unknown-language, Perl (by language hint, since `detect_language()`
reports Perl-dominant code as unknown) and AST-failure paths are tested
with spies on `is_kompress_available` and `KompressCompressor`, with
`enable_kompress` both off and on.
`docs/content/docs/code-compression.mdx` notes that the router always
turns the option off.
- `c1f8464e` `fix(router): keep passthrough code from external
compressors and stale caches`. Author: Tao Hansen. Follows review:
`_try_external_compressor` skips `PASSTHROUGH` blocks (code with
`prefer_code_aware_for_code=False`, whole or as a mixed section), which
would otherwise be offered to a prose-only external compressor as
`text/plain`. The disk cache namespace now also hashes
`_ROUTING_POLICY_REVISION`, so a same-version restart does not replay
code that Kompress damaged before this change. Tests: two
external-dispatch cases with a `text/plain` external, and one namespace
test. All three fail on the previous head.

I did not touch `CHANGELOG.md`.

## Testing

- [x] Unit tests pass (`pytest`): see below, the failures are
environment failures that also fail on `main`
- [x] Linting passes (`ruff check .`, `ruff format --check .`)
- [x] Type checking passes (`mypy
headroom/transforms/content_router.py`)
- [x] New tests added for new behaviour
- [x] Manual testing performed (real model replay, below)

### Test Output

macOS, Python 3.14. Head is now `c1f8464e`.

```text
After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean.
After the rebase onto main c05fbddd and the review fix (d8776283):
  router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped
  tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail
    (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure)
  ruff check / ruff format --check headroom tests: clean
  cargo test -p headroom-core: 1151 passed, 0 failed
Earlier run on 9df60c7c (before the rebase):
router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed:
  test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget,
  a timing test the branch does not touch; it passes 3 of 3 times when rerun alone
ruff check .:           All checks passed!
ruff format --check headroom tests: 1714 files already formatted
mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
cargo test -p headroom-core:   22 test binaries, 1151 passed, 0 failed
cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean
pytest tests (full):     31 failed, 19208 passed, 513 skipped
```

Rust bug-fix tests, fail before and pass after (the router-test commit's
source with the new Rust tests, then the live-zone commit; measured
before the rebase as `9df60c7c`, and the Rust code is unchanged since):

```text
before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on 7705a4e6)
        source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed)
after:  all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok
```

The dispatch fail-before was measured on the earlier, code-identical
branch (only documentation files differ). The dispatch test only fails
before the fix when the Kompress model is loaded (`HF_HOME` with the
model, `ORT_DYLIB_PATH` set). The single-test binary fails before the
fix on any host.

Python regression: with the new tests run against `origin/main`'s router
(no fix), 9 test cases fail (the #4214 fence test in both configs, the
repinned tests, and the fenced-code keyword test). With the fix they
pass.

### Failures in the full `pytest tests` run

The full run on the branch had 31 failures, none in router or
compression code. I compared them with pristine `origin/main`
(200c838c):

- 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path
is too long under this checkout directory (`OSError`). They fail the
same way in isolation on `main`.
- 17 other failures are in `test_proxy_health.py`,
`test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`,
`test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`,
`test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A
full run on `main` fails 29 tests, and the set differs from run to run
(the two sets overlap by 17 of 31). I re-ran the 14 failures that were
not in the `main` list in isolation on the branch: 12 pass, the other 2
are the UDS tests above.
- Known flaky test:
`tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`
failed once in a loaded run and passes on rerun. File repeated 3 times
on `596764c3` and on `origin/main`: 16 passed each time. It is timing
based and not caused by this change. It passed in the 558-test router
run above.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`,
real `kompress-v2-base` int8 ONNX model from an offline copy of the
public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the
venv's onnxruntime.
- Fence: a 1324-character valid Python script. The issue quotes only
four of its lines, so the rest is a reconstruction in the same style,
not the reporter's bytes.
- Exact command / steps: `ContentRouter.compress` on the fence followed
by about 40 lines of prose, for four configs (default;
`enable_code_aware=True`; `prefer_code_aware_for_code=False`;
`enable_code_aware=True` with lossless-then-lossy). Also
`headroom.compress` on a full Anthropic message list with the fence in a
`tool_result`.
- Observed result: Before (v0.44.0): in the default and code-aware
configs the fence comes back as 1144 characters and does not parse
(`import json, collections` becomes `json, collections`; `[s for s in
...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs
and in `headroom.compress`, the fence is byte-identical, 1324
characters, and parses. Prose control: the same router on prose still
compresses. A prose-only block went from 5709 to 5357 characters
(strategy TEXT). A prose / fence / prose block went from 12760 to 12056
characters (strategy MIXED), and the fence in it is byte-identical and
parses.
- Not tested: the reporter's exact 1324-character fence (only four lines
are public, so the tests use the reconstructed fence described above);
the Rust live-zone path end to end through the proxy with a real request
(covered by `cargo test` only); platforms other than macOS; the full
proxy on a live deployment.

## Other places where code can still reach a prose model

I checked every path from code to a prose or ML compressor, in Python
and Rust.

| Path | State |
| --- | --- |
| Python `CODE_AWARE` branch, code compressor declines or is disabled |
Fixed here: the block stays unchanged |
| Python post-strategy fallback after "no savings" (`content_router.py`,
`fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only
applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so
unchanged |
| Python lossless-then-lossy (opt-in) after the lossless fold |
`CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged |
| Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in)
| Prose-like output, not code. Unchanged |
| Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged |
| Python mixed content with fenced code (`_compress_mixed`) | Each fence
goes through the `CODE_AWARE` branch, so it is fixed by this change.
Covered by the new fence tests |
| Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before
this PR |
| Python `_experimental_compress_read`
(`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment
that runs Kompress on file reads, which can be code. Off by default.
Left as is |
| Python size gate and deadline fallbacks in `_try_ml_compressor` | Run
only after a caller chose Kompress, so code no longer reaches them |
| Python `CodeAwareCompressor` internal fallback
(`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST
failure) | Fixed in commit 4: the router always builds it with
`fallback_to_kompress=False`. Covered by Kompress-spy tests |
| Python external compressor dispatch for `PASSTHROUGH` code | Fixed in
commit 5: never offered to an external compressor |
| Disk result cache made before this change | Fixed in commit 5: a new
namespace through `_ROUTING_POLICY_REVISION` |
| Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) |
| Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It
is the single remaining call to `kompress_or_noop` |
| Rust: prose text with an embedded fenced code block | **Not
reproduced.** I built synthetic tool results of prose around the fenced
Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher
classified each one as `source_code`, so commit 3 leaves it unchanged,
and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has
no fence-aware splitting, but I found no input where that sends code to
Kompress, so this PR files no follow-up. |
| Python `config_compressor.py` docstring says YAML/TOML/INI "fall
through to the lossy prose path" | Documentation only: a stale comment,
not a code path. Config content has its own `CONFIG` strategy. Can be
refreshed in a later change |

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, in the default config code is no
longer sent to Kompress (see "Behaviour change to review"). Tokens saved
on code drop in return for code that stays valid.
- Kill switch / disable path: none added. Revert the PR to restore the
old fallback.
- Unsafe override required: no.
- Qualification impact: none expected. Compression of code by the code
compressor is unchanged.
- Rollback path: revert the three commits. No data, schema or config
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines (`ruff`, `cargo
fmt`, `clippy` clean)
- [x] I have made corresponding changes to the documentation
(`docs/content/docs/code-compression.mdx`,
`docs/content/docs/how-compression-works.mdx`)
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
- [ ] New and existing unit tests pass locally: the only failures are
environment failures that also fail on `main`, listed above

## Additional Notes

- Rust has no fence-aware splitting like the Python router has. A
synthetic prose-plus-fence repro did not reach Kompress (see the table
above), so no follow-up issue is filed.
- One stale comment remains in `content_router.py`
(`_try_ml_compressor`, the Phase 0 size-gate note lists
`CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not
changed here to keep the review commits as they are. Say if you want it
fixed in this PR.
- Reviewers: the Python change is easiest to review commit by commit.
Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and
the Rust fix.

---------

Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:25:06 -06:00
99021af20a fix(proxy): make the Rust front's native Bedrock and Vertex routes opt-in (#4851)
## Description

Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:

- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.

This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.

**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.

**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.

## Testing

- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).

### Test Output

Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):

```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and this branch. AWS credentials supplied
to the proxy through `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (fake
values), IMDS disabled. Upstream and Bedrock endpoint are a small Python
recorder on `127.0.0.1:18788` that prints the `Authorization` it
receives.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:18788 --bedrock-endpoint
http://127.0.0.1:18788` (no Bedrock/Vertex flags), then `curl -d
'<anthropic body>'
http://127.0.0.1:18787/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with no client credentials. Repeated on this branch with
`--enable-bedrock-native true`.
- Observed result (key id and signature redacted):

```text
== origin/main (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}

== this branch (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": null}

== this branch, --enable-bedrock-native true
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}
```

- Not tested: Vertex against real Google ADC and a real
`*.googleapis.com` upstream (covered by the integration tests with a
static token source; no GCP credentials on the test machine).

## Runtime Rollout Safety

- Rollout-managed feature(s): `native_bedrock` (existing). The CLI flag
already maps `false` to a rollout disable, so with the new default the
feature reports disabled unless `--enable-bedrock-native true` is
passed. Vertex is not rollout-managed.
- Minimum rollout channel: unchanged
- Stable/default behavior changed: yes, both native routes are off by
default
- Kill switch / disable path: the routes are now off unless enabled
- Unsafe override required: no
- Qualification impact: none, the qualification script already passes
`--enable-bedrock-native true`
- Rollback path: pass `--enable-bedrock-native true` /
`--enable-vertex-native true` (plus `--vertex-allow-non-google-upstream
true` if the upstream is not a Google host), or revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4851"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:07:49 -06:00
f5c4217648 fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description

The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.

This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.

Fix adapted from a proposal by HetCreep.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
  - the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.

**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.

## Testing

- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.

### Test Output

Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):

```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and from this branch. Upstream is a small
Python HTTP recorder on `127.0.0.1:18788` that prints each request it
receives. Requests are sent with curl to the machine's LAN address
(non-loopback peer) and to `127.0.0.1`.
- Exact command / steps: `HEADROOM_PROXY_TOKEN=s3cret
HEADROOM_PROXY_STATS=false headroom-proxy --listen 0.0.0.0:18787
--upstream http://127.0.0.1:18788`, then `curl
http://<lan-ip>:18787/admin/upstream` with no token, a wrong token and
the right token in `x-headroom-proxy-token`, `GET /readyz` with no token
from the LAN address, and `GET /v1/models` from `127.0.0.1` with no
token.
- Observed result:

```text
== origin/main
GET /admin/upstream (no token)                     -> 200
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 200
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (x3)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

== this branch
GET /admin/upstream (no token)                     -> 401
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 401
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (only the authenticated one)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

$ headroom-proxy --listen 0.0.0.0:18787 --upstream http://127.0.0.1:18788   # no token
headroom-proxy: Refusing to listen on 0.0.0.0:18787: with no HEADROOM_PROXY_TOKEN this proxy would relay every network peer to the upstream Headroom proxy from 127.0.0.1, where it is treated as a trusted local caller. Either set HEADROOM_PROXY_TOKEN ... or ... set HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1 to acknowledge the open bind.
(exit 2)
```

- Not tested: a full chain with the real Python proxy behind the Rust
front (the recorder stands in for it); WebSocket upgrades through the
gate are covered only by the layer wrapping the catch-all, not by a
dedicated test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes, default listen address is now
loopback, and a non-loopback listen needs a token or the acknowledgement
variable
- Kill switch / disable path: `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
restores starting on a non-loopback address without a token; `--listen
0.0.0.0:8787` restores the old address
- Unsafe override required: no
- Qualification impact: none (qualification scripts already pass
`--listen 127.0.0.1:<port>`)
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A companion PR makes the Rust front's native Bedrock and Vertex routes
opt-in; it relies on this token gate for callers on those routes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 09:47:48 +05:30
c8b3d67715 fix(tag-protector): make tag protection and restore linear in input size (#4841)
## Description

Tag protection (`protect_tags` / `restore_tags` in `headroom-core`) ran
in quadratic time on some text shapes, so a single large tool output
could pin a CPU for seconds to minutes before compression. Four places
were quadratic:

1. An opening tag that never closes (`<a ` repeated, or an unclosed
quote) made the attribute scan run to the end of the text, once per `<`.
2. A close tag with no matching open walked the whole open-tag stack
each time.
3. Each block-mode close filtered the whole span list.
4. Restore did `contains` + `replace` over the whole text once per
protected block.

All four are now linear (plus a binary search per attribute step).
Output is byte-identical to before.

Fix adapted from a proposal by HetCreep (the proposal covered 1 and 4;
this version keeps the old output for `<` inside an unquoted attribute
and also fixes 2 and 3).


## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `parse_tag_at` finds the end of an opening tag through `AttrScanner`:
an index of `>`, `"` and `'` offsets (built once per text, on first use)
plus a remembered outcome per stop. The walk outside quotes reaches the
same result from a given stop no matter which tag started it, so each
stop is resolved once. Self-closing is the byte before `>` being `/`,
which is what the old byte loop computed.
- `identify_spans` keeps a count of open frames per tag name, so a close
with no open of its name skips the stack walk; frames are dropped
through `pop_frames` to keep the counts right.
- Block-mode span pruning truncates the sorted span list at a partition
point instead of `retain` over all of it.
- `restore_tags` splices every placeholder in one left-to-right pass
(`splice_placeholders`), looking for the first `}}` only within the
longest placeholder's length after each `{{`. Lost-placeholder logging
is unchanged.
- Tests: time-bounded regression tests for each shape, an
exact-placeholder restore test, and two proptests that compare the new
attribute scan and restore with the old byte-by-byte versions.

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core --lib
tag_protector`)
- [x] Linting passes (`cargo clippy -p headroom-core --all-targets -- -D
warnings`, `cargo fmt`)
- [ ] Type checking passes (`mypy headroom`) — no Python changes
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

With the fix:

```text
$ cargo test -q -p headroom-core --lib tag_protector
test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 1011 filtered out; finished in 0.41s
```

The three new timing tests run against the `origin/main` code (debug
build), 5 s bound:

```text
transforms::tag_protector::tests::unclosed_openers_are_linear --- FAILED
transforms::tag_protector::tests::unmatched_closes_over_deep_stack_are_linear --- FAILED
transforms::tag_protector::tests::many_blocks_protect_and_restore_are_linear --- FAILED
<a  took 11.131358667s
deep stack took 24.821851583s
<zz></zz> took 67.701058667s
test result: FAILED. 34 passed; 3 failed; 0 ignored; 0 measured; 1011 filtered out; finished in 67.70s
```

## Real Behavior Proof

- Environment: macOS (Apple silicon), Rust stable, release build of a
scratch binary that includes the old module and the new module side by
side.
- Exact command / steps: (1) 400,000 random inputs built from `<`, `>`,
`/`, quotes, `{{`, `}}`, HTML and custom tags, non-ASCII bytes, each run
through old and new `protect_tags` in both modes and old and new
`restore_tags` (on output with stray `{{` inserted), asserting equal
results; (2) timing old vs new on large inputs.
- Observed result: `equivalence ok on 400000 cases`. Timings (release):

  | input | old | new |
  |---|---|---|
  | `"<a " * 40000` (120 KB) protect | 2.56 s | <1 ms |
  | `"<a \"" * 40000` (160 KB) protect | 2.34 s | 4 ms |
| `"<zz>" * 40000 + "</yy>" * 40000` (360 KB) protect | 19.2 s | 7 ms |
| `"<zz></zz>" * 40000` (360 KB) protect / restore | 0.72 s / 5.81 s |
14 ms / 6 ms |
  | `"<x/> filler " * 40000` (480 KB) restore | 14.9 s | 17 ms |
- Not tested: the Python extension was not rebuilt; Python callers go
through the same Rust functions via `headroom._core`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no (same output, less time)
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The timing tests use a loose 5 s bound so they stay stable on slow CI
runners; the fixed code runs each in milliseconds.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4841"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:26:18 +05:30
dependabot[bot] f0bdcfc9ca deps: bump the cargo-minor-patch group across 1 directory with 11 updates (#4860)
Bumps the cargo-minor-patch group with 11 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [aws-sigv4](https://github.com/smithy-lang/smithy-rs) | `1.6.4` |
`1.6.5` |
| [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.13.0` |
`1.13.2` |
| [aws-smithy-runtime-api](https://github.com/smithy-lang/smithy-rs) |
`1.19.2` | `1.21.0` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11`
|
| [toml](https://github.com/toml-rs/toml) | `1.1.8+spec-1.1.0` |
`1.2.0+spec-1.1.0` |
| [tree-sitter](https://github.com/tree-sitter/tree-sitter) | `0.27.1` |
`0.27.2` |
| [uuid](https://github.com/uuid-rs/uuid) | `1.29.0` | `1.29.2` |
| [crc32fast](https://github.com/srijs/rust-crc32fast) | `1.5.4` |
`1.5.5` |
| [rustls-pki-types](https://github.com/rustls/pki-types) | `1.15.1` |
`1.16.0` |
| [webpki-roots](https://github.com/rustls/webpki-roots) | `1.0.11` |
`1.0.12` |
| [cc](https://github.com/rust-lang/cc-rs) | `1.10.0` | `1.11.1` |


Updates `aws-sigv4` from 1.6.4 to 1.6.5
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/smithy-lang/smithy-rs/commits">compare
view</a></li>
</ul>
</details>
<br />

Updates `aws-config` from 1.13.0 to 1.13.2
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/smithy-lang/smithy-rs/commits">compare
view</a></li>
</ul>
</details>
<br />

Updates `aws-smithy-runtime-api` from 1.19.2 to 1.21.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md">aws-smithy-runtime-api's
changelog</a>.</em></p>
<blockquote>
<!-- raw HTML omitted -->
<h1>November 9th, 2026</h1>
<p><strong>New this release:</strong></p>
<ul>
<li>🎉 (client, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4965">smithy-rs#4965</a>)
Retry Behavior 2.1, as described in <a
href="https://github.com/awslabs/aws-sdk-rust/discussions/1431">aws-sdk-rust#1431</a>,
is now on by default. Customers who want the previous retry behavior
should pin <code>BehaviorVersion</code> to an older version, such as
<code>v2026_01_12</code>.</li>
</ul>
<h1>November 6th, 2026</h1>
<p><strong>New this release:</strong></p>
<ul>
<li>🎉 (client, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4718">smithy-rs#4718</a>,
<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4990">smithy-rs#4990</a>)
Add <code>aws_smithy_observability::unit_of_work</code>, which tells a
caller whether a response body was
validated against its checksum. Attach a <code>Capture</code> carrying a
<code>checksums::ChecksumCapture</code> with
<code>.customize().interceptor(..)</code>, read the body, then call
<code>try_outcome()</code>: it reads <code>Validated</code> with
the algorithm and computed checksum, <code>Mismatch</code> with both
checksums, <code>NotValidated</code> with a reason,
or <code>Incomplete</code>. The SDK reports through a new stable
contract in <code>aws-smithy-runtime-api</code>
(<code>client::unit_of_work::checksums</code>), so a capture keeps
working when the caller and the SDK resolve
different releases of <code>aws-smithy-observability</code>.
<code>aws-smithy-checksums</code> adds
<code>ChecksumBody::with_completion</code>.</li>
<li>🎉 (all) Add <code>OperationSchema</code> to
<code>aws-smithy-schema</code>, that relates an operation's shape
ID to its input, output, and error schemas and is built with a
<code>const</code>
<code>OperationSchema::builder</code>, so it can be declared as a
<code>static</code>. It also documents
<code>ShapeId::from_parts</code> as a supported constructor for shape
IDs that borrow
runtime-owned strings.</li>
<li>🐛 (client, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4986">smithy-rs#4986</a>,
<a href="https://github.com/xiazhvera"><code>@​xiazhvera</code></a>) Fix
<code>RequestChecksumCalculation::WhenRequired</code> ignoring a
caller-set checksum algorithm. With
<code>RequestChecksumCalculation::WhenRequired</code> and trait
<code>requestChecksumRequired: false</code>, the SDK now calculates a
request checksum when the caller sets the modeled request algorithm
member. If the member is not set, no checksum is calculated as
before.</li>
<li>🐛 (all, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/2833">smithy-rs#2833</a>,
<a href="https://github.com/Regalias3"><code>@​Regalias3</code></a>)
<code>ByteStream::from_path</code> and
<code>ByteStream::read_from()</code> now read files with a 64 KiB buffer
by default, up from 4 KiB. On a 64 MiB file this makes reads about 5x
faster (174 to 899 MiB/s in the new <code>bytestream_from_file</code>
benchmark). Each open file stream uses about 60 KiB more memory. To keep
the old behavior, call
<code>ByteStream::read_from().buffer_size(4096)</code>.</li>
<li>(all, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4916">smithy-rs#4916</a>,
<a
href="https://github.com/FalkWoldmann"><code>@​FalkWoldmann</code></a>)
Removed the <code>num-integer</code> dependency from
<code>aws-smithy-types</code>. <code>DateTime</code> now uses
<code>div_euclid</code> and <code>rem_euclid</code> from std, which give
the same results here since all divisors are positive.</li>
<li>🐛 (all, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4863">smithy-rs#4863</a>,
<a href="https://github.com/Regalias3"><code>@​Regalias3</code></a>)
<code>BigDecimal</code> now compares and hashes by numeric value instead
of by spelling. <code>&quot;1e2&quot;</code> equals
<code>&quot;100&quot;</code>, <code>&quot;1.0&quot;</code> equals
<code>&quot;1.00&quot;</code>, <code>&quot;+5&quot;</code> equals
<code>&quot;5&quot;</code>, and every spelling of zero, such as
<code>&quot;-0.0&quot;</code> and <code>&quot;0&quot;</code>, is equal.
<code>AsRef&lt;str&gt;</code> still returns the original string, so
serialization doesn't change.</li>
<li>🐛 (all, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4960">smithy-rs#4960</a>,
<a
href="https://github.com/franco2002lu"><code>@​franco2002lu</code></a>)
Fix <code>DateTime</code> epoch-seconds formatting and parsing of
timestamps before 1970 that have fractional seconds. Formatting -0.75
seconds produced <code>-1.25</code>, and parsing <code>-0.75</code>
produced 0.75. Timestamps at or after 1970 are unaffected.</li>
<li>🐛 (client, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4484">smithy-rs#4484</a>,
<a
href="https://github.com/thomasqueirozb"><code>@​thomasqueirozb</code></a>)
Clients for services with operations modeled with
<code>@requestCompression</code> (such as
<code>aws-sdk-cloudwatch</code>) no longer enable the
<code>http-body-0-4-x</code> feature of <code>aws-smithy-types</code>.
The request compression interceptor only uses <code>http-body</code>
1.x, so these clients no longer pull in <code>http</code> 0.2 and
<code>http-body</code> 0.4.</li>
</ul>
<p><strong>Contributors</strong>
Thank you for your contributions! ❤</p>
<ul>
<li><a
href="https://github.com/FalkWoldmann"><code>@​FalkWoldmann</code></a>
(<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4916">smithy-rs#4916</a>)</li>
<li><a href="https://github.com/Regalias3"><code>@​Regalias3</code></a>
(<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/2833">smithy-rs#2833</a>,
<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4863">smithy-rs#4863</a>)</li>
<li><a
href="https://github.com/franco2002lu"><code>@​franco2002lu</code></a>
(<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4960">smithy-rs#4960</a>)</li>
<li><a
href="https://github.com/thomasqueirozb"><code>@​thomasqueirozb</code></a>
(<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4484">smithy-rs#4484</a>)</li>
<li><a href="https://github.com/xiazhvera"><code>@​xiazhvera</code></a>
(<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4986">smithy-rs#4986</a>)</li>
</ul>
<h1>October 29th, 2026</h1>
<p><strong>Breaking Changes:</strong></p>
<ul>
<li>
<p>:warning:🎉 (server, <a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4894">smithy-rs#4894</a>,
<a
href="https://redirect.github.com/smithy-lang/smithy-rs/issues/4906">smithy-rs#4906</a>,
<a href="https://github.com/fahadzub"><code>@​fahadzub</code></a>)
Upgrade <code>aws-smithy-http-server</code> to 0.69.0 with schema-based
serialization and
multi-protocol routing for HTTP 1.x servers. Enable schema serde with
<code>codegen.schemaSerde: true</code> and regenerate your server SDK.
Review these migration
changes when upgrading:</p>
<ul>
<li>Protocol types such as <code>RestJson1</code> now carry
configuration. Replace unit-struct
construction with <code>RestJson1::default()</code>. For
request-specific serialization,
use the protocol selected by routing so its configuration is
preserved.</li>
</ul>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/smithy-lang/smithy-rs/commits">compare
view</a></li>
</ul>
</details>
<br />

Updates `flate2` from 1.1.10 to 1.1.11
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/flate2-rs/releases">flate2's
releases</a>.</em></p>
<blockquote>
<h2>v1.1.11</h2>
<h3>Fixed</h3>
<ul>
<li>fixup! Complete sync flushes across output buffer boundaries (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/581">#581</a>)</li>
<li>Separate gzip header reads from compressed data</li>
<li>Defer gzip input errors without losing read progress</li>
<li>Preserve gzip header progress on input read errors (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/571">#571</a>)</li>
</ul>
<h3>Other</h3>
<ul>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/582">#582</a>
from rust-lang/fix-lacking-buffer-drain</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/574">#574</a>
from Sharadhi-v98/decompress-reset-window-bits</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/577">#577</a>
from rust-lang/fix-loop</li>
<li>Setup trusted publishing for the flate2 crate</li>
<li>Complete sync flushes across output buffer boundaries (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/581">#581</a>)</li>
<li>Fix &quot;can by extended&quot; typos in deflate and zlib write
docs</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/579">#579</a>
from NUAAwqy/docs/fix-double-space</li>
<li>Fix double space in <code>read</code> module doc comment</li>
<li>Add missing preposition in README description</li>
<li>Cover decoder retries across the output buffer boundary</li>
<li>Drain output before refilling the buffer</li>
<li>Clarify partial-write buffering behavior</li>
<li>Avoid shifting buffered output on partial writes</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/567">#567</a>
from ndossche/more-asserts</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/566">#566</a>
from HDauven/reject-incomplete-write-decoder-streams</li>
<li>Fix encoder writes hanging after try_finish (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/576">#576</a>)</li>
<li>Avoid panicking when dropping unfinished compressors</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/565">#565</a>
from HDauven/zlib-rs-uninit-output</li>
<li>still allow resetting incomplete write decoders</li>
<li>Assert return result of reset calls</li>
<li>Reject incomplete streams in write decoders</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/573">#573</a>
from NUAAwqy/taiyi/fix-gzip-read-error-header</li>
<li>review</li>
<li>Reject incomplete streams when finishing write-side decoders</li>
<li>Preserve window size and gzip mode across Decompress::reset</li>
<li>preserve runtime detection in backend examples (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/570">#570</a>)</li>
<li>Make miniz_oxide dictionary clearing explicit on reset</li>
<li>Harden miniz_oxide decompressor reset</li>
<li>Deduplicate zlib-rs flush conversions</li>
<li>Use native zlib-rs uninitialized output</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md">flate2's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/rust-lang/flate2-rs/compare/v1.1.10...v1.1.11">1.1.11</a>
- 2026-11-06</h2>
<h3>Fixed</h3>
<ul>
<li>fixup! Complete sync flushes across output buffer boundaries (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/581">#581</a>)</li>
<li>Separate gzip header reads from compressed data</li>
<li>Defer gzip input errors without losing read progress</li>
<li>Preserve gzip header progress on input read errors (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/571">#571</a>)</li>
</ul>
<h3>Other</h3>
<ul>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/582">#582</a>
from rust-lang/fix-lacking-buffer-drain</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/574">#574</a>
from Sharadhi-v98/decompress-reset-window-bits</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/577">#577</a>
from rust-lang/fix-loop</li>
<li>Setup trusted publishing for the flate2 crate</li>
<li>Complete sync flushes across output buffer boundaries (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/581">#581</a>)</li>
<li>Fix &quot;can by extended&quot; typos in deflate and zlib write
docs</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/579">#579</a>
from NUAAwqy/docs/fix-double-space</li>
<li>Fix double space in <code>read</code> module doc comment</li>
<li>Add missing preposition in README description</li>
<li>Cover decoder retries across the output buffer boundary</li>
<li>Drain output before refilling the buffer</li>
<li>Clarify partial-write buffering behavior</li>
<li>Avoid shifting buffered output on partial writes</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/567">#567</a>
from ndossche/more-asserts</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/566">#566</a>
from HDauven/reject-incomplete-write-decoder-streams</li>
<li>Fix encoder writes hanging after try_finish (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/576">#576</a>)</li>
<li>Avoid panicking when dropping unfinished compressors</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/565">#565</a>
from HDauven/zlib-rs-uninit-output</li>
<li>still allow resetting incomplete write decoders</li>
<li>Assert return result of reset calls</li>
<li>Reject incomplete streams in write decoders</li>
<li>Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/573">#573</a>
from NUAAwqy/taiyi/fix-gzip-read-error-header</li>
<li>review</li>
<li>Reject incomplete streams when finishing write-side decoders</li>
<li>Preserve window size and gzip mode across Decompress::reset</li>
<li>preserve runtime detection in backend examples (<a
href="https://redirect.github.com/rust-lang/flate2-rs/pull/570">#570</a>)</li>
<li>Make miniz_oxide dictionary clearing explicit on reset</li>
<li>Harden miniz_oxide decompressor reset</li>
<li>Deduplicate zlib-rs flush conversions</li>
<li>Use native zlib-rs uninitialized output</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/cd06529164c5447958f309e649ffe6c9d6329865"><code>cd06529</code></a>
chore: release v1.1.11</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/6ff40985e5623187abd999105e3a3283cc9660d9"><code>6ff4098</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/582">#582</a>
from rust-lang/fix-lacking-buffer-drain</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/5f689134a7dbd42833d4dc16d9e242aa5caf94e9"><code>5f68913</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/574">#574</a>
from Sharadhi-v98/decompress-reset-window-bits</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/5582d951c1f50ceca448ec540b62043fe4220d86"><code>5582d95</code></a>
fixup! Complete sync flushes across output buffer boundaries (<a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/581">#581</a>)</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/d05c7c4af2879369e4cdc72827ee71216f868eb7"><code>d05c7c4</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/577">#577</a>
from rust-lang/fix-loop</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/5c7ee7a17b025a1fb4b94d4a929b71ab8f29a199"><code>5c7ee7a</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/583">#583</a>
from Kobzol/trusted-publishing</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/cc527c6b69c323e3073e2663307d5b810107c741"><code>cc527c6</code></a>
Setup trusted publishing for the flate2 crate</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/54f49cd9f4c36774f68cc6ab95edf62a3b6c506e"><code>54f49cd</code></a>
Merge pull request <a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/580">#580</a>
from NUAAwqy/fix/can-by-extended-typo</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/491df15a631ae05859245175943d518d3b03c126"><code>491df15</code></a>
Complete sync flushes across output buffer boundaries (<a
href="https://redirect.github.com/rust-lang/flate2-rs/issues/581">#581</a>)</li>
<li><a
href="https://github.com/rust-lang/flate2-rs/commit/a7a1161296ef71297abd51bc14941826db750e05"><code>a7a1161</code></a>
Fix &quot;can by extended&quot; typos in deflate and zlib write
docs</li>
<li>Additional commits viewable in <a
href="https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11">compare
view</a></li>
</ul>
</details>
<br />

Updates `toml` from 1.1.8+spec-1.1.0 to 1.2.0+spec-1.1.0
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/toml-rs/toml/commit/8ffe3c590d69fab440fedb2ebb396ae1596c3e34"><code>8ffe3c5</code></a>
chore: Release</li>
<li><a
href="https://github.com/toml-rs/toml/commit/fa5187836d077fa7d24ecd4a1e7d079ed2851357"><code>fa51878</code></a>
docs: Update changelog</li>
<li><a
href="https://github.com/toml-rs/toml/commit/f8ad91e30439d0549bd422481aa786f1239fe929"><code>f8ad91e</code></a>
perf(serde): Reuse key allocations (<a
href="https://redirect.github.com/toml-rs/toml/issues/1261">#1261</a>)</li>
<li><a
href="https://github.com/toml-rs/toml/commit/fe514677a7ee2bd2a5b9d29c93bb9b046eba0787"><code>fe51467</code></a>
perf(serde): Reuse key allocations</li>
<li><a
href="https://github.com/toml-rs/toml/commit/70243195af074b6561ee37f852a36921ad3f5225"><code>7024319</code></a>
perf(writer): Expand coverage of the fast path (<a
href="https://redirect.github.com/toml-rs/toml/issues/1259">#1259</a>)</li>
<li><a
href="https://github.com/toml-rs/toml/commit/be196a83fe04414306d4881866bd12c77768d9a1"><code>be196a8</code></a>
perf(writer): Expand coverage of the fast path</li>
<li><a
href="https://github.com/toml-rs/toml/commit/932704cde52ccd986a4f342f4b842fee14dba075"><code>932704c</code></a>
perf(serde): Pre-allocate serialization buffers (<a
href="https://redirect.github.com/toml-rs/toml/issues/1258">#1258</a>)</li>
<li><a
href="https://github.com/toml-rs/toml/commit/52b563d8d6a257189ef3656ab1a8d9a8e15f61b3"><code>52b563d</code></a>
perf(serde): Pre-allocate serialization buffers</li>
<li><a
href="https://github.com/toml-rs/toml/commit/e346a7955749cc7f1b141873d0695cf82ebf9ad2"><code>e346a79</code></a>
perff(writer): Add key and string fast paths (<a
href="https://redirect.github.com/toml-rs/toml/issues/1257">#1257</a>)</li>
<li><a
href="https://github.com/toml-rs/toml/commit/ab695e6bab7e16fabb89f9624153b3d83ba361e1"><code>ab695e6</code></a>
perf(writer): Add a fast path for writing string types</li>
<li>Additional commits viewable in <a
href="https://github.com/toml-rs/toml/compare/toml-v1.1.8...toml-v1.2.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `tree-sitter` from 0.27.1 to 0.27.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/tree-sitter/tree-sitter/releases">tree-sitter's
releases</a>.</em></p>
<blockquote>
<h2>v0.27.2</h2>
<h2>What's Changed</h2>
<ul>
<li>fix(cli): detect corpus parse errors from the tree by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6030">tree-sitter/tree-sitter#6030</a></li>
<li>fix(lib): only match the whole name <code>ERROR</code> in
<code>ts_language_symbol_for_name</code> by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6031">tree-sitter/tree-sitter#6031</a></li>
<li>fix(wasm): copy all supertype map entries by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6036">tree-sitter/tree-sitter#6036</a></li>
<li>fix: Windows symbol visibility backport by <a
href="https://github.com/WillLillis"><code>@​WillLillis</code></a> in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6051">tree-sitter/tree-sitter#6051</a></li>
<li>fix(lib): handle terminal ERROR nodes in
<code>ts_node_has_error</code> by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6058">tree-sitter/tree-sitter#6058</a></li>
<li>fix(web): various issues by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6059">tree-sitter/tree-sitter#6059</a></li>
<li>fix(lib): reject symbol names with embedded <code>NULL</code>
suffixes by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6073">tree-sitter/tree-sitter#6073</a></li>
<li>fix(lib): handle large child indices in reverse cursor traversal by
<a href="https://github.com/WillLillis"><code>@​WillLillis</code></a> in
<a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6075">tree-sitter/tree-sitter#6075</a></li>
<li>fix(rust): address new <code>unnecessary-box-pin</code> lint by <a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6089">tree-sitter/tree-sitter#6089</a></li>
<li>fix(generate): preserve reserved-word behavior when merging states
by <a href="https://github.com/WillLillis"><code>@​WillLillis</code></a>
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6094">tree-sitter/tree-sitter#6094</a></li>
<li>fix(make): don't remove tree-sitter dir on <code>uninstall</code> by
<a
href="https://github.com/tree-sitter-ci-bot"><code>@​tree-sitter-ci-bot</code></a>[bot]
in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6098">tree-sitter/tree-sitter#6098</a></li>
<li>backport zig by <a
href="https://github.com/clason"><code>@​clason</code></a> in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6106">tree-sitter/tree-sitter#6106</a></li>
<li>build(deps): bump wasmtime-c-api to v48.0.8 by <a
href="https://github.com/clason"><code>@​clason</code></a> in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6107">tree-sitter/tree-sitter#6107</a></li>
<li>release v0.27.2 by <a
href="https://github.com/clason"><code>@​clason</code></a> in <a
href="https://redirect.github.com/tree-sitter/tree-sitter/pull/6108">tree-sitter/tree-sitter#6108</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/tree-sitter/tree-sitter/compare/v0.27.1...v0.27.2">https://github.com/tree-sitter/tree-sitter/compare/v0.27.1...v0.27.2</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/3fb7be49145acfb07e33b61ed5689e2b6771b2e9"><code>3fb7be4</code></a>
release v0.27.2</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/a7fd3a50930cd3fe5bc12a82655dff8745bcbe6c"><code>a7fd3a5</code></a>
build(deps): bump wasmtime-c-api to v48.0.8</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/270e8855063183069acec3f7ac208b8488b137f1"><code>270e885</code></a>
fix(xtask): use zig fetch --save to update wasmtime in
build.zig.zon</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/292f96375e94f5a7ff739824c9e2413e57bcdf2e"><code>292f963</code></a>
build(zig): support Zig 0.17</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/f4739f941b55b5f72df5d0288e7b70032951bb01"><code>f4739f9</code></a>
fix(make): don't remove tree-sitter dir on <code>uninstall</code></li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/df4c34e969059481b249bc56272c270965cd1ecf"><code>df4c34e</code></a>
fix(generate): preserve reserved-word behavior when merging states</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/d0051f83bd4a7e7f1b7c5b65c24f0ff4d57a2608"><code>d0051f8</code></a>
fix(rust): address new <code>unnecessary-box-pin</code> lint</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/c2a14c724a4935849b1ae7f875e09717729b2399"><code>c2a14c7</code></a>
fix(lib): retain siblings during byte-based child lookup</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/cfebeaa2a8a80ae0490dbe1e64569128b4851495"><code>cfebeaa</code></a>
fix(lib): preserve descendant indices during reverse cursor
traversal</li>
<li><a
href="https://github.com/tree-sitter/tree-sitter/commit/bbc0e28bf9db9cbe35b894966efc1a0846e90d13"><code>bbc0e28</code></a>
fix(lib): handle large child indices in reverse cursor traversal</li>
<li>Additional commits viewable in <a
href="https://github.com/tree-sitter/tree-sitter/compare/v0.27.1...v0.27.2">compare
view</a></li>
</ul>
</details>
<br />

Updates `uuid` from 1.29.0 to 1.29.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/uuid-rs/uuid/releases">uuid's
releases</a>.</em></p>
<blockquote>
<h2>v1.29.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Support wasm64 in the js feature by <a
href="https://github.com/nickbabcock"><code>@​nickbabcock</code></a> in
<a
href="https://redirect.github.com/uuid-rs/uuid/pull/928">uuid-rs/uuid#928</a></li>
<li>Prepare for 1.29.2 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/929">uuid-rs/uuid#929</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a
href="https://github.com/nickbabcock"><code>@​nickbabcock</code></a>
made their first contribution in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/928">uuid-rs/uuid#928</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.29.1...v1.29.2">https://github.com/uuid-rs/uuid/compare/v1.29.1...v1.29.2</a></p>
<h2>v1.29.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Remove some deprecated attrs and resolve Cargo lints by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/921">uuid-rs/uuid#921</a></li>
<li>Deprecate the <code>borsh</code> feature of <code>uuid</code> in
favour of a <code>uuid</code> feature in <code>borsh</code> by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/923">uuid-rs/uuid#923</a></li>
<li>Fix up several bugs in v7 monotonicity raised by automated scanning
by <a href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/926">uuid-rs/uuid#926</a></li>
<li>Prepare for 1.29.1 release by <a
href="https://github.com/KodrAus"><code>@​KodrAus</code></a> in <a
href="https://redirect.github.com/uuid-rs/uuid/pull/927">uuid-rs/uuid#927</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/uuid-rs/uuid/compare/v1.29.0...v1.29.1">https://github.com/uuid-rs/uuid/compare/v1.29.0...v1.29.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/uuid-rs/uuid/commit/5205a6ea18425e6781f2ac09e896d88153d9cd2d"><code>5205a6e</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/929">#929</a> from
uuid-rs/cargo/v1.29.2</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/ff39846f1e065b5428fb4e893f2b7a4c4ce139bc"><code>ff39846</code></a>
prepare for 1.29.2 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/c879f612c2841f65b768db857e832a4f44d58dfc"><code>c879f61</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/928">#928</a> from
nickbabcock/wasm64</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/4b578527f9aaccc971d890bebe1f35056c726a57"><code>4b57852</code></a>
Support wasm64 in the js feature</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/0fe2dab29a99d0379c08e7b8ba49d89169f994bf"><code>0fe2dab</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/927">#927</a> from
uuid-rs/cargo/v1.29.1</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/8896d2e0f12c41fae80ac5aea96541f0f7a9e65c"><code>8896d2e</code></a>
prepare for 1.29.1 release</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/2eef37094016e6315ff46d947d7f7b91d7004a67"><code>2eef370</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/926">#926</a> from
uuid-rs/fix/timestamp-edges</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/de33fde2c6764022f6bdf8a574f5c55a1518fa87"><code>de33fde</code></a>
fix up several bugs in v7 monotonicity raised by automated scanning</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/7c7326d55fd502d890169100f64c941b753500da"><code>7c7326d</code></a>
Merge pull request <a
href="https://redirect.github.com/uuid-rs/uuid/issues/923">#923</a> from
uuid-rs/fix/deprecate-borsh</li>
<li><a
href="https://github.com/uuid-rs/uuid/commit/13e1ff03bd5fef0df42243be4c22f397b2816923"><code>13e1ff0</code></a>
deprecate the <code>borsh</code> feature of <code>uuid</code> in favour
of a <code>uuid</code> feature in <code>borsh</code></li>
<li>Additional commits viewable in <a
href="https://github.com/uuid-rs/uuid/compare/v1.29.0...v1.29.2">compare
view</a></li>
</ul>
</details>
<br />

Updates `crc32fast` from 1.5.4 to 1.5.5
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/srijs/rust-crc32fast/commit/8a93d2a70c00ab3e72beda42014b18fad8e496ea"><code>8a93d2a</code></a>
Release 1.5.5</li>
<li><a
href="https://github.com/srijs/rust-crc32fast/commit/f81fd96dba5cef51175a620dfdf9b466be62bba9"><code>f81fd96</code></a>
CI: check the aarch64 target at the MSRV and on 1.80 (<a
href="https://redirect.github.com/srijs/rust-crc32fast/issues/80">#80</a>)</li>
<li><a
href="https://github.com/srijs/rust-crc32fast/commit/0b0c895b2ff244bcc85d8977034851cbd1279c53"><code>0b0c895</code></a>
CI: check aarch64 kernel detection on known runners (<a
href="https://redirect.github.com/srijs/rust-crc32fast/issues/79">#79</a>)</li>
<li><a
href="https://github.com/srijs/rust-crc32fast/commit/548781c99892c50b0ba239c28e240b8d15688889"><code>548781c</code></a>
aarch64: fold with PMULL from 64 bytes (<a
href="https://redirect.github.com/srijs/rust-crc32fast/issues/78">#78</a>)</li>
<li>See full diff in <a
href="https://github.com/srijs/rust-crc32fast/compare/v1.5.4...v1.5.5">compare
view</a></li>
</ul>
</details>
<br />

Updates `rustls-pki-types` from 1.15.1 to 1.16.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rustls/pki-types/releases">rustls-pki-types's
releases</a>.</em></p>
<blockquote>
<h2>1.16.0</h2>
<ul>
<li><strong>Behavior change</strong> for edge case inputs:
<code>DnsName</code> now rejects names whose final label is a hex
number, like <code>0x7f000001</code> or <code>example.0x1f</code>.</li>
<li><strong>Behaviour change</strong>: <code>DnsName</code> and
<code>ServerName</code> now strip a trailing dot to canonicalize
names.</li>
<li><strong>New API</strong>: <code>DnsName::into_owned()</code> which
can avoid copies.</li>
<li>Faster <code>DnsName</code> validation.</li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>chore: replace datatracker references, update I-D -&gt; RFC9881 by
<a href="https://github.com/mcr"><code>@​mcr</code></a> in <a
href="https://redirect.github.com/rustls/pki-types/pull/115">rustls/pki-types#115</a></li>
<li>Introduce <code>DnsName::into_owned()</code> by <a
href="https://github.com/ctz"><code>@​ctz</code></a> in <a
href="https://redirect.github.com/rustls/pki-types/pull/116">rustls/pki-types#116</a></li>
<li>Correct zero-check helper name and documentation by <a
href="https://github.com/eliask"><code>@​eliask</code></a> in <a
href="https://redirect.github.com/rustls/pki-types/pull/118">rustls/pki-types#118</a></li>
<li>pem: state correct 256 MB limit in the SectionTooLarge error message
by <a href="https://github.com/herniqeu"><code>@​herniqeu</code></a> in
<a
href="https://redirect.github.com/rustls/pki-types/pull/119">rustls/pki-types#119</a></li>
<li>1.16.0: stricter and canonicalising <code>DnsName</code> by <a
href="https://github.com/ctz"><code>@​ctz</code></a> in <a
href="https://redirect.github.com/rustls/pki-types/pull/117">rustls/pki-types#117</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/mcr"><code>@​mcr</code></a> made their
first contribution in <a
href="https://redirect.github.com/rustls/pki-types/pull/115">rustls/pki-types#115</a></li>
<li><a href="https://github.com/eliask"><code>@​eliask</code></a> made
their first contribution in <a
href="https://redirect.github.com/rustls/pki-types/pull/118">rustls/pki-types#118</a></li>
<li><a href="https://github.com/herniqeu"><code>@​herniqeu</code></a>
made their first contribution in <a
href="https://redirect.github.com/rustls/pki-types/pull/119">rustls/pki-types#119</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rustls/pki-types/compare/v/1.15.1...v/1.16.0">https://github.com/rustls/pki-types/compare/v/1.15.1...v/1.16.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rustls/pki-types/commit/26de75cf1eab9143c5a97faa1effca237b401b86"><code>26de75c</code></a>
Prepare 1.16.0</li>
<li><a
href="https://github.com/rustls/pki-types/commit/3b1c732aec13b36a0bbef8ba07478ad5068b78c0"><code>3b1c732</code></a>
Test <code>DnsName</code> equality and hashing</li>
<li><a
href="https://github.com/rustls/pki-types/commit/1f931ad378e60489bc76365c74969a84b9fa333e"><code>1f931ad</code></a>
Strip a single trailing dot in <code>DnsName</code> ctors</li>
<li><a
href="https://github.com/rustls/pki-types/commit/192942a48a60a8764dc54d6e5155cbf3a3ffeaba"><code>192942a</code></a>
<code>DnsName</code> parsing rejects confusables with IPv4 addrs</li>
<li><a
href="https://github.com/rustls/pki-types/commit/d6981e74a637475d9162adbe71c230244ebe4a32"><code>d6981e7</code></a>
pem: state correct 256 MB limit in SectionTooLarge error message</li>
<li><a
href="https://github.com/rustls/pki-types/commit/481303f052129a0837e151e303957f7d56b8034c"><code>481303f</code></a>
Correct zero-check helper name and documentation</li>
<li><a
href="https://github.com/rustls/pki-types/commit/370d0a783348e99e4c43051459c25bd0d861c071"><code>370d0a7</code></a>
Introduce <code>DnsName::into_owned()</code></li>
<li><a
href="https://github.com/rustls/pki-types/commit/8b99ee82fafa5ce08f7cf1798f6df771c3228ef6"><code>8b99ee8</code></a>
Fix new clippy lints</li>
<li><a
href="https://github.com/rustls/pki-types/commit/024dc7f139c4ab4e1e2b6f4e97430a39e0dc2405"><code>024dc7f</code></a>
chore: replace datatracker references, update I-D -&gt; RFC9881</li>
<li>See full diff in <a
href="https://github.com/rustls/pki-types/compare/v/1.15.1...v/1.16.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `webpki-roots` from 1.0.11 to 1.0.12
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rustls/webpki-roots/releases">webpki-roots's
releases</a>.</em></p>
<blockquote>
<h2>1.0.12</h2>
<ul>
<li>
<p>Remove &quot;Buypass Class 2 Root CA&quot; and &quot;Buypass Class 3
Root CA&quot;. Reference: <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=2085917">https://bugzilla.mozilla.org/show_bug.cgi?id=2085917</a></p>
</li>
<li>
<p>Remove &quot;DigiCert Trusted Root G4&quot;, &quot;QuoVadis Root CA 1
G3&quot; and &quot;QuoVadis Root CA 3 G3&quot; (as they are now trusted
only for email). Reference: <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=2088266">https://bugzilla.mozilla.org/show_bug.cgi?id=2088266</a></p>
</li>
<li>
<p>Remove &quot;Certigna Root CA&quot;. Reference: <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=2085928">https://bugzilla.mozilla.org/show_bug.cgi?id=2085928</a></p>
</li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>ccadb: skip empty included subtrees by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/138">rustls/webpki-roots#138</a></li>
<li>1.0.12: track roots removed upstream by <a
href="https://github.com/ctz"><code>@​ctz</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/139">rustls/webpki-roots#139</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rustls/webpki-roots/compare/v/1.0.11...v/1.0.12">https://github.com/rustls/webpki-roots/compare/v/1.0.11...v/1.0.12</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rustls/webpki-roots/commit/b5d2438fce188490ae60193c187a3b0b02f61d54"><code>b5d2438</code></a>
1.0.12: track roots removed upstream</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/190e6f315317ce1b4e5dfd2bc3201a02e51f5271"><code>190e6f3</code></a>
ccadb: bump version to 0.2.3</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/acf073fd3202f8ca5674d72b35c31ec93d9e04ef"><code>acf073f</code></a>
ccadb: skip empty included subtrees</li>
<li>See full diff in <a
href="https://github.com/rustls/webpki-roots/compare/v/1.0.11...v/1.0.12">compare
view</a></li>
</ul>
</details>
<br />

Updates `cc` from 1.10.0 to 1.11.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/cc-rs/releases">cc's
releases</a>.</em></p>
<blockquote>
<h2>cc-v1.11.1</h2>
<h3>Fixed</h3>
<ul>
<li><em>(target-parsing)</em> wasm32-none-component (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2056">#2056</a>)</li>
</ul>
<h2>cc-v1.11.0</h2>
<h3>Added</h3>
<ul>
<li>add reproducible CodeView and source path remapping (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2071">#2071</a>)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>pass <code>.cc</code> sources to cl.exe as one
<code>-Tp&lt;path&gt;</code> argument (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2063">#2063</a>)</li>
<li>pass -march=x86-64 to GCC on x86_64 Linux (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2061">#2061</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/cc-rs/blob/main/CHANGELOG.md">cc's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/rust-lang/cc-rs/compare/cc-v1.11.0...cc-v1.11.1">1.11.1</a>
- 2026-11-06</h2>
<h3>Fixed</h3>
<ul>
<li><em>(target-parsing)</em> wasm32-none-component (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2056">#2056</a>)</li>
</ul>
<h2><a
href="https://github.com/rust-lang/cc-rs/compare/cc-v1.10.0...cc-v1.11.0">1.11.0</a>
- 2026-10-30</h2>
<h3>Added</h3>
<ul>
<li>add reproducible CodeView and source path remapping (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2071">#2071</a>)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>pass <code>.cc</code> sources to cl.exe as one
<code>-Tp&lt;path&gt;</code> argument (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2063">#2063</a>)</li>
<li>pass -march=x86-64 to GCC on x86_64 Linux (<a
href="https://redirect.github.com/rust-lang/cc-rs/pull/2061">#2061</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/401b5a63050f33d86e58b2e303c1e7b0a1e8ff19"><code>401b5a6</code></a>
chore(cc): release v1.11.1 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2087">#2087</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/f8de72570114e95407449a3609999873f148367f"><code>f8de725</code></a>
ci(deps): bump tombi-toml/setup-tombi from 1.9.7 to 1.9.8 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2085">#2085</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/38448675ac25b4ff38b90b4d16492f40b242c1a4"><code>3844867</code></a>
ci(deps): bump tombi-toml/setup-tombi from 1.9.6 to 1.9.7 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2083">#2083</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/3a0f2a34d71501d5d1aba1a60beec34da87a13ea"><code>3a0f2a3</code></a>
ci(deps): bump release-plz/action from 0.5.146 to 0.5.147 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2081">#2081</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/568f5e133597c2e5874c31b4fb877a4ac254d121"><code>568f5e1</code></a>
chore(ci): Install cuda for ubuntu-26.04 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2082">#2082</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/499d43eadfe485218bb82384112b1516dc469258"><code>499d43e</code></a>
fix(target-parsing): wasm32-none-component (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2056">#2056</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/995643d7e9d672e8e5ec9fef2767c8e37fb5b20f"><code>995643d</code></a>
ci(deps): bump tombi-toml/setup-tombi from 1.9.1 to 1.9.6 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2077">#2077</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/0e224ff8286c688db006c5a4a8c994ff3c31c12d"><code>0e224ff</code></a>
ci(deps): bump release-plz/action from 0.5.145 to 0.5.146 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2079">#2079</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/67e9f6d1824a2b1d5b3a21f71e23f5510c669a41"><code>67e9f6d</code></a>
ci(deps): update rusqlite requirement from 0.40.0 to 0.41.0 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2080">#2080</a>)</li>
<li><a
href="https://github.com/rust-lang/cc-rs/commit/fe52971c86218b0b1b7d747b67c1c515607a2d6d"><code>fe52971</code></a>
chore(cc): release v1.11.0 (<a
href="https://redirect.github.com/rust-lang/cc-rs/issues/2076">#2076</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/rust-lang/cc-rs/compare/cc-v1.10.0...cc-v1.11.1">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4860"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-13 20:30:53 -06:00
cherylandClaude e6e2ebb678 fix(smart-crusher): keep the row the question names (#4713)
## Description

Ask "Who opened issue #4475?" over a 100-issue `gh issue list` result,
and SmartCrusher cuts row #4475. `prioritize_indices` fills the budget
with errors, outliers and first/last rows, then fills by position, so
rows matching a query anchor (ID, UUID, email, hostname, quoted string)
come last. `plan_top_n` already keeps anchor matches unconditionally,
and the docs list them as preserved. This makes `smart_sample`,
`cluster_sample` and `time_series` do the same.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `apply_query_signals` returns the anchor matches; `prioritize_indices`
keeps them with the critical set.
- If an anchor matches more rows than the whole budget, it isn't
selective, so those rows compete as before. Like errors, matches can
push the result over `max_items`, by at most `max_items` rows.
- BM25 matches are unchanged. Anchor matching is the existing
`item_matches_anchors`, unchanged here.
- The `planning.rs` call sites only pass the new set; the extra lines
are rustfmt.

## Testing

- [x] New tests: errors fill the budget, no error rows, anchor matches
more rows than the budget, end to end through `crush_array`. Each fails
without the fix; `cargo mutants --in-diff`: 10/10 caught.
- [x] `cargo test --workspace`, `cargo fmt --check`, `cargo clippy
--workspace -D warnings` (1.95.0), `make test-parity` (smart_crusher
17/17). Fork CI: green.

### Test Output

```
$ cargo test -p headroom-core --lib smart_crusher
test transforms::smart_crusher::orchestration::tests::prioritize_keeps_query_match_when_errors_fill_budget ... ok
test transforms::smart_crusher::orchestration::tests::prioritize_keeps_query_match_without_error_rows ... ok
test transforms::smart_crusher::orchestration::tests::prioritize_ignores_anchor_matching_more_than_budget ... ok
test transforms::smart_crusher::crusher::tests::query_named_row_survives_lossy_crush ... ok
test result: ok. 354 passed; 0 failed; 0 ignored; 0 measured; 689 filtered out
```

## Real Behavior Proof

- Environment: macOS arm64, `compress()` from the README (0.43.1 vs this
branch built from source), gpt-oss-20b local, temperature 0.
- Exact command / steps: 12 random issues, "Who opened issue #N, and is
it open or closed?" Scripts and answers:
https://github.com/cherylaurelia/headroom-demo/tree/main/bench
- Observed result: 0.43.1 kept 2 of the 12 asked rows (2/12 correct);
this branch keeps 12/12 (10/12 correct; the 2 misses had the row in
context). About 90 more prompt tokens.
- Not tested: rows named only by title words can still be cut.

Through `headroom proxy`, Claude Code also needs
`fix/toolsearch-query-note` (its relevance query is "Tool loaded."):
this fix alone 2/12, both 12/12.

## Runtime Rollout Safety

- Rollout-managed feature(s): n/a
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, one more kept row when the
question names it
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I did **not** edit `CHANGELOG.md`

Written with Claude Code, which also ran the tests and measurements
above.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4713"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-09 14:54:45 -06:00
HetCreepandClaude Opus 5.5 5438f0035d fix(proxy): keep the Bedrock credentials provider when startup resolution fails (#4705)
## Description

The startup half of #4684, which #4696 left open ("Startup
credential-resolution failure recovery remains open on that issue").

`load_bedrock_credentials` (`crates/headroom-proxy/src/main.rs:187` on
`bb8d18a5`) built the `BedrockCredentials` source and then returned
`Err` if the first `get()` failed, so `main` never attached the source.
For the rest of the process every Bedrock request answered `500
bedrock_credentials_missing`, even after credentials became available
(an ECS task role or IMDS that was not ready yet, an SSO login done
after the proxy started), until a restart.

With this PR the loader only fails when the default chain has no
provider at all. `main` resolves once to report the state at startup,
and attaches the source whether or not that first resolution worked.
`BedrockCredentials::get` does not cache a failure, so each Bedrock
request asks the provider again until it succeeds, then the #4696
refresh-ahead cache takes over. Requests are still never forwarded
unsigned; until credentials arrive they get `500
bedrock_credentials_refresh_failed` instead of
`bedrock_credentials_missing`.

Fixes #4684

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-proxy/src/main.rs`: `load_bedrock_credentials`
returns the source without resolving; the startup block resolves once,
attaches the source either way, and logs `bedrock_credentials_loaded` or
`bedrock_credentials_unavailable` (message now says the provider is
asked again on each Bedrock request). The no-provider case is unchanged.
- `crates/headroom-proxy/src/bedrock/credentials.rs`: test
`failed_fetch_is_not_cached` (a provider that fails, then succeeds: the
second `get()` returns the credentials and the provider is called twice
in total). It pins the property `main.rs` now relies on.
-
`crates/headroom-proxy/tests/integration_bedrock_startup_credentials.rs`:
runs the real binary (`CARGO_BIN_EXE_headroom-proxy`), because the
startup decision lives in `main.rs` and the in-process
`start_proxy_with_state` harness never runs it. Credentials come from a
wiremock ECS container endpoint (`AWS_CONTAINER_CREDENTIALS_FULL_URI`)
that answers 503 until switched on; all other `AWS_*` variables are
removed and IMDS is disabled. The first invoke must get 500; after the
switch the second must get 200 and reach the wiremock upstream signed
with the late key. The proxy is killed and its temp file removed on
drop.
- `crates/headroom-proxy/src/bedrock/invoke.rs`: module doc for the
refresh-failure case also names "none resolved yet since startup".

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

This is a Rust-only change, so the Python checks above do not apply; the
Rust equivalents were run (Windows 11, Rust 1.95.0).

### Test Output

```text
cargo test -p headroom-proxy --lib --locked bedrock::credentials
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 349 filtered out

cargo fmt -p headroom-proxy -- --check                               -> exit 0
cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings -> exit 0
```

Full crate suite (`cargo test -p headroom-proxy --locked
--no-fail-fast`, Windows): 585 passed, 1 failed. The one failure is
`integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix`
(`Os { code: 10054, kind: ConnectionReset }`), a Windows-only failure in
a test this change does not touch; it fails the same way in #4694's
Windows run.

Fail before (`main.rs` from `bb8d18a5`, new integration test kept;
`cargo test -p headroom-proxy --locked --test
integration_bedrock_startup_credentials`):

```text
assertion `left == right` failed: credentials served now: {"error":{"type":"bedrock_credentials_missing","message":"AWS credentials not configured; refusing to forward unsigned"}}
  left: 500
 right: 200
test result: FAILED. 0 passed; 1 failed
```

Pass after:

```text
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
```

The unit test `failed_fetch_is_not_cached` passes on `main` too (the
cache already did not store failures); it pins the property the fix
relies on. The integration test above is the fail-before.

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0, debug builds of the
`headroom-proxy` binary at `bb8d18a5` (control) and at this branch,
offline.
- Exact command / steps: a local ECS container-credentials endpoint
(`AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:<port>/creds`)
that answers 503 until a flag file exists and then returns test
credentials (`AKIDPROOF4684`, 1 h expiry); empty `AWS_CONFIG_FILE` /
`AWS_SHARED_CREDENTIALS_FILE`, `AWS_EC2_METADATA_DISABLED=true`, no
other `AWS_*` variables. Proxy: `headroom-proxy --listen 127.0.0.1:<p>
--upstream http://127.0.0.1:1 --enable-bedrock-native true
--bedrock-endpoint http://127.0.0.1:<recorder>`, where the recorder
stores each request's `Authorization` header and answers 200. Request A:
`POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke` before the
flag exists. Then create the flag. Request B: the same request.
- Observed result: on `bb8d18a5` request B, sent after the credentials
endpoint recovered, still got 500 `bedrock_credentials_missing` with 0
upstream hits; on this branch it got 200 and reached the upstream signed
with `AKIDPROOF4684`. Details:

  | | `bb8d18a5` | this branch |
  |---|---|---|
| startup log | `bedrock_credentials_unavailable` |
`bedrock_credentials_unavailable` |
| A (no credentials yet) | 500 `bedrock_credentials_missing` | 500
`bedrock_credentials_refresh_failed` |
| B (credentials now served) | 500 `bedrock_credentials_missing`, 0
upstream hits | 200, 1 upstream hit, `Authorization: AWS4-HMAC-SHA256
Credential=AKIDPROOF4684/.../us-east-1/bedrock/aws4_request`, log
`bedrock_credentials_refreshed` |

- Not tested: real AWS (IMDS, ECS, SSO); the streaming route over HTTP
(it reads the same `bedrock_credentials` source); load. A profile-based
source that is broken at startup does not recover with this change: in a
first attempt with a `credential_process` profile, the SDK's profile
provider failed to build (this crate does not enable the
`credentials-process` feature) and then answered "profile file
credentials provider initialization error already taken" on later calls,
so the SDK keeps that failure itself. Container and IMDS sources are
asked again on each call.

## Runtime Rollout Safety

- Rollout-managed feature(s): none changed; the existing
`native_bedrock` route only.
- Minimum rollout channel: n/a (no new feature)
- Stable/default behavior changed: yes, narrowly: when credentials are
unavailable at startup, Bedrock requests now ask the provider again on
each request (and succeed once credentials appear) instead of failing
until restart; the error code while unavailable changes from
`bedrock_credentials_missing` to `bedrock_credentials_refresh_failed`.
When credentials resolve at startup nothing changes.
- Kill switch / disable path: none added; disabling `native_bedrock`
removes the route as before.
- Unsafe override required: no
- Qualification impact: none expected; a failed resolution never
forwards an unsigned request.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (see
Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- While credentials are unavailable, each Bedrock request runs one
provider attempt (for IMDS or a container endpoint, an HTTP call with
the SDK's own timeouts) before it gets the 500. That is the cost of
recovering without a restart; a maintainer may prefer a backoff or a
background retry instead. The issue's own suggestion was "Retrying the
initial load would also cover the startup-failure case".

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4705"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:32:18 -06:00
HetCreepandClaude Opus 5.5 d70a8b76a5 fix(proxy): bound the Bedrock Prometheus model label (#4694)
## Description

The Bedrock handlers in `headroom-proxy` use the `{model_id}` URL path
segment, which the client controls, as the `model` label of
`bedrock_invoke_count_total`, `bedrock_invoke_latency_seconds` and
`bedrock_eventstream_message_count_total`. Every distinct value created
a new series that stayed in the registry until exit. This clips the
label to 128 characters and folds values past 200 distinct ids into
`other`, inside the three `record_*`/`observe_*` helpers in
`observability/prometheus.rs`, so the callers in `bedrock/invoke.rs` and
`bedrock/invoke_streaming.rs` are covered without being touched. The
limits are the ones `headroom/proxy/persistent_metrics.py` already uses
(`MAX_LABEL_LENGTH = 128`, `MAX_TRACKED_MODELS = 200`).

Fixes #4685

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-proxy/src/observability/prometheus.rs`: add
`bound_label` (clip + fold past a cap) and `bounded_model` (one
process-wide set shared by the three families), and call it at the top
of `record_bedrock_invoke`, `observe_bedrock_invoke_latency` and
`record_bedrock_eventstream_message`.
- Three new unit tests in the same file: many distinct ids stay at most
cap + 1 on all three families, a long id is clipped, and `bound_label`
keeps normal ids exact and folds only past the cap.
- Left out: the issue also suggests recording after validation; that is
a handler-order change and was not needed once the helpers bound the
label.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

This is a Rust-only change, so the Python checks above do not apply; the
Rust equivalents (`cargo test`, `cargo fmt --check`, `cargo clippy -D
warnings`) were run.

### Test Output

Fail before (the three call-site lines reverted, tests kept; `cargo test
-p headroom-proxy --locked --lib observability::prometheus`):

```text
test observability::prometheus::tests::long_model_id_is_clipped ... FAILED
test observability::prometheus::tests::many_distinct_model_ids_stay_bounded_across_all_families ... FAILED
assertion failed: labels.contains(&"x".repeat(MAX_MODEL_LABEL_LEN))
bedrock_invoke_count_total: 300 distinct model labels
test result: FAILED. 5 passed; 2 failed; 0 ignored; 0 measured; 345 filtered out
```

Pass after:

```text
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 345 filtered out; finished in 0.28s
```

Full crate suite (`cargo test -p headroom-proxy --locked
--no-fail-fast`, Windows): 579 passed, 1 failed. The one failure is
`integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix`
(`Os { code: 10054, kind: ConnectionReset }`), a Windows-only failure
that is not touched by this change; it was not re-run on a clean
checkout of main in this session.

```text
cargo fmt -p headroom-proxy -- --check   -> exit 0
cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings   -> exit 0
```

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0, debug build of the
`headroom-proxy` binary at this commit, listening on 127.0.0.1:18787;
`AWS_EC2_METADATA_DISABLED=true` and no AWS credentials, so Bedrock
requests are unsigned and the closed upstream (127.0.0.1:1) refuses
them; the metric is recorded at handler entry, before the upstream call.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:1 --enable-bedrock-native true
--bedrock-endpoint http://127.0.0.1:1`; then `curl -X POST -d '{}'
http://127.0.0.1:18787/model/proof-model-$i/invoke` for i = 1..300, one
300-character model id, and `anthropic.claude-3-haiku-20240307-v1:0`;
then `GET /metrics` and count the distinct `model` values per family.
- Observed result: 201 distinct `model` values on
`bedrock_invoke_count_total` (200 ids + `other`), 201 on
`bedrock_invoke_latency_seconds_count`;
`bedrock_invoke_count_total{auth_mode="oauth",model="other",region="us-east-1"}
102` (the 100 ids past the cap, plus the 300-character id and the normal
id, which were also sent after the cap was reached); longest label seen
is 15 characters. Before the change the same unit-level flood produced
300 distinct labels.
- Not tested: a control run of the unfixed binary through the same HTTP
steps (the before-state is shown by the failing unit test only); the
streaming route and `bedrock_eventstream_message_count_total` over HTTP
(covered by the unit test, which drives the helper directly); memory
growth or scrape time under load; clipping over HTTP (the 300-character
id arrived after the cap, so it was folded; clipping is covered by
`long_model_id_is_clipped`); a real AWS upstream.

## Runtime Rollout Safety

- Rollout-managed feature(s): none changed; the code sits on the
existing `native_bedrock` route, which this PR does not gate or alter.
- Minimum rollout channel: n/a (no new feature)
- Stable/default behavior changed: yes, narrowly: after 200 distinct
model ids since process start, further new ids appear in the three
Bedrock metrics as `model="other"`, and ids longer than 128 characters
are truncated. Ids seen earlier keep their exact label.
- Kill switch / disable path: none added; it is label bounding only.
Disabling `native_bedrock` removes the metrics' source as before.
- Unsafe override required: no
- Qualification impact: none expected; no request or response path
changes, only metric label values.
- Rollback path: revert this commit; the series count returns to
unbounded growth.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (one
pre-existing Windows-only failure, see Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: not changed; the module header comment in
`prometheus.rs` still describes the label set, and the new constants
carry their own doc comments.
- The folded set is shared by the three families and is never evicted,
so the ids that arrive first keep their own label for the life of the
process (same behaviour as `bounded_model_key` in the Python tracker). A
maintainer may prefer a different cap or an allow-list of known Bedrock
model ids; this takes the existing project limits.
- `other` is allowed through even if a client literally sends `other`;
it does not use up a slot.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4694"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:31:11 -06:00
HetCreepandClaude Opus 5.5 d5644cbaca fix(tag-protector): do not treat a slash in an unquoted attribute as self-closing (#4681)
## Description

`tag_protector` protects custom workflow tags from lossy compression.
Its attribute lexer
(`crates/headroom-core/src/transforms/tag_protector.rs`, the `b'/'` arm
of the open-tag loop, about line 374 on `12864fc8`) set `self_closing =
true` on every `/` and only cleared it on whitespace or a quote. An
unquoted attribute value holding a path, for example `<file
path=/src/main.rs>`, therefore reached `>` with `self_closing` still
set. The open tag was protected as a self-closing tag on its own, and
the body plus `</file>` went to the compressor unprotected, the outcome
the protector exists to prevent (the comment at
`headroom/transforms/content_router.py` around `protect_tags` explains
why tagged blocks must stay verbatim).

With this PR, a `/` makes the tag self-closing only when it is directly
followed by `>`, the `/>` pair. `<marker path=/a/b/>` still self-closes,
and the whitespace and quote handling is unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/tag_protector.rs`: the `b'/'` arm
sets `self_closing = bytes.get(i + 1) == Some(&b'>')` instead of `true`,
with a comment.
- Same file, tests:
`unquoted_attribute_value_with_slash_is_not_self_closing` checks that
`<file path=/src/main.rs>body</file>` becomes one protected block with
no `body` or `</file>` left in the cleaned text, and that `<marker
path=/a/b/>` still self-closes.

## Testing

- [x] Unit tests pass (`pytest`) (Rust only: `cargo test -p
headroom-core --lib`; no Python changed)
- [x] Linting passes (`ruff check .`) (no Python changed; `cargo fmt
--check` and `cargo clippy -D warnings` on headroom-core instead)
- [ ] Type checking passes (`mypy headroom`) (no Python changed)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (tag_protector.rs at 12864fc8 with only the new test added)
cargo test -p headroom-core --lib tag_protector: 31 passed; 1 failed
  unquoted_attribute_value_with_slash_is_not_self_closing ... FAILED

# pass after
cargo test -p headroom-core --lib tag_protector: 32 passed; 0 failed
cargo test -p headroom-core --lib: 1034 passed; 0 failed; 1 ignored
cargo fmt -p headroom-core -- --check: clean
cargo clippy -p headroom-core --lib -- -D warnings: clean
```

## Real Behavior Proof

- Environment: before: Ubuntu (WSL2) on Windows 11, Python 3.12.15, the
`headroom._core` extension from a 2026-10-29 build (`tag_protector.rs`
has not changed since 2026-05-02, so it matches `12864fc8`); after:
Windows 11, Rust 1.95.0 (`rust-toolchain.toml`), headroom-core at
`12864fc8` plus this change.
- Exact command / steps: before: called the shipped binding
`_core.protect_tags(text, False)` on `<file
path=/src/main.rs>body</file>` and, as a control, on `<file
path="/src/main.rs">body</file>`, offline, on a copy of the extension;
after: `cargo test -p headroom-core --lib` with the new test, which
feeds the same input to the same function.
- Observed result: before, the unquoted input came back as cleaned
`{{HEADROOM_TAG_0}}body</file>` with the single block `<file
path=/src/main.rs>`, while the quoted control came back as one
placeholder covering the whole block; after, the unquoted input is
protected as one block (`<file path=/src/main.rs>body</file>`), the new
test passes and the rest of the headroom-core library tests pass.
- Not tested: the Python extension rebuilt from this branch (I could not
build the PyO3 wheel on this machine), and a full proxy request through
`ContentRouter` with a compressor that drops text.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: an open tag whose
unquoted attribute value contains `/` is no longer treated as
self-closing, so its whole block is protected.
- Kill switch / disable path: none added.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

A separate PR fixes a different defect in the same file (marker mode
drops the enclosing tag's frame on a nested close); the two changes do
not overlap.

Produced with AI assistance (Claude Code) operated by me (HetCreep). The
tests and the binding call were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4681"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:30:46 -06:00
HetCreepandClaude Opus 5.5 385d487f41 fix(tag-protector): keep the enclosing tag open after a nested close in marker mode (#4682)
## Description

With `compress_tagged_content=true` (marker mode), `tag_protector`
records each custom open and close tag as its own placeholder and keeps
the body compressible. When a close tag matched an open frame on the
stack, the code ran `stack.truncate(stack_idx)` and then `stack.pop()`
(`crates/headroom-core/src/transforms/tag_protector.rs`, about line 571
on `12864fc8`). `truncate(stack_idx)` already removes the matched frame
and everything above it, so the extra `pop` removed the frame of the
enclosing tag. That tag's close then matched nothing, was counted as an
orphan close and stayed in the text as a raw tag. Block mode already did
only the `truncate`.

This is reachable on the default batch path:
`headroom/transforms/compression_batches.py` calls
`protect_tags(envelope, compress_tagged_content=True)`, and the envelope
wraps every unit in its own `<headroom-batch-{nonce}-{id}>` tag. Any
unit that holds one custom tag pair (for example
`<tool_call>…</tool_call>`) therefore lost the placeholder for its
envelope close, which went to the router as raw text.

With this PR the extra `pop` is removed and the comment explains why.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/tag_protector.rs`: marker-mode
close matching keeps only `stack.truncate(stack_idx)`; the comment now
says that `truncate` already drops the matched frame.
- Same file, tests: `marker_mode_nested_custom_tags_keep_outer_close`
checks that `<outer><inner>body</inner></outer>` gives four
placeholders, no raw `</outer>`, zero orphan closes, and an exact round
trip through `restore_tags`.

## Testing

- [x] Unit tests pass (`pytest`) (Rust only: `cargo test -p
headroom-core --lib`; no Python changed)
- [x] Linting passes (`ruff check .`) (no Python changed; `cargo fmt
--check` and `cargo clippy -D warnings` on headroom-core instead)
- [ ] Type checking passes (`mypy headroom`) (no Python changed)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (tag_protector.rs at 12864fc8 with only the new test added)
cargo test -p headroom-core --lib tag_protector: 31 passed; 1 failed
  marker_mode_nested_custom_tags_keep_outer_close ... FAILED
  cleaned = {{HEADROOM_TAG_0}}{{HEADROOM_TAG_1}}body{{HEADROOM_TAG_2}}</outer>

# pass after
cargo test -p headroom-core --lib tag_protector: 32 passed; 0 failed
cargo test -p headroom-core --lib: 1034 passed; 0 failed; 1 ignored
cargo fmt -p headroom-core -- --check: clean
cargo clippy -p headroom-core --lib -- -D warnings: clean
```

## Real Behavior Proof

- Environment: before: Ubuntu (WSL2) on Windows 11, Python 3.12.15, the
`headroom._core` extension from a 2026-10-29 build (`tag_protector.rs`
has not changed since 2026-05-02, so it matches `12864fc8`); after:
Windows 11, Rust 1.95.0 (`rust-toolchain.toml`), headroom-core at
`12864fc8` plus this change.
- Exact command / steps: before: called the shipped binding
`_core.protect_tags(text, True)` on the shape the batch path builds,
`<headroom-batch-abc-1><tool_call>do
it</tool_call></headroom-batch-abc-1>`, offline, on a copy of the
extension; after: `cargo test -p headroom-core --lib` with the new test
on a nested pair of custom tags.
- Observed result: before, the cleaned text was
`{{HEADROOM_TAG_0}}{{HEADROOM_TAG_1}}do
it{{HEADROOM_TAG_2}}</headroom-batch-abc-1>`: three placeholders, and
the envelope close left raw; after, the nested input gives four
placeholders with no raw close tag and restores exactly, the new test
passes and the rest of the headroom-core library tests pass.
- Not tested: the Python extension rebuilt from this branch (I could not
build the PyO3 wheel on this machine), and what a real compressor then
does with the raw envelope close in `compress_batch_with_router`
(whether the batch falls back to passthrough or the envelope still
parses).

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: in marker mode, the
close tag of a tag that encloses another custom tag is now protected
instead of left raw.
- Kill switch / disable path: none added.
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

A separate PR fixes a different defect in the same file (a `/` in an
unquoted attribute value made the tag self-closing); the two changes do
not overlap.

Produced with AI assistance (Claude Code) operated by me (HetCreep). The
tests and the binding call were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4682"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:38:20 -06:00
HetCreepandClaude Opus 5.5 adc6724966 fix(proxy): refresh Bedrock AWS credentials before they expire (#4696)
## Description

`headroom-proxy` resolved AWS credentials once at startup and signed
every Bedrock request with that one `Credentials` value. Temporary
credentials (IMDS, ECS task role, SSO, assumed role) expire, so after
expiry every Bedrock request was signed with a dead key until the
process was restarted.

This keeps the credentials provider and resolves credentials per request
through a small cache that asks the provider again once the cached
credentials are within 60 seconds of their expiry, the same policy as
`vertex/adc.rs` (`REFRESH_AHEAD_SECS = 60`).

Why a cache here and not just `provide_credentials()` per request: the
provider returned by `SdkConfig::credentials_provider()` does not cache.
In aws-config 1.13.0, `ConfigLoader::load` builds it as
`SharedCredentialsProvider::new(DefaultCredentialsChain::builder()...build().await)`
(`src/lib.rs:974-983`), and `DefaultCredentialsChain::credentials` calls
the provider chain directly
(`src/default_provider/credentials.rs:72-77`; the struct holds only
`provider_chain`, built at `:175-197`). The SDK caches in the separate
`identity_cache` (`src/lib.rs:1060-1064`, `IdentityCache::lazy()`),
which SDK clients use and this proxy does not. Calling the chain on
every request would hit IMDS / the ECS endpoint / SSO each time.

Partially addresses #4684: refreshes credentials after a successful
initial resolution. Startup credential-resolution failure recovery
remains open on that issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `bedrock/credentials.rs`: `BedrockCredentials` holds a
`SharedCredentialsProvider` and the last `Credentials` it returned.
`get()` returns the cached value unless it is within 60s of expiry,
otherwise asks the provider again (lock held across the fetch, so
concurrent requests share one refresh). Credentials with no expiry
(long-lived keys) are cached for good. Logs
`event=bedrock_credentials_refreshed` per fetch.
- `AppState.bedrock_credentials` is now
`Option<Arc<BedrockCredentials>>`.
`with_bedrock_credentials(Credentials)` keeps its signature (wraps the
static value, so existing tests are unchanged); new
`with_bedrock_credentials_source(BedrockCredentials)`.
- `bedrock/invoke.rs` and `bedrock/invoke_streaming.rs` call
`get().await` per request. A refresh error returns 500 with
`event=bedrock_credentials_refresh_failed` and never forwards unsigned.
The `None` path (`bedrock_credentials_missing`) is unchanged.
- `main.rs`: `load_bedrock_credentials` still resolves once at startup
(fills the cache) so a missing provider or failed resolve is reported at
startup exactly as before (`bedrock_credentials_unavailable`, Bedrock
routes 5xx); on success it now keeps the provider instead of the one
value.
- Tests: 3 unit tests in `bedrock/credentials.rs` and a new
`tests/integration_bedrock_credentials_refresh.rs` (both invoke routes)
using a test `ProvideCredentials` impl that issues `AKID_<n>` with a
1-second expiry and counts calls. No real AWS.

## Testing

- [ ] Unit tests pass (`pytest`) - not applicable, Rust-only change
- [ ] Linting passes (`ruff check .`) - not applicable; `cargo fmt
--check` and `cargo clippy -D warnings` run instead
- [ ] Type checking passes (`mypy headroom`) - not applicable
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Fail before (source reverted, new test file kept; the test needs the new
API, so it fails at compile time):

```text
error[E0432]: unresolved import `headroom_proxy::bedrock::BedrockCredentials`
  --> crates\headroom-proxy\tests\integration_bedrock_credentials_refresh.rs:13:5
error[E0599]: no method named `with_bedrock_credentials_source` found for struct `AppState` in the current scope
  --> crates\headroom-proxy\tests\integration_bedrock_credentials_refresh.rs:66:15
error: could not compile `headroom-proxy` (test "integration_bedrock_credentials_refresh") due to 2 previous errors
```

The runtime fail-before is in the Real Behavior Proof below (unpatched
binary signs the second request with the expired key).

Pass after:

```text
$ cargo test -p headroom-proxy --locked --test integration_bedrock_credentials_refresh
test invoke_streaming_signs_with_refreshed_credentials_after_expiry ... ok
test invoke_signs_with_refreshed_credentials_after_expiry ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.33s

$ cargo test -p headroom-proxy --locked --lib bedrock::credentials
test bedrock::credentials::tests::static_credentials_without_expiry_are_returned ... ok
test bedrock::credentials::tests::expiring_credentials_are_replaced_on_next_get ... ok
test bedrock::credentials::tests::long_lived_credentials_are_cached ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 349 filtered out; finished in 0.09s
```

Full crate suite (`cargo test -p headroom-proxy --locked
--no-fail-fast`): 42 test binaries, 581 passed, 1 failed. The one
failure is
`integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix`
(`Os { code: 10054, kind: ConnectionReset, ... }` at
`integration_path_traversal.rs:48`, a raw-TCP read on the catch-all
forwarder). It is a known Windows-only failure on unmodified `main` and
does not touch the Bedrock path; I did not re-run it on `main` for this
PR.

```text
$ cargo fmt -p headroom-proxy -- --check
(no output, exit 0)
$ cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 18s
```

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0 (pinned by rust-toolchain.toml),
debug build of the `headroom-proxy` binary from `main` at 12864fc8
(before) and from this branch (after). No real AWS: a local Python HTTP
server on 127.0.0.1 stands in for the ECS container-credentials endpoint
(every call issues a new fake key `AKIDFAKE<n>` with `Expiration` 3
seconds later) and for Bedrock runtime (logs which access key id signed
each request). Real AWS config was shut out: `AWS_CONFIG_FILE` and
`AWS_SHARED_CREDENTIALS_FILE` pointed at an empty file,
`AWS_EC2_METADATA_DISABLED=true`, no `AWS_*` key variables set, so the
real `aws-config` default chain resolved through its ECS provider via
`AWS_CONTAINER_CREDENTIALS_FULL_URI=http://127.0.0.1:18923/creds`.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18924
--upstream http://127.0.0.1:18923 --bedrock-endpoint
http://127.0.0.1:18923` with the environment above, then two `curl -X
POST
http://127.0.0.1:18924/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with a minimal Anthropic-on-Bedrock body, 5 seconds apart (so the first
key has expired before the second request); run once with each binary.
- Observed result: before the patch the credentials endpoint is called
once at startup and both requests are signed with `AKIDFAKE1`, the
second one about 6 seconds after a 3-second expiry (`creds-endpoint call
1: issued AKIDFAKE1` / `bedrock-mock .../invoke: signed with AKIDFAKE1`
/ `bedrock-mock .../invoke: signed with AKIDFAKE1`). After the patch:
`creds-endpoint call 1: issued AKIDFAKE1` (startup) / `creds-endpoint
call 2: issued AKIDFAKE2` / `bedrock-mock .../invoke: signed with
AKIDFAKE2` / `creds-endpoint call 3: issued AKIDFAKE3` / `bedrock-mock
.../invoke: signed with AKIDFAKE3`, both requests HTTP 200, proxy log
shows 1 `bedrock_credentials_loaded` and 3
`bedrock_credentials_refreshed`. (The first request also refreshes
because a 3-second credential is already inside the 60-second
refresh-ahead window.)
- Not tested: real AWS, STS, IMDS, SSO or an assumed role; a real
signature check by Bedrock (the mock only reads the key id from
`Authorization`); the streaming route through the binary (covered by the
integration test only); the refresh-failure 500 path through the binary;
behaviour under concurrent requests during a refresh; Linux/macOS.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this changes how the existing native
Bedrock route obtains credentials.
- Minimum rollout channel: n/a (no flag; applies wherever
`--enable-bedrock-native` is on, which is the default).
- Stable/default behavior changed: yes, for the better on long-running
processes: credentials are re-resolved from the same provider chain
within 60s of expiry instead of never. With non-expiring credentials
(static access keys) behaviour is unchanged (one resolve at startup,
cached). Startup behaviour on missing credentials is unchanged. New
error case: if a refresh fails, the request gets 500
`bedrock_credentials_refresh_failed` instead of being signed with an
expired key.
- Kill switch / disable path: `--enable-bedrock-native false` (existing)
disables the route; no separate switch for the refresh.
- Unsafe override required: no.
- Qualification impact: none expected; Bedrock tests use the unchanged
`with_bedrock_credentials(Credentials)` helper.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no
user-facing docs describe the credential lifetime; module docs updated)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (except
the known Windows-only `integration_path_traversal` failure noted above)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Out of scope: the inbound-auth side of the Bedrock route (how the
proxy authenticates or classifies incoming client requests,
`bedrock/auth_mode_layer.rs`) is not touched.
- Left for a maintainer decision: when the startup resolve fails, the
proxy still keeps no provider and the Bedrock routes 5xx until restart,
as before (the issue mentions this too). Keeping the provider and
retrying per request would fix that, but it would also make every
Bedrock request on a box with no AWS credentials walk the whole chain
(IMDS timeouts included), so I left the startup behaviour as it is.
- When a refresh fails while the cached credentials have not yet
actually expired (inside the 60s window), this returns 500 rather than
using the still-valid cached value, matching the project's
no-silent-fallback rule. Easy to change if you prefer the fallback.
- Produced with AI assistance (Claude Code) operated by me (HetCreep);
the tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4696"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:37:19 -06:00
HetCreepandClaude Opus 5.5 08e66222c9 fix(proxy): apply max_body_bytes to chat completions and responses routes (#4693)
## Description

Fixes #4686

`POST /v1/chat/completions` and `POST /v1/responses` in
`crates/headroom-proxy` extract the request body as `Bytes`. With no
`DefaultBodyLimit` on those routes, axum's built-in 2 MiB limit applied
instead of `--max-body-bytes` (100 MB by default), so a larger request
(for example one with inline base64 images) was cut off at the proxy.
The Bedrock sub-router already sets
`DefaultBodyLimit::max(config.max_body_bytes)` for the same reason (the
comment near `proxy.rs:353-355`). This PR applies the same layer to the
two OpenAI routes.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-proxy/src/proxy.rs`:
`.layer(DefaultBodyLimit::max(state.config.max_body_bytes as usize))` on
the `post(...)` of `/v1/chat/completions` and `/v1/responses`, with a
one-line comment. Other routes are unchanged.
- `crates/headroom-proxy/tests/integration_openai_body_limit.rs` (new,
using the `tests/common` harness and a wiremock upstream), each test
looping over both routes:
- `three_mib_body_reaches_upstream_with_default_cap`: a 3 MiB POST
returns 200 and reaches the upstream.
- `body_above_configured_cap_still_413`: with `max_body_bytes = 1024`, a
4 KiB POST returns 413 and does not reach the upstream.

## Testing

- [ ] Unit tests pass (`pytest`) (no Python changed)
- [ ] Linting passes (`ruff check .`) (no Python changed; `cargo fmt
--check` and `cargo clippy -D warnings` instead)
- [ ] Type checking passes (`mypy headroom`) (no Python changed)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# fail before (proxy.rs at 12864fc8, new test file kept)
cargo test -p headroom-proxy --test integration_openai_body_limit
  three_mib_body_reaches_upstream_with_default_cap: reqwest::Error ... Os { code: 10054, kind: ConnectionReset }
    (the proxy dropped the 3 MiB upload on /v1/chat/completions)
  body_above_configured_cap_still_413: assertion `left == right` failed: /v1/chat/completions  left: 200  right: 413
    (a 4 KiB body passed although max_body_bytes = 1024)
  test result: FAILED. 0 passed; 2 failed

# pass after
test result: ok. 2 passed; 0 failed

# full crate suite on this branch (Windows 11)
cargo test -p headroom-proxy --locked --no-fail-fast: every target passes except
  integration_path_traversal::forwarder_keeps_requests_under_the_base_prefix (os error 10054, ConnectionReset),
  which also fails on unmodified main on this Windows machine (3 of 3 runs) and does not touch these routes
cargo fmt -p headroom-proxy -- --check: clean
cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings: clean
```

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0 (`rust-toolchain.toml`), debug
build of `headroom-proxy` from this branch; upstream is a PowerShell
TcpListener recorder on `127.0.0.1:18991` that logs the request line and
how much body it read (it reads only the first chunk, so its byte count
is not the full body).
- Exact command / steps: started `headroom-proxy --listen
127.0.0.1:18990 --upstream http://127.0.0.1:18991`, sent a 3 MiB JSON
`POST` to `/v1/chat/completions` and to `/v1/responses`, then restarted
it with `--max-body-bytes 1MB` and sent the same two requests.
- Observed result: with the default limit both 3 MiB requests were
forwarded (the recorder logged `POST /v1/chat/completions` and `POST
/v1/responses`; chat returned 200, responses 502 because the recorder's
canned reply is not a valid Responses body). With `--max-body-bytes
1MB`, chat returned 413 and the responses upload was closed by the proxy
before completion, and the recorder logged no third or fourth request.
Before this change the same 3 MiB requests were rejected at the proxy,
as the fail-before test shows.
- Not tested: a real OpenAI upstream, a streaming request near the
limit, Linux and macOS, and a release build.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly: requests between 2 MiB
and `--max-body-bytes` on these two routes are now forwarded instead of
rejected; the configured limit now applies to them.
- Kill switch / disable path: `--max-body-bytes` (lower it to restore a
tighter limit).
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

The layer is applied to these two routes only, matching the issue. Other
handlers that buffer the body (for example Vertex and Conversations)
were not changed.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4693"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:37:08 -06:00
HetCreepandClaude Opus 5.5 8f4122f4e5 fix(proxy): track response.-prefixed item and content-part events in Responses SSE state (#4695)
## Description

`ResponseState::apply` in
`crates/headroom-proxy/src/sse/openai_responses.rs` matched
`output_item.added`, `output_item.done`, `content_part.added` and
`content_part.done` only in their bare form. Every other arm of the same
`match` accepts both the `response.`-prefixed name and the bare name
(for example `"response.output_text.delta" | "output_text.delta"`). The
prefixed names for these four events therefore reached the `other =>`
arm: one `sse_unknown_event` warning per event and no item or
content-part state. The repository's own Python emitter
(`headroom/proxy/handlers/openai.py`) writes the prefixed names. This PR
accepts both forms in the four arms.

Fixes #4687

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-proxy/src/sse/openai_responses.rs`: the four
bare-only arms now also match `response.output_item.added`,
`response.output_item.done`, `response.content_part.added`,
`response.content_part.done` (4 lines).
- `crates/headroom-proxy/tests/sse_openai_responses.rs`: new test
`prefixed_item_and_content_part_events_update_state`; the existing
bare-name tests are unchanged.

## Testing

- [x] Unit tests pass (`pytest`): not applicable, Rust-only change;
`cargo test -p headroom-proxy` below
- [x] Linting passes (`ruff check .`): not applicable; `cargo fmt` and
`cargo clippy -D warnings` below
- [x] Type checking passes (`mypy headroom`): not applicable
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Fail before (new test against unmodified `src`, base `12864fc8`):

```text
test prefixed_item_and_content_part_events_update_state ... FAILED
thread 'prefixed_item_and_content_part_events_update_state' panicked at crates\headroom-proxy\tests\sse_openai_responses.rs:252:5:
assertion `left == right` failed
  left: ""
 right: "message"
test result: FAILED. 8 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.27s
```

Pass after:

```text
test prefixed_item_and_content_part_events_update_state ... ok
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s
```

Full crate suite, `cargo test -p headroom-proxy --locked --no-fail-fast`
(Windows 11): every test binary passes except
`integration_path_traversal`:

- `forwarder_keeps_requests_under_the_base_prefix` fails with `Os {
code: 10054, kind: ConnectionReset }` at
`integration_path_traversal.rs:48`. It fails the same way on unmodified
`main` (`12864fc8`, checked in a separate worktree) and on every rerun
here.
- `vertex_dot_segment_project_is_rejected_unsigned` failed once in the
full run with the same os error 10054 and passed on `main` and on three
reruns of the file on this branch, so I treat it as a flaky Windows
socket reset, not related to this change.

```text
cargo fmt -p headroom-proxy -- --check            exit 0
cargo clippy -p headroom-proxy --all-targets --locked -- -D warnings   exit 0
```

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0 (pinned by
`rust-toolchain.toml`), branch from `main` at `12864fc8`,
`headroom-proxy` crate; no network, no live OpenAI API.
- Exact command / steps: a throwaway integration test (deleted before
commit) fed an SSE byte stream through the real `SseFramer` and
`ResponseState::apply` with `cargo test -p headroom-proxy --test <tmp>
-- --nocapture`. The six events use the `response.`-prefixed names and
the payload shapes copied from the repository's Python emitter
(`headroom/proxy/handlers/openai.py`, around lines 1720-1745):
`response.output_item.added`, `response.content_part.added`,
`response.output_text.delta`, `response.output_text.done`,
`response.content_part.done`, `response.output_item.done`. I printed the
resulting item state.
- Observed result: before the change (library built from unmodified
`main`): `ITEM msg_1: type="" complete=false status=null parts={}
text="hello"` (only the delta, which was already matched, registered).
After the change: `ITEM msg_1: type="message" complete=true
status="completed" parts={0: ContentPartState { part_type:
"output_text", text: "hello", complete: true }} text="hello"`.
- Not tested: a live OpenAI Responses stream (no live stream was used;
the shapes come from the repository's own emitter); the `tracing`
warning count itself (the unknown-event arm cannot update state, so
state tracking is the observable proof); which consumers read item and
content-part state.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for streams that use the
prefixed names: item and content-part state is now tracked, and the
per-event `sse_unknown_event` warning no longer fires for them. The
stream bytes forwarded to the client are unchanged.
- Kill switch / disable path: none needed (no new feature or config)
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (except
the Windows-only `integration_path_traversal` failure that also fails on
`main`, see above)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- The other arms already accept both forms; this change makes the four
item and content-part arms consistent with them. No handler function
changed.
- Not checked: how much of the item and content-part state is read by
other code, so how visible the effect is to users.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4695"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:36:34 -06:00
HetCreepandClaude Opus 5.5 ced26852c6 fix(smart-crusher): keep mixed-array dict rows and exact drop count with factor_out_constants (#4692)
## Description

With `factor_out_constants=true` (off by default), SmartCrusher had two
row-loss bugs:

1. `crush_mixed_array` matched the rows that `crush_array` returned
against the original dict rows using canonical JSON. `execute_plan`
strips the constant keys from kept rows when the flag is on, so no
stripped row equalled its original, and the mixed array kept **zero**
dict rows (while the strategy string still said e.g. `dict:40->15`).
2. `crush_array` computed `dropped_count = items.len() - result.len()`
with the leading `{"_constant_fields": ...}` sentinel counted as a kept
row. The `<<ccr:... N_rows_offloaded>>` marker was one short, and when
exactly one real row was dropped no marker was written and nothing was
stored in CCR, so that row could not be retrieved.

Fixes #4683

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/smart_crusher/crusher.rs`:
- New private helper `constant_fields_sentinel(items)`: returns the
constants map when `items[0]` is the `{"_constant_fields": {...}}`
sentinel that `execute_plan` inserts.
- New private helper `strip_constant_fields(item, constants)`: applies
the same defensive strip as `execute_plan` (a key is removed only when
its value equals the recorded constant).
- `crush_array_with_source`: `dropped_count` no longer counts the
sentinel (only when `factor_out_constants` is on and the sentinel is
present).
- `crush_mixed_array` dict group: skips the sentinel, strips the
originals the same way before the canonical match, and reports the real
row count in the `dict:N->M` strategy part.
- Three regression tests (below). No behaviour change when
`factor_out_constants` is off.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

This is a Rust-only change, so I ran `cargo test`, `cargo fmt --check`
and `cargo clippy` for `headroom-core` instead of pytest/ruff/mypy.

### Test Output

New tests (in `crusher.rs` `mod tests`):
`factored_mixed_array_keeps_the_same_dict_rows`,
`factored_marker_counts_only_real_dropped_rows`,
`factored_single_dropped_row_still_writes_marker_and_stores`.

Before the fix (tests added, source change reverted), `cargo test -p
headroom-core --locked --lib factored_`:

```text
test transforms::smart_crusher::crusher::tests::factored_single_dropped_row_still_writes_marker_and_stores ... FAILED
test transforms::smart_crusher::crusher::tests::factored_marker_counts_only_real_dropped_rows ... FAILED
test transforms::smart_crusher::crusher::tests::factored_mixed_array_keeps_the_same_dict_rows ... FAILED
marker missing: ""
marker "<<ccr:148edcb9b3c4 29_rows_offloaded>>" must count 30 dropped rows
  left: [String("note 0"), String("note 1"), ..., String("note 9")]          (flag on: no dict rows)
 right: [Object {"level": Number(0), ...}, ... 15 dict rows ..., String("note 0"), ...]   (flag off)
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 1034 filtered out; finished in 0.11s
```

(The `left`/`right` lines are shortened here; the full values are 10
strings vs. 15 dict rows plus the same 10 strings.)

After the fix:

```text
test transforms::smart_crusher::crusher::tests::factored_mixed_array_keeps_the_same_dict_rows ... ok
test transforms::smart_crusher::crusher::tests::factored_single_dropped_row_still_writes_marker_and_stores ... ok
test transforms::smart_crusher::crusher::tests::factored_marker_counts_only_real_dropped_rows ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 1034 filtered out; finished in 0.85s
```

Full crate suite, `cargo test -p headroom-core --locked --no-fail-fast`:
exit 0, 21 test binaries, 1140 passed, 0 failed (lib: `test result: ok.
1036 passed; 0 failed; 1 ignored`).

`cargo fmt -p headroom-core -- --check`: exit 0. `cargo clippy -p
headroom-core --all-targets --locked -- -D warnings`: exit 0 (`Finished
dev profile`).

## Real Behavior Proof

- Environment: Windows 11 Pro, Rust 1.95.0 (pinned by
rust-toolchain.toml), branch based on `main` at `12864fc8`; Python
binding not rebuilt (no Rust toolchain in my WSL, and the Windows build
of the extension was not attempted), so the proof goes through the
public Rust API instead.
- Exact command / steps: a throwaway example (not committed) built
`SmartCrusher::without_compaction(SmartCrusherConfig {
factor_out_constants, dedup_identical_items: false, ..Default::default()
})` and called the public entry point `crush(&json, "", 1.0)` on three
inputs: 40 dict rows (two constant keys `region`/`kind`, two varying
keys) followed by 10 strings; the same 40 dict rows alone; 16 such rows.
Run with `cargo run -p headroom-core --example <probe>` once with the
source change reverted and once with it applied; for each output it
counted real dict rows (excluding the `_constant_fields` and
`_ccr_dropped` sentinels) and printed the CCR marker.
- Observed result: with the fix reverted and
`factor_out_constants=true`: mixed array `real_dict_rows_out=0` with
marker `40_items_offloaded` (control with the flag off: 15 rows,
`25_items_offloaded`); 40 dict rows: 10 kept, marker `29_rows_offloaded`
(control: `30_rows_offloaded`); 16 rows: 15 kept, `marker=None`. With
the fix and `factor_out_constants=true`: mixed array 15 dict rows,
`25_items_offloaded`; 40 rows: `30_rows_offloaded`; 16 rows:
`<<ccr:301c57390d35 1_rows_offloaded>>`, identical to the flag-off
control in all three cases.
- Not tested: the Python binding (`headroom._core`) and the proxy end to
end; retrieval of the stored payload through a retrieval tool call (the
unit test reads it back from the in-memory CCR store directly); other
crates' test suites.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; affects only the existing
`factor_out_constants` SmartCrusher option.
- Minimum rollout channel: N/A (bug fix, no new feature gate).
- Stable/default behavior changed: no. `factor_out_constants` defaults
to false, and every changed line is guarded by it; with the flag off the
output is unchanged.
- Kill switch / disable path: leave `factor_out_constants=false` (the
default).
- Unsafe override required: no.
- Qualification impact: none expected; with the flag on, mixed arrays
now keep the same dict rows as with the flag off, and markers count one
more row (the correct number).
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: N/A, no documented behaviour changes.
- Sentinel detection: the sentinel is recognised as a first element that
is an object with the single key `_constant_fields`, and only when
`factor_out_constants` is on. An input row of exactly that shape would
be ambiguous, but it is equally ambiguous to anything that reads the
crushed output today. Returning the kept indices from `crush_array`
would avoid the shape check, but it touches every `CrushArrayResult`
constructor, so I kept the smaller change.
- Out of scope, unchanged by this PR: the mixed-array canonical match
keeps every original that equals a kept row, so with duplicate rows the
mixed path can keep more dict rows than `crush_array` did (e.g.
`dict:40->10` with 15 dict rows in the output). This happens with the
flag off as well and is not touched here.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4692"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:34:39 -06:00
Phil LaFayetteandJerrettDavis a538740ee4 fix(search_compressor): keep kept rows byte-for-byte and in input order (#3982)
## Description

`SearchCompressor` rebuilt every row it kept from its parsed parts
(`file:N:content`) instead of copying it. The agent then read text that
was never in the tool output:

- a time or date in the content was taken as a line number: `12:00:01`
printed as `12:0:01`, and `2026-10-03 12:05` as `2026:10:03 12:05`;
- a grep context line (`file-N-text`, from `grep -C`/`rg -C`) was
printed with `:`, turning it into a match line;
- kept rows within a file could come out in a different order than the
input.

This PR makes the compressor copy kept rows instead of rebuilding them.
(No existing issue covers it.) It complements #3788 (merged after
0.39.1, same file), which fixed a context line's body becoming its line
marker; this PR applies on top of it without conflicts and all of
#3788's tests still pass.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

All in `crates/headroom-core/src/transforms/search_compressor.rs`:

- **Rows render byte-for-byte.** Each parsed match keeps the exact
marker text between the file name and the content (`:12:`, `-12-`), and
output rows are `file + marker + content`, so the original separator
survives.
- **A zero-led digit run is never a line number** (`00`, `05`): real
line numbers never start with 0, so `12:00:01` and dates stay content.
- **Input order is kept within each file**: kept rows stay in the order
they arrived, and files are emitted in order of first appearance (also
on equal scores under `max_files`). Rows of different files that were
interleaved in the input are still grouped by file, as before.
- **The selected occurrence is the one emitted**: rows are selected and
emitted by input index. Previously the output pass matched rows back by
(line, content), so when a later row was selected (`always_keep_last`)
and an earlier row shared its number and content (a match `a.py:12:x`
and a context row `a.py-12-x`), the earlier one was printed instead. The
(line, content) dedup and every cap are unchanged (100,000-case
differential fuzz against the previous selection: same keys, counts and
stats).
- **Indented rows**: leading whitespace is set aside before parsing (so
the row splits at its real marker, not inside its path) and restored on
output.

## Testing

- [x] New tests added
- [x] New and existing unit tests pass locally

New tests: `the_selected_occurrence_is_emitted_not_its_twin`,
`indented_rows_split_at_the_real_marker_and_render_verbatim`,
`rejects_zero_led_line_numbers`,
`compress_output_rows_are_verbatim_and_in_input_order`,
`max_files_keeps_the_earliest_files_on_equal_scores`,
`rendering_reproduces_each_parsed_line_byte_for_byte`.

### Test Output

```text
# this branch
$ cargo test -p headroom-core --lib search_compressor
test result: ok. 34 passed; 0 failed; 0 ignored; 0 measured; 932 filtered out
$ cargo test -p headroom-core --lib
test result: ok. 965 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out
$ cargo fmt -p headroom-core -- --check   # clean
$ cargo clippy -p headroom-core --lib -- -D warnings   # clean

# fails before: the new tests against origin/main's implementation (the two tests that
# exercise the new render helper cannot compile there and were left out)
test transforms::search_compressor::tests::max_files_keeps_the_earliest_files_on_equal_scores ... FAILED
test transforms::search_compressor::tests::rejects_zero_led_line_numbers ... FAILED
test transforms::search_compressor::tests::compress_output_rows_are_verbatim_and_in_input_order ... FAILED
test result: FAILED. 28 passed; 3 failed
```

## Real Behavior Proof

- Environment: Linux (WSL2), Python 3.13, headroom-ai 0.39.1 installed
with uv; the same compiled `_core.abi3.so` swapped between the stock
wheel and one built from these commits on 0.39.1.
- Exact command / steps:
`headroom._core.SearchCompressor().compress(text)` on 120 rows of `grep
-n -C1`-style output (`app.log-2-2026-10-03 12:00:01 DEBUG tick` context
rows interleaved with `app.log:3:... ERROR ...` match rows); counted
kept rows that are not byte-identical to an input row.
- Observed result: stock core: 1 of 5 kept rows not in the input
(`app.log:2:2026-10-03 12:00:01 DEBUG tick`, a context row turned into a
match row). Fixed core: 0 of 5. The fix has also been running in my
local Headroom proxy (0.39.1) since 2026-10-03.
- Not tested: a full Python wheel built from main with this change (the
behavior proof used a 0.39.1-based build of the first three commits; the
unit tests above ran on main); Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, only in that kept search rows
are now exact copies of input rows and keep their order
- Kill switch / disable path: none needed; existing search compression
settings are unchanged
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Known, out of scope (same on main): a standalone date line whose month
has no leading zero (`2026-10-03 12:05 ...`) still parses as file
`2026`, line 10. The row is still printed verbatim; only its grouping is
wrong, and it can take a file slot under `max_files`. Happy to follow up
separately.

Python-side tests (`pytest`, `ruff`, `mypy`) were not run: the change is
Rust-only and touches no Python file.

One visible effect on the Python surface: `parse_search_lines`
(crates/headroom-py) now returns each row's content with its trailing
whitespace and `\r` intact (main trimmed them). File and line numbers
are unchanged for every row tested, indented or not. This is deliberate,
as the content is what gets copied back into the output; if the legacy
Python `SearchMatch.content` should stay trimmed, `m.content.trim_end()`
in the shim restores main's behavior.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3982"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->


### Maintainer current-main requalification

Exact head `545c18c771817cd97f4f9f716714161bb0ce3b6c`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Clean mechanical synchronization with upstream main
c01cf07d94. Re-reviewed author/Devin
discussion: selected occurrence identity is fixed, ordering is within
each file with files grouped by first appearance, and the preexisting
nonzero-date classification limitation remains explicitly scoped.
Contributor Rust implementation unchanged.

Maintainer test-only reconciliation resolves the four previously
reported Python failures: assertions now require input order and
first-appearance file grouping; the two 100-row fixtures use valid
positive line numbers. The zero-line control explicitly checks parser
rejection and unchanged passthrough, while negative/malformed-row
coverage remains. Exact-branch native build reproduced all four original
failures before the fixes; all 74 Python search/text tests now pass. All
34 Rust search-compressor tests pass, as do Rust formatting,
changed-file Ruff lint/format and whitespace checks.

Native provenance verifies all 238 tracked native/build inputs and the
DLL hash against the qualified head; the subsequent commit changes only
Python tests. An independent public-Python native probe confirms the
selected context occurrence preserves indentation, dash separator,
trailing spaces and CR instead of emitting its match-row twin. This is
local Windows native/consumer validation, not a full
Python/workspace-suite or live provider claim. The previous
contract/fixture request is addressed; formal approval awaits exact-head
hosted qualification. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 09:30:30 -06:00
dependabot[bot]andJerrettDavis 74b7ac2386 deps: bump rusqlite from 0.40.2 to 0.41.1 (#4658)
Bumps [rusqlite](https://github.com/rusqlite/rusqlite) from 0.40.2 to
0.41.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rusqlite/rusqlite/releases">rusqlite's
releases</a>.</em></p>
<blockquote>
<h2>0.41.1</h2>
<h2>What's Changed</h2>
<ul>
<li>No check of PRAGMA name with SQLCipher <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1923">#1923</a>
(<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1924">#1924</a>)</li>
<li>Fix panic with a zero-argument deterministic scalar function <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1925">#1925</a>
(<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1927">#1927</a>)</li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rusqlite/rusqlite/compare/v0.41.0...v0.41.1">https://github.com/rusqlite/rusqlite/compare/v0.41.0...v0.41.1</a></p>
<h2>0.41.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Check PRAGMA <!-- raw HTML omitted --> is valid with the help of
pragma_list <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1857">#1857</a></li>
<li>Fix UtcDateTime <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1859">#1859</a></li>
<li>Bump sqlite3-parser version <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1864">#1864</a></li>
<li>Edition 2024 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1863">#1863</a></li>
<li>Use u64 for zeroblob <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1867">#1867</a></li>
<li>Fix virtual table set_result <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1868">#1868</a></li>
<li>Bump bundled SQLCipher to version 4.17.0 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1870">#1870</a></li>
<li>Refactor bind_parameter and set_result <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1871">#1871</a></li>
<li>Update prettyplease requirement from 0.2.20 to 0.3.0 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1872">#1872</a></li>
<li>Update syn requirement from 2.0.89 to 3.0.1 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1873">#1873</a></li>
<li>Fix Connection::serialize <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1874">#1874</a>
(<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1875">#1875</a>)</li>
<li>Make Statement::column_names,columns,columns_with_metadata return
Iterator <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1877">#1877</a></li>
<li>Introduce DEFAULT_NAME <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1883">#1883</a></li>
<li>Expose sqlite3_file_control <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1881">#1881</a>
(<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1882">#1882</a>)</li>
<li>Replace assert!(matches! with assert_matches! <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1851">#1851</a></li>
<li>Lower MSRV to 1.88.0 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1885">#1885</a></li>
<li>Fix clippy warnings <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1888">#1888</a></li>
<li>Apply and check MSRV <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1890">#1890</a></li>
<li>Bump minimal SQLite version to 3.45.3 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1894">#1894</a></li>
<li>sqlite3_set_clientdata / sqlite3_get_clientdata <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/977">#977</a>,
<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1708">#1708</a>
(<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1791">#1791</a>)</li>
<li>Fix discrepancy between backup and restore <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1893">#1893</a></li>
<li>Don't enable SQLITE_ENABLE_MEMORY_MANAGEMENT in the bundled build <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1900">#1900</a></li>
<li>create_aggregate_function / create_window_function are missing a
<code>Send</code> bound <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1898">#1898</a></li>
<li>use-after-free from safe code in trace_v2 / busy_handler / ... <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1899">#1899</a></li>
<li>feat: add <code>Values::get_subtype</code> to read SQLite value
subtype in vtab callbacks <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1879">#1879</a></li>
<li>Bump bindgen version to 0.73 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1903">#1903</a></li>
<li>Bump <code>sqlite-wasm-rs</code> to 0.6.1 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1904">#1904</a></li>
<li>Relax <code>sqlite-wasm-rs</code> version <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1906">#1906</a></li>
<li>Rerun bundled-sqlcipher build when <code>OPENSSL_*</code> env
changes <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1910">#1910</a></li>
<li>Bump bundled SQLite version to 3.54.0 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1913">#1913</a></li>
<li>Fix possible memory leak on error in deserialize_read_exact <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1909">#1909</a></li>
<li>Move sqlite3_wal_checkpoint_v2 binding <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1667">#1667</a>
(<a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1915">#1915</a>)</li>
<li>Bump uuid version to 1.28 <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1916">#1916</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rusqlite/rusqlite/compare/v0.40.1...v0.41.0">https://github.com/rusqlite/rusqlite/compare/v0.40.1...v0.41.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/18ee76dbb976d5ef7e37ad5a8fffae678ca21a78"><code>18ee76d</code></a>
Prepare next release</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/11ac8f515728368f19f3f5d0eb4e55928c0e6642"><code>11ac8f5</code></a>
Merge pull request <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1924">#1924</a>
from gwenn/pragma_name_check</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/4d2be948a14f6e3110c0bbb5dad67535cf086b6a"><code>4d2be94</code></a>
Merge pull request <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1927">#1927</a>
from gwenn/from_raw_parts</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/a60e76b5e86ee60442213efe79758a6c2ca8cf3d"><code>a60e76b</code></a>
Merge pull request <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1919">#1919</a>
from gwenn/0.41.0</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/f652850cd7735094d65d383626b29e9ecd5fb11d"><code>f652850</code></a>
Prepare release</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/661107f6eb5d7477c61be1221744de3ad96c53e7"><code>661107f</code></a>
Merge pull request <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1917">#1917</a>
from LucaCappelletti94/feat/sqlcipher-src</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/dd770f6f9fa800e54a3cbd584a35b1af5a0d260b"><code>dd770f6</code></a>
Fix the SQLCipher license link and match upstream style</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/2daf10a6ea333ea8810e4b248137f7e7d7eae5f8"><code>2daf10a</code></a>
Select the SQLCipher source with cfg_select</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/bb401536c47cd074d287c7426e0ed99e5d3f086f"><code>bb40153</code></a>
Merge pull request <a
href="https://redirect.github.com/rusqlite/rusqlite/issues/1916">#1916</a>
from gwenn/uuid</li>
<li><a
href="https://github.com/rusqlite/rusqlite/commit/92a7d4965b12d0bab61d0c0447d91931f1d48461"><code>92a7d49</code></a>
Bump uuid version to 1.28</li>
<li>Additional commits viewable in <a
href="https://github.com/rusqlite/rusqlite/compare/v0.40.2...v0.41.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rusqlite&package-manager=cargo&previous-version=0.40.2&new-version=0.41.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4658"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---

### Maintainer dependency assessment

The user has explicitly authorized routine dependency maintenance for
this bot PR. This is a maintainer-approved exception to the usual
requirement for a project-specific bug/security fix or required feature;
no unused upstream fix is represented as a Headroom bug fix.

Headroom already uses rusqlite for its production-default persistent
SQLite CCR store. The existing rusqlite project remains active (upstream
repository not archived; pushed 2026-11-03); 0.41.0/0.41.1 are
successive upstream releases. Reviewed both release notes and the
complete manifest/lock delta. This advances rusqlite 0.40.2 -> 0.41.1
and libsqlite3-sys 0.38.2 -> 0.39.1, with bundled SQLite 3.54.0 and
upstream bundled-build configuration changes. It retains the existing
bundled C compilation surface through cc; the lock delta adds no package
names. Cargo fetches registry sources during installation; the SQLite
backend needs no runtime network service. The Rust MSRV for upstream
0.41 is 1.88, below this validation toolchain. SQLCipher PRAGMA fixes do
not apply to our bundled, non-SQLCipher build, and the zero-argument
function fix is not claimed as an observed Headroom defect. This is not
advertised as a security remediation; the release-note/locked-source
assessment is not a comprehensive historical vulnerability audit.

### Real behavior proof (maintainer)

Windows x86_64 MSVC, rustc 1.95.0; isolated worktree and dedicated
CARGO_TARGET_DIR. The tested Cargo tree is unchanged by the final
Python-only main sync.

* `cargo test -p headroom-core --locked --lib`: 1,027 passed, 1 ignored.
* `cargo test -p headroom-core --locked --no-default-features --lib`:
978 passed, 1 ignored.
* `cargo test -p headroom-core --locked --no-default-features --test
ccr_backends -- --nocapture`: 12 passed. This executes real bundled
SQLite file open, put, drop, reopen and retrieval
(`sqlite_persists_across_proxy_restart`), plus round trip, TTL, schema
migration and backend selection checks; it is not an in-memory
substitute.

No cross-platform wheel, SQLCipher, performance or deployed multi-worker
qualification is claimed. Maintainers own CI and follow-up fixes; no
contributor action is requested. This evidence addresses the existing
dependency-assessment and native-behavior threads without inventing an
applicable project bug.

Validated branch head: `b8efb3a07977f833992046089f6ce4de0875003d`.
Includes fetched upstream main
`de1a1e0f478d1eb9257dd8dd0d4ac87c9e6fc639`.

Maintainer sync: includes upstream de1a1e0f, with the merged Click/macOS
Rust-setup fix and sqlite-vec 0.1.9 Python requirement/lock plus test
connection-lifecycle cleanup. Cargo.toml, Cargo.lock, all crates and
rust-toolchain.toml are unchanged versus the previously tested native
candidate. Exact parsed lock comparison against current main confirms
only the intended PR package versions/checksums differ; sources,
dependency edges and all other records are identical. Existing native
and combined preflight evidence carries forward (core 1,027 passed/1
ignored; real SQLite/live Redis 14 passed, none skipped; Python binding
check/build/import passed).

Additional cross-runtime proof: Windows x86_64/Python 3.12.13 process
preloaded the freshly built combined-preflight _core.dll directly
(SHA256
6dfe261cec336b73807ae3a099bdf28e3355af46a928ec4b1113207eedfc5bc7), then
loaded actual isolated sqlite-vec 0.1.9. Native JSON calls and real
vec_version/vector-distance SQL succeeded in that same process. Ran
tests/test_sqlite_vector_index.py and local relationship/backend
search/core text-search suites: 45 passed. Asserted the exact freshly
built native module remained loaded and callable after tests. This
detects local loading/PyO3/SQLite-extension coexistence regressions; it
adds no cross-platform or emitted Rust-log-record claim.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 22:48:17 -06:00
7e827285a4 fix(kompress): load ONNX Runtime through the dynamic loader guard before building a session (#4644)
<!-- maintainer-current-main -->
## Maintainer latest-head qualification
Published `01ceba05fde7de3996911dd5aa77e6e517c744f6` includes current
main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. The author's
CPU-then-loader readiness correction addresses review 5780333876. Its
injected CPU test proves the loader is not called when CPU support is
false; the missing-runtime test and both parity tests pass on Windows
with all 21 fixtures byte-identical. Fresh Rust test binaries, cargo fmt
and warnings-as-errors Clippy pass. The maintainer follow-up only
repairs the inherited Gemini fixture using a real ASGI Request with two
receive chunks; all 10 Gemini tests pass. No contributor production
logic was rewritten by the maintainer follow-up. Earlier captures retain
their original commit/platform scope. Fresh hosted CI is being qualified
at this exact head. Nothing merged.
<!-- /maintainer-current-main -->

## Description

`Kompress::build_session`
(`crates/headroom-core/src/transforms/kompress.rs`) called
`ort::Session::builder()` directly. The crate loads ONNX Runtime
dynamically (`ort-load-dynamic`), and when that load fails, `ort`
2.0.0-rc.12 deadlocks in its API-setup error path instead of returning
an error: `load_dylib_from_path` fails, builds an `ort::Error`, and
`Error::new` calls `ort::api()`, which is the `OnceLock` still being
initialised. Magika and fastembed already avoid this by calling
`dynamic_ort_loader_ready()` first (its doc comment cites the #1715 CI
hang). Kompress was the one session builder that skipped the guard.

Since #4438, the Rust side reads the Kompress model from Python's HF
cache. On a dev machine that has the model cached and no
`ORT_DYLIB_PATH`, the `kompress_parity` test stopped skipping and hung
forever. I hit it in the pre-push `make ci-precheck`: `cargo test
--workspace` sat on `kompress_parity` for 14+ minutes at 0.29 s of CPU
time, with both test threads blocked:

```text
kompress_matches_python_fixtures_byte_for_byte
  Kompress::from_files  kompress.rs:258
    ort::Session::builder → ort::environment::current
      ort::util::mutex::Mutex::lock → __psynch_mutexwait
short_input_passes_through
  Kompress::from_files → ort::Session::builder → create_environment → ort::api
    OnceLock::get_or_init → ort::setup_api → ort::load_dylib_from_path
      (load fails) ort::Error::new → ort::api
        OnceLock::get_or_init  (same lock, re-entered) → __psynch_mutexwait
```

CI doesn't see it because the model isn't in the runner's HF cache, so
the test skips there. `from_files`, `from_pretrained` and `from_cache`
all go through `build_session`, so any Rust caller hits the same hang
when ONNX Runtime can't be loaded and nothing has committed it yet. That
includes the native proxy's Kompress warm-up (`live_zone.rs` →
`Kompress::from_cache`). If Magika's guard has already committed ORT,
Kompress works. But if ORT is unavailable, Magika's guard fails cleanly
and Kompress then deadlocks instead of falling back to its deterministic
NoOp. I haven't reproduced that in the native proxy; the test below
covers the code path. Python callers are unaffected because
`headroom._ort` pins `ORT_DYLIB_PATH` before anything loads.

The fix calls the same guard before building the session:

```diff
 fn build_session(path: &Path) -> Result<Session, ...> {
+    crate::transforms::magika_detector::dynamic_ort_loader_ready()?;
     let session = Session::builder()?.commit_from_file(path)?;
```

Since main's #4559, `build_session` runs `ort_session_preconditions()`
(CPU check, then this same loader guard), so after the main merge the
fix's net change to `build_session` is nil. What remains of this PR is
the regression test and the `kompress_parity` skip below.

The guard resolves `ORT_DYLIB_PATH`, or else discovers the pip
`onnxruntime` library through `VIRTUAL_ENV` / `CONDA_PREFIX` /
`./.venv`, and commits it once per process. If neither works, it returns
an error without touching `ort`, so Kompress now fails with
`KompressError::Onnx` naming `ORT_DYLIB_PATH` instead of hanging.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/kompress.rs`: `build_session`
calls `dynamic_ort_loader_ready()` before `Session::builder()`. Both
modules sit behind the same `ml` feature.
- `crates/headroom-core/src/transforms/kompress.rs`: new `pub fn
onnx_runtime_ready()`, so harnesses outside the crate can check whether
Kompress can run. It goes through #4559's `ort_session_gate` with the
same CPU verdict and loader as `ort_session_preconditions`: on a CPU
without AVX2 it returns `cpu_unsupported: ...` without calling the
loader. `Err` is `"<reason>: <detail>"`. The unit test
`onnx_runtime_ready_checks_the_cpu_before_the_loader` drives the
injectable `onnx_runtime_ready_with` and checks the loader isn't called
when the CPU check fails.
- `crates/headroom-core/tests/kompress_parity.rs`: both tests now skip,
not fail, when the model is cached but no ONNX Runtime can be found, and
print the loader's reason. This follows `magika_idle_threads`, but gates
on the loader's own discovery instead of `ORT_DYLIB_PATH` alone, so the
test still runs wherever `VIRTUAL_ENV` / `./.venv` provides
`onnxruntime` (the pre-push hook's case).
- `crates/headroom-core/tests/kompress_ort_unavailable.rs` (new): a
minimal WordLevel `tokenizer.json`, plus `ORT_DYLIB_PATH` pointing at a
missing file. It expects `Kompress::from_files` to return
`KompressError::Onnx` naming `ORT_DYLIB_PATH` within 60 s. On timeout it
writes the reason to stderr and aborts, because the stuck thread also
wedges normal process exit. It's its own test binary because the loader
result is process-wide.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Rust-only change; no Python files touched. Rust gates:

### Test Output

```text
$ cargo test -p headroom-core --test kompress_ort_unavailable      # main 956b8855, build_session without the guard
running 1 test
Kompress::from_files deadlocked loading a missing ONNX Runtime
  process didn't exit successfully: `.../kompress_ort_unavailable-25bf304d5851aba5` (signal: 6, SIGABRT: process abort signal)
(65 s: the 60 s timeout, then abort)

$ cargo test -p headroom-core --test kompress_ort_unavailable      # this branch
test missing_onnx_runtime_is_an_error_not_a_hang ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

$ cargo fmt --all -- --check                                        # clean
$ cargo clippy --workspace -- -D warnings                           # clean
$ cargo clippy -p headroom-core --all-targets -- -D warnings        # clean
$ VIRTUAL_ENV=<repo>/.venv cargo test -p headroom-core
1118 passed; 0 failed; 3 ignored (all test binaries summed)

$ cargo test -p headroom-core --test kompress_parity -- --nocapture  # neither ORT_DYLIB_PATH nor VIRTUAL_ENV, model cached
SKIP: no ONNX Runtime to run kompress: no pip onnxruntime native library was found for Magika dynamic ONNX Runtime loading; install headroom-ai[proxy], install onnxruntime, or set ORT_DYLIB_PATH
test result: ok. 2 passed; 0 failed; ... finished in 0.00s   (both skipped)
$ VIRTUAL_ENV=<repo>/.venv cargo test -p headroom-core --test kompress_parity -- --nocapture
kompress parity: 21 fixtures matched byte-for-byte
test result: ok. 2 passed; 0 failed; ... finished in 5.82s
$ HF_HUB_CACHE=<empty dir> cargo test -p headroom-core --test kompress_parity -- --nocapture
SKIP: kompress model/tokenizer not in HF cache; run `python scripts/record_kompress_trace.py` first
```

## Real Behavior Proof

- Environment: macOS 27.2 (arm64), rustc 1.95.0, Python 3.11.17 venv
with pip `onnxruntime` 1.26.0, the Kompress model and tokenizer in
`~/.cache/huggingface/hub` (`chopratejas/kompress-v2-base` snapshot
`b1563631`). Branch is one commit on `main` `956b8855`.
- Exact command / steps: (1) the new test with the `build_session`
change reverted, then with it. (2) `cargo test -p headroom-core --test
kompress_parity` three ways: neither `ORT_DYLIB_PATH` nor `VIRTUAL_ENV`
set; `VIRTUAL_ENV=<repo>/.venv`;
`ORT_DYLIB_PATH=<venv>/onnxruntime/capi/libonnxruntime.1.26.0.dylib`.
(3) The original hang: the pre-push `make ci-precheck` (`cargo test
--workspace`) on `main` with the model cached, sampled with `sample
<pid>`.
- Observed result: without the fix the new test deadlocks and
`kompress_parity` hangs. With it, the new test passes, and
`kompress_parity` checks all 21 fixtures when ONNX Runtime can be found
and skips with the reason when it can't.
- (1) Before: deadlock, then the test's abort after 60 s (output above).
After: passes in 0.00 s with `KompressError::Onnx` naming
`ORT_DYLIB_PATH`.
  - (2) `kompress_parity`, this branch:
- neither variable set: with the first commit alone, both tests fail in
0.16 s with `Onnx { model_id: "chopratejas/kompress-v2-base", ...,
source: "no pip onnxruntime native library was found ..." }`. With the
second, both skip and print that reason. On `main` the same setup hangs
forever.
- `HF_HUB_CACHE` pointed at an empty directory: both skip with the
existing "not in HF cache" message, as before.
- `VIRTUAL_ENV` set: `kompress parity: 21 fixtures matched
byte-for-byte`, 2 passed. The guard found the venv's library by itself.
    - `ORT_DYLIB_PATH` set: 21 fixtures matched, 2 passed.
- (3) On `main`: `kompress_parity` blocked for 14+ min at 0.29 s of CPU
time, with the two stacks in the description. With this fix in the same
hook, `kompress_parity` passed with no `ORT_DYLIB_PATH` set, and `make
ci-precheck` passed.
- Not tested: Linux/Windows, the native proxy end to end, and the
pre-push hook on this exact branch.
- Linux and Windows (the guard is cross-platform and already used by
Magika and fastembed there).
- The native `headroom-proxy` binary with ONNX Runtime unavailable at
runtime (covered at the code level by the new test, but not run end to
end).
- `make ci-precheck-python` on this exact branch. I pushed it from a
worktree with no venv, so I ran the Rust gates by hand (above) instead
of the pre-push hook. There are no Python changes, and the full `make
ci-precheck` (Rust and Python) passed on a branch carrying this commit.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for Rust callers whose ONNX
Runtime fails to load: they now get `KompressError::Onnx` instead of a
deadlock. When ORT loads, the session is built exactly as before.
- Kill switch / disable path: n/a; `ORT_DYLIB_PATH` still selects the
library
- Unsafe override required: no
- Qualification impact: none; model, tokenizer and inference are
unchanged
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- No documentation change: the guard and its behaviour are already
described in `dynamic_ort_loader_ready`'s doc comment.
- Commits: the fix; `test(kompress): skip kompress_parity when no ONNX
Runtime can be found`, which adds one public function,
`kompress::onnx_runtime_ready`; and, after the main merge,
`fix(kompress): check the CPU before the loader in onnx_runtime_ready`
for the review finding below. If maintainers would rather not widen the
API, the test could match on the error instead, but that ties it to the
error's text.

## Review response (49453a11)

- Finding: `onnx_runtime_ready()` / `loadable_artifacts` called the
loader directly, bypassing #4559's CPU check. Fixed: it now runs
`ort_session_gate(onnx_runtime_supported_by_cpu(),
dynamic_ort_loader_ready)`, the CPU check first and the loader only when
it passes. `build_session` is back to #4559's single
`ort_session_preconditions()` call.
- New coverage: `onnx_runtime_ready_checks_the_cpu_before_the_loader`.
Against a version that calls the loader without the CPU check, it fails
(`called Result::unwrap_err() on an Ok value`); with the fix it passes,
together with both `ort_session_gate` tests.
- Kept: the missing-library regression `kompress_ort_unavailable`
passes. It now also accepts `cpu_unsupported`, since on a CPU without
AVX2 the gate answers before any library is loaded. Positive parity:
`kompress parity: 21 fixtures matched byte-for-byte` with `VIRTUAL_ENV`
and with `ORT_DYLIB_PATH`. With neither set: `SKIP: can't run ONNX
Runtime for kompress here: ort_loader_unavailable: no pip onnxruntime
native library was found ...`.
- Gates on `49453a11` (macOS arm64): `cargo fmt --all -- --check`,
`cargo clippy --workspace -- -D warnings`, `cargo clippy -p
headroom-core --all-targets -- -D warnings` clean; `cargo test -p
headroom-core` 1124 passed, 0 failed, 3 ignored.
- Not reproduced on non-AVX2 hardware: the CPU path is covered by the
injected unit test only.
- `test (2)` on the previous head failed in
`tests/test_gemini_memory_tail_preserved.py` (4 tests, `'_FakeRequest'
object has no attribute 'stream'`). Main `e4a60ed7`'s own CI fails the
same 4. It's a merge interaction between #4142 (Gemini now reads
`request.stream()` via `read_request_json_with_bytes`) and #4506 (the
test stubs the old `_read_request_json`), unrelated to this PR. Fix:
#4645 (test-only).


## Maintainer current-main qualification
Published `be01d4b2b68e12842f2fb4ba7ee66803e5c04dda` incorporates
upstream main `cc1eb5402aade58204b96b6b4b4d280cdd5456d2`. All three
targeted Rust tests pass on Windows: actual missing ORT path returns the
expected KompressError::Onnx in 0.01s; two parity tests pass with all 21
fixtures byte-identical using the actual venv ONNX runtime. Fresh cargo
test build, cargo fmt --all -- --check, cargo clippy -p headroom-core
--all-targets -- -D warnings, and whitespace checks pass. All three
constructors converge on the guarded build_session; the shared
process-wide loader is unchanged. Author actual before/after deadlock
evidence read; Devin has no findings. No end-to-end native proxy claim.
Contributor correctness logic was not rewritten. This independently
qualifies the real Windows Rust runtime path; it does not establish
installed native-proxy behavior. Hosted CI must finish on this exact
head before approval. Nothing merged.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-05 10:53:53 -06:00
69fbd2390d fix(ort): avoid implicit runtime discovery from the workspace (#4570)
## Description

When `ORT_DYLIB_PATH` is unset, native ONNX Runtime discovery
inadvertently searches `.venv` and `venv` directories under the current
working directory, even when neither represents an active environment.
Consequently, a caller launched inside an untrusted directory or
workspace could implicitly load a native binary from that location.

This change eliminates those two implicit roots from fallback discovery.
Workspace environments remain fully discoverable whenever explicitly
activated via `VIRTUAL_ENV` or `CONDA_PREFIX`. Standard user-install
locations and explicit `ORT_DYLIB_PATH` configuration remain unaffected.

Addresses the discovery-root behavior discussed in #4547 (hardening
fallback discovery paths when no runtime is explicitly pinned).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change

## Changes Made

- Stop implicitly searching the current workspace's `.venv` and `venv`
for native libraries.
- Cover both rejected implicit roots alongside explicitly activated
virtual environments (`VIRTUAL_ENV` / `CONDA_PREFIX`).
- Run each discovery regression in an isolated subprocess so environment
and directory mutations cannot leak into parallel test runs.

## Testing

- [x] Discovery regressions: **2 failed, 3 passed** on main; **5
passed** after the patch.
- [x] Standalone native executable exercises the public
`headroom_core::transforms::magika_detect` API against the compiled
library.
- [x] Pre-commit and commit-message hooks pass cleanly.
- [x] Pre-push verification (`make ci-precheck`): **1,676 Rust tests
passed, 3 ignored**; native extension rebuilt/imported cleanly; **178
Python tests passed, 5 skipped**; commitlint passed.

```text
Before: discovery_ignores_cwd_dot_venv FAILED
        discovery_ignores_cwd_venv FAILED
After:  5 passed
```

## Real Behavior Proof

- Environment: Windows x86-64, Rust 1.95.0, default `headroom-core`
features, `main` at `a2d4fe72381e186651466ee94e92ce9d8d1396cc`.
Standalone executable links the compiled core library and calls its
public detector without custom discovery overrides.
- Exact command / steps: Build the core library and compile the Rust
probe: `rustc --edition=2021 native_probe.rs --extern
headroom_core=<built libheadroom_core-*.rlib> -L dependency=<cargo
target>/debug/deps -o native_probe.exe`. Run from a scratch directory
containing a dummy
`.venv/Lib/site-packages/onnxruntime/capi/onnxruntime.dll` (`b"not a
native library"`) under a 15-second timeout to exercise unpinned
fallback discovery on `main` vs this patch.
- Observed result: On `main`, the non-empty detector call hangs/times
out at 15 seconds against the invalid stub DLL in cwd (clean cwd and
empty-input controls succeed). With this patch, the call returns
immediately with the expected clean initialization error because the
stub in cwd is ignored. Regressions confirm implicit cwd stubs are
excluded while explicitly activated virtual environments remain valid
candidates.
- Not tested: Valid ONNX inference, malicious native payloads, normal
Python proxy traffic, or native execution on Linux/macOS. No valid
native library was placed in the workspace.

```rust
fn main() {
    for name in ["ORT_DYLIB_PATH", "VIRTUAL_ENV", "CONDA_PREFIX", "USERPROFILE"] {
        std::env::remove_var(name);
    }
    std::env::set_var("HEADROOM_MAGIKA_INIT_TIMEOUT_SECS", "1");
    eprintln!("calling public magika_detect");
    let content = std::env::args().nth(1).unwrap_or_else(|| "hello".into());
    match headroom_core::transforms::magika_detect(&content) {
        Ok(kind) => println!("detected: {kind:?}"),
        Err(error) => println!("detector error: {error}"),
    }
    eprintln!("public call returned");
}
```

The native executable can be driven with:

```python
from pathlib import Path
import subprocess
import tempfile

executable = str(Path("native_probe.exe").resolve())
with tempfile.TemporaryDirectory() as directory:
    root = Path(directory)
    stub = root / ".venv/Lib/site-packages/onnxruntime/capi/onnxruntime.dll"
    stub.parent.mkdir(parents=True)
    stub.write_bytes(b"not a native library")
    try:
        result = subprocess.run(
            [executable, "hello"], cwd=root, capture_output=True,
            text=True, timeout=15,
        )
        print(result.returncode, result.stdout, result.stderr)
    except subprocess.TimeoutExpired:
        print("detector did not return within 15 seconds")
```

```text
Main, stub cwd, nonempty input: timeout at 15 seconds
Main, clean cwd:               normal no-runtime initialization error
Main, stub cwd, empty input:   PlainText, normal exit
Patch, stub cwd, nonempty:     normal no-runtime initialization error
```

The timeout observation reflects this specific Windows loader setup; the
contract of this patch is strictly boundary hardening for discovery
paths rather than a general loader timeout fix.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: Fallback discovery no longer
implicitly trusts unactivated `.venv` / `venv` directories under the
current working directory.
- Kill switch / disable path: Explicitly configure `ORT_DYLIB_PATH`, or
activate the environment via `VIRTUAL_ENV` / `CONDA_PREFIX`.
- Unsafe override required: no new override flag.
- Qualification impact: Any callers relying on implicit unactivated
workspace venvs will need to activate the environment or set
`ORT_DYLIB_PATH`. No new dependencies, public API, or file-format
changes.
- Rollback path: revert this patch; no migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

I implemented and verified this with Codex assistance for drafting and
local tests.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4570"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `6e4efe83bc7ced929e4769e371ed024ad213247c`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Clean mechanical update from current upstream main. The production delta
remains removal of implicit cwd .venv/venv roots from native ONNX
Runtime discovery. Explicit VIRTUAL_ENV and CONDA_PREFIX roots and other
existing trusted discovery sources are preserved. No contributor
discovery policy was rewritten and no new finding surfaced in the
reviewed scope.

Fresh native Windows/Rust 1.95 compilation ran all five named discovery
cases, including both implicit-cwd rejection tests, both
explicit-environment acceptance tests and their isolated child. The log
confirms the current worktree compiled and all five ran, avoiding the
historical stale-target binary issue. Child processes enumerate a
harmless invalid-library fixture; they never load it. This qualifies
candidate enumeration, not loading an arbitrary library, a fresh Python
extension build, ML inference or every platform. rustfmt and
main-relative whitespace checks pass. Ten shared Gemini streamed-request
fixture cases also pass, with Ruff/format clean; that test-only repair
addresses independently verified main failures.

Prior author discussion and exact-head approval were read, including the
historical 20 ms CCR-expiry test failure outside this changed file. It
is not carried forward as a discovery finding or a reason to withhold
approval. Maintainers handle the new exact-head hosted checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 10:47:01 -06:00
c6f5bf9549 fix(ccr): keep the in-memory order queue from evicting re-stored entries (#4545)
## Description

`InMemoryCcrStore` left a hash in its FIFO `order` queue when `get()`
removed the entry because it had expired. If the same hash was stored
again, `put` pushed a second copy, and at capacity
`evict_until_under_capacity` popped the stale front copy and removed the
fresh entry, the newest one, while an older entry survived. The queue
also grew by one item for every entry that expired through `get()`. Each
entry now carries a sequence number, eviction removes an entry only when
the popped queue item's sequence matches, and the queue is compacted
once it passes twice the capacity.

Fixes #4544

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/ccr/backends/in_memory.rs`:
- `Entry` gets a `seq: u64`; the store gets an `AtomicU64` counter;
`order` holds `(hash, seq)`.
- `evict_until_under_capacity` pops `(hash, seq)` and calls
`remove_if(hash, |e| e.seq == seq)`, so a stale item removes nothing and
the loop moves on.
- A fresh-key `put` takes the `order` lock first and does the capacity
check, the map insert and the queue append inside it, re-checking for a
writer that stored the same hash while it waited. So no entry is ever in
the map without its queue item while eviction runs (review round 1: two
writers missing the same hash could otherwise leave a capacity-1 store
holding 2). Lock order is `order` then a map shard everywhere; `get` and
the overwrite fast path never take `order` while holding a shard guard.
- `compact_order` drops dead queue items once the queue is longer than
`2 * max(capacity, 16)`. The in-place overwrite fast path is unchanged.
- Three tests: `reput_after_expiry_keeps_the_newest_entry`,
`order_queue_stays_bounded_when_entries_expire_through_get`, and
`concurrent_fresh_inserts_never_exceed_capacity`, which uses a
`#[cfg(test)]` hook to pause writer A between its insert and its queue
append while writer C stores another key.

## Testing

- [x] Unit tests pass (`cargo test`)
- [x] Linting passes (`cargo clippy -- -D warnings`, `cargo fmt
--check`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# fail before (original in_memory.rs, the two new tests only)
a=None b=Some("2") order_len=3
assertion `left == right` failed: fresh a was evicted instead of b
test ccr::backends::in_memory::tests::probe_reput_after_expiry_keeps_newest_entry ... FAILED
map_len=0 order_len=2000
order queue grew to 2000 while the map held 0
test ccr::backends::in_memory::tests::probe_order_queue_stays_bounded ... FAILED
test result: FAILED. 0 passed; 2 failed

# round 2, fail before (round-1 head 9ddf68cb + the hook and the concurrency test)
capacity 1 store holds 2 entries
test result: FAILED. 0 passed; 1 failed

# pass after: cargo test -p headroom-core --lib (default features)
test result: ok. 987 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 5.39s

# cargo clippy -p headroom-core --lib --tests -- -D warnings: no warnings
# cargo fmt -p headroom-core -- --check: clean
```

(The fail-before run used the tests under their draft names `probe_*`;
the bodies are the same apart from the debug `eprintln!` lines.)

## Real Behavior Proof

- Environment: Windows 11, rustup toolchain 1.95.0 (the
`rust-toolchain.toml` pin), `main` at `a2d4fe72`; a separate scratch
binary crate (`ccrprobe`, not part of this PR) that depends on
`headroom-core` by path with `default-features = false` and uses the
store only through the public `headroom_core::ccr::{CcrStore,
InMemoryCcrStore}` API, built with `cargo run --release`.
- Exact command / steps: the binary (1) stores `a` in a capacity-2 store
with a 30 ms TTL, sleeps 60 ms, calls `get("a")`, stores `b`, stores `a`
again, stores `c`, then reads all three; (2) runs 200 rounds of 8
threads x 50 `put`s over 40 keys into a capacity-8 store and records the
largest `len()` seen. Run once against this branch and once with
`in_memory.rs` from `main`.
- Observed result: on `main`: `after put(c): a=None b=Some("2")
c=Some("3") len=2` (the freshly re-stored `a` was evicted, the older `b`
kept). On this branch: `after put(c): a=Some("fresh") b=None c=Some("3")
len=2`. The concurrent run printed `max len seen = 8` on both, so the
over-capacity interleaving does not show up by chance; it is pinned
deterministically by `concurrent_fresh_inserts_never_exceed_capacity`
instead (2 entries on the round-1 head, at most 1 now).
- Not tested: the Python wheel rebuilt with this crate, the Rust proxy
binary, and the SQLite/Redis backends (unchanged).

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Internal fix to the in-memory CCR
store.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, narrowly. Eviction now removes
the oldest live entry instead of possibly the newest after an
expire-and-re-store; the queue no longer grows past twice the capacity.
Hit, miss and TTL behaviour are unchanged.
- Kill switch / disable path: None needed; revert to roll back.
- Unsafe override required: No
- Qualification impact: None expected. One extra `u64` per entry and per
queue item.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests were run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


### Maintainer current-main requalification

Exact head `48ae3713b096a6fba338cca7640424766bb0754a`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically merged upstream/main e4a60ed7 and re-reviewed the resulting
in-memory CCR implementation and existing discussion. Generation-matched
eviction, stale-queue compaction, and serialized fresh insertion retain
the contributor fix; no production logic was rewritten. All 12 native
in-memory backend tests pass on the freshly compiled current worktree,
including the synchronized concurrent-insert regression, expire/re-store
preservation, and bounded-queue checks. Rust formatting and whitespace
checks pass. The existing public-API executable reproduction remains
applicable; no fresh Python extension or full proxy runtime
qualification is claimed.

Applied the shared test-only Gemini streamed-request fixture repair for
the independently established main defect, preserving existing
assertions. All ten affected Gemini tests pass, with Ruff lint and
formatting checks clean. Formal approval awaits exact-head hosted CI.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


Maintainer commit-message qualification repair: current head
bdae501786. Corrected only the maintainer
main-sync commit message to Conventional Commit format. The complete
file tree is byte-identical to the previously tested head; exact-head
hosted checks are being refreshed. Nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-05 10:46:25 -06:00
5daaf33637 fix(kompress): run the CPU and ONNX Runtime loader guards before building a session (#4559)
## Description

The Rust Kompress init path builds an ONNX Runtime session without the
two guards the crate requires before any `ort` call. Magika
(`magika_detector.rs`) and the embedding scorer
(`relevance/embedding.rs`) already run both. Kompress did not.

Path on main `a2d4fe72`: `headroom-proxy/src/main.rs:63-68` (with
`--compression`, background `live_zone::warm_live_zone_compressors()`)
-> `live_zone.rs` `run_kompress_init()` -> `Kompress::from_cache` ->
`build_session` (`Session::builder()?.commit_from_file(path)?`).

Without the guards, as the crate's own comments describe them:
- a CPU without AVX2 hits `SIGILL` inside the precompiled ONNX Runtime
(#1723);
- an unpinned dynamic loader makes `ort` deadlock in its API-setup error
path (#1715).

Closes #4549

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `transforms/kompress.rs`: add `ort_session_gate(cpu_ok, loader)` (pure
decision) and `ort_session_preconditions()` (calls
`onnx_runtime_supported_by_cpu()` and `dynamic_ort_loader_ready()`, the
same pair magika and the embedding scorer use). `build_session` runs the
preconditions first. All three constructors (`from_files`,
`from_pretrained`, `from_cache`) go through `build_session`, so this is
the smallest place that covers every session Kompress builds.
- `transforms/live_zone.rs`: `run_kompress_init` runs the preconditions
first. On failure it settles the slot as `None` and logs
`tracing::warn!(event = "kompress_init_skipped", reason, error)`, with
`reason` = `cpu_unsupported` or `ort_loader_unavailable`. The existing
`kompress_init_complete` event is still emitted. It also skips cache
resolution when the runtime is unusable.
- `onnx_cpu.rs`: fix the stale module doc. It named two entry points and
`ort-download-binaries`; Kompress is a third entry point and Cargo.toml
uses `ort-load-dynamic`.
- Test: `ort_session_gate_rejects_unsupported_cpu_and_loader_failure`
(new seam, so there is no meaningful "fails before the fix" run).

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core --lib`)
- [x] Linting passes (`cargo clippy -p headroom-core --lib --tests -- -D
warnings`, `cargo fmt -p headroom-core -- --check`)
- [ ] Type checking passes (`mypy headroom`) (N/A, Rust change only)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib
test result: ok. 981 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 5.63s

$ cargo test -p headroom-core --lib ort_session_gate
test transforms::kompress::tests::ort_session_gate_rejects_unsupported_cpu_and_loader_failure ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 981 filtered out; finished in 0.00s

$ cargo clippy -p headroom-core --lib --tests -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 03s

$ cargo fmt -p headroom-core -- --check
(no output, exit 0)
```

## Real Behavior Proof

- Environment: Windows 11, native cargo, toolchain 1.95.0, headroom main
`a2d4fe72` plus this change, default features (`ml` on).
- Exact command / steps: `cargo test -p headroom-core --lib`; `cargo
test -p headroom-core --lib ort_session_gate`; `cargo clippy -p
headroom-core --lib --tests -- -D warnings`; `cargo fmt -p headroom-core
-- --check`.
- Observed result: 981 passed, 0 failed, 1 ignored. The gate test shows
the decision rejects a non-AVX2 CPU (`cpu_unsupported`) and a loader
error (`ort_loader_unavailable`) and passes only when both are fine.
Clippy and fmt are clean.
- Loader failure path, through the real warm path (review round 2):
Windows 11, AMD Ryzen 7 4800H (AVX2), native cargo 1.95.0, source
identical to this head `eff4992d`, default features. A temporary test
(not committed) added to `live_zone.rs` called
`ort_session_preconditions()` and then the public
`warm_live_zone_compressors()`; it ran alone in its own test process
(`--exact`, `--test-threads=1`) with `ORT_DYLIB_PATH` set to a path that
does not exist, plus `HEADROOM_OFFLINE=1`, `HF_HUB_OFFLINE=1`.
- Observed: `ort_session_preconditions()` =
`Err(("ort_loader_unavailable", "ORT_DYLIB_PATH points to a missing ONNX
Runtime library: ...\\onnxruntime.dll"))`;
`warm_live_zone_compressors()` = `false`; `KOMPRESS_INIT_RUNS` = 1, so
`run_kompress_init` ran once and took the skip branch before
`Kompress::from_cache`. The test passed in 0.01 s.
- Limits: no ONNX Runtime library was present or loaded (the missing
path is refused before any `init_from`). I did not capture the
`kompress_init_skipped` tracing event itself (no subscriber in the
crate's dev-dependencies); the branch is inferred from the precondition
result plus the init counter. The CPU-unsupported branch is covered only
by the unit test.
- Not tested: I did not reproduce the `ort` deadlock or the `SIGILL` on
a non-AVX2 CPU. I did not run the proxy with `--compression` end to end.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; Kompress init stays behind the
existing `ml` feature and `--compression`.
- Minimum rollout channel: n/a (no new feature, only a precondition
check).
- Stable/default behavior changed: only where the runtime is unusable
(no AVX2, or loader not pinned); there Kompress settles as a no-op
instead of risking a crash or hang. Working hosts behave as before.
- Kill switch / disable path: unchanged (do not pass `--compression`).
- Unsafe override required: none.
- Qualification impact: none expected.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- The guard in `build_session` also covers `from_files` and
`from_pretrained`; `run_kompress_init` checks first so it can log the
dedicated `kompress_init_skipped` event. The loader guard is
process-wide and cached (`OnceLock`), so the second check is free.
- No new `ort::init_from` call and no library is loaded by the tests.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests were run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 23:20:57 -06:00
9e8cc64b48 fix(core): case-insensitive Bearer scheme, clamp SQLite CCR TTLs, redact the Redis URL in logs (#4561)
## Description

I fix the three small `headroom-core` defects from #4551. None of them
changes a shipped path today (the only production caller of
`auth_mode::classify` coerces every result to OAuth, and the SQLite and
Redis CCR backends are only built from tests and benches), so this is
hardening before those paths get wired in.

1. `auth_mode::classify` matched the Bearer scheme case-sensitively and
did not trim. `bearer sk-ant-api03-...` was classified OAuth (should be
Payg) and `Bearer sk-ant-oat01-...` (two spaces) was classified Payg
(should be OAuth). The scheme is now matched with `eq_ignore_ascii_case`
(RFC 9110 section 11.1) and the token is `trim_start`ed.
2. SQLite CCR: `u64 as i64` on TTLs wrapped `u64::MAX` to -1, so a put
followed by a get returned `None`. A small `ccr::secs_i64` helper
saturates to `i64::MAX`, used at every non-test cast in `sqlite.rs`.
`redis.rs` had the same overflow at `conn.expire(&key, new_ttl as i64)`
(`new_ttl` comes from `max_lifetime_for`, which saturates), so I
switched it to the same helper.
3. `from_config` logged the full Redis URL at info, including
`redis://:password@host`. It now logs `scheme://host[:port]` via a small
`redact_redis_url` (userinfo, path and query dropped; a string with no
`://` becomes a placeholder). No new dependency.

Fixes #4551

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/auth_mode.rs`: case-insensitive `Bearer `
scheme, `trim_start` on the token; test
`bearer_scheme_is_case_insensitive_and_token_is_trimmed` (the four cases
from the issue plus an upper-case scheme, fake tokens only).
- `crates/headroom-core/src/ccr/mod.rs`: `pub(crate) fn secs_i64(u64) ->
i64` (saturating).
- `crates/headroom-core/src/ccr/backends/sqlite.rs`: use `secs_i64` at
the TTL/lifetime/timestamp casts; test
`huge_ttl_means_keep_not_expire_immediately` (temp dir via `tempfile`,
`ttl = u64::MAX`).
- `crates/headroom-core/src/ccr/backends/redis.rs`: `expire(&key,
secs_i64(new_ttl))`.
- `crates/headroom-core/src/ccr/backends/mod.rs`: `redact_redis_url`,
used in the `ccr_backend_initialized` log; test
`redis_url_credentials_are_not_logged`.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Rust-only change, so the Python checks above are not applicable. I ran
`cargo test -p headroom-core --lib`, `cargo clippy -p headroom-core
--lib --tests -- -D warnings`, `cargo fmt -p headroom-core -- --check`,
and `cargo check -p headroom-core --lib --features redis`.

### Test Output

Before (source fixes reverted, new tests kept):

```text
test auth_mode::inline_tests::bearer_scheme_is_case_insensitive_and_token_is_trimmed ... FAILED
test ccr::backends::sqlite::tests::huge_ttl_means_keep_not_expire_immediately ... FAILED
  left: OAuth
 right: Payg
  left: None
 right: Some("payload")
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 982 filtered out
```

After:

```text
test ccr::backends::tests::redis_url_credentials_are_not_logged ... ok
test auth_mode::inline_tests::bearer_scheme_is_case_insensitive_and_token_is_trimmed ... ok
test ccr::backends::sqlite::tests::huge_ttl_means_keep_not_expire_immediately ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 981 filtered out

cargo test -p headroom-core --lib
test result: ok. 983 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out

cargo clippy -p headroom-core --lib --tests -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s)

cargo fmt -p headroom-core -- --check   (exit 0)
cargo check -p headroom-core --lib --features redis   Finished
```

## Real Behavior Proof

- Environment: Windows 11, native Windows cargo, Rust 1.95.0 (pinned
toolchain), clone of `main` at `a2d4fe72` plus this change.
- Exact command / steps: `cargo test -p headroom-core --lib --
bearer_scheme huge_ttl redis_url`, once with the source fixes reverted
(tests kept) and once with the fixes; then the full `--lib` run, clippy
and fmt as above.
- Observed result: with the fixes reverted the bearer and SQLite tests
fail with exactly the values in the issue (`bearer ...` gives OAuth,
`ttl = u64::MAX` put-then-get gives `None`); with the fixes all pass,
983 passed in the full lib run.
- Not tested: no Redis server, so the Redis `expire` change is
compile-checked (`--features redis`) only and the redaction is covered
by a unit test of the helper, not by capturing a real log line. These
paths are not wired into production today per the issue, so there is no
end-to-end behavior to observe yet. The `Bearer` tab-separator case and
non-ASCII schemes were not exercised beyond the code reading.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for shipped paths; `classify`
gives a different result only for non-canonical Bearer headers
(lowercase scheme, extra spaces) and well-formed headers keep their
classification
- Kill switch / disable path: none needed; revert the PR
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The three fixes are independent; I can split them if you prefer one PR
per item. Existing tests in `crates/headroom-core/tests/auth_mode.rs`
were not changed and still pass.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests were run on my machine against a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4561"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 23:20:34 -06:00
Jay HemnaniandJD Davis 9355ec7019 fix(diff): renumber hunk headers after trimming context (#4319)
## Description

`DiffCompressor` trims each hunk to `max_context_lines` (2) around its
changes but keeps the hunk's original `@@ -a,b +c,d @@` header. The
header no longer describes the lines under it:

- Leading context is dropped first, so every line number read off a hunk
is off by the lines cut (one with git's default `-U3`).
- Context dropped between two changes more than 4 lines apart leaves
them next to each other. Nothing marks the gap, and every line after it
is numbered wrong.

A 400-line file with lines 10 and 17 changed, `git diff` (73 lines)
through `DiffCompressor().compress()` on 0.41.0:

```text
@@ -7,14 +7,14 @@ line 6
 line 8
 line 9
-line 10
+line 10 changed
 line 11
 line 12
 line 15
 line 16
-line 17
+line 17 changed
```

By the header, `line 8` is line 7 and `-line 17` is line 14. Anything
that maps these hunks to file lines (a model writing an edit, `git
apply`) gets the wrong place.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `reduce_context` splits a trimmed hunk at each gap and recomputes
every header from the original line numbers, one range per parent for
combined diffs, formatted the way git writes them. The section heading
is carried over unchanged.
- A hunk that loses no lines keeps its header byte-for-byte.
- Lines after the extent a header declares (the empty string from a
trailing newline, a format-patch `-- ` signature) are not numbered and
no longer count as changes.
- A format-patch signature and what follows it stay whatever the context
limit.
- A hunk with edits past the extent its header declares is malformed
input and passes through as it came.
- `hunks_kept` / `hunks_removed` still count the original hunks, so a
split can't underflow `hunks_removed`.
- Five `diff_compressor` parity fixtures recorded the old headers. In
their outputs 33 hunks had header counts that disagree with their own
lines; with this change there are none. Only `output.compressed` changes
in those files.

## Testing

- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib
test result: ok. 979 passed; 0 failed; 1 ignored

# new tests against the previous commit:
  left: ["@@ -7,14 +7,14 @@ l6", "@@ -23,6 +23,7 @@ l22"]
 right: ["@@ -8,5 +8,5 @@ l6", "@@ -15,5 +15,5 @@ l6", "@@ -24,4 +24,5 @@ l22"]
test result: FAILED. 0 passed; 2 failed

$ cargo run -p headroom-parity -- run --fixtures tests/parity/fixtures
[diff_compressor ] total=27 matched=27 skipped=0 diffed=0

$ pytest <the 12 test files that contain hunk headers> tests/test_compress*.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/parity
754 passed, 3 skipped
$ cargo clippy -p headroom-core --lib --tests -- -D warnings   # clean
```

## Real Behavior Proof

- Environment: Ubuntu, Python 3.12, headroom-ai 0.41.0 from PyPI vs this
branch built with `pip install -e .`
- Exact command / steps: for each commit, `git diff c~1 c` through
`DiffCompressor(DiffCompressorConfig(max_hunks_per_file=10**6,
max_files=10**6)).compress(...)`. Then compare every hunk (header ranges
and lines) with `git diff -U2 c~1 c`. Last 300 non-merge commits of this
repo, plus 200 of another Python repo. Merge commits the same way with
`git show -c` / `-c -U2`.
- Observed result:
  ```text
  0.41.0, this repo:    compressed=276 match_git_U2=3
  branch, this repo:    compressed=276 match_git_U2=276
  branch, second repo:  compressed=163 match_git_U2=163
  branch, git show -c:  compressed=36  match=36
  ```
  Same example after the change:
  ```text
  @@ -8,5 +8,5 @@ line 6
   line 8
   line 9
  -line 10
  +line 10 changed
   line 11
   line 12
  @@ -15,5 +15,5 @@ line 6
   line 15
  ```
- Not fully matched: `git show --cc`. 8 of 11 merges match `--cc -U2`.
In the other 3, git hides more single-parent hunks when the context
shrinks, which a context trim doesn't try to copy. The compressor keeps
every change line of the `--cc` input there.

## Runtime Rollout Safety

N/A: no rollout-managed feature, no config change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

The section heading after the second `@@` is kept from the original
hunk. git recomputes it for each hunk, but that depends on the file's
language driver, and the original heading still names the enclosing
context.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4319"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:51:06 -06:00
1bbc0ea89e test(tokenizer): hold the env lock in the invalid-repo Hub error test (#4151)
## Description

`from_pretrained_invalid_repo_returns_hub_error`
(`crates/headroom-core/src/tokenizer/hf_impl.rs`) reads
`HEADROOM_OFFLINE` through `guard_egress` without holding
`crate::test_support::env_lock()`. The `*_refuses_while_offline` tests
set that variable to `"1"` process-wide under the lock. When cargo runs
them on parallel threads, this test can land inside that window and get
`HfTokenizerError::Offline` instead of `Hub`. That happened on the
ubuntu job of #4037 at `cbbf77c9`. It also fails every time when
`HEADROOM_OFFLINE=1` is exported in the developer's shell.

The test now takes the env lock and clears any inherited
`HEADROOM_OFFLINE` for the call, then restores it. This is a test-only
change.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/tokenizer/hf_impl.rs`:
`from_pretrained_invalid_repo_returns_hub_error` holds
`crate::test_support::env_lock()`, the same lock the offline tests use.
It removes an inherited `HEADROOM_OFFLINE` around the
`HfTokenizer::from_pretrained("")` call and restores it afterwards. No
production code changes.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Deterministic reproduction of the race: export HEADROOM_OFFLINE=1, which is the state a
# parallel offline test leaves the process in. Same checkout, only hf_impl.rs swapped, forced rebuild.

# origin/main (976aa714) version of hf_impl.rs
$ HEADROOM_OFFLINE=1 cargo test -p headroom-core --lib tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error
test tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error ... FAILED
expected HfTokenizerError::Hub, got Err(Offline(OfflineEgressBlocked { purpose: "HuggingFace tokenizer download", destination: "" }))

# this branch
$ HEADROOM_OFFLINE=1 cargo test -p headroom-core --lib tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error
test tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error ... ok

$ cargo test -p headroom-core --lib tokenizer
test result: ok. 46 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings   # clean
$ cargo fmt --all -- --check                                   # clean
```

## Real Behavior Proof

- Environment: macOS arm64, Rust toolchain pinned by
`rust-toolchain.toml` (1.95.0), `headroom-core` lib tests.
- Exact command / steps: `HEADROOM_OFFLINE=1 cargo test -p headroom-core
--lib
tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error`,
run against `origin/main`'s `hf_impl.rs` and against this branch's
version in the same checkout, with a forced rebuild between the two
runs.
- Observed result: `origin/main` fails with
`Err(Offline(OfflineEgressBlocked { purpose: "HuggingFace tokenizer
download", .. }))`, which is the error from #4037's ubuntu job. This
branch passes.
- Not tested: the timing race itself under CI's thread scheduling.
Exporting `HEADROOM_OFFLINE=1` reproduces the state the race produces,
every time.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A. Test-only change.
- Stable/default behavior changed: no.
- Kill switch / disable path: N/A. No runtime code changes.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Rust-only, test-only change: the Python checks (`pytest`, `ruff`,
`mypy`) weren't run because no Python changed, and no new test was added
(an existing one was fixed). The Rust checks that were run are in Test
Output.
- Documentation: N/A.
- The branch is based on an older `main`; the one-file diff applies
cleanly on current `main` (cherry-picked onto a branch that contains it
without conflicts).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4151"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 12:35:23 -06:00
eef44f60cc fix(kompress): read the pinned snapshot from Python's one HF cache in the native proxy (#4438)
## Description

**Stacked on #4190.** This branch contains #4190's commits (it reuses
#4190's Rust HF-cache resolver). Please review only the five commits
listed under Changes Made. Once #4190 merges, I'll rebase onto main and
the diff will shrink to those commits.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now
linear, with no merge commits:

- #4190's first three commits: `e173e100`, `344cf365`, `41918cf0`.
- This PR's five commits (SHAs updated under Changes Made).
- `d9f16d39` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does. This is #4190's last commit, applied to
`crate::hf_cache` since this PR moves the resolver there. It replaces
the earlier merge `a2e89118`.

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`.
- The main-sync merges (`7e17a2d3`, `6904fd19`, `8be25204`, `79e3a5cb`,
`19b1adc5`, and #4190's merges) are linearized away.
- The Rust embedder conflicts with main's batch-size cap (#4507) were
re-resolved to the same content as before: every file this PR touches is
byte-identical between the previous head `19b1adc5` and the new head
`d9f16d39` (`git diff 19b1adc5 d9f16d39 -- <files changed vs main>` is
empty). Only the history and SHAs changed.

The native proxy's live-zone Kompress is cache-only
(`Kompress::from_cache`): it reads the files Python downloaded. Its
cache lookup (`hf_hub_roots` + `hf_cache_file` on main) disagrees with
`huggingface_hub` in several ways. When it misses, `kompress_ready`
stays false and live-zone passes text through uncompressed. When it
loads, it can load the wrong files. This is the follow-up planned in
https://github.com/headroomlabs-ai/headroom/pull/4190#issuecomment-6209746394
(Devin's note on `hf_hub_roots`). The Python proxy resolves Kompress
through `huggingface_hub` and is unaffected.

| setup | Python | native Kompress on main |
|---|---|---|
| only `HUGGINGFACE_HUB_CACHE` | found | miss |
| only `XDG_CACHE_HOME` | found | miss |
| `HF_HUB_CACHE=${ROOT}/hub` | `<ROOT>/hub` | miss |
| `HF_HUB_CACHE` set + a copy under `$HOME` | looks only in
`HF_HUB_CACHE` | loads the `$HOME` copy |
| pinned snapshot plus older/newer ones | the `_PINNED_REVISIONS` commit
| first snapshot in `read_dir` order; tokenizer and ONNX looked up
independently |

The last row matters most: every Kompress pin bump leaves the old
snapshot in the cache.

Closes # (no issue; follow-up to #4190)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

Five commits on top of #4190:

- `af191f19` (was `ddec6b88`) fix(kompress): read the pinned snapshot
from Python's one HF cache in the native proxy.
- #4190's resolver moves to a shared `crate::hf_cache` module:
`hub_cache_dir`, the `expandvars` port, `pinned_or_main`, and a
pin-aware `snapshot_path`.
- Kompress reads one snapshot of each repo, at `DEFAULT_MODEL_REVISION`
/ `DEFAULT_TOKENIZER_REVISION`. A new `tests/test_kompress_rust_pins.py`
keeps these equal to `_PINNED_REVISIONS`.
- `HEADROOM_HF_PIN=off` and unpinned repos follow `refs/main`, as in
Python.
  - `kompress_cache_miss` logs the one cache and revision it looked in.
- New `kompress::cached_artifacts` lets the parity harnesses load the
files production would load.
- `7cdc3d94` (was `5e71eeea`) fix(tokenizer):
`HfTokenizer::from_pretrained` downloads into Python's hub cache with
Python's `HF_TOKEN` / `HF_ENDPOINT`. `hf-hub`'s defaults read neither,
and with `Api::new` not even `HF_HOME`.
- `c43c5eb8` (was `5fd3a87d`) fix(hf-cache): changes to the shared Hub
client:
- The tokenizer and embedder share one Hub client, `hf_cache::hub_api`.
- Trailing slashes are stripped from `HF_ENDPOINT`, as Python's
`constants.ENDPOINT` does.
- The token is withheld from a plain-`http://` non-loopback endpoint,
with a warning.
- `91128294` (was `70e61b02`) fix(hf-cache): `hub_api(cache, purpose,
repo)` runs `guard_egress` itself, before it builds the client. A new
caller can't skip the air-gap check, and the offline-egress sweep in
`tests/test_offline_egress_chokepoint.py` sees the site as guarded. The
tokenizer still returns `HfTokenizerError::Offline`, and the embedder
keeps its messages.
- `bbee29cd` (was `14dbac29`) review follow-ups:
  - `HEADROOM_OFFLINE` wins over a missing cache directory.
  - The `hub_api` docs are scoped to the tokenizer and embedder.
  - The order test checks inside `hub_api`'s body.
  - Stale comments are fixed.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `d9f16d39` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 1002 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py \
    tests/test_hf_revision_pinning.py tests/test_kompress_rust_pins.py
247 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

Before the rebase:

```text
$ cargo test -p headroom-core -- --skip kompress_matches_python_fixtures_byte_for_byte --skip short_input_passes_through
  (HF_HUB_CACHE=$(mktemp -d), cold like CI; own CARGO_TARGET_DIR)
20 test binaries, all "test result: ok", 0 failed
$ cargo clippy --workspace --all-targets -- -D warnings      clean
$ cargo clippy -p headroom-core --no-default-features --lib -- -D warnings   clean
$ cargo fmt --all -- --check                                  clean
$ pytest tests/test_offline_egress_chokepoint.py tests/test_kompress_rust_pins.py tests/test_fastembed_pinned_snapshot.py
195 passed, 4 skipped, 2 failed
  the 2 failures are TestHeadroomCloudCompressionOffline (litellm fetching its
  pricing table from raw.githubusercontent.com); they fail identically on
  origin/main e995eb9f with the locked LiteLLM 1.101.0, and #4242 fixes them
$ ruff check / ruff format --check (0.16.8)                   clean
```

mypy is not checked: this PR changes no `headroom/` Python.

## Real Behavior Proof

- Environment: macOS 27 arm64, debug build of `headroom-core` (default
`ml` feature), `ORT_DYLIB_PATH` = onnxruntime 1.31.0, the real cached
`chopratejas/kompress-v2-base` and `answerdotai/ModernBERT-base`
snapshots at their pins, symlinked into throwaway hub caches;
huggingface_hub 1.16.1 for the Python column.
- Exact command / steps: a throwaway example (not committed) calls
`Kompress::from_cache(KompressConfig::default())` and prints the
outcome. It runs once per cache layout under `env -i` with an empty
`HOME`, built from `origin/main` (e995eb9f) and from this branch. The
Python column prints `huggingface_hub.constants.HF_HUB_CACHE` under the
same environment. In the pin-bump layout, the older (`0000…`) and newer
(`ffff…`, `refs/main`) snapshots hold garbage `tokenizer.json` / ONNX
files.
- Observed result: the branch now agrees with Python in every layout. On
main, four layouts miss where Python finds the cache, one loads a copy
Python would never read, and the pin-bump layout loads a non-pinned
snapshot. Each side was built in its own target dir.

```text
== origin/main (e995eb9f)
HF_HUB_CACHE=<T>/hub (control)               loaded              | Python HF_HUB_CACHE=<T>/hub
only HUGGINGFACE_HUB_CACHE                   cache miss (None)   | Python HF_HUB_CACHE=<T>/hub
only XDG_CACHE_HOME                          cache miss (None)   | Python HF_HUB_CACHE=<T>/xdg/huggingface/hub
HF_HUB_CACHE=${PROBE_ROOT}/hub               cache miss (None)   | Python HF_HUB_CACHE=<T>/root/hub
HF_HUB_CACHE=<empty>, copy under $HOME       loaded              | Python HF_HUB_CACHE=<T>/empty
pin + older/newer snapshots, main=newer      error: failed to load tokenizer for `answerdotai/ModernBERT-base`: expected value at line 1 column 1 | Python HF_HUB_CACHE=<T>/pinbump
== this branch (14dbac29)
HF_HUB_CACHE=<T>/hub (control)               loaded              | Python HF_HUB_CACHE=<T>/hub
only HUGGINGFACE_HUB_CACHE                   loaded              | Python HF_HUB_CACHE=<T>/hub
only XDG_CACHE_HOME                          loaded              | Python HF_HUB_CACHE=<T>/xdg/huggingface/hub
HF_HUB_CACHE=${PROBE_ROOT}/hub               loaded              | Python HF_HUB_CACHE=<T>/root/hub
HF_HUB_CACHE=<empty>, copy under $HOME       cache miss (None)   | Python HF_HUB_CACHE=<T>/empty
pin + older/newer snapshots, main=newer      loaded              | Python HF_HUB_CACHE=<T>/pinbump
```

On main, the pin-bump row read a garbage non-pinned snapshot (here it
errors; with two real model versions it would silently load the wrong
one). The branch reads the pinned snapshot.
- Not tested: a Windows build (the `ntpath` expandvars rules are covered
by the shared fixture on macOS/Linux); a live Hub download through
`hub_api` with `HF_ENDPOINT` pointing at a real mirror; the native proxy
end to end (the probe calls the same `Kompress::from_cache` the proxy's
live-zone warm-up calls).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this changes where the native
proxy's cache-only Kompress and the Rust tokenizer/embedder look for Hub
files.
- Minimum rollout channel: stable (no rollout gate).
- Stable/default behavior changed: yes. The native proxy now loads
Kompress from the cache Python uses, at the pinned snapshot. It no
longer loads from other roots or from whichever snapshot `read_dir`
returns first. Rust tokenizer downloads land in Python's hub cache.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats the pinned
repos on `refs/main`, as in Python; Kompress itself is disabled as
before.
- Unsafe override required: no.
- Qualification impact: native live-zone Kompress becomes ready on hosts
whose Python cache is configured via `HUGGINGFACE_HUB_CACHE`,
`XDG_CACHE_HOME` or `$VAR` paths; parity harnesses read the pinned
artifacts.
- Rollback path: revert this PR; no persisted state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no user-facing docs describe the native Kompress cache
lookup.
- Follow-up, not in this PR: `Kompress::from_pretrained` still builds
its own `hf_hub::api::sync::Api::new()` client behind its own guard, so
it ignores `HF_ENDPOINT` / `HF_TOKEN` and downloads into `hf-hub`'s
default cache. It has no production caller (the proxy uses
`from_cache`); moving it onto `hf_cache::hub_api` is a separate change.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4438"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 12:29:32 -06:00
0a9aa66bff test(onnx): fix magika idle-thread test on macOS with ORT 1.31 (#4194)
## Description

`tests/magika_idle_threads.rs` (added in #4189) fails on macOS with ONNX
Runtime 1.31. Building the magika session leaves 8 extra threads against
a fixed allowance of 2.

The cause is not a thread pool. ORT 1.31 registers Core ML as a built-in
EP that claims the Apple NPU/GPU (microsoft/onnxruntime#31975). The
`ort` 2.0.0-rc.12 crate sets the `MAX_EFFICIENCY` (= `PREFER_NPU`) EP
selection policy on every `SessionBuilder::new()`. So magika's session
now runs on Core ML / Apple Neural Engine (ANE), and building it starts
5–7 ANE service threads that are idle, not spinning. The fixed allowance
of 2 can't account for that.

The test now builds a pool-free control session
(`magika::Session::builder().with_intra_threads(1)`) first and uses the
threads it adds as the allowance. The control goes through the same EP
selection, so whatever the EP starts cancels out, while a reintroduced
intra-op pool still shows up on top.

As a precaution, Rust CI now also runs `cargo test` on `macos-26`. It
only ran on Ubuntu, which is why this slipped through.

Closes #4191

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/tests/magika_idle_threads.rs`: measure the
allowance against a pool-free control session instead of a fixed `2`,
with comments explaining the ORT 1.31 Core ML/ANE cause. Test-only; no
production code changes.
- `.github/workflows/rust.yml`: turn the `test` job into a matrix with
`ubuntu-latest` and `macos-26`, using the same ONNX Runtime setup step.
Ubuntu keeps its `test (ubuntu)` check name and runs the same steps as
before. macOS runs only `cargo test`; `cargo fmt` and `cargo clippy`
stay on Ubuntu. `fail-fast: false`, so one OS failing doesn't cancel the
other.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Rust: `cargo fmt --all -- --check`, `cargo clippy --workspace --
-D warnings`, `cargo test --workspace` with `ORT_DYLIB_PATH` set so the
magika tests actually run
- [ ] Unit tests pass (`pytest`): only the pre-push `ci-precheck-python`
subset was run (177 passed, 4 skipped); the full suite was not
- [x] Linting passes (`ruff check .`): no Python files changed;
pre-commit hooks passed
- [ ] Type checking passes (`mypy headroom`): N/A, no Python changes
- [x] New tests added for new functionality (updated regression test)
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

`cargo test -p headroom-core --test magika_idle_threads`, with the #4187
fix (`with_intra_threads(1)`) in place and with it temporarily reverted:

```text
# macOS arm64, ORT 1.31.0, before this PR (fixed allowance of 2)
magika session left 8 extra OS thread(s) running ... FAILED
# macOS arm64, ORT 1.31.0, this branch
test result: ok. 1 passed; 0 failed      (6/6 repeat runs)
# macOS arm64, ORT 1.29.0, this branch
test result: ok. 1 passed; 0 failed

# #4187 fix reverted, this branch: the test still catches the pool
macOS, ORT 1.31:                      12 extra vs 9 allowed -> FAILED
macOS, ORT 1.29:                       7 extra vs 2 allowed -> FAILED
Linux (Docker aarch64), ORT 1.31:      7 extra vs 1 allowed -> FAILED
```

Full Rust suite on macOS with ORT 1.31.0 (the same command the new
`macos-26` job runs):

```text
ORT_DYLIB_PATH=.../libonnxruntime.1.31.0.dylib cargo test --workspace
68 suites, 1658 passed, 0 failed
```

Pre-push `make ci-precheck` on this branch:

```text
ci-precheck-python: 177 passed, 4 skipped
✅ ci-precheck PASSED — safe to push.
```

## Real Behavior Proof

- Environment: macOS 26 arm64 (16 cores), Rust 1.95, `pip install
onnxruntime==1.31.0` and 1.29.0 dylibs via `ORT_DYLIB_PATH`; Linux via
Docker `rust:1.95` (aarch64) with pip onnxruntime 1.31.0.
- Exact command / steps: the reporter's repro from #4191 (`pip install
'onnxruntime==1.31.0'`, `export
ORT_DYLIB_PATH=<capi/libonnxruntime.*.dylib>`, `cargo test -p
headroom-core --test magika_idle_threads`), on `main` and on this
branch; then the same test with the #4187 `with_intra_threads(1)` fix
temporarily reverted, on macOS 1.31, macOS 1.29 and Linux 1.31. To find
the cause, a raw `ort` probe built the magika model with
`intra_threads=1` under the default policy and under
`with_auto_device(PreferCPU)`, counting threads with `ps -M` and listing
the EP devices.
- Observed result: fails on `main` with ORT 1.31 (8 extra vs 2) and
passes on this branch, including the reporter's exact steps. With the
#4187 fix reverted it fails on all three setups (12 vs 9, 7 vs 2, 7 vs
1), so it still guards #4187. Probe: EP devices are CPU, CoreML/NPU and
CoreML/GPU. The default policy adds 6–7 threads and takes ~420 ms to
build; `PreferCPU` adds 0 and takes ~12 ms.
- Not tested: the new `macos-26` CI job on GitHub runners (it runs on
this PR), ORT 1.30.0 at runtime, x86_64 Linux, Intel macOS, and low-core
macOS hosts. Details:
- ORT 1.30.0: the 1.30 vs 1.31 difference comes from the ORT source diff
and release notes, plus the reporter's data.
- Low-core macOS hosts: with only 2–4 cores the control allowance can
absorb a small pool, so the guard is weaker there.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A
- Stable/default behavior changed: no. The PR changes only the test and
CI; runtime code is untouched.
- Kill switch / disable path: N/A
- Unsafe override required: no.
- Qualification impact: adds one macOS CI job (only when Rust paths
change).
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Not fixed here: on macOS with ORT ≥ 1.31, headroom's magika detection
runs through Core ML on the ANE instead of the CPU EP. That comes from
`ort`'s default policy; nobody chose it. Visible cost: a one-off ~400 ms
session build plus 5–7 idle threads. Per-call latency and output parity
were not measured. Pinning it to CPU needs
`with_auto_device(PreferCPU)`, which magika 1.1.0's `Builder` doesn't
expose. An environment-level CPU EP does not override the policy. This
is raised on #4191 as a possible follow-up issue on the `magika` crate.
- Option 1 from the issue (count CPU-burning threads instead) was
rejected: the #4187 pool only spun briefly after each inference, so idle
CPU was ~0% with or without it and the check wouldn't catch a
regression.
- This change was developed with GitHub Copilot CLI assistance; all
measurements above were run locally by me.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4194"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Romin Parekh <rominparekh@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:31:23 -06:00
Jay HemnaniandJD Davis 4b788fd04d fix(diff): keep combined-diff changes made against a later parent (#4318)
## Description

In a merge diff (`git show -c`, `git diff --cc`) every line has one +/-
column per parent, so a change against the second parent is written `
-old` / ` +new`. `DiffCompressor` only treated lines starting with `+`
or `-` as changes. These lines were counted as context and trimmed
whenever they sat more than `max_context_lines` from a first-column
change. A hunk holding only such a change lost the change entirely, and
the footer under-counted additions and deletions.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `parse_diff` and `reduce_context` classify combined-diff lines by all
of their parent columns (the parent count comes from the `@@@` header).
- Regular `@@` diffs are classified exactly as before.
- New test `combined_diff_second_parent_changes_survive_context_trim`.

## Testing

- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib
test result: ok. 976 passed; 0 failed; 1 ignored

# new test against the old code:
" -old_from_p2" dropped as context:
test result: FAILED. 0 passed; 1 failed

$ cargo clippy -p headroom-core --lib --tests -- -D warnings   # clean
$ pytest tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py tests/parity
212 passed, 2 skipped
```

## Real Behavior Proof

- Environment: Ubuntu, Python 3.12, headroom-ai 0.41.0 from PyPI vs this
branch built with `pip install -e .`
- Exact command / steps: a 400-line file changed on two branches,
merged, then one more line changed in the merge. `git show -c HEAD` (120
lines) passed through `DiffCompressor().compress(...)`, and the +/-
lines in all parent columns counted before and after.
- Observed result:
  ```text
0.41.0: changed lines in: 28 out: 26 lost: [' -line 200', ' +line 200
main']
  branch:  changed lines in: 28 out: 28 lost: []
  ```
On 0.41.0 the `@@@ -197,7 -197,7 +197,7 @@@` hunk comes out as two
context lines and nothing else. The footer goes from `+12 -12 lines` to
`+13 -15 lines`, which matches a count of the input.
- Not tested: octopus merges (`@@@@`); the same code path handles them.

## Runtime Rollout Safety

N/A: no rollout-managed feature, no config change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

The existing combined-diff tests build their input with `\` string
continuations, which strip each line's leading whitespace, so their `
-...` lines were really `-...` lines. The new test builds its lines
explicitly.

The hunk headers are still not updated after trimming. That is a
separate PR on top of this one.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4318"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:58:03 -06:00
Mansurshakh JaparovandJerrettDavis 93279f3208 fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description

Closes #3697.

An unchanged SmartCrusher result now keeps the original bytes and
reports no structural compression. Formatting-only savings come from the
existing Rust JsonMinifier, not from pretending the JSON value changed.

The minifier preserves duplicate keys, order, escapes and number
literals. A mixed result cannot use one section's CCR marker to hide
unmarked loss in another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace
savings through JsonMinifier in Python and the eligible native live
zone.
- Keep all records and JSON literals; do not store originals or inject
CCR markers for lossless minification.
- Preserve mixed routing and reject uncovered lossy sections in both
tool-result formats. Explicit no-CCR behavior is unchanged.
- Replace incidental forwarding tests with native minification and
complete-record regressions; document the distinction.

## Testing

- [ ] Entire Python suite — not claimed.
- [x] Four consumer/native modules: `python -m pytest -q
tests/test_recursive_json.py tests/test_transforms_content_router.py
tests/test_transforms/test_smart_crusher_rust_parity.py
tests/test_transforms/test_smart_crusher_lossless_default.py`
- [x] Changed-file Ruff checks; normal commit hooks, including mypy and
commitlint.
- [x] Manual public-API runs, with and without the HTML extra.

### Test Output

Consumer/native modules:

```text
collected 136 items
============================= 136 passed in 36.43s =============================
```

Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native
build, Python checks, commitlint):

```text
======================== 180 passed, 1 warning in 8.35s ========================
✅ ci-precheck PASSED — safe to push.
To https://github.com/oxura/headroom.git
   64688217..0002af3e  HEAD -> fix/3697-unchanged-json-passthrough
```

The first push stopped on an inherited commit footer because
`origin/main` was stale. Fetching the actual upstream ref fixed the lint
range; the normal push then passed. No source merge, lint-rule change or
hook bypass.

Hosted checks on `0002af3e` now pass: all four Python shards, Windows
permissions, smoke tests, native wrappers, parity and Docker e2e.
[Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks).

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual
extension built from this checkout. No provider billing calls.
- Exact command / steps: the earlier localhost `/v1/compress` and
TransformPipeline runs exercised the original fix. For this follow-up,
call `ContentRouter.compress` on a data-only HTML page containing 60
complete records, with `smart_crusher_lossless_only=True`,
`smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run
it once without trafilatura, then with the locked 2.0.0 extra. The new
public-API test contains the full input/config. Separately call
`route_embedded_json` on a compact array followed by a padded array,
with Unicode and CRLF surroundings.
- Observed result: both HTML modes keep all 60 records and the
surrounding markup, with no unrecoverable section. The unchanged first
JSON span does not stop the second span's compaction; surrounding text
stays exact. Final Codecov is 96%; the two remaining lines are the
deferred HTML fallback return. The real no-HTML-extra API run executed
both, and local coverage also exercises the unchanged-span branch.
- Not tested: live provider acceptance, invoices or production cache hit
rates. No native eligibility or admission gates changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing JSON routing.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: eligible formatting savings return as
json_minifier; standalone SmartCrusher remains byte-exact on no-ops.
- Kill switch / disable path: existing routing configuration and
HEADROOM_LIVE_ZONE_DISABLE_ARMS.
- Unsafe override required: no.
- Qualification impact: no-op, formatting, structural and CCR paths
exercised separately.
- Rollback path: revert the minifier integration; no persistence
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

Addresses the [minifier
review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285).
The maintainer’s main merge is retained. The latest follow-up only
changes consumer tests.

AI-assisted; no subagents.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 21:32:42 -06:00
9bb4b130ff fix(relevance): load the Rust default embedder from Python's pinned snapshot (#4190)
## Description

Parent status: #4152 has merged. The pinned-loader and batch-cap merge
resolution was reviewed at e3a495cbc5. The
original split/stack history is retained below.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now four
linear commits on top of main, with no merge commits:

- `bfae4ade` fix(relevance): load the Rust default embedder from
Python's pinned snapshot
- `6fbae380` fix(relevance): look up and download the Rust embedder in
one HF cache dir
- `70afece8` fix(relevance): expand ~ and $VAR in the Rust HF cache path
like huggingface_hub
- `62b792f0` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`. Its files matched main; the only difference was
`uv.lock`, where main's later `filelock>=4.2.1` entry was kept.
- The six main-sync merges (`185dc9b7`, `052977b9`, `029c72ac`,
`4405b12c`, `8b0a10e8`, and the reviewed `e3a495cb`) are linearized
away, so those SHAs are no longer on the branch.
- `embedding.rs` conflicted again with main's batch-size cap (#4507). I
resolved it to the same content as the reviewed merge: every file this
PR touches is byte-identical between `e3a495cb` and the new head
`62b792f0` (`git diff e3a495cb 62b792f0 -- <files changed vs main>` is
empty). The review and qualification below still apply to the content;
only the history and SHAs changed.

Rust `EmbeddingScorer::try_new` used `EmbeddingModel::BGESmallENV15`.
fastembed-rs maps that to a different repo and file from what Python
runs: `Xenova/bge-small-en-v1.5` `onnx/model.onnx` (fp32), always on
`main`. So the two languages never ran the same ONNX file, and the
"byte-equal across languages" claim in both modules was wrong.

This PR makes `try_new` load the same pinned
`Qdrant/bge-small-en-v1.5-onnx-Q` snapshot that #4152 makes Python load.
It adds a cross-language parity fixture checked from both sides, and
replaces the parity claim with measured numbers.

Nothing in production calls `EmbeddingScorer::try_new` today:
`create_scorer` and `HybridScorer` use `default()`, which never loads a
model. This keeps the Rust scorer correct and pinned for when it is
wired in.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

(This PR's own commits only.)

- `crates/headroom-core/src/relevance/embedding.rs`:
- `try_new` loads the snapshot through
`TextEmbedding::try_new_from_user_defined`, with CLS pooling and static
quantization, which is what fastembed-rs uses for `BGESmallENV15Q`.
- It uses exactly one hub cache directory, resolved like Python's
`huggingface_hub.constants.HF_HUB_CACHE`: `HF_HUB_CACHE` →
`HUGGINGFACE_HUB_CACHE` → `HF_HOME/hub` →
`XDG_CACHE_HOME/huggingface/hub` → `~/.cache/huggingface/hub`, with a
leading `~` expanded. It looks up and downloads only there, so a stale
snapshot in a lower-priority cache is never used. The home directory
comes from `std::env::home_dir`, which falls back to the user database
when `HOME` is unset, as Python's `expanduser` does.
- It finds the snapshot as `snapshots/<sha>` in that directory.
`hf_hub::Cache` resolves through `refs/<rev>`, which Python's
`huggingface_hub` doesn't write for a commit SHA.
- On a miss it downloads with `Repo::with_revision`. `guard_egress` sits
in front of the Hub client and runs only on a cache miss, so a
pre-seeded air-gapped host now loads; previously Rust refused even with
a warm cache.
- `HEADROOM_HF_PIN=off` floats it on `main`, with the same values as
Python's `_resolve_revision`.
  - `try_new_with_model` is unchanged.
- Parity fixture: `tests/parity/record_embedding.py` and
`tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json` (6 texts
× 384 dims, recorded on ORT 1.31.0). The fixture is written one
embedding row per line, so it's 22 lines. Python checks it in CI
(`test-extras` sets `RUN_FASTEMBED_TESTS=1`); Rust checks it under
`RUN_FASTEMBED_TESTS=1`.
- `tests/test_fastembed_pinned_snapshot.py`: the Rust constants name the
same repo, SHA and files as Python; the fixture was recorded at the pin;
plus the gated Python parity check.
- 10 new Rust unit tests in `embedding.rs`: the lookup ignores other
commits and needs every file; the download URL requests the pinned
revision; the offline guard applies on a miss only;
`HEADROOM_HF_PIN=off` floats; the cache directory follows
`huggingface_hub`'s order variable by variable (including an empty
variable and `~`); with no `HOME` it still resolves through the user
database; and the review case: an empty explicit `HF_HUB_CACHE` next to
a complete pinned snapshot and an older `refs/main` under `HOME` finds
neither, for the pinned revision or `main`, while both are found once
`HF_HUB_CACHE` is unset.
- `tests/test_offline_egress_chokepoint.py`: the Rust sweep's regex now
also matches `ApiBuilder::from_cache` / `from_env`. It missed the new
download client, which is guarded; it flags nothing else in `crates/`.
The user-defined model load, which only reads bytes already on disk, is
allowlisted as `cache-only`.
- Docs: the "byte-equal" wording in both embedder modules,
`pyproject.toml` and `onnx_runtime.py` is replaced with measured numbers
and the cross-language keep-equal notes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `62b792f0` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 991 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_hf_revision_pinning.py
235 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

The 7 offline-egress failures noted below no longer reproduce here.

Before the rebase:

```text
$ cargo test -p headroom-core --lib
test result: ok. 985 passed; 0 failed; 1 ignored      # see the note on #4151 below

$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean

$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.31.0 dylib> cargo test -p headroom-core --lib relevance::embedding
test relevance::embedding::tests::fastembed_matches_parity_fixture ... ok
test result: ok. 24 passed; 0 failed

$ RUN_FASTEMBED_TESTS=1 pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_relevance.py \
    tests/test_stateless_writers.py tests/test_hf_revision_pinning.py
71 passed

$ pytest -q tests/test_offline_egress_chokepoint.py
7 failed, 132 passed, 3 skipped
# The 7 are the OTel / Langfuse / Headroom Cloud offline tests, whose extras aren't in this venv;
# the same 7 fail on origin/main here. The Rust sweep tests pass.
```

**Note on #4151:** with `ORT_DYLIB_PATH` set on macOS, a full `cargo
test -p headroom-core --lib` fails
`tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error`
most of the time (5/5 on `afdb60d3`, before the cache change too).
`a_cached_snapshot_is_not_refused_offline` holds `HEADROOM_OFFLINE=1`
under the env lock while ONNX Runtime loads, which is slow there, and
the tokenizer test reads the variable without the lock. That's the race
#4151 fixes; with its commit applied, 6/6 full runs pass. Linux CI
passes either way.

The cache-precedence test was checked against the bug: with the lookup
changed back to also scanning `~/.cache`, it fails on the first case
(pinned revision, empty `HF_HUB_CACHE`).

## Real Behavior Proof
- Exact command / steps: additional maintainer verification on the
main-synced head: cargo test -p headroom-core --locked hf_hub_cache_dir;
cargo test -p headroom-core --locked
expand_vars_matches_python_posixpath_and_ntpath; python -m pytest
tests/test_fastembed_pinned_snapshot.py -q; repeat its fixture cases
with huggingface-hub==2.2.0.
- Observed result: three Rust tests passed across both POSIX and Windows
expansion fixtures and cache-directory selection; Python passed 46 tests
with one gated skip using Hugging Face Hub 1.16.1, and all 36 shared
fixture cases passed separately with locked Hub 2.2.0.

- Environment: macOS arm64; Rust fastembed 5.17.3 with hf-hub 0.5.0;
repo `.venv` (fastembed 0.8.0, onnxruntime 1.23.2) and a second
interpreter on onnxruntime 1.31.0.
- A cold-cache Rust load with `HF_HUB_CACHE=<empty dir>` wrote
`snapshots/52398278…` plus `refs/52398278…` and passed the parity check.
Per-component max |Δ| against the fixture:
  - Rust vs Python on the same ORT (1.31.0): ≤ 1e-6.
  - Python on ORT 1.23.2 vs 1.31.0: ≤ 4.1e-4.
  - The fixture's tolerance is 1e-3.
- For reference, Xenova's fp32 export (what Rust used to load) is ≤
4.7e-4 from the fixture. The old cross-language gap was real but
numerically small, and this fixture bounds drift; it cannot tell the two
exports apart. Model identity is enforced by the pinned SHA and the unit
tests.
- The fixture's new one-row-per-line layout was re-emitted from the
recorded data, not re-recorded: the parsed JSON is identical.
- Not tested: Windows, and Linux outside CI. Rust's gated tests in CI
(they need the model and an ORT dylib, so they stay manual behind
`RUN_FASTEMBED_TESTS=1`). Rust `HybridScorer` thresholds against the
Qdrant export.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no production path.
`EmbeddingScorer::try_new` loads the Qdrant export instead of Xenova's,
and now loads offline from a warm cache instead of refusing, but nothing
in production calls it today.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches
are still unpinned (there's no Rust `HEADROOM_HF_PIN`), and Rust's gated
embedding tests could get a CI job once one ships an ORT dylib.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4190"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer qualification

The environment-reference mismatch from the previous review is
addressed. The shared corpus covers the reported explicit-cache case,
platform-specific variable syntax, and the two-stage home/hub expansion
ordering. The Rust test runs both POSIX and Windows expansion rules;
Python validates the same corpus against posixpath, ntpath, and fresh
huggingface_hub.constants imports.

A clean current-main sync was published as
052977b9e88ae3bbda96797615b64ee350a2f695; it does not change the
contributor's resolver or fixture files. Qualification used Windows,
Hugging Face Hub 1.16.1 and an isolated locked 2.2.0 target. No model
download or numerical embedding-parity result is claimed by these path
tests; the pre-existing gated numerical-parity test remained skipped.

Hosted checks on the pushed head must finish before approval. No
contributor CI action requested or PR merged.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 21:31:59 -06:00
4d7765c84b fix(pipeline): store the pipeline input in CCR behind offload markers (#4576)
## Description

Fixes #4550. Two contract mismatches in the Rust `CompressionPipeline`
(`crates/headroom-core/src/transforms/pipeline/`):

1. Offloads ran on the post-reformat buffer and stored it in CCR, so
with `LogTemplate` in front of `LogOffload` the entry behind the marker
was the templated text, not the pipeline input. That contradicts
`traits.rs` ("original is in supplied store") and `ccr/mod.rs`
("lossless end-to-end").
2. `LogTemplate` was documented as lossless, but
`split_whitespace`/`lines()` drop indentation, tabs, whitespace runs and
`\r`, and a literal `<*>` constant printed exactly like the wildcard.

`CompressionPipeline` has no production caller yet, so this fixes the
contract before it is wired.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `OffloadTransform::apply` takes `ccr_source: &str`; the orchestrator
passes the original pipeline input. Same approach as the SmartCrusher
`ccr_source` fix (#2694).
- All six offloads (`log_offload`, `json_offload`, `diff_offload`,
`diff_noise`, `search_offload`, `prose_field`) key and store
`ccr_source` and still compress `content`. `json_offload`'s prose hook
passes each leaf as its own source, since its marker refers to that
leaf.
- `LogCompressor`, `DiffCompressor` and `SearchCompressor` gain
`compress_with_ccr_source`; the existing entry points delegate with
`content` as the source, so their behavior is unchanged.
- I did not overwrite the existing key with the original instead:
`CcrStore::put` assumes same hash = same content, and two inputs that
differ only in whitespace reformat to the same text.
- `LogTemplate`: the doc now says whitespace is normalized and the exact
original is recoverable only through CCR when an offload runs. A
constant token that is `<*>` behind zero or more backslashes gets one
more leading backslash, so the header is unambiguous.
- Tests: `ccr_stores_pipeline_input_not_reformatted_text` (issue repro:
LogTemplate + LogOffload, `store.get(key) == input`),
`literal_wildcard_constant_header_differs_from_variable_position`.
`lossless_round_trip_via_template_and_variants` is renamed to
`token_round_trip_via_template_and_variants` and compares token lists.

Not done: a whitespace-preserving `LogTemplate` (separators stored in
the variant table). Possible follow-up.

## Testing

- [x] Unit tests pass
- [x] Lint/format checks pass

```
$ cargo fmt -p headroom-core -- --check        # exit 0
$ cargo clippy -p headroom-core --all-targets -- -D warnings   # exit 0
$ cargo test -p headroom-core --lib
test result: FAILED. 981 passed; 1 failed; 1 ignored
  (failure: tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error, unrelated, HF hub network)
$ cargo test -p headroom-core --lib from_pretrained_invalid_repo
test result: ok. 1 passed; 0 failed
```

## Real Behavior Proof

- Environment: Windows 11, Rust 1.95.0, headroom-core default features,
branch based on upstream main a2d4fe72
- Exact command / steps: cargo test -p headroom-core --lib
ccr_stores_pipeline_input_not_reformatted_text (pipeline LogTemplate +
LogOffload + InMemoryCcrStore on 300 INFO + 300 DEBUG + 1 ERROR log
lines, the repro from the issue)
- Observed result: steps_applied contains log_template and log_offload,
and the stored CCR value equals the input byte for byte (before the fix
it started with "[Template T1: ...")
- Not tested: a live proxy/SDK run. `CompressionPipeline` has no
production caller yet (outside `transforms/pipeline/`, tests and benches
it is only re-exported from `transforms/mod.rs`; headroom-proxy,
headroom-py and the Python package do not use it), so there is no
runtime path to observe and the in-process pipeline run above is the
closest real behavior available. Also not run: integration tests that
reach Kompress (live_zone_dispatch, live_zone_kompress_async_init,
live_zone_disable_arms, live_zone_arm_properties plain-text arm,
kompress_parity) hang locally without the cached model; the other
integration test files pass

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, CompressionPipeline has no production
caller
- Stable/default behavior changed: no; compressor entry points
unchanged, pipeline unused in proxy/py
- Kill switch / disable path: n/a, no runtime path uses it
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4576"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 19:47:07 -06:00
Kenneth WongandKenneth Wong 31f1304026 fix(privacy): turn off ONNX Runtime's Microsoft telemetry by default (#4602)
## Description

Since 1.31.0, ONNX Runtime's Linux and macOS builds include Microsoft's
1DS telemetry client
(`onnxruntime/core/platform/posix/telemetry_1ds.cc`, absent through
1.30.0). It is **on by default**. `OneDsTelemetry::Initialize` turns it
off only when a CI variable (`GITHUB_ACTIONS`, `TF_BUILD`, ...),
`ORT_RUNNING_UNIT_TESTS` or `ORT_DISABLE_TELEMETRY` is set. Otherwise
ORT:

- writes a device id under `~/Library/Application
Support/Microsoft/DeveloperTools/.onnxruntime/` on macOS, and sends a
SHA-256 of it with each event;
- queues events in an `onnxruntime.db` next to it and uploads them to
`https://mobile.events.data.microsoft.com/OneCollector/1.0`;
- sends a `SessionCreation` event for each session, with
`modelFileName`, `modelGraphName`, `modelProducerName`,
`modelGraphHash`, `modelWeightHash`, execution provider ids and
versions, and hardware vendor and device types.

Headroom requires `onnxruntime>=1.24.0` on Python 3.11+ and currently
resolves 1.31.1. The proxy loads ORT twice: the Python package
(Kompress, image router) and the dylib the Rust core loads through
`ORT_DYLIB_PATH` (Magika, embeddings). Each copy initializes the
uploader. In my runs the proxy opened a TLS connection to Microsoft
about 11 s after start, while the models loaded, and sent about 4 KB.
`HEADROOM_OFFLINE` doesn't touch it, and
`docs/content/docs/security-model.mdx` doesn't list this destination. CI
doesn't see it because GitHub Actions sets `GITHUB_ACTIONS`, which ORT
treats as CI. The same client is in the manylinux wheels (checked in
`onnxruntime-1.31.1-cp311-cp311-manylinux_2_28_aarch64.whl`), so Docker
deployments are affected too.

ORT reads `ORT_DISABLE_TELEMETRY` once per copy, when that copy
initializes, and then never creates the uploader or the device id. So
`import headroom` now sets it with
`os.environ.setdefault("ORT_DISABLE_TELEMETRY", "1")`, in the same place
as the existing `ensure_ort_dylib_pinned()` hook, before anything in the
package can load ORT. An explicit caller value wins, including `0` for
anyone who wants ORT's telemetry on.

Design choice: this is off always, not only under `HEADROOM_OFFLINE` (as
`pin_crewai_telemetry_off` does for CrewAI). The events describe the
user's models and hardware and go to a third party the user never chose.
Headroom's own beacon is opt-out and documented, but this upload was
neither documented nor tied to any Headroom setting. If maintainers
prefer offline-only, `_OfflineEnvPin("ORT_DISABLE_TELEMETRY", "1")` in
`headroom/offline.py` is a small change to this PR.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/_ort.py`: `disable_ort_telemetry_by_default()` sets
`ORT_DISABLE_TELEMETRY=1` unless the variable is already set.
- `headroom/__init__.py`: calls it at package import, next to
`ensure_ort_dylib_pinned()`.
- `crates/headroom-core/src/transforms/magika_detector.rs`:
`init_ort_from_path` commits the environment with
`.with_telemetry(ort_telemetry_opted_in(ORT_DISABLE_TELEMETRY))`. The
`ort` crate's default is `true`, and it calls `EnableTelemetryEvents`.
With this change, telemetry is off when the variable is unset or set to
a value ORT treats as disabled, and on for any other explicit value, so
`ORT_DISABLE_TELEMETRY=0` re-enables it for Rust-loaded models too. This
only stops later events, after the uploader has started, so it is a
backup for Rust-only callers; the environment variable is the real fix.
A unit test covers each case.
- `tests/test_transforms/test_ort_dylib.py`: the default is set; a
caller's value (`0`) is kept; a subprocess `import headroom` leaves
`ORT_DISABLE_TELEMETRY=1` with `onnxruntime` not yet imported.
- `docs/content/docs/security-model.mdx`: a row for
`mobile.events.data.microsoft.com` in the network egress table.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_transforms/test_ort_dylib.py -k importing          # without the headroom/__init__.py change
E   AssertionError: assert {'ort_loaded'...'value': None} == {'ort_loaded'... 'value': '1'}
1 failed, 12 deselected
$ pytest -q tests/test_transforms/test_ort_dylib.py tests/test_package_init_lazy.py tests/test_offline_egress_chokepoint.py \
    tests/test_transforms/test_detect_fallback_1123.py tests/test_router_external_dispatch.py
223 passed, 2 skipped in 57.30s
$ ruff check / ruff format --check on the changed .py files (ruff 0.16.8)
All checks passed!
$ mypy headroom/_ort.py
headroom/memory/adapters/embedders.py:389: error: Unsupported operand types for - ("None" and "int")   # same single error on main
$ cargo clippy --all-features --tests -D warnings / cargo fmt --check -p headroom-core
clean
$ cargo test -p headroom-core --all-features --lib ort_telemetry
1 passed
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), Python 3.11.17, onnxruntime 1.31.1
(CoreML, Azure and CPU providers), no CI variables set; `origin/main` at
`fc1ba891` and this branch, each in its own worktree with the same
`headroom-ai[all]` environment.
- Exact command / steps: (1) a bare script that creates one CPU
`InferenceSession` for `chopratejas/technique-router-onnx` and idles 60
s, with `ORT_DISABLE_TELEMETRY` unset, then set to `1`. (2)
`PYTHONPATH=<worktree> python -m headroom.cli proxy --port 8799
--no-telemetry` with `HEADROOM_UPDATE_CHECK=off` and
`ORT_DISABLE_TELEMETRY` unset, running until Kompress reports healthy.
Both record every TCP flow of the process with `nettop -m tcp -p <pid>
-L 0 -s 1`, and remote owners come from `whois`.
- Observed result: (1) unset: one TLS flow to a Microsoft Corporation
address (4.150.223.107:443, about 9.5 KB in and 4 KB out), with
`onnxruntime.db` touched; set to `1`: no TCP flows, and the database is
untouched. (2) `main`: 2 of 2 runs open a flow to Microsoft
(74.151.238.96, 40.74.98.197) about 11 s after start, with about 9.5 KB
in and 4 KB out; the collector rotates across Azure addresses. This
branch: no flow to Microsoft in 6 runs; only the Anthropic upstream,
plus litellm's GitHub pricing fetch in some runs.
- Not tested: Linux and Windows at runtime. The Linux wheel contains the
same client (`OneDsTelemetry` and the collector URL appear in
`libonnxruntime.so.1.31.1`), but I only measured macOS. Windows uses
ORT's separate ETW provider, which doesn't read this variable.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, process environment default
- Stable/default behavior changed: yes; ONNX Runtime no longer uploads
usage telemetry to Microsoft from Headroom processes unless
`ORT_DISABLE_TELEMETRY` is set to a false value
- Kill switch / disable path: `ORT_DISABLE_TELEMETRY=0` restores ORT's
own default
- Unsafe override required: no
- Qualification impact: none; inference paths, providers and models are
unchanged
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

A host that imports `onnxruntime` before `headroom` has already
initialized ORT, so it has to set the variable itself. This is the same
limitation `pin_litellm_cost_map_local` documents.

In one early run, the fixed proxy did open the Microsoft flow. I
couldn't reproduce it in six later runs, and its log reports a different
computed `__version__` (`0.43.0-dev` against `0.42.1-dev` for the same
tree), so the import state of that run is uncertain. Setting the
variable before the process starts was clean in every run, for both the
bare script and the proxy.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
2026-11-03 17:01:04 -06:00
dependabot[bot]andJerrettDavis 038af599e1 deps: Bump lru from 0.18.5 to 0.19.0 (#4382)
## Why this version

Upgrade the native proxy's bounded session caches from lru 0.18.5 to
0.19.0 to retain the upstream iterator soundness correction. The
official 0.19.0 changelog says "Require Sync for sendable shared
iterators." In source, `Send for Iter<'a, K, V>` now requires `K: Sync,
V: Sync`, replacing `K: Send, V: Send`: the iterator yields shared
references, which may cross threads only when their referents are Sync.

Headroom uses this crate for the shared `BetaStickyState` session
beta-token union and `DriftState` structural-hash history. Both live in
`Arc<Mutex<LruCache<...>>>` and are cloned into concurrent request
handlers. They use String / Vec<String> / StructuralHash values; they do
not transfer shared iterators across threads. Thus this updates a
concurrency-sensitive dependency to its corrected upstream contract; it
does not claim that current Headroom calls demonstrate an exploit or
require a new API.

The patch preserves the 0.19 target and adds the dependency rationale
beside its declaration, removes an obsolete lru 0.12 documentation
reference, and mechanically merges current main so the current security
workflows and fixes are included.

## Compatibility and installation

This is a pre-1.0 minor bump, so compatibility was checked rather than
assumed. The only dependency introduced by lru remains hashbrown 0.17.1.
The crate's MSRV is 1.85.0 in both 0.18.5 and 0.19.0; this bump does not
raise it. Headroom's workspace rust-version=1.80 already understates
that dependency floor; its pinned toolchain is 1.95.0, used for this
validation. Correcting the workspace-wide MSRV declaration is outside
this dependency patch.

Native installation already builds the ONNX/image/AWS dependency stack
and may fetch build dependencies and ONNX Runtime artifacts; lru adds no
build script or model download. This was a fresh isolated native target
build using the existing registry/artifact caches, not a clean
offline-install certification. Runtime validation disables model
downloads with HEADROOM_OFFLINE/HF_HUB_OFFLINE and uses only local fake
upstreams, with no cloud inference or credentials.

## Validation

- `cargo test -p headroom-proxy --lib cache_stabilization --locked`: 105
passed, including concurrent_unions_lose_no_tokens (8 threads, 50
updates each), LRU eviction and recency checks.
- `cargo fmt --all -- --check`: passed.
- `git diff --check`: passed.
- `cargo test -p headroom-proxy --test integration_cache_drift --test
integration_cache_control --test integration_e3_anthropic_cache_control
--test integration_e4_openai_cache_key --locked`: 20 passed across four
binaries (1/8/5/6).
- `cargo build -p headroom-proxy --locked`: passed with Rust 1.95.0 on
Windows MSVC.
- `python scripts/verify_native_session_caches.py --proxy
target/debug/headroom-proxy`: passed. A separately spawned native binary
forwarded 12 requests to an ephemeral loopback fake Anthropic upstream
without credentials. Sequential requests retained beta-a,beta-b; eight
concurrent same-session requests retained all token-0 through token-7 in
the next request. Logs recorded both first-seen and changed-system drift
events. Both proxy and fake upstream were cleaned up. Compression
interception was enabled with compression-mode off; this exercises the
actual session caches and byte passthrough, not model compression or
cloud behavior.
- The standard-library runtime proof is committed as
`scripts/verify_native_session_caches.py`; no private artifacts are
required. Security/CI approval and remote branch update remain separate
gates; this local proof does not certify hosted checks.

Official upstream references:
https://github.com/jeromefroe/lru-rs/blob/0.19.0/CHANGELOG.md and
https://github.com/jeromefroe/lru-rs/blob/0.19.0/src/lib.rs .

## Reproduce the native loopback proof

From the repository root, with its pinned Rust toolchain and Python 3:

```sh
cargo build -p headroom-proxy --locked
python scripts/verify_native_session_caches.py --proxy target/debug/headroom-proxy
```

On Windows use `--proxy target/debug/headroom-proxy.exe`. If
`CARGO_TARGET_DIR` is set, pass the binary under that directory instead.
Optional `--log /path/to/proxy.log` preserves logs; otherwise logs and
the child working directory are temporary. The script uses only Python's
standard library, strips provider/Headroom configuration environment
variables, disables model egress, bypasses caller HTTP proxy settings,
disables stats persistence, and starts only its own loopback fake
upstream and native child. It returns nonzero on readiness, forwarding,
sticky-union, or drift-event failures and cleans up its processes.

This proof was rerun after making the harness portable: all 12 upstream
requests passed. Current main was fetched again and remains
`74277d47ecfbca349ddb99f81094c4dd30539d43`, already included in this
candidate.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 08:45:55 -06:00
dependabot[bot]andJerrettDavis b9049d703a deps: Bump the cargo-minor-patch group across 1 directory with 12 updates (#4553)
## Purpose

Retain the complete selected 12-version Cargo group and qualify its
compatibility with Headroom. The AWS release moves its config,
credential, Smithy runtime/schema and SDK dependencies together to a new
coordinated compatibility line; upstream explicitly warns against
upgrading only part of that line. Existing proxy credential resolution
and SigV4 signing must continue to work with that coherent set.

This is a compatibility-qualified grouped update. Companion updates are
not individually claimed to fix a Headroom bug where upstream only added
unused APIs.

## Selected versions and project relevance

| Package | Selected update | Relevance and qualification |
| --- | --- | --- |
| tokio | 1.53.3 → 1.54.1 | Proxy HTTP/SSE/WebSocket execution and
timers; includes upstream timer-handle regression revert. Local native
proxy/SSE/WSS tests exercise runtime. |
| aws-config | 1.12.0 → 1.13.0 | Required companion for new AWS
schema/runtime compatibility line; production environment credential
chain exercised with explicit fake credentials. |
| aws-credential-types | 1.3.0 → 1.3.1 | Same coordinated AWS line,
production credential conversion and SigV4 signing exercised. |
| aws-smithy-runtime-api | 1.19.0 → 1.19.2 | Same coordinated AWS line;
timeout finalizer behavior changed upstream. Local signing/wire tests do
not establish SSO/STS timeout behavior. |
| sha2 | 0.11.0 → 0.11.1 | Existing cache/SigV4-related payload hashing;
upstream unsafe-op lint fix. Local signed payload SHA256 verified; no
claimed cryptographic vulnerability fixed. |
| redis | 1.7.1 → 1.8.0 | Optional multiworker CCR backend; upstream TLS
error/secret redaction and subscription correctness fixes. Production
backend roundtrip exercised against local RESP fixture; cluster/pubsub
are not Headroom backend usage. |
| tree-sitter | 0.27.0 → 0.27.1 | Code compressor parser/error recovery
fixes. Reconstructed 9 samples × 8 pinned languages compare old/new node
kinds, complete row/column spans, ERROR/MISSING: 72/72 identical. |
| uuid | 1.28.0 → 1.29.0 | Full-group companion target; new NonNilUuid
constructors are unused. Existing request-ID generation exercised
through real proxy calls and both response x-request-id values parsed as
UUIDv4; no invented bug/security rationale. |
| crc32fast | 1.5.2 → 1.5.3 | Full-group companion target; upstream
total-length API is unused. Existing Bedrock EventStream
checksum/framing path exercised by streaming integration suite. |
| rustls-native-certs | 0.8.4 → 0.8.5 | Native trust-store handling;
avoids duplicate Unix certificate-file reads. Windows build and WSS
fail-closed behavior exercised; Unix store traversal not locally
exercised. |
| webpki-roots | 1.0.10 → 1.0.11 | Updated bundled CA trust set (CCADB
CA removals); WSS verification tests exercise existing explicit TLS
configuration. No public CA endpoint contacted. |
| cc | 1.8.0 → 1.10.0 | Native C/C++ dependency builds; upstream fixes
MSVC archive object paths and cl.exe wrapper handling. Actual clean
Windows MSVC native dependency compilation performed. |

## Maintenance and install surface

The packages remain maintained by their existing upstream projects:
Tokio, AWS/Smithy, RustCrypto, redis-rs, tree-sitter, uuid-rs,
srijs/rust-crc32fast, rustls and rust-lang/cc-rs. Release/compare links
in the original Dependabot body identify the reviewed releases; AWS
authoritative release notes:
https://github.com/awslabs/aws-sdk-rust/blob/main/CHANGELOG.md (October
20–30, 2026). UUID comparison confirms its change only adds NonNilUuid
constructors; it is not presented as a security fix.

No new package source/registry is introduced. Cargo resolves registry
downloads at build time; cc executes the native compiler for existing
C/C++ dependencies (tree-sitter grammars and other existing native
crates). The lock update changes the transitive AWS SDK/schema/runtime
family together. Existing ONNX runtime loading/model behavior is not
altered by this PR. Redis is feature gated and adds runtime network only
when explicitly configured. AWS credentials can normally resolve through
environment, profiles, SSO/STS or metadata; qualification uses only
explicit fake environment credentials, NUL config files and disabled
IMDS. Normal configured proxy upstream traffic and native trust-store
reads remain existing runtime surfaces.

Pinned qualification compiler: Rust 1.95.0 on Windows/MSVC. Resolved
proxy AWS floor is 1.94.1; core/Python/parity graph includes
tree-sitter-language floor 1.90. Both already existed in the base lock,
so no global rust-version change is made; inherited workspace
rust-version=1.80 is not a valid full-workspace source-build promise.
Fresh base/head Windows-filtered metadata comparison confirms unchanged
floors: core/Python/parity1.90,proxy1.94.1,simulators1.86(ICU). These
are dependency-declared effective floors, not measured lower-toolchain
builds. No lower-toolchain build was attempted.

## Real behavior proof

Windows, Rust1.95.0 x86_64-pc-windows-msvc, isolated worktree, dedicated
CARGO_TARGET_DIR, CARGO_BUILD_JOBS=2. No actual AWS/provider
credentials, profiles, cloud inference or model downloads used.

- `cargo test --locked -p headroom-core -p headroom-proxy` with explicit
existing x64 ONNX Runtime DLL (`ORT_DYLIB_PATH`) and
`HEADROOM_OFFLINE=1`, `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`:
**1,659 passed, 0 failed, 3 ignored; exit 0**, across 61 test/doc-test
executables. Initial run without the Windows loader path was diagnosed
loader-blocked and terminated; its separate failed log is retained.
- `cargo test --locked -p headroom-proxy --test
integration_bedrock_invoke --test integration_bedrock_streaming --test
integration_ws_tls --test integration_sse`: 23passed,0failed
(8invoke,11streaming,2SSE,2WSS).
- `python G:/git/headroom/.codex/dep4553-runtime.py`: built
headroom-proxy.exe on ephemeral ports with compression disabled;
passthrough and native Bedrock routes each return local200, both bodies
byte equal; AWS environment provider loaded fake credentials and SigV4
Credential prefix/payload SHA256 checked; both generated response
x-request-id headers validated UUIDv4. Synthetic wire fixture, not real
AWS service emulation.
- `python G:/git/headroom/.codex/dep4553-redis-wire.py`: temporary
integration test calls production RedisCcrStore using redis1.8.0; local
RESP fixture validates PING,miss,UTF8 payload write/read,TTL refresh and
replacement; 1passed. Synthetic protocol server; supplemented by `python
G:/git/headroom/.codex/dep4553-redis-real.py`: real Redis7Alpine
ephemeral loopback Docker service with persistence disabled; production
backend2testsPASS,including real TTLrefresh andexpiry. Owned
--rmcontainer stopped automatically. TLS/cluster/failover/pubsub not
tested.
- Standalone canary under `.codex/dep4553-canary`: exact engine
versions0.27.0 and0.27.1, same eight pinned grammars,72/72 AST snapshots
identical. Historical canary source was absent; reconstructed inputs are
recorded, including second distinct valid fixture per language (explicit
Cfunctionpointer sample), CRLF,truncated,error,concatenated andUnicode
cases.
- Isolated Python3.12 env: recorder regression9tests passed; Ruff
checks/format passed. Original official fixture generator reproduced
KeyError(bash). Minimal3line helper fix raises ValueError, matching
existing detector unsupported-language contract; all8pinned grammars
remain covered. Official generator then recorded30fixtures.
- Generator revealed12preexisting Python-reference fixture differences.
Exact main74277d47 source extracted independently, same Python
environment +same helper fix:30/30 regenerated semantic fixture objects
match PR source, excluding recorded_at only. Preserve original repo
fixture bytes; do not mask unrelated Python-reference drift by replacing
goldens. Baseline/regenerated files and comparisons retained as
artifacts.

## Remaining gates / limitations

Initial broad Cargo suite was loader-blocked: source live_zone.rs
documents indefinite Windows initialization without ORT_DYLIB_PATH.
Controlled termination of exact owned testPID717412 produced
authoritative exit1 for original session37711. Distinct corrected
configuration uses existing verified x64ORT1.26.0 DLL(API24), explicit
ORT_DYLIB_PATH and offline guards; corrected full core+proxy
qualification completed exit0,1659passed,0failed,3ignored. This is an
environment diagnosis, not a demonstrated dependency regression. Hosted
checks must be rerun on the exact final remote SHA; prior shard4 red is
not claimed fixed. Canonical fixture rerecord was executed and its
baseline-identical semantic drift documented, not silently normalized
away. No Linux/macOS build, lower MSRV build, real AWS SSO/STS/IMDS,
public CA endpoint, Redis TLS/cluster/failover, production provider
inference or model download was tested. No remote push, PR edit/review
or merge performed by this qualification task.

## Public reproduction

The committed standard-library harness reproduces the full 72-case
parser comparison and launches the actual native proxy against
credentialless loopback fixtures. It stores both complete AST snapshots
and inputs when `--artifacts-dir` is provided. No private `.codex`
artifacts are needed.

Windows, from a clean checkout with Rust 1.95/MSVC and Python 3.11+:

```powershell
cargo build --locked -p headroom-proxy --target-dir target/qualification
python scripts/qualify_cargo_dependencies.py all --binary target/qualification/debug/headroom-proxy.exe --artifacts-dir target/qualification-proof --canary-target-dir target/qualification-canary
```

Linux/macOS use the same commands with the binary path
`target/qualification/debug/headroom-proxy` (omit `.exe`). Those
platforms are reproduction instructions, not platforms locally qualified
here. The canary compiles standalone exact engine versions0.27.0/0.27.1
against all eight production grammar pins, so Cargo registry
downloads/native compilers may be needed. Runtime checks use fake AWS
credentials, platform null config files, disabled IMDS, offline model
guards, compression off, and local HTTP endpoints only.

The source-comment-required Python fixture rerecord is independently
reproducible:

```text
python -m pip install tree-sitter==0.25.2 tree-sitter-python==0.25.0 tree-sitter-javascript==0.25.0 tree-sitter-typescript==0.23.2 tree-sitter-go==0.25.0 tree-sitter-rust==0.24.2 tree-sitter-java==0.23.5 tree-sitter-c==0.24.2 tree-sitter-cpp==0.23.4
python scripts/record_code_compressor_fixtures.py
```

Use an isolated development Python environment with the project
development dependencies installed. The recorder helper fix in this PR
preserves its existing eight-grammar scope; unsupported detector
candidates are skipped through the existing ValueError contract.
Regeneration currently exposes baseline Python-reference drift as
documented above; inspect semantic differences rather than replacing
goldens blindly.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 08:43:53 -06:00
a2d4fe7238 fix(relevance): cap embedding batch size so ORT's arena stays small (#4507)
## Description

The embedding relevance scorer calls fastembed's `embed()` at its
default batch of 256. fastembed pads each batch to its longest text. Its
ONNX session keeps ONNX Runtime's CPU arena on, and the arena is never
returned to the OS. A relevance split over a large tool output therefore
sizes the arena to the biggest batch it has ever scored, and the memory
stays pinned for the life of the proxy.

Found on an Extra Headroom user's macOS proxy (wheel 0.41.0, 4 days
uptime): phys_footprint was 3.94 GB, and 2.5 GB of it was arena regions
in the BFC doubling pattern (32, 64, 128 and 256 MB, then 1 GB and 1
GB), all dirty and swapped out. The tracked caches (`/debug/memory`)
totalled 10 MB. The periodic `malloc_zone_pressure_relief` trim cannot
release arena memory.

This PR caps both scorers at 16 texts per ONNX run: Python
`EmbeddingScorer._encode` and the Rust `EmbeddingScorer::score_batch`.
That retains about 10x less memory, runs faster because each batch
carries less padding, and leaves keep/drop decisions unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/relevance/embedding.py`: new `EMBED_BATCH_SIZE = 16`, passed
as `batch_size` to `model.embed()`.
- `crates/headroom-core/src/relevance/embedding.rs`: `score_batch`
passes the scorer's `batch_size` instead of `None`. It is chosen once in
`try_new_with_model`: `Some(EMBED_BATCH_SIZE)` (16), or `None` (one
batch, as before) for models fastembed marks `QuantizationMode::Dynamic`
(e.g. `AllMiniLML6V2Q`), whose `embed` rejects a batch size smaller than
the input because each run calibrates its own int8 range.
- `tests/test_relevance_extra.py`: asserts that `_encode` passes the
cap. The test fails without the fix (`assert [{}] == [{'batch_size':
16}]`).
- Real-model tests in both languages
(`fastembed_batches_past_the_cap_keep_each_score_with_its_item` and
`test_batches_past_the_cap_keep_each_score_with_its_item`): 40 texts of
mixed lengths span three batches with the one related item in the last
batch. That item must still rank first, and every batched score must
match the item scored on its own.
- Rust dynamic-quantized coverage:
`dynamic_quantized_models_embed_in_one_batch` (runs in CI, no model)
pins the per-model choice;
`fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap`
(real `AllMiniLML6V2Q`) scores 15, 16 and 40 items plus the context and
requires no inference-error fallback and the related item ranked first.
It fails on 92707844 at 16 items with fastembed's "Dynamic quantization
cannot be used with batching" error.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_relevance_extra.py tests/test_relevance.py tests/test_relevance_split.py -q
63 passed, 4 skipped in 1.75s
$ uvx ruff check headroom/relevance/embedding.py tests/test_relevance_extra.py
All checks passed!
$ uvx ruff format --check headroom/relevance/embedding.py tests/test_relevance_extra.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/relevance/embedding.py
Success: no issues found in 1 source file
$ cargo fmt --all -- --check   (clean)
$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.27 dylib> cargo test -p headroom-core --lib relevance::embedding -- --test-threads=1
test result: ok. 17 passed; 0 failed   (real models, incl. the 40-text multi-batch test and AllMiniLML6V2Q at 15/16/40 items)
$ cargo test -p headroom-core --lib relevance::embedding
test result: ok. 17 passed; 0 failed   (model tests skip without RUN_FASTEMBED_TESTS)
$ cargo test -p headroom-core --no-default-features --lib relevance::embedding
test result: ok. 8 passed; 0 failed
$ cargo clippy -p headroom-core --all-targets -- -D warnings
Finished (no warnings)
$ FASTEMBED_CACHE_PATH=<cache> uv run --frozen --extra dev --with fastembed==0.9.1 pytest tests/test_relevance.py -k TestEmbeddingScorer
4 passed   (real model, incl. the 40-text multi-batch test)
$ uv run --frozen --extra dev pytest tests/test_*content_router*.py tests/test_relevance*.py -q
236 passed, 7 skipped
```

## Real Behavior Proof

- Environment: macOS 27 on an M2 (16 GB), the installed headroom-ai
0.41.0 runtime (fastembed 0.8.0, onnxruntime 1.27),
BAAI/bge-small-en-v1.5 from the local cache; for the end-to-end run, a
scratch proxy started with Extra Headroom's flags (`--no-http2
--no-rate-limit --log-messages`, isolated HOME, telemetry off) in front
of a local mock Anthropic upstream.
- Exact command / steps: (1) bench: 120 records cut from a 7k-line Rust
source file by the real `segment()`, `EmbeddingScorer.score_batch` 3
times per batch size in a fresh process each, measured with `footprint`;
(2) equivalence: `plan_relevance_split` with `create_scorer("hybrid")`
over 24 cases (8 real corpora of 5 to 411 records: Rust source, proxy
log, grep output, git log, transcript JSONL, long-line markdown, cargo
output, a tiny log; 3 queries each), batch 16 and 256 in separate
processes; (3) end to end: 13 Claude Code shaped requests (question,
Bash `tool_use`, a 2k to 145k token `tool_result`) through the scratch
proxy, batch 16 and 256.
- Observed result: identical splits in 24 of 24 cases at 1.7x less time;
end to end, one relevance split grew the batch-256 proxy by 10.4 GB (to
an 11 GB footprint) against 0.99 GB at batch 16, with 12 of 13 forwarded
bodies byte-identical.

  | Check | Batch 256 (before) | Batch 16 (this PR) |
  |---|---|---|
| Bench, 120 records: time / retained | 16.8 s / +3769 MB | 8.7 s / +339
MB (max score diff 7.8e-5, 0 flips) |
| Equivalence, 24 cases: total split time | 1295 s | 746 s (24/24
identical runs, worst score diff 4.8e-5) |
  | End to end: proxy footprint growth | +10.4 GB | +0.99 GB |
| End to end: 16k-token search output | hit the 30 s compression
timeout, forwarded uncompressed | compressed in 20.4 s, 5,338 tokens
saved |
- Not tested: Linux and Windows; memory and timing of the Rust scorer
(its real-model tests pass, but no proxy path loads its model in
0.41.0); turning the arena off instead, which on macOS retained between
+76 and +1277 MB across runs at batch 256 (libmalloc keeps the freed
blocks), so the batch size is the dependable lever.

Rust, dynamic-quantized model (added for review, 135c2b66). This is a
real-model cargo test (real ONNX model through fastembed-rs 5.17.3,
onnxruntime 1.27, macOS 27 on M2), not a proxy run: in the 0.41.0 wheel
the proxy builds the Rust `HybridScorer::default()`, which never loads
an embedding model, so no proxy request reaches this code.

```text
$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.27 dylib> cargo test -p headroom-core --lib relevance::embedding::tests::fastembed_dynamic_quantized -- --test-threads=1
# on 92707844 (test added, source unchanged):
panicked at crates/headroom-core/src/relevance/embedding.rs:569:17:
16 items: Embedding: inference failed: Dynamic quantization cannot be used with batching. [...]
test result: FAILED. 0 passed; 1 failed
# on 135c2b66:
test relevance::embedding::tests::fastembed_dynamic_quantized_model_scores_on_both_sides_of_the_cap ... ok
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; embedding inference now runs in
batches of 16 texts instead of 256 (scores match within 1e-4, no
keep/drop changes measured). Rust models fastembed marks
dynamic-quantized (e.g. `AllMiniLML6V2Q` via `try_new_with_model`) still
run as one batch, as before.
- Kill switch / disable path: none needed for a batch-size constant;
revert the commit
- Unsafe override required: no
- Qualification impact: none expected; relevance-split decisions were
identical on the measured input
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation: N/A, since no user-facing setting changed. The arena is
left on: the batch cap bounds it, and turning it off alone did not
reliably lower retained memory on macOS. Extra Headroom will carry this
as a temporary patch against 0.41.0 until a release includes it.

Python has no equivalent constraint: fastembed 0.9.1 has no quantization
check on `embed`; `_embed_documents`
(`fastembed/text/onnx_text_model.py`) splits every model's input with
`iter_batch`, and the only batch-size check is `>= 1`
(`fastembed/common/utils.py`). Its previous default of 256 already split
every model's input the same way.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 23:45:34 -05:00
703755e873 fix(pricing): add Claude Haiku 5.5 rates, 1M context and its 100K long-prompt card (#4037)
## Current maintainer qualification

Reviewed head 595803d786 after the
current-main sync. The PR-specific pricing implementation is unchanged
from the previous reviewed head.

Independent validation: 202 focused Python pricing/provider/savings
tests passed, four skipped. The Rust proxy binary built from this head.
Through the running Rust HTTP proxy and a separate loopback upstream,
requests carrying 100,000 and 100,001 input tokens produced the expected
native-ledger request costs of $0.0105 and $0.0525005 respectively,
including 1,000 output tokens. The default stable rollout channel was
used, with no unsafe feature override.

The HTTP replay uses controlled upstream usage; it does not claim live
Anthropic inference or independently verified provider billing. The
previously recorded missing native HTTP boundary check is now covered.

Current main is an ancestor. Hosted checks on this head are terminal: 49
successful, 2 skipped, with no failures. The prior shared Rust advisory
hold is cleared. Final merge approval remains human-only; nothing has
been merged by this review.
## Description

Claude Haiku 5.5 (`claude-haiku-5-5`) is released and had no rows in
Headroom's built-in Anthropic tables or the Rust proxy's vendored
LiteLLM snapshot. This PR adds it, including its second rate card for
prompts over 100K tokens, and makes every cost path read a model's
long-context threshold from its own catalog row instead of assuming
200K.

**Why the missing rows mispriced instead of failing:** without LiteLLM
data for the model, `AnthropicProvider` falls through to the `haiku`
pattern default ($0.80 / $4, 200K), which is 8x the real input rate and
a fifth of the real window. The Rust `context_window_for` is
exact-match, so the id fell through to the 128K default.

Haiku 5.5's rate card ([Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing),
matching upstream LiteLLM's `claude-haiku-5-5` rows): $0.10 / $0.50
input / output and $0.01 cache read for prompts up to 100K tokens; every
rate is 5x ($0.50 / $2.50 / $0.05) above 100K. 1M context, 128K max
output.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All values in $ / MTok (input / output / cache read).

| Model | Before (fallback) | After | Context |
|---|---|---|---|
| `claude-haiku-5-5`, prompt <= 100K | *(no row: `haiku` tier default)*
$0.80 / $4 / $0.08 | **$0.10 / $0.50 / $0.01** | 200K -> **1M** |
| `claude-haiku-5-5`, prompt > 100K | $0.80 / $4 / $0.08 | **$0.50 /
$2.50 / $0.05** | |

- `headroom/providers/anthropic.py`:
- Adds Haiku 5.5 to `ANTHROPIC_PRICING` and `ANTHROPIC_CONTEXT_LIMITS`.
- The long-context premium becomes a small per-model table,
`_LONG_CONTEXT_TIERS` (prefixes, threshold, multipliers), so Haiku 5.5's
100K card sits next to the existing Sonnet 4 / 4.5 200K tier. The Sonnet
tier's threshold and multipliers are unchanged.
  ```diff
   _apply_long_context_premium(model, pricing, input_tokens)
- if input_tokens <= 200K or model not in Sonnet 4 / 4.5: return pricing
  -  return pricing * Sonnet premium
  +  for prefixes, threshold, premium in _LONG_CONTEXT_TIERS:
  +    if model starts with prefixes:
+ return pricing if input_tokens <= threshold else pricing * premium
  +  return pricing
  ```
- `crates/headroom-proxy/data/model_prices_and_context_window.json`:
appends the 18 upstream LiteLLM `claude-haiku-5-5` keys verbatim
(first-party, Bedrock regional/global/Mantle/GovCloud, Vertex, Azure,
OpenRouter, Perplexity). No existing entry changes or is removed;
extended rather than re-pulled for the reasons in #3841.
- **Per-model long-context tiers on the stats paths** (Devin finding 1).
Each catalog row publishes at most one `*_above_<N>k_tokens` tier (100K,
128K, 200K, 256K or 272K); the threshold now comes from the row, not a
shared 200K constant.
- `crates/headroom-proxy/src/observability/pricing.rs`: the price book
stores a `PriceCard` (base card + optional long-context card parsed from
the row's tier fields; a missing tier field falls back to that rate's
base price). `lookup` still returns the base card; `lookup_card` returns
both, and `PriceCard::for_prompt` picks one.
- `crates/headroom-proxy/src/observability/ledger.rs`: `record` picks
the card from the billed prompt (uncached + cache read + cache write)
before pricing spend and savings.
- `headroom/pricing/counterfactual.py`: `long_context_threshold(model)`
/ `long_context_tier(info)`; `resolve_rates` reads the row's own tier,
including its combined 1h + long-tier write rate where published.
`CacheMix.is_long_context` takes the threshold.
- `headroom/proxy/cost.py`, `headroom/proxy/savings_tracker.py`: bucket
and price at the model's own threshold.
- Side effect on other tiered rows, both intended: the Rust stats now
bill Sonnet 4 / 4.5 above 200K at its long card (the Python cost card
already did), and both paths switch GPT-5.x at 272K and the 128K-tier
rows at 128K with their own rates. Before, those rows crossed a 200K
line, found no `*_above_200k_tokens` fields and stayed on the base card.
- **Compression that crosses a tier credits the whole avoided premium**
(Devin re-review). When compression keeps a prompt under the threshold
its uncompressed form would have passed, the saving is the uncompressed
bill minus the actual bill, not the removed tokens at the cheap rate.
Rust: `PriceCard::compression_savings_usd` (input and output); Python:
`long_context_premium_avoided_usd`, used by
`estimate_request_savings_usd` and by `CostTracker` (which also adds the
output premium). The list companion still prices only the removed
tokens, so budget enforcement stays monotonic. The warm-prefix test in
`tests/test_cost_card_attribution.py` sent 200K with 10K saved, which
crosses Sonnet 4.5's tier; its counts are halved (same mix) so it stays
inside one tier.
- **Short ids no longer inherit a newer release** (Devin finding 2).
`get_context_limit` and `_get_pricing` matched substrings in both
directions, so `claude-haiku-5` resolved to Haiku 5.5 and
`claude-sonnet-4` to Sonnet 4.6's 1M window. `_is_release_of` accepts a
longer id containing the known one (dated or provider-wrapped) or an
undated / `-latest` alias of a dated key, and rejects ids one version
segment apart in either direction.
- `crates/headroom-proxy/src/compression/model_limits.rs`: the 1M-window
test now covers `claude-haiku-5-5`, `us.anthropic.claude-haiku-5-5` and
`vertex_ai/claude-haiku-5-5`.
- Rust tests: Haiku 5.5 base and >100K cards, per-model threshold
(Sonnet 4.5 at 200K), flat-rated model, missing tier field; ledger bills
150K in / 5K out at the long card, and cached tokens count toward the
threshold.
- `tests/test_providers/test_anthropic.py`: Haiku 5.5 price, dated-id
and 1M-context cases, cost cases at and above the 100K threshold, the
`claude-haiku-5` regression and an `_is_release_of` table.
- `tests/test_cost_card_attribution.py`,
`tests/test_savings_provider_matrix.py`: Haiku 5.5 cost card and savings
at 90K vs 150K.
- `tests/test_counterfactual_pricing.py`: the long-tier 1h test now
reads the published combined rate, and a synthetic row covers the
derived case.
- `tests/test_provider_model_fallback.py`: the "future haiku" probe
moves from `claude-haiku-5-20260101` to `claude-haiku-6-20270101`,
mirroring the opus probe change in #3841.
- `tests/test_thinking_compactor_version_gate.py`: `claude-haiku-5-5` is
treated as a Claude 5 model.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`) (ruff 0.16.8, on the changed
files)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) pytest -q <119 pricing / cost / savings / stats / provider / learn modules>
2097 passed, 18 skipped, 5 warnings

$ cargo test -p headroom-proxy
577 passed, 0 failed (41 suites)
$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
(clean)
$ cargo fmt -p headroom-proxy --check
(clean)

$ ruff check <changed .py files>          # ruff 0.16.8
All checks passed!
$ ruff format --check <changed .py files>
(clean)
$ mypy headroom --ignore-missing-imports  # mypy 1.20.2
Success: no issues found in 566 source files
```

`HOME` is isolated because a developer `~/.headroom/models.json` leaks
into `TestAnthropicConfigLoading`.

## Real Behavior Proof

- Environment: macOS, the repo `.venv`. `origin/main` and this PR head
were each checked out as a git worktree and run with the same
interpreter and an isolated `HOME`.
- Exact command / steps: a script built `AnthropicProvider()`, stubbed
`estimate_cost_from_tokens` and `_get_litellm_clients` to force the
built-in fallback (the LiteLLM-unavailable or offline path), then priced
two requests and read the context limit. For the Rust side it applied
`context_window_for` semantics (exact key, `mode == "chat"`,
`max_input_tokens`, 128K default) to each worktree's vendored JSON. For
the review fixes, a second script ran against the first PR commit
(`b0503796`) and this head: the fallback lookups for short ids, a
`CostTracker` 150K-in / 5K-out Haiku 5.5 turn with 10K saved, and
`resolve_rates(..., long_context=True)`; the Rust ledger case is the
`long_prompt_bills_at_the_models_long_context_card` test.
- Observed result: Haiku 5.5 now prices and sizes correctly on every
path, and short ids keep their tier default (tables below).
  ```text
origin/main this PR
context limit (Python fallback) 200,000 (inferred 'haiku' tier)
1,000,000
50K in / 5K out $0.060000 $0.007500
150K in (120K cached) / 5K out $0.053600 $0.033500 (100K+ card)
Rust context_window_for 128,000 1,000,000
  ```
Hand check of the last row: 30K × $0.50 + 120K × $0.05 + 5K × $2.50 per
MTok = $0.0335.

  Review fixes, first PR commit vs this head:
  ```text
b0503796 this head
get_context_limit("claude-haiku-5") 1,000,000 200,000 ('haiku' tier)
_get_pricing("claude-haiku-5") $0.10 / $0.50 / $0.01 $0.80 / $4 / $0.08
get_context_limit("claude-sonnet-4") 1,000,000 (Sonnet 4.6) 200,000
(Sonnet 4)
CostTracker 150K in / 5K out: output cost $0.002500 $0.012500
CostTracker 10K saved: cache-aware savings $0.001000 $0.005000
resolve_rates(haiku-5-5, long).uncached $0.10 / MTok $0.50 / MTok
Rust ledger 150K in / 5K out $0.0175 (base card, computed) $0.0875
(ledger unit test)
  ```

Compression crossing the tier, Haiku 5.5, 110K uncached compressed to
90K (unit tests on this head; the "before" column is the old rule's
arithmetic):
  ```text
removed tokens at base this head
Rust ledger compression savings $0.002 $0.046
estimate_request_savings_usd["compression"] $0.002 $0.046
CostTracker cache-aware savings (1K output) $0.002 $0.048 (incl. $0.002
output premium)
  ```
- Not tested: live provider API billing. The controlled native HTTP
replay now covers the 100K boundary, as documented above.
- The Rust proxy end to end with a live >100K Haiku 5.5 request; the
ledger is covered by unit tests only.
- Bedrock-style ids (`us.anthropic.claude-haiku-5-5`) get the base card
without the 100K premium on the Python fallback path, the same
limitation the Sonnet 4 / 4.5 tier already has.
  - No live API call was made.

## Maintainer follow-up qualification

Latest qualification head: `f885cc4c8afa574d867820583886687b0bae12a9`.
The pricing code and the 413-Python/73-Rust/TCP-replay validation below
are unchanged from `a45ae8c3e`. The first hosted shard-3 run exposed
five existing main integration failures:
`test_proxy_dependency_gate_matches_onnx_platform_markers` still patched
a removed eager-import hook. The follow-up aligns those existing tests
with lazy `find_spec` discovery and its missing-spec return value. The
five failures reproduce locally before; all eight dependency-contract
cases and 91 CLI/stateless/logging tests pass after, with
Ruff/format/whitespace passing. The current-head CI must finish before
approval.

Published maintainer follow-up at `a45ae8c3e` (normal push from the
unchanged `c326e2046` head): forward generated `output_tokens` into the
shared compression estimator through Prometheus and the durable lifetime
recorder, and add the long-card minus base-card output premium only when
compression crosses the model's threshold. `output_tokens_saved` remains
a separate output-shaping layer. The documented unpriced direct tracker
fallback is preserved.

Integrated current main `463c1a43e30fc33cb256a3f455daf28043e53b05` to
resolve the pricing conflict. The resolution retains canonical
model-specific cache billing, independent cache-rate provenance and
explicit-zero rates, while probing missing billed cache rates at the
model's own long-context threshold. New regressions cover both contracts
together.

Verification after the conflict resolution:

- 413 Python tests passed; one known Windows directory-fsync test
deselected after it failed identically on the unchanged PR head; three
existing warnings. This includes 20 new
output-premium/boundary/metadata/merged-cache-billing cases, pricing,
provider fallback, persistent metrics, cache TTL, tool-schema and output
savings suites.
- 73 Rust observability tests passed: `cargo test -p headroom-proxy
--lib observability --locked`.
- Ruff lint/format and whitespace checks pass. Focused Linux-platform
mypy passes for the four pricing/metrics modules. Full local mypy in
this extras-rich environment reports 19 errors in
`relevance/embedding.py`, `memory/backends/direct_mem0.py`,
`integrations/langchain/chat_model.py` and
`integrations/crewai/agents.py`; it is not claimed green.
- Before the fix, both public priced recorders report $0.046 for
110K→90K input with 1K generated output; after, both report $0.048.
Noncrossing and no-output controls pass, and separately saved output is
not conflated with the avoided premium.

Actual Windows/Python 3.12 TCP replay: real uvicorn proxy, separate
loopback HTTP upstream, and the real content-router/SmartCrusher
pipeline. A 304,903-character JSON tool result becomes 139,003
characters, saving 55,974 locally counted tokens. The controlled
upstream reports 90,000 input and 1,000 output tokens. On the unchanged
PR head the lifetime estimator reports $0.063987 and fails; with the fix
and after the main integration it reports the expected $0.065987. The
persistent Prometheus compression metric matches, and output-shaping
savings remain zero. This exercises real compression and handlers with
controlled usage/rates; no live provider billing or Rust HTTP runtime is
claimed.

Current-head GitHub CI and edited review bodies still require
qualification before approval.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: cost estimates and the context window
reported for `claude-haiku-5-5` and the 18 added LiteLLM keys;
long-prompt spend and savings on tiered models now use each model's own
threshold; short ids such as `claude-haiku-5` fall back to their tier
default instead of a newer release
- Kill switch / disable path: n/a (data tables)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a: no
user-facing docs list per-model prices)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a

## Additional Notes

The 100K card applies on every path: the Python fallback
(`_LONG_CONTEXT_TIERS`), LiteLLM's own estimate, the cost card and
savings counterfactual, and the Rust stats ledger.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4037"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 17:54:57 -05:00
Mansurshakh JaparovandJerrettDavis 023dd1bc54 fix(logs): count omitted levels from dropped lines (#4141)
## Description

Closes #4070.

The omission marker previously counted severities in the whole input,
including retained errors. It now counts ERROR/FAIL/WARN/INFO only on
dropped line identities. Selection and whole-input result.stats are
unchanged. If no dropped line has a recognized severity, the marker is
simply `[N lines omitted]`, not a severity breakdown. CCR retains the
exact original.

## Type of Change

- [x] Bug fix (non-breaking)
- [x] Documentation update

## Changes Made

- Keep the existing retained-line identity set and a single-pass fixed
stack histogram; no omitted-line collection or repeated full scans.
- Restore complete structured-marker assertions in Rust and Python,
including bounded failure names, repeated ids, whole-input totals, and
non-severity omissions.
- Restore parity fixture `3bc015edc0a36387.json` using its original
input, explicit configuration and input digest. Output and compression
ratio were re-recorded through the freshly built native extension rather
than manually copied. New runtime-frame defaults remain implicit in the
legacy configuration.
- Remove the constructor-only result-copy test; retain consumer behavior
regressions.
- Document omitted-versus-input counts and the exact bare-marker form in
the existing text/log guide. No manual changelog change.

## Testing

- [x] Scoped Rust and Python tests passed
- [x] Normal touched-file Ruff and repository mypy hooks passed
- [x] Actual native recorder and parity CLI exercised

Head `6c472f6b` includes upstream main `f1bc2609`.

- Focused Rust log-compressor tests: **41 passed**.
- Python public-transform tests: **11 passed**, using the freshly built
native extension.
- Actual parity CLI: **20 matched, 0 skipped, 0 differences**, including
the restored original-input fixture:

```text
cargo run --locked -p headroom-parity --bin parity-run -- run --fixtures tests/parity/fixtures --only log_compressor
[log_compressor  ] total=20 matched=20 skipped=0 diffed=0
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.14, native extension freshly
built from this worktree.
- Exact command / steps: Re-record the original log fixture through
LogCompressor.compress with its original explicit configuration, then
run the actual parity CLI shown above.
- Observed result: The real native recorder produced `[297 lines
omitted: 297 INFO]` while whole-input statistics remained 300 INFO / one
ERROR / 305 total / eight selected. The exact original was unchanged and
the measured compression ratio is `0.024777201057682892`. The original
fixture digest remains
`3bc015edc0a36387194d075139845c120a308afb314bae436ab454906b431d84`.

Normal commit and full pre-push hooks passed without bypass: Rust
formatting, warnings-as-errors Clippy, workspace tests, native editable
rebuild/import verification, **181 critical Python tests**, and
branch-range commitlint. The critical Python run reported five
dependency/provider warnings, no failures. These local results do not
claim that hosted checks have finished.

- Not tested: New Windows/macOS runs, live-provider sessions and the
full Python suite.

Earlier before/after actual-native CCR qualification showed the retained
ERROR was not counted as omitted, whole-input totals stayed unchanged,
and retrieval recovered the exact original. That historical
implementation evidence is in the review discussion; the explicit
results above qualify this follow-up head. No new Windows/macOS or
live-provider qualification is claimed.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing LogCompressor and CCR; no new
feature or gate.
- Minimum rollout channel: Unchanged (stable); no channel promotion.
- Stable/default behavior changed: Yes, omitted-severity accounting is
corrected; selection and stored originals remain unchanged.
- Kill switch / disable path:
ContentRouterConfig(enable_log_compressor=False) disables the log route;
existing CCR settings disable retrieval separately.
- Unsafe override required: No.
- Qualification impact: The scoped native/Python/parity proof and
required normal push gate above cover this marker path. No new
platform/live-provider qualification is claimed.
- Rollback path: Revert the fix; no schema or stored-original migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
- [x] Existing style and documentation conventions followed
- [x] Consumer-visible regressions and actual native/CLI paths exercised
- [x] Normal hooks passed; no bypass
- [x] No manual CHANGELOG edit

AI-assisted: omp / GPT-6.1 Sol. This review follow-up was completed
without subagents. No manual human-validation or full Python-suite
claim; scoped tests and the required push gate are reported separately.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:45:24 -07:00
1be699f68e fix(deps): patch OpenClaw Undici and narrow E2E audit exceptions (#4453)
## Description

Includes the approved #4358 as its prerequisite. This PR targets main so
the complete integration workflows qualify the combined repair; its diff
will shrink to this follow-up once #4358 lands. Replace OpenClaw's
affected Undici copy instead of excepting it, and reduce the E2E audit
exceptions from seven advisory/path pairs to three.

OpenClaw 2026.10.2 lists Undici in `bundleDependencies`. Version
2026.10.6 removes it from that bundle, allowing a targeted, same-major
override to install fixed Undici 8.12.3. npm 12.2.1 still bundles its
own affected copies; this PR retains only those existing, expiring
exceptions.

## Type of Change

- [x] Bug fix

## Changes Made

- Pin the existing E2E OpenClaw CLI to 2026.10.6 and override only its
Undici dependency to 8.12.3.
- Regenerate the E2E lockfile; keep Codex and the npm override
unchanged.
- Remove every OpenClaw exception and the now-stale advisory, preserving
the parent's exact bundled-version checks and 2026-11-15 expiry.
- Update the Dockerfile comment to the actual Node requirement: 24.22.1
in the Node 24 line. Its NodeSource installation already selects that
line.

## Dependency and install-surface assessment

This updates an existing upstream-maintained CLI for a security fix,
rather than adding a replacement package. The fixed Undici remains in
the same major line. The graph is used only by the Docker wrap E2E
harness and is not shipped in Headroom's wheel or SDK. Normal
installation runs OpenClaw's existing lifecycle scripts and fetches
packages from the registry; those scripts were inspected, then exercised
inside a fresh Linux container without host credentials. No application
compression or routing logic changes.

## Testing

- [x] Manual testing performed
- [x] Dependency installation and runtime resolution verified

### Test Output

```text
OpenClaw runtime resolves Undici 8.12.3
OpenClaw 2026.10.6 (fe06181)
Installed audit: only npm bundled copies remain
{ info: 0, low: 0, moderate: 1, high: 1, critical: 0, total: 2 }
parent: 9 audit-guard cases passed
child: 9 audit-guard cases passed
git diff --check: passed
```

The guard accepts the current scoped findings and rejects a new
advisory, wrong path, expired exception, stale exception, audit error,
changed version, non-bundled copy, and missing inherited-advisory entry.
A raw audit remains nonzero for npm's bundled dependencies; this PR does
not claim those packages are patched.

## Real Behavior Proof

- Environment: fresh Linux amd64 Docker container,
`node:24.22.1-bookworm-slim`, Node 24.22.1 / npm 11.19.1, normal `npm
ci` with lifecycle scripts enabled.
- Exact command / steps: copy the two committed package files into a
fresh container volume; run `npm ci --no-audit --no-fund`; resolve
`undici/package.json` using
`createRequire('/work/node_modules/openclaw/package.json')` and assert
version 8.12.3; run `node node_modules/openclaw/openclaw.mjs --version`
and `--help`; run `npm audit --json` and assert that every remaining
affected node is under `node_modules/npm/`.
- Observed result: installation, exact runtime-resolution assertion,
version, and help all succeeded. Both installed-tree and lockfile audits
leave only npm's bundled Undici 6.29.0 and ip-address 10.7.2. The parent
workflow's guard passes with the narrowed exceptions and rejects all
eight negative cases.
- Not tested: a billed provider request or live production deployment.
Hosted Docker agent/integration scenarios have now passed. Hosted
qualification on head 63c3abc5b3 is
complete: all reported checks are terminal with no failures or pending
checks, including the full Security, build, test, and Docker integration
workflows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; E2E dependency data only.
- Minimum rollout channel: existing E2E test environment.
- Stable/default behavior changed: this follow-up patches only the
harness CLI. The included parent retains its explicitly documented
affected-Pillow decoder guard.
- Kill switch / disable path: revert this dependency commit.
- Unsafe override required: none; the override selects a compatible
fixed Undici 8.x patch. The parent's existing, bounded npm exceptions
remain explicit.
- Qualification impact: package installation, CLI startup, runtime
dependency resolution, and audit guarding were verified in isolation.
- Rollback path: revert the four-file change.

## Review Readiness

- [x] I performed self-review
- [x] This PR is ready for human merge review; the complete hosted check
set passed

## Checklist

- [x] I did not edit CHANGELOG.md.
- [x] The exception scope is reduced, not expanded.
- [x] No contributor correctness logic was modified.

Python lint/type checks and new production unit tests are not applicable
to these dependency/data changes. Independent human approval is
required; this is an author-side qualification record.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai>
2026-10-23 23:11:35 -07:00
dependabot[bot]andJerrettDavis 8d468d455c deps: Bump base64 from 0.22.1 to 0.23.1 (#4029)
## Description

Qualify base64 0.23.1 for the Rust proxy and correct the dependency
comment.

## Type of Change

- [x] Bug fix / dependency security and qualification

## Changes Made

Synced with upstream `main`
(`486afaac7426fa1d6735fe2ce6ae37630d9d11ae`). No contributor
implementation logic changed. Tested head:
`78852ccc95669d4c666812bf7155055eb81d2a12`. Includes the reviewed
dependency-floor fix from #4239 because the vulnerable version was still
permitted by shipping extras.

## Dependency assessment

base64 0.23.1 fixes the upstream tests/build on non-SIMD architectures
following 0.23.0. The crate is maintained by the existing base64
upstream; 0.23 adds SIMD engine code (including the default simd-unsafe
feature), while Headroom's existing decoder call sites are unchanged.
The HTTP stack may independently retain base64 0.22; the comment now
explains this correctly. No new network service or runtime download is
introduced. Also applies the reviewed Python MCP CVE-2026-59951 floor.

## Testing

cargo test --locked -p headroom-proxy --lib: 340 passed. cargo audit
--file Cargo.lock: passed. Branch dependency contract: 1 passed; uv lock
--check passed. Non-x86 builds are covered by hosted CI rather than this
Windows host.

### Test Output

```text
cargo test --locked -p headroom-proxy --lib: 340 passed. cargo audit --file Cargo.lock: passed. Branch dependency contract: 1 passed; uv lock --check passed. Non-x86 builds are covered by hosted CI rather than this Windows host.
```

## Real Behavior Proof

- Environment: Windows, PowerShell; installed Python/uv, Node/npm/Bun
and Rust tooling as applicable. No live provider/model used.
- Exact commands / steps:

```text
cargo test --locked -p headroom-proxy --lib
cargo audit --file Cargo.lock
uv lock --check
python -m pytest -q tests/test_mcp_dependency_contract.py
git diff --check
```

- Observed result: cargo test --locked -p headroom-proxy --lib: 340
passed. cargo audit --file Cargo.lock: passed. Branch dependency
contract: 1 passed; uv lock --check passed. Non-x86 builds are covered
by hosted CI rather than this Windows host.
- Not tested: live model/provider traffic and full Docker E2E locally;
hosted cross-platform checks are running on the pushed head. Read hosted
results before merging. Focused checks are not a claim of complete
platform qualification.

## Runtime Rollout Safety

- Rollout-managed features: none added or enabled.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: dependency versions/ranges and
associated install metadata only; no feature rollout defaults changed.
- Kill switch / disable path: existing optional extras and plugin opt-in
paths remain available.
- Unsafe override required: no runtime qualification bypass. Security
dependency override is documented above where applicable.
- Qualification impact: hosted checks and the focused evidence above
must pass; no existing gate is removed.
- Rollback path: revert this dependency change and restore its prior
lockfiles together; retain the patched MCP minimum or another reviewed
security remediation.

## Review Readiness

- [x] Maintainer self-review and focused verification performed.
- [x] All hosted checks passed on the reviewed head. Ready for final
human merge review.

## Checklist

- [x] No contributor implementation logic rewritten.
- [x] Committed lockfiles checked against the changed manifests.
- [x] No manual CHANGELOG.md changes.
- [x] Known limitations recorded above.

## Original Dependabot update

Bumps [base64](https://github.com/marshallpierce/rust-base64) from
0.22.1 to 0.23.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md">base64's
changelog</a>.</em></p>
<blockquote>
<h1>0.23.1</h1>
<ul>
<li>Make the tests build again on non-SIMD architectures</li>
</ul>
<h1>0.23.0</h1>
<ul>
<li>Added more consts for preconfigured configs and engines</li>
<li>Make DecodeError::InvalidLastSymbol more clear by including the
decoded value</li>
<li>Added SIMD-accelerated engines behind the default-on
<code>simd-unsafe</code> feature: <code>Simd</code> picks the best
instruction set at runtime (AVX2 on <code>x86_64</code>, NEON on
<code>aarch64</code>) and falls back to the scalar
<code>GeneralPurpose</code> engine, while <code>Avx2</code> and
<code>Neon</code> target one instruction set with no runtime
detection and work in <code>no_std</code>. The engines support the
standard and URL-safe alphabets.</li>
<li>Update MSRV to 1.71.0</li>
<li>Add support for custom padding symbols</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/069bf7067b949f5c0a92b6ceb82492920502f2c2"><code>069bf70</code></a>
v0.23.1</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/6ab1fb0a5843013557a52c45c84b91f5d1bb87af"><code>6ab1fb0</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/310">#310</a>
from musicinmybrain/test-on-non-simd-arches</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/7cffce6f971acdf48f83112cbdd63bd61125ba06"><code>7cffce6</code></a>
Fix testing on architectures without unsafe SIMD support</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/e34f9a08c5c89a4641350ac22033f3fa4f5d4d97"><code>e34f9a0</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/308">#308</a>
from atouchet/com</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/e9240c9a01e0a4934c5619e74740caa6d1f67ce9"><code>e9240c9</code></a>
Remove outdated comment</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/9e9220a4166f628de7c8803289e120ae1e944f78"><code>9e9220a</code></a>
v0.23.0</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/870326ec592eebde9d6bfe4c5d8130c591273e9c"><code>870326e</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/306">#306</a>
from marshallpierce/mp/trailing-bits-docs</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/fbec5f1050f9fc16e6a826ebabaa2b7b0644bd67"><code>fbec5f1</code></a>
Document no trailing trailing bits</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/0a23549968f059b53cf39e96eba8f46779f322a7"><code>0a23549</code></a>
Merge pull request <a
href="https://redirect.github.com/marshallpierce/rust-base64/issues/305">#305</a>
from marshallpierce/mp/edition-2021</li>
<li><a
href="https://github.com/marshallpierce/rust-base64/commit/f10b7e20614135aa61289140683fc93e5a45d338"><code>f10b7e2</code></a>
Update deps &amp; edition</li>
<li>Additional commits viewable in <a
href="https://github.com/marshallpierce/rust-base64/compare/v0.22.1...v0.23.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=base64&package-manager=cargo&previous-version=0.22.1&new-version=0.23.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4029"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-15 22:01:32 -05:00
2016dc6d60 fix(onnx): stop magika's ORT intra-op pool from spinning idle cores (#4189)
## Description

The Rust Magika detector
(`crates/headroom-core/src/transforms/magika_detector.rs`) built its
ONNX session with `Session::new()`. ONNX Runtime then sized the intra-op
pool to the core count, and those threads spin-wait between inferences.
The proxy calls this detector for every content-routing decision through
`headroom._core.detect_content_type`, so under steady light load the
pool burns roughly 2 cores. #2540 fixed this for the Python sessions by
setting `session.intra_op.allow_spinning=0`, but the `magika` 1.1.0
`Builder` doesn't expose session config entries, so that key can't be
set here.

The fix builds the session with
`Session::builder().with_intra_threads(1)`. With one intra-op thread,
ONNX Runtime doesn't create a pool at all, so nothing is left to spin.
The inter-op pool only exists in parallel execution mode, which isn't
used here, so it needs no change. I confirmed that adding
`with_inter_threads(1)` changes nothing.

Closes #4187

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `magika_detector.rs`: replace `Session::new()` with
`Session::builder().with_intra_threads(1).build()`, with a one-line
comment explaining why.
- `tests/magika_idle_threads.rs`: new regression test. It counts OS
threads (`/proc/self/task` on Linux, `ps -M` on macOS) before and after
the first `magika_detect` call and asserts that at most 2 new threads
remain. Those 2 are ORT's always-on telemetry worker and, on macOS, a
libdispatch worker; both are blocked rather than spinning. The test is
in its own file so it runs in its own process, because the session is a
process-global singleton. It skips when the ONNX Runtime library isn't
available, matching the existing magika tests.

## Testing

- [x] Rust: `cargo fmt --all -- --check`, `cargo clippy --workspace --
-D warnings`, `cargo test --workspace` (with `ORT_DYLIB_PATH` set so the
magika tests actually run)
- [ ] Unit tests pass (`pytest`): only the pre-push `ci-precheck-python`
subset was run (177 passed, 4 skipped); the full suite was not
- [x] Linting passes (`ruff check .`): no Python files changed;
pre-commit hooks passed
- [ ] Type checking passes (`mypy headroom`): N/A, no Python changes
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

New test: fails on `main`, passes with the fix (macOS arm64, 16 cores,
onnxruntime 1.29 dylib):

```text
# main
magika session left 9 extra OS thread(s) running (before=2, after=11)
test result: FAILED. 0 passed; 1 failed
# this branch
test result: ok. 1 passed; 0 failed
```

The same test on Linux (Docker `rust:1.95-bookworm`, aarch64, 8 CPUs,
pip onnxruntime 1.31.0):

```text
# main
magika session left 8 extra OS thread(s) running (before=2, after=10)
test result: FAILED. 0 passed; 1 failed
# this branch
test result: ok. 1 passed; 0 failed
```

Full Rust gate on this branch (`make ci-precheck` via the pre-push
hook):

```text
cargo test --workspace: 68 suites, 1658 passed, 0 failed, 3 ignored
ci-precheck-python: 177 passed, 4 skipped
✅ ci-precheck PASSED — safe to push.
```

## Real Behavior Proof

**1. Linux: CPU under light load (reproduces the issue's ~180%)**

- Environment: Docker `rust:1.95-bookworm` on Apple Silicon (Linux 6.10
aarch64, 8 CPUs), `pip install onnxruntime` (1.31.0), `ORT_DYLIB_PATH`
pointed at the pip library.
- Exact command / steps: a release-build probe binary calls
`headroom_core::transforms::magika_detector::magika_detect` on a ~5.6 KB
Python snippet, first 200 times back-to-back (latency), then once every
50 ms for 10 s ("trickle", like a proxy under light load). Process CPU
was read from `/proc/<pid>/stat` (utime+stime) over a 9 s window during
the trickle, and threads from `/proc/<pid>/task`. Built from `main` and
from this branch in the same container.
- Observed result: on `main` the probe ran at ~191% of one core during
the trickle; on this branch ~24%. Threads 9 → 2, idle CPU ~0% in both,
back-to-back latency 2.4 → 8.4 ms. Full table:

| | `main` | this branch |
|---|---|---|
| OS threads after init | 9 | 2 |
| CPU during 50 ms trickle | **~191% of one core** | **~24% of one
core** |
| CPU over 10 s fully idle | ~0% | ~0% |
| Average detection latency (back-to-back) | 2.4 ms | 8.4 ms |

**2. macOS: the real proxy (`headroom proxy`) on the `/v1/compress`
path**

- Environment: macOS 26 arm64 (16 cores), Python 3.12 `.venv` from `uv
sync --extra dev --extra proxy`, onnxruntime 1.26.0. Extension built
from `main`, then from this branch, with `uv pip install -e .`.
- Exact command / steps: `headroom proxy --port 18787 --code-aware` with
a throwaway `HOME`. A client sent one `/v1/compress` request every 200
ms, each with a unique ~6 KB code tool result from a non-excluded tool.
The router log confirmed `cache_miss` / `compressed=1`, so the native
detector ran. During the load, `sample <pid> 10 1` was taken and
top-of-stack samples were summed per library: `libonnxruntime.*.dylib`
(Rust `ort`) vs `onnxruntime_pybind11_state.so` (Python ORT, i.e.
Kompress).
- Observed result: Rust-ORT worker-pool threads 7 → 1, CPU inside the
Rust ORT dylib 0.11 → 0.00 cores, `/v1/compress` p50 latency unchanged
(840 vs 846 ms). Full table:

| | `main` | this branch |
|---|---|---|
| Rust-ORT worker-pool threads (stacks in `ThreadPoolTempl::WorkerLoop`)
| 7 | 1 (the calling thread) |
| Proxy OS threads after warm-up | 44 | 37 |
| CPU inside the Rust ORT dylib during load | 0.11 cores, mostly
`SpinPause` / `WorkerLoop` | 0.00 cores |
| `/v1/compress` p50 latency | 840 ms | 846 ms (noise; dominated by
Kompress) |

On macOS the spin is smaller than on Linux, but the profile shows the
same pool spinning in `SpinPause`, and with the fix it's gone.

- Not tested: x86_64 Linux, Windows, the full proxy on Linux, very high
request rates, very low-core hosts, and `--no-magika`. Details:
- x86_64 Linux, and Windows (Windows defaults to the pure-Python
detector, so it's unaffected unless `HEADROOM_DETECT_BACKEND=rust` is
set).
- The full proxy on Linux; the Linux numbers come from the detector
probe, not the proxy.
  - Very high request rates where per-call latency could matter.
- Very low-core hosts (1–2 cores): the regression test allows 2 idle
background threads, so a 1–2 worker pool can fit inside that allowance
there. CI's 4-vCPU runner is above it.
- The optional `--no-magika` flag suggested in the issue, which is left
out of scope.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A
- Stable/default behavior changed: the Rust Magika session now uses one
intra-op thread. Detection results are unchanged; single-call latency
rises (about 2.4 → 8.4 ms on Linux aarch64, about 1.3 → 1.9 ms on macOS)
in exchange for not spinning idle cores.
- Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` already
bypasses the native detector.
- Unsafe override required: no.
- Qualification impact: none expected; well within the <50 ms transform
budget.
- Rollback path: revert this commit.

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, which is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Trade-off: one intra-op thread makes each Magika call slower (+~6 ms
worst case measured), but stops ~2 cores of spin under light load. A
fuller fix would set `allow_spinning=0` while keeping a pool, which
needs the `magika` crate to expose session config entries; that would be
an upstream change and a dependency bump.
- `--no-magika` from the issue isn't included; it's a feature and should
be raised separately per CONTRIBUTING.
- This change was developed with GitHub Copilot CLI assistance; all
measurements above were run locally by me.

---------

Co-authored-by: Romin Parekh <rominparekh@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-13 16:23:42 -07:00
Tejas Chopra 47f8710817 refactor(transforms): drop the Python mirror of the Rust search selector (#4103)
## Description

`SearchCompressor.compress()` runs entirely in Rust. The class still
carried a Python re-implementation of the scorer, selector and
formatter, plus a parser wrapper, kept only so old tests could call
them. Production never ran them, and the module's own docstring noted
the Python error-boost keywords had drifted from Rust. All removed names
are private (leading underscore).

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Remove `_parse_search_results`, `_score_matches`, `_select_matches`,
`_format_output`, `_cjk_bigrams` and `_is_cjk_char`.
- Parser tests now call `headroom._core.parse_search_lines`, the live
Rust parser, through a small test helper.
- These tests are converted to `compress()`-level tests, because Rust
only unit-tests error boosting and bigram extraction: context-word
overlap, error/warning/TODO boost, `context_keywords`, the short-word
filter, `boost_errors=False`, the CJK query bigram boost, and the
combined parse/score/select/format test.
- `test_score_capped_at_one` is dropped: the cap cannot be observed
through `compress()`.
- The `_cjk_bigrams` unit test is dropped: Rust covers it in
`cjk_bigrams_from_runs`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_search_compressor.py tests/test_transforms_search_compressor.py tests/test_search_compressor_cjk.py tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_content_router.py tests/test_error_detection.py tests/test_signals_keyword_parity.py
170 passed in 7.31s
$ ruff check / format: All checks passed;  mypy search_compressor.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) a real `SearchCompressor.compress()` keeping one match
per file, with a CJK query whose bigrams match only one row.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The direct check keeps the CJK row ranked first by the
Rust scorer. Output below.
- Not tested: Nothing beyond the listed files.

```text
branch: cleanup/transforms-search-drop-python-mirror @ 0384d610
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51340.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -c "<SearchCompressor(max_matches_per_file=1, ...).compress(3 rows, context=CJK query)>"
src/a.py:20:认证令牌已过期需要重新登录
[... and 2 more matches in src/a.py]
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. Only unused private methods are
removed.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

`adaptive_sizer.py` stays. The log-mirror PR still needs it until both
PRs land, and then it can go in a one-file follow-up.
`error_detection.PRIORITY_PATTERNS_SEARCH` is now unused in product code
but is in an `__all__`, so it is left for a later deprecation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-10 08:59:40 -07:00
Tejas ChopraandClaude Opus 5.5 a20c76c26b refactor(transforms): drop the Python mirror of the Rust log selector (#4096)
## Description

`LogCompressor.compress()` runs entirely in Rust. The class still
carried a Python re-implementation of the line parser, scorer, selector,
deduper, context adder and formatter, plus the pytest short-summary
helpers and a `_store_in_ccr` shim. Production never ran them, and
`_store_in_ccr` still had the #816 bug that the live path fixed (no
`explicit_hash`). All removed names are private.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Remove the private helpers listed above: about 400 lines.
- Format-detection tests now call `headroom._core.detect_log_format`,
which `compress()` uses.
- Level counts, and the JS and rustc stack-trace shapes, become
`compress()`-level tests. The stack-trace tests run with
`error_context_lines=0`, so only trace detection can keep the frames.
- Two Rust unit tests are added for behaviour no Rust test pinned:
`level_classifier_covers_every_level` (FAIL, DEBUG, TRACE, unknown) and
`score_line_orders_levels_and_boosts`.
- Helper tests whose behaviour Rust already unit-tests are deleted:
dedupe, pytest short-summary recognition and CRLF, the malformed
diagnostic, cap precedence and reservation, omission naming, select
first/last, and the Go/Rust/.NET/Java trace openers.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib log_compressor
test result: ok. 40 passed; 0 failed
$ pytest tests/test_log_compressor.py tests/test_transforms_log_compressor.py tests/test_transforms_stack_traces.py tests/test_ccr_rust_marker_hash_bridge.py tests/test_transforms_content_router.py tests/test_adaptive_sizer.py tests/test_transforms_search_compressor.py tests/test_search_compressor.py
223 passed in 7.52s
$ ruff check / format: All checks passed;  mypy log_compressor.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) a real `LogCompressor.compress()` with
`error_context_lines=0` on 100 filler lines, a JS stack trace, and 100
more filler lines.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The direct check keeps exactly the three trace lines
and reports the level counts from Rust. Output below.
- Not tested: Nothing beyond the listed files.

```text
branch: cleanup/transforms-log-drop-python-mirror @ b82ef1b7
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51445.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -c "<LogCompressor(error_context_lines=0).compress(100 filler + JS stack trace + 100 filler)>"
LogFormat.GENERIC {'errors': 1, 'fails': 0, 'info': 0, 'selected': 3, 'total': 203, 'warnings': 0}
Error: Connection failed
    at Connection.connect (src/db.js:42:15)
    at async main (src/index.js:10:5)
[200 lines omitted: 1 ERROR]
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. Only unused private methods are
removed.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

`adaptive_sizer.py` is kept on purpose. On main, the SearchCompressor
helpers still import it, and those are removed in a separate PR. Delete
it in a one-file follow-up once both PRs land.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:58:41 -07:00
Tejas ChopraandClaude Opus 5.5 eaa7308a52 refactor(transforms): remove the unused summarize_dropped_items (#4093)
## Description

`compression_summary.summarize_dropped_items` was SmartCrusher's
dropped-item categoriser. SmartCrusher now runs in Rust and emits no
dropped-item summary (`smart_crusher/crusher.rs`), and nothing in Python
called the function. Only tests did, including two LLM eval files. The
module has no `__all__`, is not re-exported, and the function is not
documented or imported by any consumer repo.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Remove `summarize_dropped_items`, its private helpers (`_item_key`,
`_categorize_by_fields`, `_find_notable_items`, `_common_keys`) and
their constants.
- Fix the module docstring, which still said SmartCrusher used it.
`summarize_compressed_code` (used by CodeCompressor) stays.
- Delete `test_compression_summary_hard_eval.py`,
`test_compression_summary_tool_eval.py` and
`test_compression_summary_eval.py`. The last file's one code-summary
test moves into `test_compression_summary.py`.
- Drop the dropped-item cases from the other two test files.
- Evidence: `rg -n summarize_dropped_items headroom crates plugins sdk
examples benchmarks scripts docs wiki` plus the consumer repos finds
only the definition and a Rust doc comment, now reworded.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression_summary.py tests/test_compression_summary_integration.py tests/test_transforms/test_code_compressor.py
109 passed, 1 skipped in 11.24s
$ ruff check / format: All checks passed;  mypy compression_summary.py: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) import check: `summarize_compressed_code` still works
and `summarize_dropped_items` is gone.
- Observed result: The proxy serves both requests (200) with the same
savings as main. The live summary helper works, and the removed one
raises ImportError. Output below.
- Not tested: The LLM integration test (it skips without
ANTHROPIC_API_KEY).

```text
branch: cleanup/transforms-remove-summarize-dropped-items @ bc0dc2f6
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-51402.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -c "from headroom.transforms.compression_summary import summarize_compressed_code; print(summarize_compressed_code([...2 bodies...], 2)); from headroom.transforms.compression_summary import summarize_dropped_items"
2 bodies compressed: authenticate(), refresh()
ImportError: cannot import name 'summarize_dropped_items' from 'headroom.transforms.compression_summary' (headroom/transforms/compression_summary.py)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: No. The removed function was not
called at runtime.
- Kill switch / disable path: Not applicable; nothing that runs changes.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:58:19 -07:00
Suliman AbdulrazzaqandJerrettDavis 3db1a94443 fix(smart-crusher): keep string cells in json columns from decoding as other types (#4053)
## Description

The SmartCrusher lossless table (`csv-schema`) types a column `json`
when its cells mix types, for example `[26]{id:int,v:json?}`.
`_decode_cell` in `headroom/transforms/compaction_codec.py` reads every
cell of a `json` column with `json.loads` and only keeps the raw text
when that fails. The Rust encoder did not know the column type when it
rendered a string cell, so strings that spell a JSON value came back as
that value:

- the string `null` was written `"null"` and decoded as `None`
- the string `true` was written `true` (byte-identical to the bool) and
decoded as `True`
- the string `1` was written `1` and decoded as the int `1`
- `NaN` decoded as a float nan, and the string `"q"` (a string that
carries its own quotes) decoded as `q`

A string that parses as a JSON array or object (`[1]`, `{"a":1}`) never
reaches the formatter as a string: the classifier already un-escapes it
(`CellClass::StringifiedJson`), which is existing behavior and is not
changed here.

This changes the encoder only. In a column whose type tag is `json`, a
string cell is now written as a JSON string literal (then CSV-quoted)
when `json.loads` could parse it into something else. The decoder
already returns the string for that form, so it is not touched.
Everything else renders as before.

"Could parse into something else" is a conservative superset: after
trimming JSON white space (space, tab, newline, carriage return), the
text starts with `[`, `{`, `"`, `-` or an ASCII digit, or is exactly
`true`, `false`, `null`, `NaN` or `Infinity`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

-
`crates/headroom-core/src/transforms/smart_crusher/compaction/formatter.rs`:
`write_table` works out once which columns have type tag `json` and
passes that flag to `format_cell`. For a `Value::String` cell in such a
column, `is_ambiguous_json_text` decides whether it is rendered as
`csv_quote(serde_json::to_string(s))`. Non-string cells, the
`string`/`int`/`float`/`bool` columns, the JSON formatter and the
Markdown-KV formatter are unchanged. Bucketed tables go through
`write_table` too, so they get the same treatment.
- Rust tests in the same file:
`csv_formatter_json_column_quotes_strings_that_parse_as_json` (13
strings, exact wire output, with a `string` column as control),
`csv_formatter_json_column_leaves_other_strings_alone`, and
`csv_formatter_json_column_tells_strings_from_values`, which goes
through `compact` + `CsvSchemaFormatter` so the column type really comes
out as `json`, and checks that the string `"true"` and the bool `true`
no longer share a wire form.
- `tests/test_transforms/test_compaction_codec.py`: one test that
decodes a hand-written block using the new wire form next to the bare
forms for real values. It documents the contract the Rust change relies
on and passes with or without this change, since the decoder is not
changed.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` on the touched test file; `cargo fmt`
and `cargo clippy -D warnings` in CI)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Rust, GitHub Actions run on our fork (toolchain 1.95.0)
https://github.com/SulimanAbdulrazzaq/headroom/actions/runs/37829441316
  cargo fmt --all -- --check                                 ok
  cargo clippy -p headroom-core --all-targets -- -D warnings ok
  cargo test -p headroom-core smart_crusher                  345 passed; 0 failed
    (342 on main in run 37827119554; the 3 new tests are the difference)
  csv_formatter_json_column_leaves_other_strings_alone ... ok
  csv_formatter_json_column_quotes_strings_that_parse_as_json ... ok
  csv_formatter_json_column_tells_strings_from_values ... ok

# Python, 3.12
$ pytest tests/test_transforms/test_compaction_codec.py -q
17 passed   (16 existing + 1 new)
$ ruff check tests/test_transforms/test_compaction_codec.py          All checks passed!
$ ruff format --check tests/test_transforms/test_compaction_codec.py 1 file already formatted
```

## Real Behavior Proof

- Environment: Rust core built and tested on `ubuntu-latest` in GitHub
Actions with Rust 1.95.0 (the repository pin). Python 3.12.13 on Windows
11 for the decoder test and the reproducer.
- Exact command / steps: a script crushes 26 records `{"id": i, "v":
...}` whose `v` cycles through the strings `null`, `true`, `1`, `NaN`
and other values, decodes the result with `expand_compacted`, and
compares each row with its input. The encoded header is
`[26]{id:int,v:json?}`. In CI: `cargo test -p headroom-core
smart_crusher`.
- Observed result: before the change (the Rust core of the installed
`headroom-ai` 0.40.0 wheel), 8 of 26 rows differ, for example `'null'
(str) -> None`, `'true' (str) -> True`, `'1' (str) -> 1`, `'NaN' (str)
-> nan`. After the change the new Rust test shows the string `null` in a
`json` column is written `"""null"""` and the string `hello` is still
written `hello`; in the real-path test (through `compact` and
`CsvSchemaFormatter`) the rows are `1,"""null"""`, `2,null`,
`3,"""true"""`, `4,true`, `5,"""1"""`, `6,1`, `7,hello`, so each string
and each value has its own wire form. The Python test shows the decoder
returns `'null'`, `None`, `'true'`, `True`, `'1'`, `1`, `'NaN'` for such
a block.
- Not tested: the 26-row Python reproducer against a freshly built core
(the installed wheel on my machine is older and cannot see the Rust
change), and the Python wheel end to end. The Python decoder was not
changed.

### Maintainer native qualification (2026-10-08)

- Environment: Windows 11, Rust/MSVC, Python 3.12.13. This head contains
current `main`; contributor formatter logic was not modified.
- Exact commands: `cargo test -p headroom-core --lib
transforms::smart_crusher::compaction --locked`; `cargo build -p
headroom-py --features extension-module --locked`; then the PR's
`tests/test_transforms/test_compaction_codec.py` suite using that newly
built extension. A standalone native SmartCrusher/Python decoder probe
also encoded and decoded 100 records containing mixed
string/null/bool/numeric cells.
- Observed: **73 Rust compaction tests and 17 Python codec tests
passed**. The 100-record probe preserved every value and Python type,
including literal strings `null`, `true`, `false`, `1`, `NaN`,
`Infinity`, `-Infinity`, whitespace, quotes, empty strings, and embedded
newlines. No native encoder/decoder boundary was mocked.
- Scope: JSON-container-looking strings (`[1]`, `{"a":1}`) retain the
classifier's pre-existing `StringifiedJson` behavior. They are
explicitly outside this formatter repair; the author clarified that
limitation in the test. This PR fixes mixed-column primitive-literal
strings that actually reach the formatter, and does not claim a new
container-string round-trip contract.
- Hosted CI is tracked by the maintainer at this exact head; approval
remains pending until all applicable current checks pass.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: the lossless table text changes only
for string cells that look like JSON literals inside `json`-typed
columns. All other cells and all other column types render as before.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit. Older decoders read the new form
correctly, because it is a plain quoted JSON string, so nothing else
needs to change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Rust tests ran in CI on a fork, not locally, because the machine used
for development has no linker for the Rust toolchain. `mypy` and the
full Python suite were not run, only the touched test file. No other
test or fixture in the repository asserts the wire form of a string in a
`json` column (searched `crates`, `tests`, `docs` and `headroom` for
`json?` and `:json`).

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-08 22:40:54 -05:00
Yasser SheikhandJerrettDavis 70b21c12c4 feat(proxy): native spend/savings stats + dashboard for the Rust proxy (#2321)
# feat(proxy): native spend/savings stats + dashboard for the Rust proxy

## Description

The Rust proxy has no cost observability of its own today: `/stats` and
`/dashboard` only exist on the Python proxy, so anyone running
`headroom-proxy` standalone (the Bedrock/Vertex lanes especially) is
flying blind on spend.

This adds it natively. Every lane the Rust proxy forwards — Anthropic
`/v1/messages`, OpenAI Chat + Responses, all four native Bedrock routes,
Vertex `rawPredict`/`streamRawPredict` — records per-request cost/usage
telemetry into one in-process ledger, served from `GET /stats`, `GET
/stats/timeseries`, `GET /stats/events`, and an embedded,
dependency-free `GET /dashboard`.

This is a ground-up redesign of #1057 (approved but never merged, now
badly stale against `main`), not a rebase of it. Supersedes #1057. The
"Design" section below states exactly what was kept and what was
deliberately dropped, so it can be reviewed on its own terms.

### What it measures today

Spend, input/output/cache-read/cache-write tokens, prompt-cache hit rate
and its NET dollar benefit, per-provider and per-model breakdowns,
failures, latency, and hourly/daily history that survives restarts.

Two numbers are computed carefully rather than the obvious way, because
the obvious way flatters the product:

- **Spend counts output.** Input and output are priced separately
(`input_cost_usd` / `output_cost_usd`, summed into `total_cost_usd`)
because output bills at its own rate — typically 4-5x input. Counting
input alone would understate a short-prompt/long-answer turn by most of
its cost.
- **Cache savings are NET**, not the gross read discount: the read
discount minus the cache-write premium (writes bill *above* list price),
floored at $0 per request. Reporting gross reads overstates the benefit
on write-heavy traffic.

### What reads 0 today — stated up front

`tokens_saved` / `compression_savings_usd` will read **0 on real
traffic**, because the Rust live-zone per-type compressors are still
no-ops on `main` — there is nothing to record yet. Verified against the
real proxy: a 13.8 KB body with a large `tool_result`, sent with
`--compression --compression-mode live_zone`, produced `tokens_before:
0` (dispatcher returned `NoCompression`), which is the invariant
`live_zone_mode_with_valid_body_returns_no_compression_pr_b2` pins. The
recording path is wired and unit-tested against `Outcome::Compressed`,
and starts populating on its own the moment those compressors land — no
changes needed here. Stating it up front so `Saved $0` isn't read as a
bug.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Design — how this differs from #1057

- **One fold, many views.** A single `Totals::fold` is the only place
request accounting happens; lifetime, since-boot session, per-provider,
per-model, hourly and daily buckets are that same fold under different
keys, so the views cannot disagree. #1057 (like the Python tracker it
ported) maintained each aggregate separately.
- **Its own schema, not Python emulation.** `/stats` is a documented
Rust-native payload (`version: 1` on disk). No `schema_version: 5` file
compat, no `display_session` idle-window emulation, no `projects`
passthrough. The five keys external unified-stats consumers already read
from this endpoint (`summary.api_requests`, `requests.failed`,
`requests.cached`, `tokens.saved`, `tokens.proxy_compression_saved`) are
kept and pinned by a test.
- **No federation flag.** #1057's `--upstream-stats-url` supplemental
merge is dropped (YAGNI — the deployed consumer merges in the opposite
direction).
- **No new price data file.** USD valuation reuses the LiteLLM table
already vendored for `compression::model_limits`. #1057 shipped a
second, models.dev-shaped price book.
- **Purpose-built dashboard.** One self-contained HTML file designed for
what the Rust proxy actually measures, instead of embedding the Python
template and rendering its ~20 Python-only panels as zeros.
- **Not gated on compression.** `--compression` defaults to **off**, so
recording gated behind the compression buffer (as #1057 had it) records
nothing under default config. Here the model is sourced from the
*response* when the request body isn't buffered, so spend/cache
observability works on a pure passthrough deployment — with zero change
to the byte path.

## Correctness rules (carried from #1057's review, all test-pinned)

- **Deferred recording** — a request records only when its response is
fully observed, so streaming token counts come from real SSE/EventStream
usage frames instead of 0.
- **Failed upstreams accrue nothing** — counted in `requests`/`failed`
only; retries can't inflate savings.
- **No disk I/O on the request path** — `record()` folds under a mutex
and sets a dirty flag; a background task (and shutdown) writes an atomic
snapshot (temp → fsync → rename → fsync dir), re-arming the dirty flag
on failure.
- **Bounded everything** — per-model cardinality caps with `"other"`
overflow, length-clamped client strings, a 50M-token per-event ceiling,
bounded capture queues/buffers, bounded warn-once set.
- **Honest pricing** — exact-match only; a generic id misses (and WARNs)
rather than mispricing. Unknown ⇒ $0, never a blended fallback.
- **Bedrock regional pricing respected** — the unmodified id is tried
first, because `eu.anthropic.claude-3-5-haiku…` genuinely bills $0.25/M
against the base id's $0.80/M. Geo-stripping is only a fallback for
regions the table doesn't list.

## Pre-existing bugs fixed along the way

Both were found while wiring the recording hooks; both affect `main`
today, independently of stats.

- **Bedrock geo-prefix coverage**: `au.`, `jp.`, and `us-gov.`
cross-region inference profiles weren't recognized, so Anthropic models
invoked through them silently skipped compression — the same failure
mode #953 reported and #999 fixed for `eu.`/`us.`/`apac.`/`global.`. The
list is now complete (all seven published geos) and shared
(`bedrock::vendor::GEO_PREFIXES`) between the vendor gate and pricing,
so the two can't drift apart again.
- **EventStream `Skip` frame stranding**: the streaming parser pulled a
fresh upstream chunk after each unknown (`Skip`ped) event type, delaying
— and at stream end **dropping** — frames already buffered. It now
drains the buffer to exhaustion before awaiting more bytes.
Regression-pinned by
`skip_in_final_chunk_does_not_strand_buffered_frames`.

## Changes Made

- `crates/headroom-proxy/src/observability/ledger.rs` — the store:
`Totals` fold, bounded maps/buckets, atomic snapshot persistence,
payload builders
- `crates/headroom-proxy/src/observability/pricing.rs` — deterministic
lookup over the vendored LiteLLM table (no new data file)
- `crates/headroom-proxy/src/observability/capture.rs` — `PendingRecord`
(deferred finalization), usage extraction for every wire shape, bounded
tee tasks off the byte path
- `crates/headroom-proxy/src/observability/stats_http.rs` +
`dashboard.html` — HTTP surface + embedded self-contained UI
- `crates/headroom-proxy/src/proxy.rs`, `bedrock/invoke.rs`,
`bedrock/invoke_streaming.rs`, `vertex/raw_predict.rs` — per-lane
recording hooks
- `crates/headroom-proxy/src/config.rs` — `--stats` (default on),
`--stats-path`
- `crates/headroom-proxy/src/bedrock/vendor.rs` — shared `GEO_PREFIXES`
(the geo-coverage fix above)
- `crates/headroom-proxy/src/sse/framing.rs` — 4 MiB accumulator cap
(telemetry-side; a stream with no event terminator can no longer grow
the buffer unboundedly)

`--stats=false` restores byte-for-byte pre-feature behaviour (paths
tunnel upstream, nothing recorded) — pinned by
`stats_disabled_falls_through_to_upstream`.

## Testing

- [x] `cargo fmt --all -- --check` passes
- [x] `cargo clippy --workspace -- -D warnings` passes (0 warnings)
- [x] `cargo test --workspace` passes
- [x] New tests added for new functionality (53 new tests across unit +
integration)
- [x] Manual testing performed (live AWS Bedrock, all four native lanes
— see Real Behavior Proof)

Unit: fold-consistency across views, failed-no-savings, pricing
(regional/revision/path forms, collision-miss, untracked-region
fallback), USD math, bounded maps, token-ceiling clamp,
bidi/control-char stripping, atomic flush with no temp litter,
corrupt-file preservation, oversized-state rejection, negative-USD
clamping on load, restart round-trip, week/month rollups, hostile field
lengths. Integration (wiremock): endpoint contracts, `--stats=false`
tunneling, deferred SSE capture, non-streaming capture, OpenAI
cached-token normalisation, failed-upstream accounting, hostile-usage
clamping, all four Bedrock lanes, Vertex, dashboard self-containment.

The persisted-state tests are deliberately adversarial rather than
round-trip only, since the ledger reads a file an operator can
hand-edit: malformed bucket keys (including multi-byte ones that would
panic a naive `&str` slice) are dropped before the age-trim runs,
hostile model-id keys are re-clamped on load, and an implausibly large
state file is treated as corrupt instead of allocated.

### Test Output

```text
$ cargo fmt --all -- --check
  (clean)

$ cargo clippy -p headroom-proxy --all-targets -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo nextest run -p headroom-proxy
     Summary [4.396s] 488 tests run: 488 passed, 0 skipped

$ cargo nextest run -p headroom-core -p headroom-simulators -p headroom-parity
     Summary [49.535s] 967 tests run: 967 passed, 1 skipped

$ cargo clippy --workspace -- -D warnings
  exit=0   0 warnings, 0 errors

$ cargo test --workspace --doc
  ok, 0 failed
```

## Real Behavior Proof

- Environment: live AWS Bedrock, staging account (SSO profile — name
redacted), `eu-central-1`, `anthropic.claude-3-haiku-20240307-v1:0`,
release binary, compression off (shipped default)
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:8790
--enable-bedrock-native --bedrock-region eu-central-1` then one request
per lane (`/invoke`, `/converse`, `/invoke-with-response-stream` with
`Accept: application/vnd.amazon.eventstream`, `/converse-stream`), one
deliberate failure (nonexistent model), `SIGTERM`, restart, re-read
`/stats` — full transcript, raw proxy logs, and the persisted ledger are
attached below
- Observed result: all four lanes captured usage (input tokens = 48 = 4
× 12; spend $0.000012 = 48 × $0.25/M, exact for Bedrock Haiku 3); the
failure recorded `failed: true` with 0 tokens and $0; lifetime totals
survived the restart while session totals reset; dashboard rendered
against this live data (screenshots attached)
- Not tested: live Vertex (covered by wiremock integration tests only —
no GCP staging account available); live OpenAI Chat/Responses lanes
(same wiremock coverage); dashboard on Windows browsers

The four Bedrock lanes are the ones worth proving live, because two of
them had no parsing before this PR:

| Lane | Wire shape | Captured |
|---|---|---|
| `/invoke` | sync, snake_case `usage` | 12 in / 4 out |
| `/converse` | sync, **camelCase** `usage` | 12 in / 4 out |
| `/invoke-with-response-stream` | **binary EventStream passthrough**,
base64-wrapped chunks | 12 in / 4 out |
| `/converse-stream` | SSE-translated; usage lives in the camelCase
`metadata` frame the Anthropic state machine drops | 12 in / 4 out |

```text
requests      : 5 (failed: 1)
input tokens  : 48
output tokens : 16
cache read/wr : 0 / 0
input spend   : $0.00001200      # 48 × $0.25/M
output spend  : $0.00002000      # 16 × $1.25/M — output bills 5x here
TOTAL spend   : $0.00003200      # what /dashboard reports as "Spend"
savings       : $0.00000000      # compression off -> 0 by design
avg latency   : 256 ms
by provider   : {'bedrock': 5}
by model      : {'anthropic.claude-3-haiku-20240307-v1:0': 4,
                 'anthropic.claude-nonexistent-v99:0': 1}
persistence   : True | healthy: True
```

Those two spend lines are the reason output pricing is in this PR:
**output was 62.5% of the real bill** on this run. Reporting input alone
would have called a $0.000032 spend `$0.000012`.

The exposure split was exercised against the same live proxy:

```text
$ curl /stats/events  -H 'Host: 127.0.0.1:18787'   -> 200   (local tooling works)
$ curl /stats/events  -H 'Host: attacker.com'      -> 404   (DNS-rebound: peer IS loopback)
$ curl /stats         -H 'Host: attacker.com'      -> recent_requests: null
                                                      persistence.path: null
                                                      requests.total still served
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(RUST_DEV.md, CHANGELOG.md)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Known limitations

1. Compression savings read 0 until the live-zone compressors land
(above).
2. Hourly resolution is 48 h, daily ~13 months; older traffic folds into
lifetime totals only. `/stats/events` is an in-memory ring of 100 and is
not persisted — this is a cost ledger, not an audit log.
3. The vendored price table lacks some legacy direct-API ids
(`claude-3-5-sonnet-20241022` is absent while its Bedrock form is
present); those record $0 with a WARN until refreshed via
`scripts/refresh_model_limits.sh`.
4. `output_cost_usd` is absent from state files written before this
change, so it loads as 0: historical spend keeps reading input-only
until those buckets age out (48 h hourly / ~400 d daily). New traffic is
priced in full immediately. Deliberately no migration shim — the field
defaults cleanly and the alternative is retro-pricing history against
today's price table, which would invent numbers.
5. A non-streaming response body over 2 MiB isn't buffered for usage
parsing (the request still records, compression-side numbers only).
Streaming responses have no such cap — they parse incrementally.
6. The endpoints carry no auth, so exposure is split by tier instead
(matching the Python proxy): aggregates are public like `/metrics`,
while per-request rows (`recent_requests`, `/stats/events`) and the
ledger's filesystem path require a local caller — loopback peer **and**
loopback `Host`, the latter being the DNS-rebinding defence. Remote
viewers of `/dashboard` therefore see totals but not the request feed,
and the page says so. Opening that tier to a trusted reverse proxy is
#1959's call to make for both proxies; this PR doesn't invent a second
scheme.
7. Copilot traffic isn't proxied by the Rust binary; it reaches this
dashboard only via external unified-stats layers (unchanged).

## Screenshots (if applicable)

Attached below: live-test terminal transcript, dashboard rendered
against the live Bedrock data (session + lifetime scopes, before/after
restart), raw proxy logs, and the persisted ledger file.

## Additional Notes

- Related issues/PRs: supersedes #1057; completes the geo-prefix
coverage from #999 (design discussion in #953 — the multi-vendor adapter
work proposed there remains out of scope).
- `requests.cached` counts requests whose response reported
`cache_read_tokens > 0`, matching the Python tracker's derivation, so
unified-stats consumers see consistent semantics across both proxies.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2321"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-07 20:58:22 -05:00
Mansurshakh JaparovandJD Davis a42fb2b9a1 fix(router): preserve human-readable Git status losslessly (#4020)
## Description

Addresses the still-reproducible Git status corruption in #3652. The
aligned `ls`/`ps`/`df` portions are already protected in current main;
this PR does not claim them as new fixes. Human-readable Git status is
still classified as prose and can lose staged/unstaged/untracked paths
through Kompress.

This is a narrow content-preservation bug fix, not the separate policy
proposal to exclude all `Bash` outputs by default.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Recognize the standard English branch/detached-HEAD header plus Git
status sections/entries as structured tabular content.
- Preserve this non-rectangular `git_status` format verbatim instead of
forcing it into CSV or a prose model.
- Keep Python detection and the native Rust content/live-zone route
aligned; Rust detection uses iterator-based scans without
collection/cloning of the input.
- Leave ordinary prose, bullet lists, and other Bash output on their
existing routes.
- Guard Git status before native model classification and before
external CSV/wildcard compressor dispatch; retain the locked native
Magika production chain.
- Replace the stale `PLAIN_TEXT` status expectation with
`format=git_status` and `parse_tabular(status) is None`; keep
prose/git-log negative cases.
- Document the English-format scope and lossless/no-savings tradeoff.

## Testing

- [x] Unit tests pass (`pytest`), including the consumer-visible
lossless regression.
- [x] Native regression tests, changed-file lint, configured pre-commit
mypy and manual production runtime checks pass.

### Test Output

```text
Review correction:
Python router/external-dispatch regressions: 121 passed, 2 skipped
Existing tabular parser/detection suite: 73 passed in 7.76s
Native Git-status regressions: 3 passed
Final mixed-content correction (router, external dispatch, tabular suites):
184 passed, 2 skipped in 52.14s
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate used
the explicitly installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, with cached-model assertions enabled.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 27.72s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_transforms_content_router.py
cargo test -p headroom-core git_status
cargo test -p headroom-core ordinary_bullet
```

Python content-router suite: **84 passed, 2 skipped**, with an existing
Hugging Face deprecation warning. Native focused regressions: **2 Git
status tests passed**, **1 ordinary-bullet negative passed**. The native
extension was rebuilt and the build/import verification passed. The
final deterministic consumer regression uses a ready lossy prose
compressor so a cold model cannot accidentally mask missing protection;
it asserts direct and tool-result output bytes, not internal call
counts.

The deterministic regression fails against the unchanged baseline with
missing status rows. After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran the status regression
plus main's six small-MCP result tests: **7 passed**, and reran the
native Git status cases: **2 passed**.

Changed-file Ruff and formatting passed. The installed pre-commit hooks,
including `mypy headroom` and Commitlint, passed for this commit.
Full-tree Ruff is locally blocked by executable filesystem permissions
on five unchanged OAuth2 plugin files (tracked 100644); they are not
modified. A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit environment passed. No unrelated source/permissions were
changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; actual Git
fixture, real content router and Anthropic token counter, rebuilt Rust
extension, **real ONNX Kompress fully preloaded before each run**, no
mocked compression backend in this smoke.
- Exact command / steps: initialize a throwaway Git repo; commit 120
files named `modified path NNN α.txt`; modify those, stage 120 new files
named `staged path NNN β.txt`, and add 120 untracked files named
`untracked path NNN γ.txt`. Capture:
  ```sh
LC_ALL=C git -c core.quotePath=false -c color.status=false status
--branch
  ```
Load the actual model with
`KompressCompressor().preload(allow_download=True)`, run
`ContentRouter(ContentRouterConfig()).compress(status)`, then run the
same router's `apply` on an Anthropic `Bash` tool-use/tool-result pair
using
`AnthropicProvider(warn=False).get_token_counter('claude-sonnet-4-5')`
and model limit 200000. Compare output bytes to the same captured
status.
- Same input in both runs: **12,469 bytes**, SHA256
`78ed6a39f2f994159099f8d4c0fe052ca09a37fa6d9f6248e8c5ebb655b19a32`.
- Before: detected as `text`; direct and Bash tool-result routes both
returned **10,748 bytes**, neither byte-exact; transform
`router:tool_result:text`.
- Observed result: after the fix, detected as `tabular`; both routes
returned **12,469 bytes**, both byte-exact; `router:noop`.
- External-compressor review proof against published head `3b035027`: a
real Git repository with 120 untracked paths produced the same
**3,415-byte** input (SHA256
`e0164d25f3f47428686d102a73018d0e00bcda53ff9674e658aa7128c038896b`).
With an operational row-trimming plugin selected for `text/csv` and
`*/*`, direct and tool-output paths both shrank to **318 bytes** before;
both preserve all **3,415 bytes** after. The same plugin still
compresses ordinary CSV. This separate plugin smoke uses the production
Tiktoken counter and disables Kompress to isolate external dispatch.
- Final mixed-content review: on published `539669c6`, appended JSON
selected `MIXED` and changed both direct and Bash tool-result bytes; the
fenced-code example was already preserved. The strategy selector now
honors `git_status` before mixed splitting. Both appended JSON and
fenced code preserve the complete payload on both routes with a ready
lossy compressor available.
- Not tested: localized Git status formats, every platform-specific Git
variant, or a paid upstream model response. The recognizer deliberately
covers standard English status output. The local real compression path
was exercised after model warmup; the initial cold-model no-op was not
treated as proof.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; content-routing data-preservation
fix.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: recognized Git status is no longer
compressed lossily, sacrificing savings for path/state integrity.
- Kill switch / disable path: existing global Bash exclusion can bypass
compression; no new flag.
- Unsafe override required: no.
- Qualification impact: Python/Rust route parity for the recognized
status format only.
- Rollback path: revert this commit; no persistent schema migration.
Global Bash exclusions remain unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing tabular lossless route reused; compression docs updated.
- [x] Consumer regression and native positive/negative tests added.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

This PR completes the remaining Git status defect, but intentionally
does not close #3652's separate Bash-default policy request. Remote CI
is reported by GitHub, not assumed from local tests.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:57 -05:00
Laura MoneyandJerrettDavis a05717f6ec fix(proxy): redact upstream error detail and add opt-in /metrics loopback gate (#2589)
## Description

Two response-path information-exposure hardenings for the native Rust
reverse proxy (`crates/headroom-proxy`). Both are defensive and
non-breaking; neither adds a user-facing capability nor changes
forwarding behaviour, and defaults are unchanged.

1. **Redact upstream error detail from client responses.**
`ProxyError::into_response` returned `self.to_string()` as the client
body, which for `reqwest` upstream errors interpolates the upstream
URL/host/port. It now returns a generic, stable message per status while
the full detail is retained in the server-side `tracing::warn!` log.
`413` intentionally stays descriptive (no internal detail; retry-on-413
clients rely on recognising it).
2. **Opt-in loopback gate for `/metrics`.** New
`--metrics-require-loopback` / `HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK`
(default `false`). When enabled, the Prometheus scrape returns `403` to
non-loopback peers. This is defense-in-depth for deployments that bind a
non-loopback `--listen`. Default-off preserves current behaviour
exactly.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `error.rs`: client-facing bodies are now generic per status; full
detail stays in the log only.
- `config.rs`: add `--metrics-require-loopback` /
`HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK` (default `false`).
- `proxy.rs`: mount `/metrics` in its own sub-router so the optional
`require_metrics_loopback` layer gates only that route (403 to
non-loopback peers when enabled).
- `RUST_DEV.md`: document the new flag in the configuration table.
- Tests: new `tests/integration_metrics_loopback.rs` plus unit tests for
the gate (v4/v6 loopback and non-loopback) and for error-body redaction.

## Testing

<!-- This PR touches only the Rust `headroom-proxy` crate; the
pytest/ruff/mypy items are Python and N/A here (see Additional Notes).
Validated with cargo on the repo-pinned toolchain. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo fmt --all -- --check          # clean
$ cargo clippy --workspace -- -D warnings   # 0 warnings
$ cargo test --workspace
test result: ok. 1432 passed; 0 failed; 0 ignored   (aggregate across all suites)
$ make test-parity
[tokenizer/smart_crusher/content_detector/text_crusher/diff/log/search] diffed=0
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Rust 1.95.0 (repo-pinned
`rust-toolchain.toml`), `headroom-proxy` debug binary run directly (no
mocks), dead upstream `http://127.0.0.1:9`.
- Exact command / steps: started with
`HEADROOM_PROXY_METRICS_REQUIRE_LOOPBACK=true --listen 0.0.0.0:18787`,
then `curl http://127.0.0.1:18787/metrics`, `curl
http://192.168.1.184:18787/metrics` (LAN IP, non-loopback), and `curl -X
POST http://127.0.0.1:18787/v1/messages -d
'{"model":"x","messages":[]}'`; then restarted with the gate off and
repeated the non-loopback `/metrics` scrape.
- Observed result: loopback `/metrics` returned `200`; non-loopback
`/metrics` returned `403` with body `metrics endpoint restricted to
loopback`; the dead-upstream POST returned `502` with client body
`upstream connection failed` while the server log retained the full
detail (`... error sending request for url
(http://127.0.0.1:9/v1/messages)`), i.e. the upstream URL is logged, not
returned to the client; with the gate off, the non-loopback `/metrics`
scrape returned `200` (default behaviour unchanged).
- Not tested: live cloud upstreams (Bedrock/Vertex/OpenAI/Anthropic real
endpoints); the Python proxy path (unchanged by this PR); Windows/Linux
(macOS only).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A (server-side behaviour; see Real Behavior Proof for command output).

## Additional Notes

- This PR modifies only the Rust `headroom-proxy` crate (plus its docs),
so the Python-oriented Testing checkboxes (`pytest`, `ruff`, `mypy`) are
N/A; validation was via `cargo fmt`/`clippy`/`test` and `make
test-parity` on the repo-pinned toolchain, per the Test Output above.
- No new dependencies. Both changes are non-breaking and leave all
defaults unchanged (the `/metrics` gate is opt-in).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2589"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-05 15:29:31 -05:00
Tejas ChopraandClaude Opus 5 119d1a1939 fix(offline): make HEADROOM_OFFLINE a real air-gap via one chokepoint (#3729)
## Description

**Revised 2026-10-05 after review: merged current `main`, routed main's
new TLS-diagnostic egress (`doctor --network` and the TLS-error chain
re-probe) through `guard_egress`, made the test socket trap let loopback
through so the async offline tests run on Windows, and closed five
model-loader findings from Devin Review.**

`HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several
egress paths never consulted `headroom/offline.py`. This PR makes
`guard_egress` the single chokepoint every Headroom-initiated outbound
connection goes through, with a per-site meta-test that fails on any new
unguarded egress. Full detail of every revision is under *Changes Made*.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change) — outbound calls that previously worked with
`HEADROOM_OFFLINE=1` set now refuse; see *Behaviour change customers can
see*
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### This revision (2026-10-05, after @JerrettDavis's review of
2026-09-24 and comment of 2026-10-03)

* **Merged current `main`.** Conflicts in `docs/content/docs/proxy.mdx`
(kept main's new rows, this PR's offline row),
`headroom/image/onnx_router.py` (both imports) and
`headroom/proxy/server.py` lifespan (main's rust-core passthrough block,
then this PR's `_configure_observability_or_refuse()`).
* **`doctor --network` / TLS diagnostics now go through the guard.**
`tls_diagnostics.probe_endpoint` calls `guard_egress` before both the
certificate-chain handshake and the `httpx.Client` GET; `headroom
doctor` turns the refusal into one `network · skipped: HEADROOM_OFFLINE
is set, no endpoint was contacted` row (status `skip`, so it does not
fail the doctor) instead of a page of "unreachable". `_probe_uncached` —
the chain handshake used both by doctor and by the request path's
TLS-error re-probe (`describe_upstream_failure`) — guards before its
socket; being diagnostic-only on the request path, it reports `skipped:
…` and the TLS explanation is still produced without the chain. No
allowlist widening: the inventory test passes because the `httpx.Client`
site is now dominated by a guard.
* **Socket trap is Windows-safe.** `no_sockets` now passes connects
whose address is a loopback **IP literal** (the Windows Proactor
self-pipe is a `127.0.0.1` listen + connect) and still fails every other
address, including hostnames such as `localhost`. `TestSocketTrap`
reproduces the Windows path on every platform by forcing
`socket._fallback_socketpair` (the implementation Windows uses for
`socket.socketpair()`) under `asyncio.run()`, and pins that outbound
IPv4/IPv6 `connect` / `connect_ex` / `create_connection` are still
caught.
* **Devin Review findings (`40e2d449`).**
* **Kompress tokenizer bypass.** Cached ONNX weights plus an uncached
ModernBERT tokenizer fell through to an unguarded
`from_pretrained(local_files_only=False)`. That call is now guarded and
becomes `KompressModelNotCached`, the same translation `_hf_artifact`
uses.
* **Kompress ONNX candidates.** An uncached first candidate no longer
ends the loop. The refusal is logged once, then the remaining candidates
are checked cache-only, so a cached fp32 fallback still loads offline.
* **fastembed.** `huggingface_hub` reads `HF_HUB_OFFLINE` once, at
import, into `constants.HF_HUB_OFFLINE`. The cache-only attempt
therefore forces that constant as well as the env var, and restores it
afterwards.
* **Image router.** It now resolves every artifact before publishing the
session, which doubles as the "loaded" flag. A refused secondary
artifact no longer leaves a half-loaded router that never retries.
* **Eval runner.** The local Ollama provider is no longer refused, for
the same operator-endpoint reason as the Ollama embedder. Anthropic and
OpenAI are still refused.
* **Sweep of main for other new egress since this PR was written:** the
scanner's only new finding was `tls_diagnostics.py`. Manually checked
the rest of the diff: the new `reqwest` client in
`crates/headroom-proxy` and the keep-alive `httpx.AsyncClient` in
`proxy/server.py` are upstream forwarding (`user-traffic`); the new HF
tokenizer allowlist in `tokenizers/huggingface.py` sits behind the
existing `is_offline()` gate (`gated`); `port_discovery.py`,
`upstream_pinning.py`, `license_env.py` and `credential_policy.py` open
no new external connections.

### A-2 (S2): the air-gap guarantee was not real

`HEADROOM_OFFLINE=1` is documented as an air-gap switch, but several
egress paths never consulted `headroom/offline.py`. A customer who set
it and believed they were air-gapped was wrong.

**This PR has been revised after an adversarial review found that the
first version's guarantee was largely unenforced.** The review is
reproduced in the fixes below; read the *What this does and does not
promise* section for the honest scope.

### The chokepoint

`headroom/offline.py` provides `guard_egress(purpose, destination)` and
a named `OfflineEgressBlocked`. It refuses loudly — a silent skip is
indistinguishable from success at the call site, which is how these
holes survived. It is always called **before the HTTP client is
constructed**, not merely before the request, so no pooled connection is
ever opened. `crates/headroom-core/src/offline.rs` is the same switch,
same env var, same rules, for the Rust core.

#### Paths routed through it

| path | previously |
| --- | --- |
| `headroom/transforms/kompress_remote.py` | built an `httpx.Client` and
`POST`ed every compressible block to the remote Kompress endpoint |
| `headroom/observability/metrics.py` | started an `OTLPMetricExporter`
plus a background `PeriodicExportingMetricReader` timer |
| `headroom/observability/tracing.py` | started an `OTLPSpanExporter` to
Langfuse Cloud — the metric exporter's twin, missed the first time |
| `headroom/onnx_runtime.py` | `hf_hub_download` on a cache miss: the
**Python** half of the HF fetch, every ONNX model in the tree resolves
through it |
| `crates/…/tokenizer/hf_impl.rs` | fetched `tokenizer.json` from
huggingface.co |
| `crates/…/transforms/kompress.rs` | fetched the **Kompress model**
from huggingface.co |
| `crates/…/relevance/embedding.rs` | fastembed fetched ONNX weights
from huggingface.co |

The three Rust sites are the same shape and the same argument:
`HF_HUB_OFFLINE` only shortcuts the *cache lookup*, and on a miss
`hf-hub` still dials out. The first version of this PR guarded one of
the three and left the other two open.

Remote Kompress is guarded twice — in `__init__`, and again at the top
of `compress()` **outside** the fail-open `try`, because the router
caches one compressor for the process lifetime and `compress()` converts
every exception into a silent passthrough.

`onnx_runtime.hf_hub_download_local_first` is guarded on the **network
fallback only**, so a pre-seeded air-gapped cache still resolves. The
three Rust fetches refuse even on a warm cache: `hf-hub` 0.5 exposes no
cache-hit/cache-miss split at that layer, so "guard before the client
exists" and "serve from the cache" cannot both hold.
`Kompress::from_files` / `HfTokenizer::from_file` are the network-free
paths for a pre-seeded Rust deployment. Splitting the Rust fetches
cache-first is the obvious follow-up and is **not** done here.

### The refusal now survives the fail-open handlers

The original docstring said "anything that catches broad `Exception`
around an egress call should re-raise this". **Nothing did** — there
were zero re-raise sites in the package — and the refusal was swallowed
on the way out of a request by `thinking_compactor._memo_compact`
(wrapped directly around the guarded `kompress.compress` call),
`ContentRouter._kompress_model_ready` (which reported a policy refusal
as "model ready"), the startup preload, and the OpenAI
reasoning-compaction path. A full `/v1/messages` request with
`HEADROOM_OFFLINE=1` + `HEADROOM_KOMPRESS_ENDPOINT` returned **200
uncompressed** with `guard_egress` having fired twice.

**`OfflineEgressBlocked` now derives from `BaseException`, not
`Exception`.** A convention that has to be re-applied at every future
`except Exception:` in a 100k-line codebase is not a guarantee; the type
hierarchy is. Same reason `KeyboardInterrupt` and `SystemExit` sit
outside `Exception`: a policy decision is control flow, not a runtime
error to degrade around. All four sites propagate with no edit to any of
them.

⚠️ **This is a behaviour change for callers.** `except Exception` no
longer catches it anywhere, including in third-party code. `except
BaseException` still does; those are swept by `TestBroadHandlerSweep`,
which requires each to re-raise or carry a written reason.
`content_router`'s native-detect degrade path now re-raises it
explicitly, alongside the cancellation `BaseException`s it already let
through.

#### Where the refusal deliberately becomes a degradation

Escaping `except Exception` is the point, but it made one class of
caller worse. Fetching **public model weights** is not data leaving the
box, and an air-gapped proxy with a cold HF cache would have started
failing user requests where it previously degraded — punishing the
request for a decision the operator made about the host. So each
optional model loader translates the refusal into its own "model
unavailable" error, **explicitly**, with the switch named in the log and
the refusal chained:

| boundary | translated to | the caller's existing degradation |
| --- | --- | --- |
| `kompress_compressor._hf_artifact` | `KompressModelNotCached` | non-ML
compression path |
| `image/onnx_router._hf_artifact` | `RuntimeError` | `ImageCompressor`
→ `Technique.PRESERVE` |
| `memory/adapters/embedders` ONNX load | `RuntimeError` | local
embedder unavailable |
| the two Kompress download daemon threads | logged, `False`/return |
keeps whatever is cached |

Behaviour for a cold air-gapped cache is exactly what it was. What
changed is that the log now names the switch, and the translation is a
reviewed decision at a named boundary rather than something inherited
from whichever `except Exception` happened to be in the stack. The
daemon threads matter separately: an unhandled `BaseException` there
reaches `threading.excepthook` as a bare traceback on every air-gapped
boot.

Paths that **send data off the box** — remote Kompress, OTLP, Langfuse,
the beacon — are untouched and still refuse loudly.

### Startup refuses a contradictory configuration

`configure_otel_metrics` was called **outside** `lifespan`'s `try`,
above the line that sets `app.state.startup_error`, so an
`OfflineEgressBlocked` escaped uvicorn as an unhandled error — killing a
proxy that had been serving traffic, with a traceback instead of an
explanation.

It now runs through `_configure_observability_or_refuse()`, which turns
a refusal into the shape `_check_rust_core` uses ten lines above: what
is wrong, how to fix it, **exit 78** (`EX_CONFIG`). A preflight covers
the remote-Kompress contradiction the same way. So the ordinary case is
settled at startup with an actionable message, and `BaseException` is
the backstop rather than the user experience.

```
FATAL: OTLP metric export (HEADROOM_OTEL_METRICS_ENABLED) needs outbound network access, but HEADROOM_OFFLINE=1 forbids it.
    refused: HEADROOM_OFFLINE is set: refusing outbound network access for OTLP metric export to …
    fix:     set HEADROOM_OTEL_METRICS_EXPORTER=console or scrape /metrics (both stay on-box), or unset HEADROOM_OTEL_METRICS_ENABLED
```

### Python/Rust normalisation parity

Python trimmed with `str.strip()`, Rust with `str::trim()`. Not the same
set: `str.isspace()` includes U+001C–U+001F, the Unicode `White_Space`
property does not, so `HEADROOM_OFFLINE=$'\x1c1'` read as **offline to
Python and online to Rust** — one variable, one half of the process
air-gapped. Both sides now trim an enumerated `TRIM_CHARS` /
`_TRIM_CHARS`, and the parity test compares the trim sets and a shared
normalisation table, not just the accepted tokens. (Case folding needed
no change: no non-ASCII character lowercases into any character of
`1`/`true`/`yes`/`on`.)

### The meta-test, rebuilt

The first version was neutered by one line:

```python
if "guard_egress" in source:
    continue
```

A whole-file exemption keyed on the string appearing *anywhere*,
including in a docstring — the exact blanket exemption the allowlist's
own comment says must never happen. It made the site-count assertion
unreachable for every guarded file, so "adding a second client trips
this test" only held for files with **no** guard at all. Two bypasses
were demonstrated against it: a second unguarded `httpx.Client` in an
already-guarded file, and a brand-new unguarded module with the word in
its docstring.

It now decides **per egress site**:

* **AST, not text.** Comments and docstrings vouch for nothing.
`self.client.post()` is no longer read as `requests.post`, and the
`httpx.post` example in `compress.py`'s docstring is not a site.
* **Dominance.** A site is guarded only when a `guard_egress` call is in
its block or an enclosing one, earlier in that block, in the same
function. A guard in a sibling `if`/`except` arm, in a nested `def`, or
after the site does not count. A guard in a helper the function calls
does not count either — conservative on purpose.
* **`TestSiteScannerRules`** tests the scanner itself, pinning both
demonstrated bypasses plus the one-line `def f(): return
httpx.Client().post(...)` that the old `startswith("def ")` skip made
invisible. A meta-test nobody tests is a comment.
* **Wider callee set:** module-level `httpx` verbs
(`headroom/cli/mcp.py` was a live unmatched site), `hf_hub_download`
(skipping `local_files_only=True`, which opens no socket), fastembed's
`TextEmbedding`, the OTLP exporters, and the OpenAI/Anthropic SDK
clients.
* **`crates/` is swept too** (`TestRustEgressChokepointCoverage`). It
was outside the scan entirely, which is how two Rust downloads stayed
open. Text-level rather than AST — no Rust parser here — so "guarded"
means a `guard_egress` earlier in the same `fn` at no deeper indent.
Weaker, and labelled as such; the runtime assertions live in each
crate's `#[test]`s.
* **Callees are resolved through the module's import table before
matching.** The scanner compared the text `ast.unparse` produced, which
is whatever local name the module bound — so `import httpx as h` hid a
client from a test whose entire job is to notice clients. That was not
hypothetical: `copilot_auth.py` does `from urllib import request as
urllib_request`, so its GitHub calls rendered as
`urllib_request.urlopen` and **five call sites, including the shared
`_urlopen` helper every Copilot request goes through, were invisible**.
`_canonical_callee` rewrites the leading name first; the alias defence
cuts both ways, so `guard_egress as ge` still counts as a guard. Imports
are collected from the whole tree, not just module level, because half
the egress here is imported inside the function that uses it.
* Allowlist counts now mean **unguarded** sites only, so a file can
legitimately guard some of its egress and still carry an entry for the
rest.
* **There is no longer a category for "known violation".**
`test_no_category_permits_a_known_violation` pins the permitted set to
`loopback` / `gated` / `user-traffic` / `operator-endpoint` /
`cache-only` and rejects "out of scope" as a reason.

### The offline policy, in plain words

`HEADROOM_OFFLINE=1` **refuses every connection Headroom itself decides
to make to a destination Headroom itself chose.** That is the rule, not
a sample list: telemetry, update check, license/usage reporter, OTLP and
Langfuse export, HuggingFace / Kompress / fastembed model and tokenizer
downloads (Python and Rust), release-binary and codebase-memory-mcp
downloads, eval dataset downloads and the provider SDK clients the eval
harness drives, GitHub Copilot device-flow auth and token exchange, the
Anthropic / Codex / Copilot subscription pollers, both OpenAI embedders,
both Headroom Cloud compression integrations, and the TLS diagnostics
(`headroom doctor --network` endpoint checks and the certificate-chain
re-probe after an upstream TLS failure).

**Four things are deliberately still allowed, and they are the complete
set of exceptions.** Every one of them is either physically incapable of
leaving the box or is not Headroom-initiated traffic:

| category | what it covers | why it survives the switch |
| --- | --- | --- |
| `user-traffic` | `proxy/server.py` (2 sites) | the caller's own
request forwarded to the upstream the operator configured. A proxy that
refuses this is not a proxy; an air-gapped install points it at an
on-prem endpoint. |
| `operator-endpoint` | `memory/adapters/embedders.py` (1 site) | the
Ollama embedder, whose address comes entirely from operator config and
defaults to `http://localhost:11434`. Exactly one thing qualifies today,
and a hard-coded internet host may never hide here. |
| `loopback` | 8 files, 12 sites | hard-coded `127.0.0.1` probes against
the operator's own local proxy (`headroom doctor`, the installers, the
MCP sidecar, the test harness). |
| `gated` | 3 files, 3 sites | the beacon, the update check and the
usage reporter are already unreachable behind an `is_offline()` check,
so no connection is ever built. |

Plus one mechanical exemption, `cache-only` (`relevance/embedding.py`, 1
site): the first of the two fastembed loads runs with `HF_HUB_OFFLINE`
forced to `1`, so `huggingface_hub` resolves from the local cache or
raises **without opening a socket**; the network retry directly below it
calls `guard_egress`. That is what lets a pre-seeded air-gapped host
keep loading the model while a cold one refuses.

#### What changed since the review

The previous revision of this PR recorded **nine egress-capable files**
under an `unguarded` allowlist category meaning "reachable under
`HEADROOM_OFFLINE`, out of scope for A-2": Copilot auth/token exchange,
three subscription pollers, the two binary/CBM downloads, the
eval-dataset fetch, and both Headroom Cloud compression integrations —
plus the eval harness's LLM clients, both OpenAI embedders and the
fastembed weights. @JerrettDavis blocked on exactly that, and he was
right: recording that a switch does not do what it says is not the same
as making it do it.

**All of them are now guarded, and the `unguarded` category is
deleted.** `test_no_category_permits_a_known_violation` pins the five
permitted categories by name and rejects "out of scope" as an allowlist
reason, so re-introducing a bless-it-and-move-on category takes a
deliberate edit with a reviewer looking at it.

#### ⚠️ Behaviour change customers can see

**Outbound calls that previously worked with `HEADROOM_OFFLINE=1` set
now refuse.** That is the point of the PR, but it is a real change for
anyone who set the flag and kept using these features:

* `headroom auth` / `headroom copilot-auth login` cannot complete a
GitHub device flow, and a cached Copilot OAuth token cannot be exchanged
for an API token. A Copilot subscription is unusable on an air-gapped
box regardless, but the failure now arrives at the moment of the
attempt.
* `headroom install` will not download release binaries or
codebase-memory-mcp. Pre-stage them, or install with the flag unset.
* The Anthropic / Codex / Copilot subscription and quota panels stay
empty, with the reason logged and (for Copilot quota) shown in the
panel's own error slot.
* Headroom Cloud compression mode in the ASGI middleware and the LiteLLM
callback refuses instead of falling through to local compression —
loudly, because silently compressing locally would hide that the
deployment is no longer doing what it was configured to do.
* The eval harness refuses to build provider clients or download BFCL.
* The OpenAI embedders (memory adapter and direct mem0 backend) refuse.
Use the ONNX or Ollama embedder on an air-gapped box.
* `EmbeddingScorer` raises "model unavailable" on a **cold** fastembed
cache. A warm one is unaffected.

#### Refusals reach a human, never a traceback

Following the pattern the earlier commits set for the Kompress download
threads, the model layer and the startup banner:

* **CLI:** `OfflineAwareGroup` (`headroom/cli/main.py`) is the one place
that translates the refusal into a Click `Error:` line, for every
subcommand at once. `OfflineEgressBlocked` stays a `BaseException`
everywhere inside, so nothing else can downgrade it; only the last frame
before the terminal is allowed to make it readable.
* **Background pollers:** each catches `OfflineEgressBlocked`
specifically, calls `offline.note_refusal` (WARNING once per purpose,
DEBUG after, so a 30-second poller does not flood the log), and returns
its ordinary "no data" value. An un-caught `BaseException` out of a bare
`create_task` surfaces as "Task exception was never retrieved" whenever
the GC gets to it — a traceback with no explanation in it.
* **Model loaders:** `EmbeddingScorer._get_model` translates into a
named "model unavailable" `RuntimeError`, the way
`kompress_compressor._hf_artifact` and `image/onnx_router._hf_artifact`
already do.

### 🚩 Two decisions that need a founder call

Both were implemented the **safe** way — refuse, loudly and legibly —
because that is the defensible default for a switch an operator sets to
mean "nothing leaves this box". Neither is obviously right, and a
permissive default should not be chosen quietly:

1. **Should OTLP / telemetry export be permitted under
`HEADROOM_OFFLINE`?** Today it refuses, and a contradictory config
(`HEADROOM_OTEL_METRICS_ENABLED=1` + offline) exits 78 at startup with
an explanation. The argument for permitting it is that a regulated
customer's OTLP collector is usually in-cluster, and losing
observability is a real operational cost. The argument against — and the
reason it is refused today — is that the guard **cannot reliably tell an
in-cluster collector from an internet host**: DNS, service meshes and
egress proxies all blur it, so any exemption is really "trust the
endpoint the config names". A customer with an in-cluster collector
currently has to scrape `localhost:8787/metrics` or use
`HEADROOM_OTEL_METRICS_EXPORTER=console`. If we want to permit it, the
honest shape is an explicit second flag (e.g.
`HEADROOM_OFFLINE_ALLOW_OTLP=1`) rather than widening the policy.
2. **Should the subscription pollers refuse, or degrade silently?**
Today all three refuse and say so once per purpose at WARNING. The
argument for silence is that these are cosmetic panels and a WARNING on
an air-gapped box is noise about a decision the operator already made.
The argument for the current behaviour is that a usage panel that
silently stops updating is exactly the "why is this empty?" confusion
the switch was supposed to end, and once-per-purpose logging bounds the
noise. Changing this is a one-line change in `offline.note_refusal`.

### Tests

`tests/test_offline_egress_chokepoint.py` — 110 tests. Per-path socket
assertions (a `no_sockets` fixture booby-traps
`socket.create_connection`, `socket.socket.connect` and `connect_ex`, so
asserting "no socket" rather than "raises" catches a guard placed after
the client is built), the scanner-rule tests, the two coverage sweeps,
the broad-handler sweep, the startup-refusal tests, the model-loader
degradation tests, the cross-language parity tests, and the doc-claim
test. Plus 8 Rust tests in `headroom-core`.

### Fail-before / pass-after

Per fix, work committed, then sources reverted while keeping the new
tests, run, restore. No `git stash`.

**Meta-test (1).** Added a probe module whose docstring mentions
`guard_egress` and a second unguarded `httpx.Client` to the
already-guarded `kompress_remote.py`. The old meta-test: `3 passed`. The
new one:

```
  headroom/_a2_probe_tmp.py
        line 7: httpx.Client().get("https://evil.example.com")
        (not guarded, not allowlisted)
  headroom/transforms/kompress_remote.py
        line 154: return httpx.Client(timeout=1.0)
        (not guarded, not allowlisted)
```

**Rust guards (2).** Guard calls commented out, tests kept:

```
panicked at crates/headroom-core/src/transforms/kompress.rs:688:
expected an offline refusal, got a network error: no loadable ONNX artifact in
`headroomlabs-ai/this-model-does-not-exist-a2` (tried [...]): request error: http status: 401
```

The model id cannot exist, so a `Hub` error can only come from a real
round-trip. That **401 is the proof a socket was opened to
huggingface.co with `HEADROOM_OFFLINE=1` set**, and `Offline` can only
come from returning before `Api::new()`.

**Refusal / startup (3, 4).**
`headroom/{offline,proxy/server,transforms/content_router}.py` reverted
to the PR tip, tests kept: `11 failed, 6 passed` — including
`test_thinking_compactor_does_not_swallow_it`,
`test_kompress_model_ready_does_not_report_a_refusal_as_ready`,
`test_no_broad_handler_swallows_the_refusal` and all five
`TestStartupRefusal` tests.

**New Python guards (5).**
`headroom/{onnx_runtime,observability/tracing}.py` reverted to
`origin/main`, tests kept: `4 failed` — the two socket tests plus both
coverage assertions.

**Normalisation (6).** Both `offline` modules reverted to the PR tip,
parity tests kept: `6 failed`, with `HEADROOM_OFFLINE="\x1c1"` reading
as offline in Python and not in Rust.

**Docs (7).** Both doc files reverted, test kept:

```
AssertionError: docs/metrics-technical-guide.md says 'disables all outbound traffic', but
_EGRESS_ALLOWLIST still records these as reachable under HEADROOM_OFFLINE: [...16 files...]
```

#### This revision (the review fixes)

Committed first, then `git checkout origin/main --
headroom/{binaries,cli/main,copilot_auth,graph/installer,integrations/asgi,integrations/litellm_callback,memory/adapters/embedders,memory/backends/direct_mem0,relevance/embedding}.py
headroom/{evals,subscription}/` with the new tests kept:

```
23 failed, 79 passed
```

The 23 are the five Copilot-auth socket tests, the four poller tests,
the two install-download tests, the BFCL test, both Headroom Cloud
tests, the two cloud-embedder tests, the four fastembed tests, both
CLI-translation tests — and
`test_every_egress_site_is_guarded_or_allowlisted`, which lists every
reverted file as "(not guarded, not allowlisted)".

The aliased-import blind spot, isolated (canonicalisation stubbed back
to the identity, i.e. the pre-fix behaviour, against the same inputs):

```
before:  INVISIBLE: import httpx as h
         INVISIBLE: from urllib import request as urllib_request
         INVISIBLE: from openai import AsyncOpenAI as AO
         INVISIBLE: import httpx as h (inside a function)
         aliased guard recognised: [False]
         copilot_auth.py sites seen: 4
after:   DETECTED (all four); aliased guard recognised: [True]
         copilot_auth.py sites seen: 6
```

**Pass-after (this revision):** `102 passed` in
`tests/test_offline_egress_chokepoint.py`; `391 passed, 86 skipped`
across the copilot-auth / copilot-quota / codex-rate-limits / binaries /
integrations / CLI subsets; `544 passed, 15 skipped` across the
relevance / embedding / mem0 / asgi / litellm / subscription / installer
subsets; `160 passed, 6 skipped` across the evals and CLI doctor/learn
subsets; `59 passed` across the proxy-startup subset; `cargo test -p
headroom-core offline` → `8 passed`.

**Pass-after (earlier revisions):** `70 passed` (pytest), `942 passed`
(`cargo test -p headroom-core --lib`), `720 passed, 4 skipped` across
the offline/otel/kompress/telemetry/onnx/tracing/relevance subsets, and
`1868 passed, 30 skipped` across the
kompress/offline/image/embedder/memory subsets after the model-loader
boundary change. The one failure in the full run,
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
reproduces unchanged at the PR tip and is unrelated. Cost/pricing tests
are the known `origin/main` litellm breakage (#3722) and were not run.

### Checks

- `uvx ruff@0.16.7 check .` → All checks passed
- `uvx ruff@0.16.7 format --check .` → 1622 files already formatted
- `cargo fmt --check` → clean; `cargo clippy -p headroom-core
--all-targets` → no warnings
- `CHANGELOG.md` untouched
- CodeQL: the four high alerts were all in
`tests/test_offline_egress_chokepoint.py` and are fixed honestly rather
than suppressed. Three `py/incomplete-url-substring-sanitization` host
assertions (`"host" in url`, which is also true of
`https://host.attacker.test`) now parse with `urllib.parse.urlsplit` and
compare the hostname exactly — which is what the tests meant. The
`py/redos` regex scraping Rust `char` literals had a nested `(...)+`
that can backtrack exponentially; a Rust char literal is exactly one
character, so it is now a single bounded alternation.

### Follow-ups (not in this PR)

1. ~~Guard the 16 `unguarded` files, each with its own user-facing
message.~~ Done in this PR; the `unguarded` category is deleted.
2. Split the three Rust HF fetches cache-first so a pre-seeded
air-gapped cache resolves without egress, the way
`hf_hub_download_local_first` already does on the Python side.
3. The Rust sweep is a text heuristic. An AST-level dominance check (via
`syn`, in a Rust test) would make the two halves symmetric.
4. The control-plane heartbeat lives in a different repository and is
still out of scope.
5. The scanner has no entry for transformers `from_pretrained` /
`SentenceTransformer`. The Kompress tokenizer site Devin found is
guarded now, but `models/ml_models.py` (SigLIP, technique router,
sentence-transformers) and the Kompress PyTorch backend still rely on
`HF_HUB_OFFLINE` alone, which `apply_offline_env` sets with `setdefault`
and only in the proxy process. Each needs the cache-first split the
tokenizer has, plus a scanner entry.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3729"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Testing

- [x] Unit tests pass (`pytest`) — subsets below
- [x] Linting passes (`ruff check .`) — touched files, `ruff@0.16.7`
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# this revision
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest \
    tests/test_offline_egress_chokepoint.py tests/test_cli_doctor.py tests/test_corporate_tls.py -q
245 passed
# after the Devin fixes
$ pytest tests/test_offline_egress_chokepoint.py + kompress/onnx_router/relevance/embedding/evals subsets
361 passed, 6 skipped
# fail-before for the Devin fixes (4 sources reverted, tests kept): 6 failed + the fallback test (fails once its fake
# raises OSError instead of LocalEntryNotFoundError, which conftest converts to a skip)

# fail-before, TLS guard (tls_diagnostics.py + doctor.py reverted, tests kept)
FAILED TestDoctorNetworkOffline::test_endpoint_check_refuses_before_any_connection
FAILED TestDoctorNetworkOffline::test_doctor_reports_the_network_checks_as_skipped
FAILED TestDoctorNetworkOffline::test_chain_probe_opens_no_socket
FAILED TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted
   headroom/proxy/tls_diagnostics.py line 473: with httpx.Client(...)  (not guarded, not allowlisted)
# with only tls_diagnostics.py reverted, the doctor test fails on content, not on a missing helper:
   [('net api.anthropic.com', 'fail'), ('net api.openai.com', 'fail'), ...] != [('network', 'skip')]

# fail-before, socket trap (old fixture body, new tests kept)
FAILED TestSocketTrap::test_event_loop_wakeup_pipe_is_not_egress
   SocketOpened: outbound connection attempted while offline: (<socket ...>, ('127.0.0.1', 50151))
# — the same csock.connect inside socketpair() that Jerrett's Windows trace shows.

$ uvx ruff@0.16.7 check / format --check (touched files)
All checks passed! / 4 files already formatted
```

Earlier-revision test evidence is in *Fail-before / pass-after* under
*Changes Made*.

## Real Behavior Proof

- Environment: macOS, Python 3.14, this branch (merged with `main` at
`befdb52f`), no proxy running.
- Exact command / steps: `HEADROOM_OFFLINE=1 headroom doctor --network
--port 59999`, run for real. A second run went through
`headroom.cli.main` with a `sys.addaudithook` recording every
`socket.getaddrinfo` / `socket.connect`.
- Observed result: the doctor table shows `network · skip · skipped:
HEADROOM_OFFLINE is set, no endpoint was contacted`, with the refusal
naming `https://api.anthropic.com/v1/models` as the hint. The audit hook
recorded only `127.0.0.1` lookups and connects (the doctor's own proxy
probes on :59999 and :8787) and **zero external lookups or connects**.
Exit code 2 comes from the unrelated "proxy not reachable" failures.
- Not tested: Windows itself. The Windows mechanism is reproduced by
forcing `socket._fallback_socketpair`, which is the code Windows runs
for `socket.socketpair()`; the CI Windows job (`test-windows-newline`)
only runs `windows_newline`-marked tests and was not extended here.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; behaviour keyed on the existing
`HEADROOM_OFFLINE` switch.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only when `HEADROOM_OFFLINE` is set
(refusals listed above). Unset, nothing changes.
- Kill switch / disable path: unset `HEADROOM_OFFLINE`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Two founder decisions remain flagged above (OTLP under offline; pollers
loud vs silent). Windows CI coverage for this module is not added: no
unit-test job runs the full module on Windows today, and adding one is a
workflow change better made separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 15:20:39 -05:00
dependabot[bot]andJD Davis fee53b4daa deps: bump webpki-roots from 0.26.11 to 1.0.8 (#3905)
Bumps [webpki-roots](https://github.com/rustls/webpki-roots) from
0.26.11 to 1.0.8.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rustls/webpki-roots/releases">webpki-roots's
releases</a>.</em></p>
<blockquote>
<h2>1.0.8</h2>
<ul>
<li>Remove <code>SecureSign Root CA12</code> root; see <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=2031105">https://bugzilla.mozilla.org/show_bug.cgi?id=2031105</a></li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>Test MSRV with 1.70 by <a
href="https://github.com/ctz"><code>@​ctz</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/122">rustls/webpki-roots#122</a></li>
<li>Take semver-compatible dependency updates by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/123">rustls/webpki-roots#123</a></li>
<li>1.0.8: remove <code>SecureSign Root CA12</code> root by <a
href="https://github.com/ctz"><code>@​ctz</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/124">rustls/webpki-roots#124</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rustls/webpki-roots/compare/v/1.0.7...v/1.0.8">https://github.com/rustls/webpki-roots/compare/v/1.0.7...v/1.0.8</a></p>
<h2>1.0.7</h2>
<p>For their April 2026 root store changes, Mozilla has made more
changes than usual:</p>
<blockquote>
<p>These changes are part of Mozilla’s ongoing root store maintenance
under the Mozilla Root Store Policy (MRSP), including <a
href="https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#74-root-ca-lifecycles">§7.4</a>
(Root CA Lifecycles) and <a
href="https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#753-transition-plan-for-existing-roots">§7.5.3</a>
(Transition Plans). They reflect a combination of lifecycle-based
transitions, CA operator requests, and alignment with intended
certificate usage, including retiring older or less suitable root
certificates, enforcing clear separation of trust purposes (e.g., TLS
vs. S/MIME), and reducing unnecessary trust surface in the Web PKI
ecosystem. Collectively, these actions help to ensure that root
certificates are relied upon only for their intended and actively
maintained use cases, or are retired in accordance with established
distrust timelines.</p>
</blockquote>
<p>This removes:</p>
<ul>
<li>CN=Certigna O=Dhimyotis</li>
<li>CN=COMODO Certification Authority O=COMODO CA Limited</li>
<li>CN=DigiCert Assured ID Root CA O=DigiCert Inc
OU=www.digicert.com</li>
<li>CN=DigiCert Global Root CA O=DigiCert Inc OU=www.digicert.com</li>
<li>CN=DigiCert High Assurance EV Root CA O=DigiCert Inc
OU=www.digicert.com</li>
<li>CN=FIRMAPROFESIONAL CA ROOT-A WEB O=Firmaprofesional SA</li>
<li>CN=GTS Root R2 O=Google Trust Services LLC</li>
<li>CN=QuoVadis Root CA 2 O=QuoVadis Limited</li>
<li>CN=QuoVadis Root CA 3 O=QuoVadis Limited</li>
<li>CN=Secure Global CA O=SecureTrust Corporation</li>
<li>CN=SecureTrust CA O=SecureTrust Corporation</li>
<li>CN=SwissSign Gold CA - G2 O=SwissSign AG</li>
<li>CN=TeliaSonera Root CA v1 O=TeliaSonera</li>
<li>CN=Trustwave Global Certification Authority O=Trustwave Holdings,
Inc.</li>
<li>CN=Trustwave Global ECC P256 Certification Authority O=Trustwave
Holdings, Inc.</li>
<li>CN=Trustwave Global ECC P384 Certification Authority O=Trustwave
Holdings, Inc.</li>
<li>O=certSIGN OU=certSIGN ROOT CA</li>
</ul>
<p>See <a
href="https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/o1VliD70ctg/m/pY0JBzTlAQAJ?pli=1">their
announcement</a> for more details.</p>
<h2>What's Changed</h2>
<ul>
<li>Take semver-compatible dependency updates by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/116">rustls/webpki-roots#116</a></li>
<li>Take semver-compatible dependency updates by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/117">rustls/webpki-roots#117</a></li>
<li>Take semver-compatible updates by <a
href="https://github.com/ctz"><code>@​ctz</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/118">rustls/webpki-roots#118</a></li>
<li>Prepare 1.0.7 by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/120">rustls/webpki-roots#120</a></li>
<li>Update dependencies by <a
href="https://github.com/djc"><code>@​djc</code></a> in <a
href="https://redirect.github.com/rustls/webpki-roots/pull/119">rustls/webpki-roots#119</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/rustls/webpki-roots/compare/v/1.0.6...v/1.0.7">https://github.com/rustls/webpki-roots/compare/v/1.0.6...v/1.0.7</a></p>
<h2>1.0.6</h2>
<p>&quot;e-Szigno TLS Root CA 2023&quot; added, see <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=1873057">https://bugzilla.mozilla.org/show_bug.cgi?id=1873057</a></p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/rustls/webpki-roots/commit/50729cd212ccd332be193856258fa93da2876d6a"><code>50729cd</code></a>
1.0.8: remove <code>SecureSign Root CA12</code> root (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/124">#124</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/c965c541f62aff7436b709f6ec14ab545cb670f0"><code>c965c54</code></a>
Take semver-compatible dependency updates (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/123">#123</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/a83a3c9e689f00c84183ca10d96256a2c0bb3f6d"><code>a83a3c9</code></a>
Move MSRV to 1.70 (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/122">#122</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/be948464fd5907af6227213a066743a161221688"><code>be94846</code></a>
Update dependencies (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/119">#119</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/76476f91c903fa034b51d5eaa22e3f7a40c04176"><code>76476f9</code></a>
Prepare 1.0.7 (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/120">#120</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/ed392f437cc8ffbe286226a6588e80f5f0da2ce6"><code>ed392f4</code></a>
Take semver-compatible updates (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/118">#118</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/ec8b744a63fa01c7d3ea2e87bb694dca330f9770"><code>ec8b744</code></a>
Take semver-compatible dependency updates (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/117">#117</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/84a0c23ef25cfe07b65ac9e31892f001fbdaf0c0"><code>84a0c23</code></a>
Take semver-compatible dependency updates (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/116">#116</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/c97def9fb9fcf4c6f687a6f11c376c021b4811a1"><code>c97def9</code></a>
webpki-roots: 1.0.6 (<a
href="https://redirect.github.com/rustls/webpki-roots/issues/115">#115</a>)</li>
<li><a
href="https://github.com/rustls/webpki-roots/commit/d30d2482e935c2a95d415e9623dfcef22cba0745"><code>d30d248</code></a>
Trigger CI workflow on merge groups</li>
<li>Additional commits viewable in <a
href="https://github.com/rustls/webpki-roots/compare/v/0.26.11...v/1.0.8">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=webpki-roots&package-manager=cargo&previous-version=0.26.11&new-version=1.0.8)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3905"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 09:58:13 -05:00
49f69be283 feat(live-zone): wire the SourceCode and PlainText dispatch arms (#3227)
## Description

`dispatch_compressor` in
`crates/headroom-core/src/transforms/live_zone.rs` has two arms that
return `NoOp` under a `TODO(PR-B4)` marker — `ContentType::SourceCode`
and `ContentType::PlainText` — even though both compressors they need
(`CodeAwareCompressor`, `Kompress`) already ship in the same crate. Any
source code or prose that reaches the live zone is classified, gated,
and then dropped on the floor.

This PR wires those two arms:

- `SourceCode` → `CodeAwareCompressor`. When the compressor returns its
input unchanged (its own declines: min-token floor, unknown language,
parse-error revert, ratio guard) the block falls back to cached
Kompress. That is an **exact-passthrough** fallback only: a rewrite that
changes bytes without shrinking tokens is left to the existing
tokenizer-rejection gate in `compress_one_block`, which forwards the
original. This is a deliberate divergence from the Python router, which
re-runs Kompress on any no-shrink result; the code and the module doc
say so rather than claiming parity.
- `PlainText` → `Kompress::from_cache`. Cache-only by construction: on a
cold HuggingFace cache (or a `--no-default-features` build) the arm is a
deterministic no-op, never a download and never an error on the request
path.
- The two arms' byte thresholds move off their placeholder `512` to the
values `REALIGNMENT/04-phase-B-live-zone.md` specifies: `2048` for
source code, `5120` for plain text. The tokenizer gate and the threshold
plumbing themselves already shipped; this PR only fills the two slots.
- `HEADROOM_LIVE_ZONE_DISABLE_ARMS` — a kill switch so an operator can
force any arm to a no-op without rebuilding. Shipped as the rollback for
the two new arms, but checked once at the top of `dispatch_compressor`,
so it applies to every arm.

**Relationship to #1155.** That PR, open since June and
reviewer-approved, wires the same two slots. I found it only after this
work was complete, and I am filing this as an alternative rather than a
duplicate because the two differ in specific ways, listed under
*Additional Notes*. If the maintainers prefer to land #1155, I am happy
to close this and port whichever of the differences below are wanted
onto it.

Closes #2949.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

Non-breaking: both arms were no-ops, so no request that compressed
before compresses differently now; the thresholds that move were pinned
as unused; the switch defaults to unset; `ContentType` gains `FromStr` /
`ALL` / `natural_name()` additively.

## Changes Made

- `crates/headroom-core/src/transforms/live_zone.rs` — the two arms, the
exact-passthrough fallback, thresholds `2048` / `5120`, the generic kill
switch (`parse_disabled_arms` is a pure `pub fn`; the env read uses
`var_os` and is latched on first dispatch), `threshold_for` made `pub
const fn` so tests read the real constants, and the Kompress pieces
cfg-gated on `ml` instead of lint-suppressed. The Kompress slot is an
atomic `UNINIT → INITIALIZING → READY` state machine (review feedback):
the request path never runs or waits on the ~261 MB session build — it
dispatches `NoOp` until the slot settles — construction runs exactly
once, on a background thread or inline via the blocking
`warm_live_zone_compressors()`, and a cold-cache or failed load settles
as a terminal no-op, logged once as `kompress_init_complete {
kompress_ready }` and never re-paid.
- `crates/headroom-core/src/transforms/content_detector.rs` —
`ContentType::ALL`, `natural_name()`, and `impl FromStr` accepting both
the log-tag spelling (`text`, `search`, …) and the natural spelling
(`plain_text`, `search_results`, …). The switch parses through this;
there is no second name table.
- `crates/headroom-proxy/src/config.rs` — doc comments only: the
`compression-mode` / `auth-mode-policy-enforcement` text still described
the live-zone dispatcher as unimplemented.
- `crates/headroom-proxy/src/main.rs` — startup warmup: with
`--compression` on, a background thread runs
`warm_live_zone_compressors()` so steady-state traffic meets a settled
slot; completion is logged as `live_zone_warmup_complete {
kompress_ready }`. A byte-pipe proxy (compression off) never loads the
model.
- `crates/headroom-core/tests/` — `live_zone_dispatch.rs` flips the B3
"source code is a no-op" pin to the routing assertion and adds the
Kompress routing test; `live_zone_arm_properties.rs` adds no-panic and
determinism properties over generators that actually reach both arms
(the earlier generators reached an arm in ~4.5% of cases and SourceCode
never), byte-fidelity tests for both arms (SHA-256 over the bytes
outside the rewritten block), a reach-fraction instrumentation test, and
`FromStr` round-trips over `ContentType::ALL`;
`live_zone_disable_arms.rs` / `live_zone_kompress_absent.rs` are
single-test binaries for the env-var paths;
`live_zone_disable_arms_parsing.rs` covers the parser with no
environment mutation; `live_zone_kompress_async_init.rs` is a new
single-test binary pinning the non-blocking contract — 8
barrier-synchronized threads against a virgin slot must each return in <
500 ms as `NoOp`, the construction counter must read exactly 1, and a
post-warmup dispatch must compress via `kompress` (it fails against the
previous code at 22.1 s of first-dispatch blocking);
`live_zone_kompress_absent.rs` also pins that a cold-cache settle
reports not-ready and that a post-settle dispatch stays the same
deterministic no-op; `tests/common/mod.rs` holds the shared fixtures,
and its cache probe now runs the production warmup itself rather than
building a throwaway model.
- `crates/headroom-core/benches/live_zone_dispatch.rs` (+ `Cargo.toml`
bench entry) — criterion bench for the four dispatch shapes; not wired
into CI.
- `docs/content/docs/configuration.mdx` — environment-variable row for
`HEADROOM_LIVE_ZONE_DISABLE_ARMS`.

## Testing

- [ ] Unit tests pass (`pytest`) — N/A, Rust-only change; the equivalent
gate is `cargo test --workspace`, below
- [ ] Linting passes (`ruff check .`) — N/A; equivalent is `cargo clippy
--workspace -- -D warnings`, below
- [ ] Type checking passes (`mypy headroom`) — N/A
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Re-run at the tip of this branch after the review-requested non-blocking
init landed (the run recorded here supersedes the pre-review one).

```text
$ cargo fmt --all -- --check
(clean, exit 0)

$ cargo clippy --workspace -- -D warnings
    Finished `dev` profile [unoptimized] target(s)
(exit 0, no warnings)

$ cargo check -p headroom-core --no-default-features
    Finished `dev` profile [unoptimized] target(s)
(exit 0 — the async-init slot compiles under no-ml too: the warmup exists and reports not-ready)

$ cargo test --workspace          # CARGO_PROFILE_TEST_DEBUG=0 to fit the host's disk; affects backtraces only
     Running tests\live_zone_kompress_async_init.rs
test result: ok. 1 passed; 0 failed; 0 ignored
     Running tests\live_zone_arm_properties.rs
test result: ok. 7 passed; 0 failed; 0 ignored
     Running tests\live_zone_disable_arms.rs
test result: ok. 1 passed; 0 failed; 0 ignored
     Running tests\live_zone_disable_arms_parsing.rs
test result: ok. 5 passed; 0 failed; 0 ignored
     Running tests\live_zone_dispatch.rs
test result: ok. 9 passed; 0 failed; 0 ignored
     Running tests\live_zone_kompress_absent.rs
test result: ok. 1 passed; 0 failed; 0 ignored
     Running tests\live_zone_thresholds.rs
test result: ok. 2 passed; 0 failed; 0 ignored
     ... (58 test binaries in total)
workspace totals: 1512 passed; 0 failed; 3 ignored
```

The new `live_zone_kompress_async_init` test was watched fail against
the pre-review code before the fix was written: the first dispatch
blocked 22.1 s on the synchronous session build and returned `Modified`.
It now passes in 3.3 s including the real model build.

Note on `--all-targets`: `cargo clippy --workspace --all-targets -- -D
warnings` fails on two lints in files this PR does not touch
(`code_compressor.rs:1907` `manual_range_contains`,
`log_compressor.rs:1780` `field_reassign_with_default`), both
pre-existing on `main`. The CI gate (`rust.yml`) runs without
`--all-targets` and passes.

## Real Behavior Proof

- Environment: Windows 11 (10.0.26200); rustc 1.95.0 (the pinned
toolchain); `headroom-proxy` built from this branch in the `dev`
profile; warm HuggingFace cache (`chopratejas/kompress-v2-base` +
`answerdotai/ModernBERT-base`); `ORT_DYLIB_PATH` pointed at the
`onnxruntime` 1.28.0 DLL. Upstream set to an unreachable port on purpose
— the dispatcher logs its decision before the upstream call, so no
provider is needed and nothing is billed; every request therefore ends
in a 502 after the proxy has already done its work.
- Exact command / steps: start the proxy three times, each with a
different `HEADROOM_LIVE_ZONE_DISABLE_ARMS`, and POST four `tool_result`
bodies to `/v1/messages` with an `x-api-key` header (classified `Payg`,
the full-compression policy): a 10-function Python module (3,716 B),
repetitive prose (8,090 B), a 200-element JSON array of dicts (11,490 B
— SmartCrusher's shape, a control arm this PR does not touch), and prose
below the 5,120 B threshold (900 B). The proxy's own `anthropic
live-zone dispatch` JSON log lines, trimmed to the relevant fields:

  ```text
$ headroom-proxy --listen 127.0.0.1:8797 --upstream http://127.0.0.1:9
--compression --compression-mode live_zone --log-level info
python_source_10fn 3716 B ->
{"decision":"compressed","live_zone_strategies":["code_compressor"],"live_zone_block_original_tokens":1062,"live_zone_block_compressed_tokens":495,"body_bytes_in":4291,"body_bytes_out":2027}
plain_prose 8090 B ->
{"decision":"compressed","live_zone_strategies":["kompress"],"live_zone_block_original_tokens":2311,"live_zone_block_compressed_tokens":2173,"body_bytes_in":8262,"body_bytes_out":7776}
json_array_200 11490 B ->
{"decision":"compressed","live_zone_strategies":["smart_crusher"],"live_zone_block_original_tokens":3283,"live_zone_block_compressed_tokens":1410}
prose_below_threshold 900 B ->
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1}

$ HEADROOM_LIVE_ZONE_DISABLE_ARMS=source_code,plain_text headroom-proxy
... (same flags, port 8798)
python_source_10fn 3716 B ->
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1}
plain_prose 8090 B ->
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1}
json_array_200 11490 B ->
{"decision":"compressed","live_zone_strategies":["smart_crusher"],"live_zone_block_original_tokens":3283,"live_zone_block_compressed_tokens":1410}
prose_below_threshold 900 B ->
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1}

$ HEADROOM_LIVE_ZONE_DISABLE_ARMS=json_array headroom-proxy ... (same
flags, port 8799)
python_source_10fn 3716 B ->
{"decision":"compressed","live_zone_strategies":["code_compressor"],...,"live_zone_block_compressed_tokens":495}
plain_prose 8090 B ->
{"decision":"compressed","live_zone_strategies":["kompress"],...,"live_zone_block_compressed_tokens":2173}
json_array_200 11490 B ->
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1}
prose_below_threshold 900 B ->
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1}
  ```

- Observed result: with the switch unset, source code routes to
`code_compressor` (1,062 → 495 tokens) and prose to `kompress` (2,311 →
2,173 tokens) while the pre-existing SmartCrusher arm and the
byte-threshold gate behave exactly as before. With
`source_code,plain_text` disabled, both new arms are silent and
SmartCrusher is unaffected. With `json_array` disabled, SmartCrusher is
silent and the new arms still fire — the switch is generic, not
special-cased to the two arms it was written for.
**Non-blocking init (added for the review).** Same environment and
unreachable upstream. Start the proxy, POST the same > 5,120 B prose
`tool_result` twice: once ~0.3 s after startup, while the model warmup
is still running, and once after the warmup log line. The proxy's own
log, in order, trimmed:

  ```text
  07:23:16.966  "headroom-proxy starting"
07:23:17.254 anthropic live-zone dispatch
{"decision":"no_change","reason":"no_block_compressed","live_zone_blocks":1,...}
└─ request 1, during warmup: the block is live and above threshold, the
arm answers NoOp, and the request returns ~0.29 s after startup —
nothing waited on the build
  07:23:18.729  {"event":"kompress_init_complete","kompress_ready":true}
07:23:18.729 {"event":"live_zone_warmup_complete","kompress_ready":true}
07:23:22.910 anthropic live-zone dispatch
{"decision":"compressed","live_zone_strategies":["kompress"],"live_zone_block_original_tokens":2774,"live_zone_block_compressed_tokens":2613,...}
└─ request 2, after the slot settled: the same body compresses
  ```

- Not tested: a round trip to a live provider with a compressed body (no
provider was involved); Linux/macOS hosts; the cold-cache fall-open path
against the running proxy (it is covered by
`live_zone_kompress_absent.rs`, which forces an empty cache, but only
in-process); a `--no-default-features` *test* run (that configuration
was `cargo check`ed only); the lazy request-path init against the
running proxy — the startup warmup wins that race by design, and the
lazy path is pinned in-process by `live_zone_kompress_async_init.rs`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this PR registers nothing in
`headroom-core/src/rollout.rs` (`Feature::{NativeBedrock,
OpenAiResponsesStreaming, CanaryProbe}` are untouched). The two arms sit
behind the existing operator switch `--compression --compression-mode
live_zone` (default `off`), which is exactly how the four arms already
shipped are gated. Happy to add a registry entry if the maintainers want
the live-zone arms channel-managed.
- Minimum rollout channel: n/a — no registry entry; reachable on every
channel, and only when the operator has turned live-zone compression on.
- Stable/default behavior changed: no. Defaults stay `--compression` off
and `--compression-mode off`, under which the body is a byte-faithful
passthrough. With live-zone compression on, two arms that returned
`NoOp` now compress; the thresholds that move (`512` → `2048` / `5120`)
were pinned as unused. One deliberate timing change under compression-on
(review feedback): the Kompress session loads once at startup in the
background instead of synchronously on the first qualifying request, and
requests that arrive before it settles dispatch as `NoOp` — no request
ever waits on the build.
- Kill switch / disable path:
`HEADROOM_LIVE_ZONE_DISABLE_ARMS=source_code,plain_text` forces the two
arms to a no-op without a rebuild (per arm, latched per process,
documented in `configuration.mdx`); `--compression-mode off` disables
the whole dispatcher. The Kompress arm is additionally inert on any host
without the model in its HuggingFace cache.
- Unsafe override required: no.
- Qualification impact: none — `qualification_eligible()` reads the
rollout channel and feature flags, neither of which this PR touches.
- Rollback path: set the kill switch and restart, or revert the PR;
there is no persisted state, no data migration, and no wire-format
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — proxy log output is in the proof above.

## Additional Notes

**How this differs from #1155** (from reading its diff, not its
description):

| | #1155 | this PR |
|---|---|---|
| SourceCode compressor declines (returns input) | `NoOp` | falls back
to cached Kompress |
| Byte thresholds | unchanged at `512` | `2048` / `5120` per
`REALIGNMENT/04` |
| Kompress enablement | opt-in flag `--enable-kompress` /
`HEADROOM_PROXY_ENABLE_KOMPRESS`, default off | cache-only: fires iff
the model is already cache-resident; no flag |
| Operator off-switch | process-wide Kompress gate |
`HEADROOM_LIVE_ZONE_DISABLE_ARMS`, per content type, every arm |
| Non-blocking model load | yes (added during its review) | yes — atomic
`UNINIT → INITIALIZING → READY` slot plus a startup warmup; requests
during initialization dispatch as `NoOp` (added during review here as
well) |
| Tests | routing tests | routing + property (no-panic, determinism) +
byte-fidelity for both arms + reach instrumentation + parser tests |

The default-off Kompress gate in #1155 exists because the model is ~261
MB; this PR takes the other position — the proxy already refuses to
download on the request path, so "present in the cache" is the opt-in —
but that is a product decision for the maintainers. The non-blocking
load was a genuine improvement this PR did not have; review here
requested the same property and it is now implemented (atomic init slot,
startup warmup, `NoOp` while initialization is in flight). Either PR
could still absorb the other's remaining pieces.

**Two remaining `std::env::set_var` calls** live in
`live_zone_disable_arms.rs` and `live_zone_kompress_absent.rs`, each a
single-test binary, because the input under test is the environment
itself. The parsing contract is covered without any environment mutation
(`live_zone_disable_arms_parsing.rs`). Plumbing the switch through
`config.rs` as a clap `env =` argument and handing it to the dispatcher
as a parameter would remove both; I left that out to keep
`compress_anthropic_live_zone`'s signature unchanged in this PR and
would do it as a follow-up if wanted.

**Strategy tags stay `&'static str`** (`"code_compressor"`,
`"kompress"`) to match the existing `STRATEGY_*` constants. A `Strategy`
enum carried in `BlockAction::Compressed` would be the idiomatic form
but changes a public type used across the proxy; out of scope here,
happy to follow up.

**`proptest` case counts** in `live_zone_arm_properties.rs` are 40 / 20
rather than the ~1000 the parser fuzz tests use: the generators now
actually reach the compressors (a tree-sitter parse or an ONNX inference
per case), and 64/32 measured ~60 s on the test host. The reach-fraction
test is what guards against the generators regressing back to sizes that
never clear a threshold.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 12:26:24 -05:00
ANIRUDDHA ADAKandJD Davis f3f2e0077a fix(search): stop a context line's body from becoming its line marker (#3788)
## Description

Closes #3545

I could not reproduce the exact `Line 476: … 497: 535: 556: …` output in
the
report, and I think the reason is worth stating up front, because it
changes what
this PR does.

That shape cannot come out of the search compressor. `parse_match_line`
splits
**one input line at a time**, and `format_output` emits **one match per
line**, so
no code path can move line 497's number onto line 476's body. Feeding
the
shipped build a body that genuinely contains those numbers reproduces
the
reported string exactly:

```
input   app/settings.py:476:settings: settings({contact_precedence: 497: 535: 556:
        app/settings.py:497:...
        app/settings.py:535:...
        app/settings.py:556:...
output  app/settings.py:476:settings: settings({contact_precedence: 497: 535: 556:
        [... and 3 more matches in app/settings.py]
```

— i.e. the numbers were part of line 476's own content all along, and
497/535/556
were dropped as ordinary matches. `Line N:` rows are not parsed as
matches at all
and pass through untouched, which matches @TigerkidYang's finding on
`main`.

**But the class of bug the issue is about is real, and it is in the
parser.** Here
is the same failure with different numbers, on a shape that grep emits
routinely —
a `-`-separated *context* line (`path-line-content`) whose body carries
a
whitespace-free `name:N:` reference:

```
input   app/settings.py-476-a:7:b:8:c
parsed  ("app/settings.py-476-a", 7, "b:8:c")
correct ("app/settings.py",      476, "a:7:b:8:c")
```

Three separate falsehoods in one row:

1. the path swallows the real `-476-` marker and invents
`app/settings.py-476-a`,
   a file that does not exist;
2. **the reported line number is the body's `7`, not the row's `476`** —
this is
   the manufactured line↔content association the issue is about;
3. the body is truncated to `b:8:c`.

Worse, the bogus path becomes the grouping key in
`parse_search_results`, so
every match in that file is filed under a nonexistent path.

This is not speculative. On the shipped `headroom-ai==0.38.0` build,
over a
generated corpus of tricky paths/line numbers/bodies:

```
$ uv run --with headroom-ai==0.38.0 python probe_parse.py
checked 2394 colon-form rows; 0 mismatched
dash-form mismatches: 279
```

`path:line:content` (match lines — the dominant shape) is clean. Every
one of the
279 failures is a `path-line-content` context line with a
whitespace-free
`name:N:` in its body.

### Root cause

`parse_match_line` tries the **colon tier first** and it commits
unconditionally
on the first `:\d+:` whose prefix has no whitespace. Its own doc comment
(search_compressor.rs:646-651) claims:

> The no-whitespace guard keeps a `foo.rs:12:` reference *inside the
body* of a
> `-`-separated context line from hijacking the parse.

The guard only holds when the body happens to contain whitespace ahead
of the
reference. `a:7:b` has none, so the colon tier takes it, and the dash
tier —
which reads the row correctly — is never consulted.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/src/transforms/search_compressor.rs`
- `parse_match_line` now hands priority back to the dash tier for
exactly one
shape: the dash tier *positively confirmed* a boundary that sits before
the
colon marker, **and** nothing path-like (separator or extension dot)
lies
    between the two. The colon triplet is then body text, not a marker.
- The colon tier keeps priority everywhere else, so `path:line:content`
rows
    and rows whose **path** holds the dash triplet are untouched —
    `logs/2026-05-03/app.log:12:ERROR` and
`migrations/20240101-002-add_users.sql:12:SELECT` still resolve
correctly,
    because the dash tier never confirmed a boundary there.
- `scan_match_line` becomes `scan_marker`, returning the marker it
settled on
**plus a flag** for whether that marker was a confirmed boundary
(`chosen`)
    or merely the leftmost fallback (`first`). The caller needs that
    proven-vs-guessed distinction; it is the whole safety condition.
- Doc comment updated — it previously asserted a guarantee the code did
not
    provide.
- New unit test
`dash_context_line_wins_over_a_colon_marker_in_its_body`.
- `tests/test_search_compressor.py`: new `TestContextLineBodyReference`
class
  (4 tests) covering the repro, the extension-less path variant, the
must-not-regress colon row with a dated path, and a 12-row sweep
asserting
  every surviving coordinate and body.

**Deliberately not changed:** the `Dockerfile-3-1-` shape is genuinely
ambiguous
in text (`Dockerfile-3` line 1, or `Dockerfile` line 3) and keeps its
existing
reading. The dash tier's comments already document that trade-off, and
this PR
does not widen or narrow it.

## Testing

- [x] New tests added for new functionality
- [ ] Unit tests pass (`pytest`) — see the note below
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] Manual testing performed — against the shipped wheel and a
compiled harness

**Why `pytest` is unticked:** the Python tests need the compiled
`headroom._core`
extension, which requires a checkout of the repo. I worked remote-only,
so I
could not run them. I validated every assertion they make by running the
identical expectations as a Rust test against the patched parser (below,
case
`python_test_expectations`) — but that is not the same as having run
pytest, and
I would rather say so than imply otherwise. The Rust unit test in the
crate is
exercised by the repo's `cargo` CI.

I also could not run the 8 crate-level tests that need the compressor
types
(`select_respects_per_file_cap_and_global_cap`, the four `ccr_*` tests,
etc.).
They are untouched by this diff and CI will run them.

### Rust: the parser compiled and run for real

The parser is self-contained, so I extracted the functions I touched
plus all 20
parser-only tests **verbatim** from the patched file and compiled them
with the
real toolchain (rustc 1.93.1, `--target x86_64-pc-windows-gnu`; the repo
pins
1.95.0 in `rust-toolchain.toml`):

```
$ rustc --test --edition 2021 --target x86_64-pc-windows-gnu -o parser_harness.exe parser_harness.rs
$ ./parser_harness.exe
running 20 tests
test tests::dash_context_line_wins_over_a_colon_marker_in_its_body ... ok
test tests::body_line_reference_does_not_hijack_a_context_line ... ok
test tests::dash_tier_does_not_run_past_the_path_into_the_body ... ok
test tests::date_stamped_path_is_not_misread_as_line_number_marker ... ok
test tests::dated_path_inside_a_context_body_is_not_hijacked ... ok
test tests::parses_context_line_for_extensionless_path ... ok
test tests::fixed_in_3e2_handles_windows_path_with_backslash ... ok
test tests::ambiguous_dash_line_falls_back_to_the_leftmost_marker ... ok
...
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
```

With **only** `parse_match_line` reverted to the pre-fix three-tier
chain and
nothing else changed, the new test fails exactly as the bug predicts:

```
thread 'tests::dash_context_line_wins_over_a_colon_marker_in_its_body' panicked:
  left: Some(("app/settings.py-476-a", 7, "b:8:c"))
 right: Some(("app/settings.py", 476, "a:7:b:8:c"))
test result: FAILED. 19 passed; 1 failed; 0 ignored
```

All 19 pre-existing parser tests still pass in both runs — dated paths,
CVE
paths, Windows drive letters, dashes in filenames, the ambiguous
fallback, the
`foo.rs:12:`-with-whitespace case, and CJK.

### Rust: blast radius over a generated corpus

I also transcribed the tier logic and diffed current-order vs
patched-order
across `19 paths × 9 line numbers × 17 bodies` in both separators:

```
colon-form (2907 rows):  current-order wrong=0     patched wrong=0
dash-form  (2907 rows):  current-order wrong=612   patched wrong=153
```

All 153 remaining are the single `Dockerfile-3` path, and they are
**identical
before and after** — the documented ambiguity, unchanged. Every
unambiguous
mis-parse is fixed and no colon-form row changes.

### Lint / format

```
$ uv run --with "ruff==0.16.8" ruff check --line-length 100 --target-version py310 \
    --select E,W,F,I,B,C4,UP --ignore E501,B008,B905 tests/test_search_compressor.py
All checks passed!
$ uv run --with "ruff==0.16.8" ruff format --check --line-length 100 \
    --target-version py310 tests/test_search_compressor.py
1 file already formatted
$ rustfmt --edition 2021 --check crates/headroom-core/src/transforms/search_compressor.rs
(no diff; exit 0)
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.15, rustc 1.93.1. No repo checkout
for this
branch (remote-only contribution), so no end-to-end proxy run and no
`cargo test`
  on the full crate.
- Exact command / steps: `uv run --with headroom-ai==0.38.0 python
probe_parse.py`
against the shipped compiled parser, then `rustc --test --edition 2021
--target
x86_64-pc-windows-gnu -o parser_harness.exe parser_harness.rs` on the
patched
parser extracted verbatim, then `./parser_harness.exe`. Both command
sets are
  quoted in full under *Testing*.
- Observed result: the defect reproduces in the **shipped** build —
  `parse_search_lines("app/settings.py-476-a:7:b:8:c")` returns
`("app/settings.py-476-a", 7, "b:8:c")`, i.e. a wrong line number (7 for
476), a
path that does not exist, and a truncated body. Over a generated corpus
it reports
`checked 2394 colon-form rows; 0 mismatched` / `dash-form mismatches:
279`. After
the change the same input returns `("app/settings.py", 476,
"a:7:b:8:c")` and the
harness reports `20 passed; 0 failed`, with the 19 pre-existing parser
tests still
  green.
- Not tested: the full `cargo test` suite for the crate; the 8
crate-level
compressor tests that need the compressor types; `pytest` (needs the
built Rust
extension); a live proxy request carrying a context line; ripgrep `-C`
output
  end-to-end through the proxy.

If you want the last row closed before merge, the check is a `grep -rn
-C2`
whose context lines contain a `name:N:` reference, run through
`headroom compress` — the parsed coordinates should be the row's own.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: the change is in the Rust core, so it ships
with the
  next wheel.
- Stable/default behavior changed: yes, for one input shape only. Rows
that
previously mis-parsed now parse correctly; every other row is
byte-identical
(0 changes across 2907 colon-form rows, and the only changed dash-form
rows
  are the 459 that were wrong).
- Kill switch / disable path: revert this commit.
- Unsafe override required: none.
- Qualification impact: none. No config, flag, or metric changes.
- Rollback path: revert; the mis-parse returns.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — see
*Testing*
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- **On the report itself:** if you can still get the original failure,
the raw
tool result as your harness delivered it (plus `headroom inspect --last
1`) is
what would settle it. My working theory is the `grep_numbers_in_content`
case
above — the numbers were in line 476's own content and 497/535/556 were
dropped as ordinary matches, which reads exactly like fusion. If the raw
  result shows a genuinely different transform (e.g. `smart_crusher` /
`text_block`), this fix will not address it and the issue should stay
open on
  that thread.
- The parser is a deliberate three-tier design with carefully documented
trade-offs, so I kept the change to a single override condition rather
than
reordering the tiers. Reordering outright is the obvious alternative and
it
  **regresses** real rows — `logs/2026-05-03/app.log:12:ERROR` and
`migrations/20240101-002-add_users.sql:12:SELECT` both break — which is
why the
  override is gated on a *confirmed* dash boundary.
- The `chosen.is_some()` flag is the safety property: the permissive
tier's
leftmost fallback never counts as a confirmed boundary, so a line the
parser
  merely guessed at keeps today's behaviour.
- The sibling defect raised in this thread (Kompress dropping ` or `
from code)
  was already fixed by #3687, so it is out of scope here.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-30 11:14:05 -05:00