Commit Graph
21 Commits
Author SHA1 Message Date
Jay HemnaniandJD Davis 9355ec7019 fix(diff): renumber hunk headers after trimming context (#4319)
## Description

`DiffCompressor` trims each hunk to `max_context_lines` (2) around its
changes but keeps the hunk's original `@@ -a,b +c,d @@` header. The
header no longer describes the lines under it:

- Leading context is dropped first, so every line number read off a hunk
is off by the lines cut (one with git's default `-U3`).
- Context dropped between two changes more than 4 lines apart leaves
them next to each other. Nothing marks the gap, and every line after it
is numbered wrong.

A 400-line file with lines 10 and 17 changed, `git diff` (73 lines)
through `DiffCompressor().compress()` on 0.41.0:

```text
@@ -7,14 +7,14 @@ line 6
 line 8
 line 9
-line 10
+line 10 changed
 line 11
 line 12
 line 15
 line 16
-line 17
+line 17 changed
```

By the header, `line 8` is line 7 and `-line 17` is line 14. Anything
that maps these hunks to file lines (a model writing an edit, `git
apply`) gets the wrong place.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `reduce_context` splits a trimmed hunk at each gap and recomputes
every header from the original line numbers, one range per parent for
combined diffs, formatted the way git writes them. The section heading
is carried over unchanged.
- A hunk that loses no lines keeps its header byte-for-byte.
- Lines after the extent a header declares (the empty string from a
trailing newline, a format-patch `-- ` signature) are not numbered and
no longer count as changes.
- A format-patch signature and what follows it stay whatever the context
limit.
- A hunk with edits past the extent its header declares is malformed
input and passes through as it came.
- `hunks_kept` / `hunks_removed` still count the original hunks, so a
split can't underflow `hunks_removed`.
- Five `diff_compressor` parity fixtures recorded the old headers. In
their outputs 33 hunks had header counts that disagree with their own
lines; with this change there are none. Only `output.compressed` changes
in those files.

## Testing

- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib
test result: ok. 979 passed; 0 failed; 1 ignored

# new tests against the previous commit:
  left: ["@@ -7,14 +7,14 @@ l6", "@@ -23,6 +23,7 @@ l22"]
 right: ["@@ -8,5 +8,5 @@ l6", "@@ -15,5 +15,5 @@ l6", "@@ -24,4 +24,5 @@ l22"]
test result: FAILED. 0 passed; 2 failed

$ cargo run -p headroom-parity -- run --fixtures tests/parity/fixtures
[diff_compressor ] total=27 matched=27 skipped=0 diffed=0

$ pytest <the 12 test files that contain hunk headers> tests/test_compress*.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/parity
754 passed, 3 skipped
$ cargo clippy -p headroom-core --lib --tests -- -D warnings   # clean
```

## Real Behavior Proof

- Environment: Ubuntu, Python 3.12, headroom-ai 0.41.0 from PyPI vs this
branch built with `pip install -e .`
- Exact command / steps: for each commit, `git diff c~1 c` through
`DiffCompressor(DiffCompressorConfig(max_hunks_per_file=10**6,
max_files=10**6)).compress(...)`. Then compare every hunk (header ranges
and lines) with `git diff -U2 c~1 c`. Last 300 non-merge commits of this
repo, plus 200 of another Python repo. Merge commits the same way with
`git show -c` / `-c -U2`.
- Observed result:
  ```text
  0.41.0, this repo:    compressed=276 match_git_U2=3
  branch, this repo:    compressed=276 match_git_U2=276
  branch, second repo:  compressed=163 match_git_U2=163
  branch, git show -c:  compressed=36  match=36
  ```
  Same example after the change:
  ```text
  @@ -8,5 +8,5 @@ line 6
   line 8
   line 9
  -line 10
  +line 10 changed
   line 11
   line 12
  @@ -15,5 +15,5 @@ line 6
   line 15
  ```
- Not fully matched: `git show --cc`. 8 of 11 merges match `--cc -U2`.
In the other 3, git hides more single-parent hunks when the context
shrinks, which a context trim doesn't try to copy. The compressor keeps
every change line of the `--cc` input there.

## Runtime Rollout Safety

N/A: no rollout-managed feature, no config change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

The section heading after the second `@@` is kept from the original
hunk. git recomputes it for each hunk, but that depends on the file's
language driver, and the original heading still names the enclosing
context.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4319"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:51:06 -06:00
eef44f60cc fix(kompress): read the pinned snapshot from Python's one HF cache in the native proxy (#4438)
## Description

**Stacked on #4190.** This branch contains #4190's commits (it reuses
#4190's Rust HF-cache resolver). Please review only the five commits
listed under Changes Made. Once #4190 merges, I'll rebase onto main and
the diff will shrink to those commits.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now
linear, with no merge commits:

- #4190's first three commits: `e173e100`, `344cf365`, `41918cf0`.
- This PR's five commits (SHAs updated under Changes Made).
- `d9f16d39` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does. This is #4190's last commit, applied to
`crate::hf_cache` since this PR moves the resolver there. It replaces
the earlier merge `a2e89118`.

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`.
- The main-sync merges (`7e17a2d3`, `6904fd19`, `8be25204`, `79e3a5cb`,
`19b1adc5`, and #4190's merges) are linearized away.
- The Rust embedder conflicts with main's batch-size cap (#4507) were
re-resolved to the same content as before: every file this PR touches is
byte-identical between the previous head `19b1adc5` and the new head
`d9f16d39` (`git diff 19b1adc5 d9f16d39 -- <files changed vs main>` is
empty). Only the history and SHAs changed.

The native proxy's live-zone Kompress is cache-only
(`Kompress::from_cache`): it reads the files Python downloaded. Its
cache lookup (`hf_hub_roots` + `hf_cache_file` on main) disagrees with
`huggingface_hub` in several ways. When it misses, `kompress_ready`
stays false and live-zone passes text through uncompressed. When it
loads, it can load the wrong files. This is the follow-up planned in
https://github.com/headroomlabs-ai/headroom/pull/4190#issuecomment-6209746394
(Devin's note on `hf_hub_roots`). The Python proxy resolves Kompress
through `huggingface_hub` and is unaffected.

| setup | Python | native Kompress on main |
|---|---|---|
| only `HUGGINGFACE_HUB_CACHE` | found | miss |
| only `XDG_CACHE_HOME` | found | miss |
| `HF_HUB_CACHE=${ROOT}/hub` | `<ROOT>/hub` | miss |
| `HF_HUB_CACHE` set + a copy under `$HOME` | looks only in
`HF_HUB_CACHE` | loads the `$HOME` copy |
| pinned snapshot plus older/newer ones | the `_PINNED_REVISIONS` commit
| first snapshot in `read_dir` order; tokenizer and ONNX looked up
independently |

The last row matters most: every Kompress pin bump leaves the old
snapshot in the cache.

Closes # (no issue; follow-up to #4190)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

Five commits on top of #4190:

- `af191f19` (was `ddec6b88`) fix(kompress): read the pinned snapshot
from Python's one HF cache in the native proxy.
- #4190's resolver moves to a shared `crate::hf_cache` module:
`hub_cache_dir`, the `expandvars` port, `pinned_or_main`, and a
pin-aware `snapshot_path`.
- Kompress reads one snapshot of each repo, at `DEFAULT_MODEL_REVISION`
/ `DEFAULT_TOKENIZER_REVISION`. A new `tests/test_kompress_rust_pins.py`
keeps these equal to `_PINNED_REVISIONS`.
- `HEADROOM_HF_PIN=off` and unpinned repos follow `refs/main`, as in
Python.
  - `kompress_cache_miss` logs the one cache and revision it looked in.
- New `kompress::cached_artifacts` lets the parity harnesses load the
files production would load.
- `7cdc3d94` (was `5e71eeea`) fix(tokenizer):
`HfTokenizer::from_pretrained` downloads into Python's hub cache with
Python's `HF_TOKEN` / `HF_ENDPOINT`. `hf-hub`'s defaults read neither,
and with `Api::new` not even `HF_HOME`.
- `c43c5eb8` (was `5fd3a87d`) fix(hf-cache): changes to the shared Hub
client:
- The tokenizer and embedder share one Hub client, `hf_cache::hub_api`.
- Trailing slashes are stripped from `HF_ENDPOINT`, as Python's
`constants.ENDPOINT` does.
- The token is withheld from a plain-`http://` non-loopback endpoint,
with a warning.
- `91128294` (was `70e61b02`) fix(hf-cache): `hub_api(cache, purpose,
repo)` runs `guard_egress` itself, before it builds the client. A new
caller can't skip the air-gap check, and the offline-egress sweep in
`tests/test_offline_egress_chokepoint.py` sees the site as guarded. The
tokenizer still returns `HfTokenizerError::Offline`, and the embedder
keeps its messages.
- `bbee29cd` (was `14dbac29`) review follow-ups:
  - `HEADROOM_OFFLINE` wins over a missing cache directory.
  - The `hub_api` docs are scoped to the tokenizer and embedder.
  - The order test checks inside `hub_api`'s body.
  - Stale comments are fixed.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `d9f16d39` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 1002 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py \
    tests/test_hf_revision_pinning.py tests/test_kompress_rust_pins.py
247 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

Before the rebase:

```text
$ cargo test -p headroom-core -- --skip kompress_matches_python_fixtures_byte_for_byte --skip short_input_passes_through
  (HF_HUB_CACHE=$(mktemp -d), cold like CI; own CARGO_TARGET_DIR)
20 test binaries, all "test result: ok", 0 failed
$ cargo clippy --workspace --all-targets -- -D warnings      clean
$ cargo clippy -p headroom-core --no-default-features --lib -- -D warnings   clean
$ cargo fmt --all -- --check                                  clean
$ pytest tests/test_offline_egress_chokepoint.py tests/test_kompress_rust_pins.py tests/test_fastembed_pinned_snapshot.py
195 passed, 4 skipped, 2 failed
  the 2 failures are TestHeadroomCloudCompressionOffline (litellm fetching its
  pricing table from raw.githubusercontent.com); they fail identically on
  origin/main e995eb9f with the locked LiteLLM 1.101.0, and #4242 fixes them
$ ruff check / ruff format --check (0.16.8)                   clean
```

mypy is not checked: this PR changes no `headroom/` Python.

## Real Behavior Proof

- Environment: macOS 27 arm64, debug build of `headroom-core` (default
`ml` feature), `ORT_DYLIB_PATH` = onnxruntime 1.31.0, the real cached
`chopratejas/kompress-v2-base` and `answerdotai/ModernBERT-base`
snapshots at their pins, symlinked into throwaway hub caches;
huggingface_hub 1.16.1 for the Python column.
- Exact command / steps: a throwaway example (not committed) calls
`Kompress::from_cache(KompressConfig::default())` and prints the
outcome. It runs once per cache layout under `env -i` with an empty
`HOME`, built from `origin/main` (e995eb9f) and from this branch. The
Python column prints `huggingface_hub.constants.HF_HUB_CACHE` under the
same environment. In the pin-bump layout, the older (`0000…`) and newer
(`ffff…`, `refs/main`) snapshots hold garbage `tokenizer.json` / ONNX
files.
- Observed result: the branch now agrees with Python in every layout. On
main, four layouts miss where Python finds the cache, one loads a copy
Python would never read, and the pin-bump layout loads a non-pinned
snapshot. Each side was built in its own target dir.

```text
== origin/main (e995eb9f)
HF_HUB_CACHE=<T>/hub (control)               loaded              | Python HF_HUB_CACHE=<T>/hub
only HUGGINGFACE_HUB_CACHE                   cache miss (None)   | Python HF_HUB_CACHE=<T>/hub
only XDG_CACHE_HOME                          cache miss (None)   | Python HF_HUB_CACHE=<T>/xdg/huggingface/hub
HF_HUB_CACHE=${PROBE_ROOT}/hub               cache miss (None)   | Python HF_HUB_CACHE=<T>/root/hub
HF_HUB_CACHE=<empty>, copy under $HOME       loaded              | Python HF_HUB_CACHE=<T>/empty
pin + older/newer snapshots, main=newer      error: failed to load tokenizer for `answerdotai/ModernBERT-base`: expected value at line 1 column 1 | Python HF_HUB_CACHE=<T>/pinbump
== this branch (14dbac29)
HF_HUB_CACHE=<T>/hub (control)               loaded              | Python HF_HUB_CACHE=<T>/hub
only HUGGINGFACE_HUB_CACHE                   loaded              | Python HF_HUB_CACHE=<T>/hub
only XDG_CACHE_HOME                          loaded              | Python HF_HUB_CACHE=<T>/xdg/huggingface/hub
HF_HUB_CACHE=${PROBE_ROOT}/hub               loaded              | Python HF_HUB_CACHE=<T>/root/hub
HF_HUB_CACHE=<empty>, copy under $HOME       cache miss (None)   | Python HF_HUB_CACHE=<T>/empty
pin + older/newer snapshots, main=newer      loaded              | Python HF_HUB_CACHE=<T>/pinbump
```

On main, the pin-bump row read a garbage non-pinned snapshot (here it
errors; with two real model versions it would silently load the wrong
one). The branch reads the pinned snapshot.
- Not tested: a Windows build (the `ntpath` expandvars rules are covered
by the shared fixture on macOS/Linux); a live Hub download through
`hub_api` with `HF_ENDPOINT` pointing at a real mirror; the native proxy
end to end (the probe calls the same `Kompress::from_cache` the proxy's
live-zone warm-up calls).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this changes where the native
proxy's cache-only Kompress and the Rust tokenizer/embedder look for Hub
files.
- Minimum rollout channel: stable (no rollout gate).
- Stable/default behavior changed: yes. The native proxy now loads
Kompress from the cache Python uses, at the pinned snapshot. It no
longer loads from other roots or from whichever snapshot `read_dir`
returns first. Rust tokenizer downloads land in Python's hub cache.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats the pinned
repos on `refs/main`, as in Python; Kompress itself is disabled as
before.
- Unsafe override required: no.
- Qualification impact: native live-zone Kompress becomes ready on hosts
whose Python cache is configured via `HUGGINGFACE_HUB_CACHE`,
`XDG_CACHE_HOME` or `$VAR` paths; parity harnesses read the pinned
artifacts.
- Rollback path: revert this PR; no persisted state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no user-facing docs describe the native Kompress cache
lookup.
- Follow-up, not in this PR: `Kompress::from_pretrained` still builds
its own `hf_hub::api::sync::Api::new()` client behind its own guard, so
it ignores `HF_ENDPOINT` / `HF_TOKEN` and downloads into `hf-hub`'s
default cache. It has no production caller (the proxy uses
`from_cache`); moving it onto `hf_cache::hub_api` is a separate change.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4438"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 12:29:32 -06:00
Mansurshakh JaparovandJerrettDavis 93279f3208 fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description

Closes #3697.

An unchanged SmartCrusher result now keeps the original bytes and
reports no structural compression. Formatting-only savings come from the
existing Rust JsonMinifier, not from pretending the JSON value changed.

The minifier preserves duplicate keys, order, escapes and number
literals. A mixed result cannot use one section's CCR marker to hide
unmarked loss in another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace
savings through JsonMinifier in Python and the eligible native live
zone.
- Keep all records and JSON literals; do not store originals or inject
CCR markers for lossless minification.
- Preserve mixed routing and reject uncovered lossy sections in both
tool-result formats. Explicit no-CCR behavior is unchanged.
- Replace incidental forwarding tests with native minification and
complete-record regressions; document the distinction.

## Testing

- [ ] Entire Python suite — not claimed.
- [x] Four consumer/native modules: `python -m pytest -q
tests/test_recursive_json.py tests/test_transforms_content_router.py
tests/test_transforms/test_smart_crusher_rust_parity.py
tests/test_transforms/test_smart_crusher_lossless_default.py`
- [x] Changed-file Ruff checks; normal commit hooks, including mypy and
commitlint.
- [x] Manual public-API runs, with and without the HTML extra.

### Test Output

Consumer/native modules:

```text
collected 136 items
============================= 136 passed in 36.43s =============================
```

Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native
build, Python checks, commitlint):

```text
======================== 180 passed, 1 warning in 8.35s ========================
✅ ci-precheck PASSED — safe to push.
To https://github.com/oxura/headroom.git
   64688217..0002af3e  HEAD -> fix/3697-unchanged-json-passthrough
```

The first push stopped on an inherited commit footer because
`origin/main` was stale. Fetching the actual upstream ref fixed the lint
range; the normal push then passed. No source merge, lint-rule change or
hook bypass.

Hosted checks on `0002af3e` now pass: all four Python shards, Windows
permissions, smoke tests, native wrappers, parity and Docker e2e.
[Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks).

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual
extension built from this checkout. No provider billing calls.
- Exact command / steps: the earlier localhost `/v1/compress` and
TransformPipeline runs exercised the original fix. For this follow-up,
call `ContentRouter.compress` on a data-only HTML page containing 60
complete records, with `smart_crusher_lossless_only=True`,
`smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run
it once without trafilatura, then with the locked 2.0.0 extra. The new
public-API test contains the full input/config. Separately call
`route_embedded_json` on a compact array followed by a padded array,
with Unicode and CRLF surroundings.
- Observed result: both HTML modes keep all 60 records and the
surrounding markup, with no unrecoverable section. The unchanged first
JSON span does not stop the second span's compaction; surrounding text
stays exact. Final Codecov is 96%; the two remaining lines are the
deferred HTML fallback return. The real no-HTML-extra API run executed
both, and local coverage also exercises the unchanged-span branch.
- Not tested: live provider acceptance, invoices or production cache hit
rates. No native eligibility or admission gates changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing JSON routing.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: eligible formatting savings return as
json_minifier; standalone SmartCrusher remains byte-exact on no-ops.
- Kill switch / disable path: existing routing configuration and
HEADROOM_LIVE_ZONE_DISABLE_ARMS.
- Unsafe override required: no.
- Qualification impact: no-op, formatting, structural and CCR paths
exercised separately.
- Rollback path: revert the minifier integration; no persistence
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

Addresses the [minifier
review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285).
The maintainer’s main merge is retained. The latest follow-up only
changes consumer tests.

AI-assisted; no subagents.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 21:32:42 -06:00
9bb4b130ff fix(relevance): load the Rust default embedder from Python's pinned snapshot (#4190)
## Description

Parent status: #4152 has merged. The pinned-loader and batch-cap merge
resolution was reviewed at e3a495cbc5. The
original split/stack history is retained below.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now four
linear commits on top of main, with no merge commits:

- `bfae4ade` fix(relevance): load the Rust default embedder from
Python's pinned snapshot
- `6fbae380` fix(relevance): look up and download the Rust embedder in
one HF cache dir
- `70afece8` fix(relevance): expand ~ and $VAR in the Rust HF cache path
like huggingface_hub
- `62b792f0` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`. Its files matched main; the only difference was
`uv.lock`, where main's later `filelock>=4.2.1` entry was kept.
- The six main-sync merges (`185dc9b7`, `052977b9`, `029c72ac`,
`4405b12c`, `8b0a10e8`, and the reviewed `e3a495cb`) are linearized
away, so those SHAs are no longer on the branch.
- `embedding.rs` conflicted again with main's batch-size cap (#4507). I
resolved it to the same content as the reviewed merge: every file this
PR touches is byte-identical between `e3a495cb` and the new head
`62b792f0` (`git diff e3a495cb 62b792f0 -- <files changed vs main>` is
empty). The review and qualification below still apply to the content;
only the history and SHAs changed.

Rust `EmbeddingScorer::try_new` used `EmbeddingModel::BGESmallENV15`.
fastembed-rs maps that to a different repo and file from what Python
runs: `Xenova/bge-small-en-v1.5` `onnx/model.onnx` (fp32), always on
`main`. So the two languages never ran the same ONNX file, and the
"byte-equal across languages" claim in both modules was wrong.

This PR makes `try_new` load the same pinned
`Qdrant/bge-small-en-v1.5-onnx-Q` snapshot that #4152 makes Python load.
It adds a cross-language parity fixture checked from both sides, and
replaces the parity claim with measured numbers.

Nothing in production calls `EmbeddingScorer::try_new` today:
`create_scorer` and `HybridScorer` use `default()`, which never loads a
model. This keeps the Rust scorer correct and pinned for when it is
wired in.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

(This PR's own commits only.)

- `crates/headroom-core/src/relevance/embedding.rs`:
- `try_new` loads the snapshot through
`TextEmbedding::try_new_from_user_defined`, with CLS pooling and static
quantization, which is what fastembed-rs uses for `BGESmallENV15Q`.
- It uses exactly one hub cache directory, resolved like Python's
`huggingface_hub.constants.HF_HUB_CACHE`: `HF_HUB_CACHE` →
`HUGGINGFACE_HUB_CACHE` → `HF_HOME/hub` →
`XDG_CACHE_HOME/huggingface/hub` → `~/.cache/huggingface/hub`, with a
leading `~` expanded. It looks up and downloads only there, so a stale
snapshot in a lower-priority cache is never used. The home directory
comes from `std::env::home_dir`, which falls back to the user database
when `HOME` is unset, as Python's `expanduser` does.
- It finds the snapshot as `snapshots/<sha>` in that directory.
`hf_hub::Cache` resolves through `refs/<rev>`, which Python's
`huggingface_hub` doesn't write for a commit SHA.
- On a miss it downloads with `Repo::with_revision`. `guard_egress` sits
in front of the Hub client and runs only on a cache miss, so a
pre-seeded air-gapped host now loads; previously Rust refused even with
a warm cache.
- `HEADROOM_HF_PIN=off` floats it on `main`, with the same values as
Python's `_resolve_revision`.
  - `try_new_with_model` is unchanged.
- Parity fixture: `tests/parity/record_embedding.py` and
`tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json` (6 texts
× 384 dims, recorded on ORT 1.31.0). The fixture is written one
embedding row per line, so it's 22 lines. Python checks it in CI
(`test-extras` sets `RUN_FASTEMBED_TESTS=1`); Rust checks it under
`RUN_FASTEMBED_TESTS=1`.
- `tests/test_fastembed_pinned_snapshot.py`: the Rust constants name the
same repo, SHA and files as Python; the fixture was recorded at the pin;
plus the gated Python parity check.
- 10 new Rust unit tests in `embedding.rs`: the lookup ignores other
commits and needs every file; the download URL requests the pinned
revision; the offline guard applies on a miss only;
`HEADROOM_HF_PIN=off` floats; the cache directory follows
`huggingface_hub`'s order variable by variable (including an empty
variable and `~`); with no `HOME` it still resolves through the user
database; and the review case: an empty explicit `HF_HUB_CACHE` next to
a complete pinned snapshot and an older `refs/main` under `HOME` finds
neither, for the pinned revision or `main`, while both are found once
`HF_HUB_CACHE` is unset.
- `tests/test_offline_egress_chokepoint.py`: the Rust sweep's regex now
also matches `ApiBuilder::from_cache` / `from_env`. It missed the new
download client, which is guarded; it flags nothing else in `crates/`.
The user-defined model load, which only reads bytes already on disk, is
allowlisted as `cache-only`.
- Docs: the "byte-equal" wording in both embedder modules,
`pyproject.toml` and `onnx_runtime.py` is replaced with measured numbers
and the cross-language keep-equal notes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `62b792f0` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 991 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_hf_revision_pinning.py
235 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

The 7 offline-egress failures noted below no longer reproduce here.

Before the rebase:

```text
$ cargo test -p headroom-core --lib
test result: ok. 985 passed; 0 failed; 1 ignored      # see the note on #4151 below

$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean

$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.31.0 dylib> cargo test -p headroom-core --lib relevance::embedding
test relevance::embedding::tests::fastembed_matches_parity_fixture ... ok
test result: ok. 24 passed; 0 failed

$ RUN_FASTEMBED_TESTS=1 pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_relevance.py \
    tests/test_stateless_writers.py tests/test_hf_revision_pinning.py
71 passed

$ pytest -q tests/test_offline_egress_chokepoint.py
7 failed, 132 passed, 3 skipped
# The 7 are the OTel / Langfuse / Headroom Cloud offline tests, whose extras aren't in this venv;
# the same 7 fail on origin/main here. The Rust sweep tests pass.
```

**Note on #4151:** with `ORT_DYLIB_PATH` set on macOS, a full `cargo
test -p headroom-core --lib` fails
`tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error`
most of the time (5/5 on `afdb60d3`, before the cache change too).
`a_cached_snapshot_is_not_refused_offline` holds `HEADROOM_OFFLINE=1`
under the env lock while ONNX Runtime loads, which is slow there, and
the tokenizer test reads the variable without the lock. That's the race
#4151 fixes; with its commit applied, 6/6 full runs pass. Linux CI
passes either way.

The cache-precedence test was checked against the bug: with the lookup
changed back to also scanning `~/.cache`, it fails on the first case
(pinned revision, empty `HF_HUB_CACHE`).

## Real Behavior Proof
- Exact command / steps: additional maintainer verification on the
main-synced head: cargo test -p headroom-core --locked hf_hub_cache_dir;
cargo test -p headroom-core --locked
expand_vars_matches_python_posixpath_and_ntpath; python -m pytest
tests/test_fastembed_pinned_snapshot.py -q; repeat its fixture cases
with huggingface-hub==2.2.0.
- Observed result: three Rust tests passed across both POSIX and Windows
expansion fixtures and cache-directory selection; Python passed 46 tests
with one gated skip using Hugging Face Hub 1.16.1, and all 36 shared
fixture cases passed separately with locked Hub 2.2.0.

- Environment: macOS arm64; Rust fastembed 5.17.3 with hf-hub 0.5.0;
repo `.venv` (fastembed 0.8.0, onnxruntime 1.23.2) and a second
interpreter on onnxruntime 1.31.0.
- A cold-cache Rust load with `HF_HUB_CACHE=<empty dir>` wrote
`snapshots/52398278…` plus `refs/52398278…` and passed the parity check.
Per-component max |Δ| against the fixture:
  - Rust vs Python on the same ORT (1.31.0): ≤ 1e-6.
  - Python on ORT 1.23.2 vs 1.31.0: ≤ 4.1e-4.
  - The fixture's tolerance is 1e-3.
- For reference, Xenova's fp32 export (what Rust used to load) is ≤
4.7e-4 from the fixture. The old cross-language gap was real but
numerically small, and this fixture bounds drift; it cannot tell the two
exports apart. Model identity is enforced by the pinned SHA and the unit
tests.
- The fixture's new one-row-per-line layout was re-emitted from the
recorded data, not re-recorded: the parsed JSON is identical.
- Not tested: Windows, and Linux outside CI. Rust's gated tests in CI
(they need the model and an ORT dylib, so they stay manual behind
`RUN_FASTEMBED_TESTS=1`). Rust `HybridScorer` thresholds against the
Qdrant export.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no production path.
`EmbeddingScorer::try_new` loads the Qdrant export instead of Xenova's,
and now loads offline from a warm cache instead of refusing, but nothing
in production calls it today.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches
are still unpinned (there's no Rust `HEADROOM_HF_PIN`), and Rust's gated
embedding tests could get a CI job once one ships an ORT dylib.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4190"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer qualification

The environment-reference mismatch from the previous review is
addressed. The shared corpus covers the reported explicit-cache case,
platform-specific variable syntax, and the two-stage home/hub expansion
ordering. The Rust test runs both POSIX and Windows expansion rules;
Python validates the same corpus against posixpath, ntpath, and fresh
huggingface_hub.constants imports.

A clean current-main sync was published as
052977b9e88ae3bbda96797615b64ee350a2f695; it does not change the
contributor's resolver or fixture files. Qualification used Windows,
Hugging Face Hub 1.16.1 and an isolated locked 2.2.0 target. No model
download or numerical embedding-parity result is claimed by these path
tests; the pre-existing gated numerical-parity test remained skipped.

Hosted checks on the pushed head must finish before approval. No
contributor CI action requested or PR merged.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 21:31:59 -06:00
dependabot[bot]andJerrettDavis b9049d703a deps: Bump the cargo-minor-patch group across 1 directory with 12 updates (#4553)
## Purpose

Retain the complete selected 12-version Cargo group and qualify its
compatibility with Headroom. The AWS release moves its config,
credential, Smithy runtime/schema and SDK dependencies together to a new
coordinated compatibility line; upstream explicitly warns against
upgrading only part of that line. Existing proxy credential resolution
and SigV4 signing must continue to work with that coherent set.

This is a compatibility-qualified grouped update. Companion updates are
not individually claimed to fix a Headroom bug where upstream only added
unused APIs.

## Selected versions and project relevance

| Package | Selected update | Relevance and qualification |
| --- | --- | --- |
| tokio | 1.53.3 → 1.54.1 | Proxy HTTP/SSE/WebSocket execution and
timers; includes upstream timer-handle regression revert. Local native
proxy/SSE/WSS tests exercise runtime. |
| aws-config | 1.12.0 → 1.13.0 | Required companion for new AWS
schema/runtime compatibility line; production environment credential
chain exercised with explicit fake credentials. |
| aws-credential-types | 1.3.0 → 1.3.1 | Same coordinated AWS line,
production credential conversion and SigV4 signing exercised. |
| aws-smithy-runtime-api | 1.19.0 → 1.19.2 | Same coordinated AWS line;
timeout finalizer behavior changed upstream. Local signing/wire tests do
not establish SSO/STS timeout behavior. |
| sha2 | 0.11.0 → 0.11.1 | Existing cache/SigV4-related payload hashing;
upstream unsafe-op lint fix. Local signed payload SHA256 verified; no
claimed cryptographic vulnerability fixed. |
| redis | 1.7.1 → 1.8.0 | Optional multiworker CCR backend; upstream TLS
error/secret redaction and subscription correctness fixes. Production
backend roundtrip exercised against local RESP fixture; cluster/pubsub
are not Headroom backend usage. |
| tree-sitter | 0.27.0 → 0.27.1 | Code compressor parser/error recovery
fixes. Reconstructed 9 samples × 8 pinned languages compare old/new node
kinds, complete row/column spans, ERROR/MISSING: 72/72 identical. |
| uuid | 1.28.0 → 1.29.0 | Full-group companion target; new NonNilUuid
constructors are unused. Existing request-ID generation exercised
through real proxy calls and both response x-request-id values parsed as
UUIDv4; no invented bug/security rationale. |
| crc32fast | 1.5.2 → 1.5.3 | Full-group companion target; upstream
total-length API is unused. Existing Bedrock EventStream
checksum/framing path exercised by streaming integration suite. |
| rustls-native-certs | 0.8.4 → 0.8.5 | Native trust-store handling;
avoids duplicate Unix certificate-file reads. Windows build and WSS
fail-closed behavior exercised; Unix store traversal not locally
exercised. |
| webpki-roots | 1.0.10 → 1.0.11 | Updated bundled CA trust set (CCADB
CA removals); WSS verification tests exercise existing explicit TLS
configuration. No public CA endpoint contacted. |
| cc | 1.8.0 → 1.10.0 | Native C/C++ dependency builds; upstream fixes
MSVC archive object paths and cl.exe wrapper handling. Actual clean
Windows MSVC native dependency compilation performed. |

## Maintenance and install surface

The packages remain maintained by their existing upstream projects:
Tokio, AWS/Smithy, RustCrypto, redis-rs, tree-sitter, uuid-rs,
srijs/rust-crc32fast, rustls and rust-lang/cc-rs. Release/compare links
in the original Dependabot body identify the reviewed releases; AWS
authoritative release notes:
https://github.com/awslabs/aws-sdk-rust/blob/main/CHANGELOG.md (October
20–30, 2026). UUID comparison confirms its change only adds NonNilUuid
constructors; it is not presented as a security fix.

No new package source/registry is introduced. Cargo resolves registry
downloads at build time; cc executes the native compiler for existing
C/C++ dependencies (tree-sitter grammars and other existing native
crates). The lock update changes the transitive AWS SDK/schema/runtime
family together. Existing ONNX runtime loading/model behavior is not
altered by this PR. Redis is feature gated and adds runtime network only
when explicitly configured. AWS credentials can normally resolve through
environment, profiles, SSO/STS or metadata; qualification uses only
explicit fake environment credentials, NUL config files and disabled
IMDS. Normal configured proxy upstream traffic and native trust-store
reads remain existing runtime surfaces.

Pinned qualification compiler: Rust 1.95.0 on Windows/MSVC. Resolved
proxy AWS floor is 1.94.1; core/Python/parity graph includes
tree-sitter-language floor 1.90. Both already existed in the base lock,
so no global rust-version change is made; inherited workspace
rust-version=1.80 is not a valid full-workspace source-build promise.
Fresh base/head Windows-filtered metadata comparison confirms unchanged
floors: core/Python/parity1.90,proxy1.94.1,simulators1.86(ICU). These
are dependency-declared effective floors, not measured lower-toolchain
builds. No lower-toolchain build was attempted.

## Real behavior proof

Windows, Rust1.95.0 x86_64-pc-windows-msvc, isolated worktree, dedicated
CARGO_TARGET_DIR, CARGO_BUILD_JOBS=2. No actual AWS/provider
credentials, profiles, cloud inference or model downloads used.

- `cargo test --locked -p headroom-core -p headroom-proxy` with explicit
existing x64 ONNX Runtime DLL (`ORT_DYLIB_PATH`) and
`HEADROOM_OFFLINE=1`, `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`:
**1,659 passed, 0 failed, 3 ignored; exit 0**, across 61 test/doc-test
executables. Initial run without the Windows loader path was diagnosed
loader-blocked and terminated; its separate failed log is retained.
- `cargo test --locked -p headroom-proxy --test
integration_bedrock_invoke --test integration_bedrock_streaming --test
integration_ws_tls --test integration_sse`: 23passed,0failed
(8invoke,11streaming,2SSE,2WSS).
- `python G:/git/headroom/.codex/dep4553-runtime.py`: built
headroom-proxy.exe on ephemeral ports with compression disabled;
passthrough and native Bedrock routes each return local200, both bodies
byte equal; AWS environment provider loaded fake credentials and SigV4
Credential prefix/payload SHA256 checked; both generated response
x-request-id headers validated UUIDv4. Synthetic wire fixture, not real
AWS service emulation.
- `python G:/git/headroom/.codex/dep4553-redis-wire.py`: temporary
integration test calls production RedisCcrStore using redis1.8.0; local
RESP fixture validates PING,miss,UTF8 payload write/read,TTL refresh and
replacement; 1passed. Synthetic protocol server; supplemented by `python
G:/git/headroom/.codex/dep4553-redis-real.py`: real Redis7Alpine
ephemeral loopback Docker service with persistence disabled; production
backend2testsPASS,including real TTLrefresh andexpiry. Owned
--rmcontainer stopped automatically. TLS/cluster/failover/pubsub not
tested.
- Standalone canary under `.codex/dep4553-canary`: exact engine
versions0.27.0 and0.27.1, same eight pinned grammars,72/72 AST snapshots
identical. Historical canary source was absent; reconstructed inputs are
recorded, including second distinct valid fixture per language (explicit
Cfunctionpointer sample), CRLF,truncated,error,concatenated andUnicode
cases.
- Isolated Python3.12 env: recorder regression9tests passed; Ruff
checks/format passed. Original official fixture generator reproduced
KeyError(bash). Minimal3line helper fix raises ValueError, matching
existing detector unsupported-language contract; all8pinned grammars
remain covered. Official generator then recorded30fixtures.
- Generator revealed12preexisting Python-reference fixture differences.
Exact main74277d47 source extracted independently, same Python
environment +same helper fix:30/30 regenerated semantic fixture objects
match PR source, excluding recorded_at only. Preserve original repo
fixture bytes; do not mask unrelated Python-reference drift by replacing
goldens. Baseline/regenerated files and comparisons retained as
artifacts.

## Remaining gates / limitations

Initial broad Cargo suite was loader-blocked: source live_zone.rs
documents indefinite Windows initialization without ORT_DYLIB_PATH.
Controlled termination of exact owned testPID717412 produced
authoritative exit1 for original session37711. Distinct corrected
configuration uses existing verified x64ORT1.26.0 DLL(API24), explicit
ORT_DYLIB_PATH and offline guards; corrected full core+proxy
qualification completed exit0,1659passed,0failed,3ignored. This is an
environment diagnosis, not a demonstrated dependency regression. Hosted
checks must be rerun on the exact final remote SHA; prior shard4 red is
not claimed fixed. Canonical fixture rerecord was executed and its
baseline-identical semantic drift documented, not silently normalized
away. No Linux/macOS build, lower MSRV build, real AWS SSO/STS/IMDS,
public CA endpoint, Redis TLS/cluster/failover, production provider
inference or model download was tested. No remote push, PR edit/review
or merge performed by this qualification task.

## Public reproduction

The committed standard-library harness reproduces the full 72-case
parser comparison and launches the actual native proxy against
credentialless loopback fixtures. It stores both complete AST snapshots
and inputs when `--artifacts-dir` is provided. No private `.codex`
artifacts are needed.

Windows, from a clean checkout with Rust 1.95/MSVC and Python 3.11+:

```powershell
cargo build --locked -p headroom-proxy --target-dir target/qualification
python scripts/qualify_cargo_dependencies.py all --binary target/qualification/debug/headroom-proxy.exe --artifacts-dir target/qualification-proof --canary-target-dir target/qualification-canary
```

Linux/macOS use the same commands with the binary path
`target/qualification/debug/headroom-proxy` (omit `.exe`). Those
platforms are reproduction instructions, not platforms locally qualified
here. The canary compiles standalone exact engine versions0.27.0/0.27.1
against all eight production grammar pins, so Cargo registry
downloads/native compilers may be needed. Runtime checks use fake AWS
credentials, platform null config files, disabled IMDS, offline model
guards, compression off, and local HTTP endpoints only.

The source-comment-required Python fixture rerecord is independently
reproducible:

```text
python -m pip install tree-sitter==0.25.2 tree-sitter-python==0.25.0 tree-sitter-javascript==0.25.0 tree-sitter-typescript==0.23.2 tree-sitter-go==0.25.0 tree-sitter-rust==0.24.2 tree-sitter-java==0.23.5 tree-sitter-c==0.24.2 tree-sitter-cpp==0.23.4
python scripts/record_code_compressor_fixtures.py
```

Use an isolated development Python environment with the project
development dependencies installed. The recorder helper fix in this PR
preserves its existing eight-grammar scope; unsupported detector
candidates are skipped through the existing ValueError contract.
Regeneration currently exposes baseline Python-reference drift as
documented above; inspect semantic differences rather than replacing
goldens blindly.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 08:43:53 -06:00
Mansurshakh JaparovandJerrettDavis 023dd1bc54 fix(logs): count omitted levels from dropped lines (#4141)
## Description

Closes #4070.

The omission marker previously counted severities in the whole input,
including retained errors. It now counts ERROR/FAIL/WARN/INFO only on
dropped line identities. Selection and whole-input result.stats are
unchanged. If no dropped line has a recognized severity, the marker is
simply `[N lines omitted]`, not a severity breakdown. CCR retains the
exact original.

## Type of Change

- [x] Bug fix (non-breaking)
- [x] Documentation update

## Changes Made

- Keep the existing retained-line identity set and a single-pass fixed
stack histogram; no omitted-line collection or repeated full scans.
- Restore complete structured-marker assertions in Rust and Python,
including bounded failure names, repeated ids, whole-input totals, and
non-severity omissions.
- Restore parity fixture `3bc015edc0a36387.json` using its original
input, explicit configuration and input digest. Output and compression
ratio were re-recorded through the freshly built native extension rather
than manually copied. New runtime-frame defaults remain implicit in the
legacy configuration.
- Remove the constructor-only result-copy test; retain consumer behavior
regressions.
- Document omitted-versus-input counts and the exact bare-marker form in
the existing text/log guide. No manual changelog change.

## Testing

- [x] Scoped Rust and Python tests passed
- [x] Normal touched-file Ruff and repository mypy hooks passed
- [x] Actual native recorder and parity CLI exercised

Head `6c472f6b` includes upstream main `f1bc2609`.

- Focused Rust log-compressor tests: **41 passed**.
- Python public-transform tests: **11 passed**, using the freshly built
native extension.
- Actual parity CLI: **20 matched, 0 skipped, 0 differences**, including
the restored original-input fixture:

```text
cargo run --locked -p headroom-parity --bin parity-run -- run --fixtures tests/parity/fixtures --only log_compressor
[log_compressor  ] total=20 matched=20 skipped=0 diffed=0
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.13.14, native extension freshly
built from this worktree.
- Exact command / steps: Re-record the original log fixture through
LogCompressor.compress with its original explicit configuration, then
run the actual parity CLI shown above.
- Observed result: The real native recorder produced `[297 lines
omitted: 297 INFO]` while whole-input statistics remained 300 INFO / one
ERROR / 305 total / eight selected. The exact original was unchanged and
the measured compression ratio is `0.024777201057682892`. The original
fixture digest remains
`3bc015edc0a36387194d075139845c120a308afb314bae436ab454906b431d84`.

Normal commit and full pre-push hooks passed without bypass: Rust
formatting, warnings-as-errors Clippy, workspace tests, native editable
rebuild/import verification, **181 critical Python tests**, and
branch-range commitlint. The critical Python run reported five
dependency/provider warnings, no failures. These local results do not
claim that hosted checks have finished.

- Not tested: New Windows/macOS runs, live-provider sessions and the
full Python suite.

Earlier before/after actual-native CCR qualification showed the retained
ERROR was not counted as omitted, whole-input totals stayed unchanged,
and retrieval recovered the exact original. That historical
implementation evidence is in the review discussion; the explicit
results above qualify this follow-up head. No new Windows/macOS or
live-provider qualification is claimed.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing LogCompressor and CCR; no new
feature or gate.
- Minimum rollout channel: Unchanged (stable); no channel promotion.
- Stable/default behavior changed: Yes, omitted-severity accounting is
corrected; selection and stored originals remain unchanged.
- Kill switch / disable path:
ContentRouterConfig(enable_log_compressor=False) disables the log route;
existing CCR settings disable retrieval separately.
- Unsafe override required: No.
- Qualification impact: The scoped native/Python/parity proof and
required normal push gate above cover this marker path. No new
platform/live-provider qualification is claimed.
- Rollback path: Revert the fix; no schema or stored-original migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
- [x] Existing style and documentation conventions followed
- [x] Consumer-visible regressions and actual native/CLI paths exercised
- [x] Normal hooks passed; no bypass
- [x] No manual CHANGELOG edit

AI-assisted: omp / GPT-6.1 Sol. This review follow-up was completed
without subagents. No manual human-validation or full Python-suite
claim; scoped tests and the required push gate are reported separately.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:45:24 -07:00
Raphael Malikian c1fc84ade6 fix(smart-crusher): disclose dropped scalar-array items with a retrievable ccr sentinel (#3692)
## Description

On default config, `SmartCrusher` silently truncated long
string/number/mixed arrays down to `max_items_after_crush` and emitted
no disclosure at all — valid JSON with no marker, no sentinel, and
untouched sibling count fields. This contradicts the repo's own "Safety
first: never drop user/assistant content" principle.

The dict-array path already disclosed drops via a `{"_ccr_dropped":
...}` sentinel; scalar arrays had no equivalent. This PR gives them one:
when the `StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, it appends `"… N more items <<ccr:HASH
N_items_offloaded>>"` and stores the full original array in the CCR
store (same SHA256-12 hashing the Python bridge and retrieval use), so
the marker round-trips through `headroom_retrieve`. No-op when nothing
is dropped or the marker gate is off.

Closes #3650

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

| File | What changed |
|---|---|
| `crates/headroom-core/src/transforms/smart_crusher/crusher.rs` | New
`append_scalar_drop_sentinel()`: when
`StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, append the sentinel and store the full
original array in the CCR store. No-op when nothing is dropped or the
marker gate is off. |
| `headroom/transforms/smart_crusher.py` | `is_ccr_sentinel()` now
recognizes the string sentinel shape (previously only the dict shape),
so downstream code can filter it like the dict sentinel; docstring
updates. |
| `tests/test_transforms/test_smart_crusher_issue_3650.py` | New
regression tests: sentinel is emitted with the right drop count, carries
a retrievable CCR marker whose hash resolves to the full original array,
and is suppressed when the marker gate is off. |
| `crates/.../smart_crusher/crusher.rs` (tests module) | Rust regression
tests mirroring the above (sentinel emission, marker retrieval of the
full 120-item array, gate-off suppression). |
| `tests/parity/fixtures/smart_crusher/*.json` (3 files) | Recorded
outputs updated — strategy strings now reflect the trailing sentinel. |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
cargo test --lib transforms::smart_crusher  → 337 passed, 0 failed
  (includes the new #3650 Rust regression tests)

pytest tests/test_transforms/test_smart_crusher_issue_3650.py \
       tests/test_transforms/test_smart_crusher_bugs.py \
       tests/test_transforms/test_smart_crusher_ccr_roundtrip.py \
       tests/test_transforms/test_smart_crusher_rust_parity.py \
       tests/test_transforms/test_content_router.py  → 106 passed
  (includes 18 Rust-parity fixture tests)

ruff check and ruff format --check on the touched Python files → clean
```

## Real Behavior Proof

- Environment: Linux container, Python 3.12, editable install from
`main` with the maturin-built native `_core` module (no ONNX model
involved — this path is pure Rust crusher).
- Exact command / steps: the repro snippet from #3650, verbatim
(`ContentRouter(ContentRouterConfig())`, 120 slugs + `"total": 120`).
- Observed result: **before** → 16 of 120 kept, `"&lt;&lt;ccr:" in out`
False; **after** → 16 kept + `'… 104 more items &lt;&lt;ccr:4f03caae30bb
104_items_offloaded&gt;&gt;'`, marker present, and the marker's hash
retrieves all 120 originals from the CCR store. The `total` sibling
field is left untouched — the truncation is now disclosed in-band next
to it rather than silently contradicting it.
- Not tested: the ONNX/Kompress model path from #3673 (a separate issue;
this fix is model-independent), Windows/macOS install flows, and the
`lossless_only=True` path (unchanged: nothing dropped there, so no
sentinel is emitted).

## Runtime Rollout Safety

- Rollout-managed feature(s): CCR marker gate (`enable_ccr_marker`)
- Minimum rollout channel: n/a (open-source library; applies to default
config where the marker gate is on)
- Stable/default behavior changed: Yes — truncated scalar arrays now
carry a trailing sentinel string instead of being silently shortened;
sibling count fields untouched
- Kill switch / disable path: set `enable_ccr_marker=False` — no
sentinel emitted, gate-off path unchanged
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — library behavior; observed output captured under Real Behavior
Proof.

## Additional Notes

Raphael Malikian — rtmalikian@gmail.com

If you'd like similar help with your own project or codebase, I'm open
to collaborate — feel free to reach out.

*This PR was prepared with AI assistance and reviewed by the author
before submission.*

---------

Signed-off-by: Raphael Malikian <rtmalikian@gmail.com>
2026-09-24 11:19:23 -05:00
Tejas ChopraandClaude Opus 5 96dfc7005f chore: finish the org and docs-domain migration (#3389)
Two stale identities left over from moves that were only half applied.

## GitHub org

`install/state.py` already migrated Docker images to
`ghcr.io/headroomlabs-ai/headroom` and carries a
`_DEPRECATED_IMAGE_REPO` constant that rewrites the old path on load —
so this move was decided long ago. Everything else still said
`chopratejas/headroom`.

Rewritten across packaging (`pyproject.toml`, `Cargo.toml`), both plugin
marketplaces, the agent-hooks manifests, install scripts, CI templates,
crate docs and tests.

**Two deliberate exclusions:**
- **`ghcr.io/chopratejas/headroom` is untouched everywhere.** It is the
*input* to `_migrate_deprecated_image()` and to the tests covering it.
Rewriting it would silently disable the migration.
- **`CHANGELOG.md` keeps its 238 historical references.** GitHub's
rename redirect resolves them, and rewriting would bury this diff in
noise.

**HuggingFace model ids are NOT affected and were never in scope.**
`chopratejas/kompress-base`, `kompress-v2-base`, `kompress-finance`,
`technique-router`, `technique-router-onnx` and
`siglip-image-encoder-onnx` are account-scoped identifiers on a
different service. All six were validated against the HF API before this
change:

| Model | HTTP |
|---|---|
| `kompress-base` | 200 |
| `kompress-v2-base` | 200 |
| `technique-router` | 200 |
| `technique-router-onnx` | 200 |
| `siglip-image-encoder-onnx` | 200 |
| `kompress-finance` | **401 (private)** |

The 401 is unrelated to this migration, but worth knowing:
`kompress_compressor.py:1255` references it, so that path is not
reachable by an OSS user without access.

**One functional fix, not cosmetic:** `_marketplace_source()`
(`headroom/cli/init.py`) returned the old org as its fallback, so plugin
installs were pointed at the retired repo.

## Docs domain

The site moved to `docs.headroomlabs.ai`, but eleven references still
named the vercel.app host — including **`llms.txt` (27 links)**, which
is served at `docs.headroomlabs.ai/llms.txt` and read by LLM crawlers,
the **dashboard's own Documentation link**, and a `headroom wrap` help
message. The docs app's canonical-URL fallbacks were fixed separately in
#3388.

## Also folded in

`RUST_DEV.md` corrections found while verifying the above:
- `make build-wheel` points at `Cargo.toml`, not `pyproject.toml`
- the Rust toolchain is pinned, not tracking `stable`
- `headroom-simulators` was missing from the workspace listing
- the Python `CompressionStore` defaults to **SQLite**, not in-memory,
when `HEADROOM_CCR_BACKEND` is unset (verified by execution) — this one
matters for multi-worker reasoning

And the install scripts pointed at `docs/docker-install.md`, a path that
no longer exists; they now point at the live docs page.

## Verification

297 tests pass, including the plugin-manifest, install-state and
release-workflow suites that assert on these exact strings. `ruff check`
clean. Unrelated lockfile churn (`uv.lock`, both `package-lock.json`) is
deliberately excluded.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-02 15:53:35 +05:30
Ruben A. e530de5ad2 feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154)
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the
AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for
Python, JavaScript, TypeScript, Go, Rust, Java, C and C++.

Parity-only, like #1153. Nothing calls it: the only references outside the
module are the pub mod / pub use declarations in transforms/mod.rs, and
live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched
and no Python source changes, so the engine is unreachable from the shipped
package. #1155 wires it into live-zone dispatch.

Every grammar is pinned with '=' to the exact version of the corresponding
Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means
the same grammar.js, hence the same generated parser.c, hence node-for-node
identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8
languages confirmed identical node-type and line-span trees at these pins;
bumping any pin requires re-running it and re-recording the fixtures.

Ships 30 recorded parity fixtures, a CodeCompressorComparator in
headroom-parity, and scripts/record_code_compressor_fixtures.py.

Verified byte-identical to the recorded Python output:

  [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0

Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped
(cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under
ONNX Runtime 1.24.4 (see #2591).

Also verified cargo check -p headroom-core --no-default-features passes, so the
static-musl path stays intact.
2026-07-27 09:21:57 -07:00
Ruben A. 83e27e5036 feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress
ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the
kompress-v2-base ONNX model through ort with a cache-only loader that never
touches the network.

Parity-only. Nothing calls it: the only references outside the module are the
pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries
TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve
prose compression. The pyo3 bridge is untouched and no Python source changes, so
the new engine is unreachable from the shipped package. #1155 wires it up.

Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and
scripts/record_kompress_fixtures.py.

Verified byte-identical to the recorded Python output:

  [kompress] total=21 matched=21 skipped=0 diffed=0

That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead
of erroring, which is why these fixtures had never been run. In CI the model is
absent from the HF cache, so the comparator errors and the fixtures report
Skipped rather than hanging.

Also gates the module behind the ml feature, matching magika_detector: kompress.rs
uses ort, which is optional = true, so an unconditional pub mod broke
cargo check --no-default-features (the static-musl path). CI does not catch that
class of break because cargo test --workspace only builds default features.
2026-07-27 08:17:53 -07:00
Zhenjia ZHOUandClaude Opus 4.8 4035c04187 feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)
## Description

`TextCrusher` (the native extractive prose compressor added in #1171)
only handled ASCII: `split_segments` split on `.!?`+whitespace and
`tokens` split on whitespace/alphanumeric runs. CJK
(Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a
whole CJK paragraph collapsed into **one segment / one token** — it
passed through at ~0% compression, and BM25 relevance + salience scored
zero terms.

This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU
(`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path,
with a length fallback for terminator-sparse runs, a local BM25 over the
ICU word tokens, and ICU-token salience. Dispatch is on **content
only**, so pure-ASCII text is byte-identical to before — the shared
`BM25Scorer` and the ASCII path are untouched.

It also adds a committed, reproducible answer-retention eval
(`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI
regression gate, so the improvement below is permanently verifiable
rather than a one-off measurement.

Extends #1171.

## Type of Change

- [x] Bug fix (CJK passed through near-uncompressed)
- [x] New feature (CJK segmentation / relevance support)
- [x] Performance improvement (CJK now compresses; ICU segmenters
cached, not rebuilt per call)

## Changes Made

- `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK
punctuation, full/half-width forms).
- `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory
length fallback (whitespace / CJK punctuation / hard cap) for
terminator-sparse runs; ASCII path unchanged.
- `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path
unchanged.
- `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII
`BM25Scorer` scores zero terms for CJK and is parity-locked, so this is
an intentional separate scorer (documented in code).
- CJK salience uses ICU tokens (whitespace-split gave one giant "word" →
zero salience).
- `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for
space-free text.
- ICU segmenters resolved once in `static LazyLock` (compiled_data is
static) instead of rebuilt per call.
- New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency
below).
- `benchmarks/i18n_compression_eval.py` +
`tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko
answer-retention eval — a deterministic needle CI gate (always-runs, no
external data), real-transcript fidelity with CJK-aware salient, and
optional `multi-wiki-qa` natural-data retention (loaded via the
`[evals]` `datasets` extra, skipped if absent; data never vendored —
CC-BY-NC-SA).

## Testing

- [x] Unit tests pass (`pytest` + `cargo test`)
- [x] Linting passes (`ruff check`/`format` on the new eval + test —
clean)
- [ ] Type checking passes (`mypy headroom`) — N/A, the only Python
added is a benchmark + test, not `headroom/` source
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

```text
$ cargo test -p headroom-core --lib text_crusher
running 12 tests
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out

$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py
15 passed

$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py
6 passed   # deterministic zh/ja/ko needle CI gate

$ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py   # both clean
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Python in a uv venv,
`headroom-core` built via `uv pip install -e .` (maturin), branch
`feat/cjk-text-compression`.
- Exact command / steps: built `_core`, then ran a mixed
Chinese+Japanese doc (no spaces, `。` terminators) through
`TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated
answer-retention on the public CMRC2018 Chinese QA dev set (bury the
gold-answer paragraph among 25 distractors, query = the question,
compress to 30%, check the gold answer survives), and end-to-end through
`ContentRouter`.
- Observed result: a mixed Chinese+Japanese doc compressed 189 → 78
tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence
surviving — before this change the same doc was a single segment → 100%
passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention
under 30% compression rose 34% → 93% (multiple seeds). End-to-end
through `ContentRouter` on real CJK content, aggregate savings rose 16%
→ 40%. Pure-ASCII (English) output stayed byte-identical (the English
parity fixtures did not move). Demo terminal output:

    ```text
    ORIGINAL  tokens= 189  chars=189
    COMPRESS  tokens=  78  ratio=0.41  segments kept 3/8
    QUERY-RELEVANT sentence survived: True
    --- compressed output (verbatim kept CJK sentences) ---
    认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。
    请求重试使用指数退避并设置最大次数上限。
    数据备份每天凌晨执行并保留最近三十天的快照。
    ```
The committed eval now demonstrates this across all three CJK languages.
The deterministic needle gate (in CI via
`tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has
TextCrusher keep the query-relevant needle while truncate/random drop it
in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang),
query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs
**25–41%** for the truncate/random baselines:

    ```text
=== Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3)
===
      lang    text_crusher  truncate  random
      zh-cn           74%       25%     38%
      ja              70%       31%     39%
      ko              50%       26%     41%
    ```
Korean is measurably weaker (ICU has no Korean dictionary and falls back
to UAX#29 word-breaking) — still well above baselines, and scoped as a
follow-up.
- Not tested: the live proxy HTTP path (validated at the `ContentRouter`
/ `TextCrusher` layer, not via a running proxy); no-space Korean
(standard Korean is space-delimited and is covered); non-CJK SE-Asian
scripts (out of scope).

## Dependency (per CONTRIBUTING supply-chain policy)

`icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`:

- **Why this package (vs. ourselves / existing deps):** CJK needs
dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored
slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91%
bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X
covers zh/ja/ko in one crate. The existing `unicode-segmentation` does
UAX#29 only (no CJK dictionary), so it can't word-segment space-free
CJK.
- **Who maintains it:** the official `unicode-org` ICU4X project; active
release cadence (2.2 in 2025); no known CVEs.
- **Install surface:** ~13 new pure-Rust crates, no build scripts, no
native code, no build/runtime network. `compiled_data` bundles locale
data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled
— LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in
`libm` for nothing.
- **Why this version:** 2.x is the stabilized ICU4X API (1.x used a
different data-provider model); floored at 2.2 (Cargo.lock pins the
patch) since segmenter boundaries are observable in output and bumps
should be deliberate.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings (clippy + fmt clean)
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md

## Additional Notes

- **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with
`headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a
separate local scorer because the shared one's tokenizer is ASCII-only.
The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper
over `_core`), so there is no Python mirror to keep in sync; the parity
fixtures stay green (only the CJK `unicode` fixture was re-recorded,
intentionally; English fixtures unchanged).
- **Known by-design gap (not a bug):** CJK content + a pure-ASCII query
yields no token overlap, so relevance falls back to recency + salience
(cross-script query matching is unsupported).
- The Python added is a benchmark
(`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/`
runtime source — both are `ruff`-clean; `mypy headroom` is unaffected.
- **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa`
(CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if
absent — the dataset is never vendored into the repo, and the always-run
CI gate (Part C) uses only our own deterministic data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 19:58:48 +00:00
Zhenjia ZHOUandClaude Opus 4.8 6c68ff4e9f perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description

On a cold-start large context, kompress (ModernBERT ONNX) runs
**synchronously on the request thread** — ~200–300s for ~1M tokens. It
blows the 30s compression budget, leaks a non-preemptible worker, and
cascades (executor saturation → queue timeouts on healthy requests); on
timeout the request is forwarded **uncompressed** after eating 30s. This
adds four layered, **default-off, fail-open** mitigations so the request
path is never blocked on ML compression.

Closes #1171

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default
50000): route oversized text away from ModernBERT (→ LogCompressor /
TextCrusher / passthrough) at the single `_try_ml_compressor` boundary.
- **Phase 1 — cooperative deadline**
(`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run
self-terminates at the next chunk boundary past the budget, keeping the
unprocessed tail verbatim.
- **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native
Rust** extractive prose compressor in
`crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as
`headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the
shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25,
and ships record/replay parity fixtures (mirroring the SmartCrusher
Rust-core + Python-shim pattern).
- **Phase 3 — off-path compression**
(`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately
and compress in a per-process background drain; a byte-identical cache
hit on a later turn means the request never blocks on ML.
- Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG
entry, and docstrings documenting the fail-open limits.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`, new modules)
- [x] New tests added for new functionality
- [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed
on real traffic in earlier iterations; Phase 3 off-path is unit- +
byte-identity-tested, not yet live-validated)

### Test Output

```text
$ pytest tests/test_transforms/ tests/test_cache/ \
    tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q
501 passed, 37 skipped in 40.33s

$ cargo test -p headroom-core --lib text_crusher
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out

$ ruff check <changed files>
All checks passed!

$ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py
Success: no issues found in 2 source files
```

New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS +
TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim
tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3
byte-identity round-trip; TextCrusher unit + parity.

## Real Behavior Proof

- Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv
pip install -e .`.
- Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy`
commands shown under Test Output; quality eval `python
benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`.
- Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on
changed/new modules. Quality eval: TextCrusher keeps ~94% of buried
SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed
run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT
takes minutes (fast-vs-slow contrast, not a same-input run).
- Not tested: Phase 3 off-path on live traffic; multi-worker
(per-process by design — see Additional Notes).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **All four features are off by default and fail-open** — with the env
flags unset the paths are no-ops for realistic inputs; on any error the
request is forwarded (compressed if possible, else verbatim), never
dropped. A full background queue / duplicate key surfaces as
`deferred:dropped`.
- **Known limits (documented in `background_compression.py`):** Phase 3
is per-process, in-memory, and token-mode-only — these are
**lost-savings, never lost-correctness**, and consistent with the
project's existing per-process compression cache + sticky-session
multi-worker model. The startup multi-worker warning now names off-path
background compression.
- Phase 2 reuses the existing BM25 scorer; reuse did not improve
answer-retention over a Python prototype (query-awareness dominates) —
its value is the Rust speed + repo-conventional Rust-core/Python-shim
shape.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 10:48:06 -05:00
0ce68dedd7 fix(proxy): restore Codex usage headers on WS and streaming SSE transports (#577) (#794)
## Description

Codex's subscription/rate-limit window (the `x-codex-*` headers) was
being
**stripped on every transport Codex actually uses**, so session/weekly
usage
never reached the Codex CLI's own `/status` display, Headroom
`/stats`/dashboard,
or any consumer that sniffs the client-facing handshake. This PR
restores it on
**both** the WebSocket and streaming-SSE paths — the two halves of #577
— in one
place.

Fixes #577

**Supersedes #582 and #590.** This PR incorporates #582's SSE fix
(carried verbatim
with a `Co-authored-by` trailer) and additionally forwards the window
onto the client
`101` on the WS path, which #582/#590's capture-only WS code cannot do.
Both can be
closed as superseded once this merges — GitHub closing keywords only
auto-close
issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual
close.

### WebSocket (`gpt-5.4+`)

OpenAI delivers `x-codex-*` **only** on the upstream WS handshake
response, never
in data frames. `handle_openai_responses_ws` accepted the client WS
*before* it
connected upstream and never read `upstream.response.headers`, so the
window was
dropped. This reorders the handler to **connect upstream first**,
extract the
`x-codex-*` subset, then **accept the client WS with those headers
attached** to
the `101`, and refresh the Python state for `/stats` parity.

### Streaming SSE (incorporated from #582, @m16khb)

Codex CLI almost always streams. `streaming.py` neither captured
`x-codex-*` into
`CodexRateLimitState` nor forwarded it — the forwarded-header filter
matched only
the substring `"ratelimit"`, which `x-codex-*` does not contain. This
calls
`update_from_headers()` **before** the `>=400` early-return (so a
streaming 429/5xx
still refreshes the window, matching the non-streaming handlers) and
widens the
forward filter to pass `x-codex-*`.

> Credit: the SSE fix is @m16khb's work from #582, carried here verbatim
with a
> `Co-authored-by` trailer so the maintainer gets a single PR covering
both
> transports. This supersedes #582/#590's **WS** capture (which only
writes
> `/stats`); the connect-before-accept reorder additionally forwards the
window to
> the client `101`, which capture-only cannot do. #590's optional
snapshot
> persistence is intentionally left out (separable; hot-path sync write;
doesn't
> help the `101`-sniff consumers).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `openai.py`: add `_extract_codex_handshake_headers()` (strictly
`x-codex-*`, via
`raw_items()` to avoid `MultipleValuesError`; never
`set-cookie`/`authorization`).
- `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry
loop runs
before `accept()`; `accept(headers=...)` carries the forwarded window;
first
client frame read afterward. HTTP fallback preserved; it now also
refreshes
  `/stats` from the HTTP response headers.
- `streaming.py`: capture `x-codex-*` on all statuses + widen the
forwarded-header
  filter (from #582).

### Diff-size note

The bulk of the `openai.py` line count is **whitespace-only
relocation**: the relay
block dedents one level out of the old per-attempt `async with`. Logical
change is
~290 lines. **Review with `?w=1`.** In API-key mode the handshake
carries no
`x-codex-*`, so the accept-header list is empty and the path behaves
exactly as
before — the fix only activates for ChatGPT-subscription auth.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

- WS: `test_ws_connect_happens_before_accept`,
`test_ws_forwards_codex_headers_to_client_accept`
(only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded;
`/stats` refreshed),
`test_ws_connect_failure_falls_back_to_http`,
`test_ws_first_frame_timeout_after_connect_closes_upstream`.
- Fallback: `test_fallback_refreshes_codex_rate_limit_state`.
- SSE:
`test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`,
  `test_codex_rate_limit_captured_on_streaming_429` (from #582).
- Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real
proxy + fake
upstream + real `websockets` client and reads the client `101` — closes
the gap
the unit tests stub (that uvicorn/starlette actually write
`accept(headers=...)`).

## Test Output

```
$ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \
                tests/test_ws_http_fallback.py \
                tests/test_openai_codex_ws_lifecycle.py \
                tests/test_openai_codex_ws_timings.py \
                tests/test_codex_rate_limits.py -q
63 passed in 0.83s

$ .venv/bin/python tests/e2e_ws_codex_usage_headers.py
[codex-hdr-e2e] client 101 headers:
    x-codex-primary-used-percent: 42
    x-codex-primary-window-minutes: 300
    x-codex-secondary-used-percent: 7
    x-codex-secondary-window-minutes: 10080
[codex-hdr-e2e] /stats reflects codex window (primary-used=42)
=== CODEX-HDR E2E ALL GREEN ===

$ uv run ruff check . && uv run ruff format --check <touched files>
All checks passed!
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **Why connect-before-accept (not capture-only).** Once `accept()`
sends the `101`,
headers can no longer be added; the `x-codex-*` window only exists after
we connect
upstream. Capturing into Python state (as #582/#590's WS code does)
fixes `/stats`
but not the Codex CLI's native display or any `101`-sniffing consumer —
those need
  the headers *on the client handshake*, which requires the reorder.
- **Security.** Forwarding is filtered strictly to `x-codex-*`;
`set-cookie`,
`authorization`, and all other upstream headers are never forwarded to
the client
  (asserted by both the unit test and the e2e).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Contract Schemas

Per maintainer request: a JSON Schema (draft 2020-12) artifact
enshrining the OpenAI
interaction expectations this changeset relies on, so drift is
detectable later.

Committed following the repo's parity convention:
- schema:
`tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json`
- test: `tests/test_codex_openai_contract_parity.py` binds the schema to
the **live code**
in both directions, so drift fails CI rather than living only in this
description -
every declared `x-codex-*` header must be consumed by
`parse_codex_rate_limits`, and
`_extract_codex_handshake_headers` must forward exactly the declared
subset and never
`set-cookie`/`authorization`. No new dependency (does not pull in
`jsonschema`).

It covers, as `$defs`:

- `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` -
the upstream
`x-codex-*` header family (full superset, with per-header wire pattern +
the parsed
semantic type) the WS and SSE captures read. Source of truth:
`parse_codex_rate_limits`.
- `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny**
contract: only
`x-codex-*` may be forwarded; `set-cookie`/`authorization` are
explicitly forbidden
  (`propertyNames` + `not`).
- `ClientForwardedStreamingHeaders` - the wider SSE forward set
(`*ratelimit*` OR `x-codex*`).
- `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` -
the WS frame
  envelopes and the unwrapped HTTP-fallback POST body.
- `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity
tests assert.

Validated with `jsonschema` (Draft202012 `check_schema` passes; positive
instances from
the e2e validate; negative instances - a leaked `set-cookie`, a fallback
body still
carrying a top-level `type` - are correctly rejected).

<details>
<summary><code>codex-openai-interaction.schema.json</code> (draft
2020-12)</summary>

```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json",
  "title": "Codex <-> OpenAI interaction contracts (PR #794)",
  "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).",
  "$defs": {
    "OpenAICodexWindowHeaders": {
      "title": "x-codex-*-{primary,secondary} window headers",
      "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.",
      "type": "object",
      "properties": {
        "x-codex-primary-used-percent": {
          "type": "string",
          "pattern": "^\\d+(?:\\.\\d+)?$",
          "x-headroom-parsed-type": "float (0-100, NaN-guarded)",
          "description": "Percent of the primary window consumed. Gates creation of the primary window."
        },
        "x-codex-primary-window-minutes": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int",
          "description": "Primary window size in minutes."
        },
        "x-codex-primary-reset-at": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int (Unix epoch seconds)",
          "description": "Absolute reset time of the primary window."
        },
        "x-codex-secondary-used-percent": {
          "type": "string",
          "pattern": "^\\d+(?:\\.\\d+)?$",
          "x-headroom-parsed-type": "float (0-100, NaN-guarded)",
          "description": "Percent of the secondary window consumed. Gates creation of the secondary window."
        },
        "x-codex-secondary-window-minutes": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int"
        },
        "x-codex-secondary-reset-at": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int (Unix epoch seconds)"
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexCreditsHeaders": {
      "title": "x-codex-credits-* headers",
      "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.",
      "type": "object",
      "properties": {
        "x-codex-credits-has-credits": {
          "type": "string",
          "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$",
          "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)",
          "description": "Gates creation of the credits snapshot."
        },
        "x-codex-credits-unlimited": {
          "type": "string",
          "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$",
          "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)"
        },
        "x-codex-credits-balance": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Free-form server string, e.g. \"$5.00\"."
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexMetaHeaders": {
      "title": "x-codex meta headers",
      "type": "object",
      "properties": {
        "x-codex-limit-name": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"."
        },
        "x-codex-promo-message": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Server announcement. Also gates snapshot creation when present."
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexRateLimitHeaders": {
      "title": "Full x-codex-* header family OpenAI may emit",
      "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.",
      "type": "object",
      "allOf": [
        { "$ref": "#/$defs/OpenAICodexWindowHeaders" },
        { "$ref": "#/$defs/OpenAICodexCreditsHeaders" },
        { "$ref": "#/$defs/OpenAICodexMetaHeaders" }
      ],
      "additionalProperties": true
    },
    "WSUpstreamHandshakeResponse": {
      "title": "OpenAI WS handshake (101) response headers consumed by the WS fix",
      "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.",
      "$ref": "#/$defs/OpenAICodexRateLimitHeaders"
    },
    "StreamingUpstreamResponseHeaders": {
      "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix",
      "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).",
      "$ref": "#/$defs/OpenAICodexRateLimitHeaders"
    },
    "ClientForwardedHandshakeHeaders": {
      "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)",
      "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.",
      "type": "object",
      "propertyNames": {
        "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-"
      },
      "not": {
        "anyOf": [
          { "required": ["set-cookie"] },
          { "required": ["Set-Cookie"] },
          { "required": ["authorization"] },
          { "required": ["Authorization"] }
        ]
      },
      "additionalProperties": { "type": "string" }
    },
    "ClientForwardedStreamingHeaders": {
      "title": "Headers forwarded to the client on the streaming SSE path",
      "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.",
      "type": "object",
      "propertyNames": {
        "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]"
      },
      "additionalProperties": { "type": "string" }
    },
    "WSClientRequestFrame": {
      "title": "Client -> proxy WS data frame (Responses API over WS)",
      "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.",
      "type": "object",
      "properties": {
        "type": { "const": "response.create" },
        "response": {
          "type": "object",
          "properties": {
            "model": { "type": "string", "description": "e.g. gpt-5.4" },
            "input": {
              "description": "String prompt or Responses-API structured input array.",
              "type": ["string", "array"]
            },
            "stream": { "type": "boolean" }
          },
          "required": ["model"],
          "additionalProperties": true
        }
      },
      "required": ["type", "response"],
      "additionalProperties": true
    },
    "WSRelayEvent": {
      "title": "proxy -> client WS data frame (relayed Responses API event)",
      "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.",
      "anyOf": [
        {
          "title": "lifecycle event",
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "examples": [
                "response.created",
                "response.output_item.added",
                "response.completed"
              ]
            },
            "response": { "type": "object", "additionalProperties": true }
          },
          "required": ["type"],
          "additionalProperties": true
        },
        {
          "title": "error event",
          "type": "object",
          "properties": {
            "type": { "const": "error" },
            "error": {
              "type": "object",
              "properties": { "message": { "type": "string" } },
              "required": ["message"],
              "additionalProperties": true
            }
          },
          "required": ["type", "error"],
          "additionalProperties": true
        }
      ]
    },
    "HTTPFallbackRequestBody": {
      "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback",
      "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.",
      "type": "object",
      "properties": {
        "model": { "type": "string" },
        "stream": { "const": true },
        "input": { "type": ["string", "array"] }
      },
      "required": ["model", "stream"],
      "not": { "required": ["type"] },
      "additionalProperties": true
    },
    "CodexRateLimitStatsOutput": {
      "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)",
      "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.",
      "type": "object",
      "properties": {
        "limit_id": { "const": "codex" },
        "limit_name": { "type": ["string", "null"] },
        "primary": { "$ref": "#/$defs/CodexWindowDict" },
        "secondary": { "$ref": "#/$defs/CodexWindowDict" },
        "credits": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "properties": {
                "has_credits": { "type": "boolean" },
                "unlimited": { "type": "boolean" },
                "balance": { "type": ["string", "null"] }
              },
              "required": ["has_credits", "unlimited", "balance"],
              "additionalProperties": false
            }
          ]
        },
        "promo_message": { "type": ["string", "null"] },
        "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." }
      },
      "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"],
      "additionalProperties": false
    },
    "CodexWindowDict": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "properties": {
            "used_percent": { "type": "number" },
            "window_minutes": { "type": ["integer", "null"] },
            "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." },
            "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." },
            "seconds_until_reset": { "type": ["integer", "null"] }
          },
          "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"],
          "additionalProperties": false
        }
      ]
    }
  }
}
```

</details>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: m16khb <m16khb@gmail.com>
2026-06-09 15:55:53 -05:00
chopratejas 967b0db439 fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.

Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py

Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
  candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
  becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py

Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
  preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
  the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.

Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
  `intelligent_context` fields; hoist `output_buffer_tokens` to top
  level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
  and RollingWindow imports + branch; pipeline is CacheAligner →
  ContentRouter (smart_routing) or CacheAligner → SmartCrusher
  (legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
  `--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
  `_apply_compression`, drop RollingWindowConfig dep. Threshold is
  now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
  exports of deleted symbols.

Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
  empty-dict to falsy → callers passing `environ={}` accidentally
  pulled from os.environ. Use `environ if environ is not None else
  os.environ`.

Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
  the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
  byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
  `\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
  handler attached to the named logger, so the assertion is
  order-independent (proxy `_setup_file_logging` flips
  `headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
  before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
  monkeypatch.delenv every provider key so the BYOK error
  actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
  pytest.mark.skip — proxy currently has no :embedContent route;
  feature gap, not regression.

Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.

Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 12:23:17 -07:00
chopratejas 21989e3640 fix(rust): port MessageScorer to Rust + parity harness (PR-A)
Direct port of `headroom.transforms.scoring.MessageScorer` (459 LOC).
Foundation piece for the IntelligentContext port (PR-B onward).

What's wired:
- Deterministic factors fully ported: recency (exp-decay), forward
  references (tool_call_id graph), token density (unique/total).
- External-dep factors gated behind traits: `EmbeddingProvider` and
  `ToinProvider`. No concrete impls yet — both default to neutral
  values matching Python's `embedding_provider=None` / `toin=None`.
  PR-A1 wires fastembed; PR-A2 plugs in a PyO3 ToinProvider.
- ScoringWeights + MessageScore with serde + BTreeMap-ordered
  breakdown for stable JSON.

Parity:
- 13 fixtures recorded from Python, byte-equal under the comparator.
- Floats rounded to 5 decimals on both sides — absorbs f32-vs-f64
  drift in the weighted sum without masking real bugs.

Drive-by: re-fix three pre-existing clippy errors in
smart_crusher/crusher.rs that re-emerged with new test additions
(field_reassign_with_default + dead hash_array_for_ccr).
2026-05-01 10:06:56 -07:00
chopratejas 625290cfbc chore(rust): port ContentDetector to Rust + parity harness + PyO3 bridge
Faithful port of `headroom/transforms/content_detector.py` into
`headroom-core`. Same regex patterns, dispatch order, confidence
formulas, and line-count caps; lockstep with Python via 21 recorded
parity fixtures (every dispatch branch exercised).

- crates/headroom-core/src/transforms/content_detector.rs: regex-only
  detector (no ML); ContentType/DetectionResult mirror Python's enum +
  dataclass surface; metadata uses serde_json::Map for clean PyO3
  bridging.
- Tie-break in code detection: track scores in first-match insertion
  order (matches Python dict iteration semantics on `max()` ties).
- TypeScript second pattern is start-anchored — Python's
  `pattern.match(line)` is start-anchored, but the regex crate's
  `is_match` is unanchored, so the literal `:` prefix is required for
  parity.
- crates/headroom-parity: ContentDetectorComparator + universal f64
  normalization in `compare_fixture` (serde_json's lossy parse vs
  full-precision serialize creates a 1-ULP asymmetry that broke
  comparison; round-trip the actual through to_string/from_str).
- crates/headroom-py: detect_content_type/is_json_array_of_dicts and
  PyDetectionResult exposed via PyO3 with GIL released during scan.
- tests/parity/recorder.py: new `_wrap_function` for free-function
  recording; content_detector hook + 21 varied inputs covering JSON
  arrays, diffs, HTML, search, build/log, six languages, and fallbacks.

Sets up PR2 (ContentRouter scaffold) to call into Rust ContentDetector
in-process via the bridge.
2026-04-27 23:57:39 -07:00
chopratejas da7716a95a chore(rust): SmartCrusher CCR marker injection + walker unification
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:

1. CCR-Dropped marker is now injected into process_value's lossy-path
   output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
   appended to the kept-items array. Previously the store held the
   original but no pointer reached the prompt -- the retrieval contract
   was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
   array-of-dicts shape so downstream iteration with x.get(...) keeps
   working.

2. Walker / process_value drift removed. process_value gains a
   Value::String arm that handles stringified-JSON containers (parse,
   recurse, re-encode) and opaque blobs (CCR marker + store) -- same
   semantics walker.rs has always had, now reachable from the main
   crush() pipeline.

3. Opaque-string CCR now stores originals. DocumentCompactor gains an
   Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
   store.put when one is configured. Same hash regardless of store
   presence -- runtime contract is stable across configurations.
   Same wiring is shared between walker.rs and process_value via the
   extracted helper.

5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
   compacted-json string. Routes through the crusher's existing CCR
   store, so ccr_get resolves both row-drop and opaque-string hashes.

Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
  nested-array marker, opaque-string roundtrip, stringified-JSON
  recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
  both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
  -- their lossy outputs now carry the sentinel; Rust + Python both
  match the new bytes (parity-run smart_crusher: 17/17)
2026-04-27 20:25:22 -07:00
chopratejas 43d1aa0329 parity(smart_crusher): byte-equal harness — 17 fixtures green
Adds the SmartCrusher half of the Rust-vs-Python parity harness. Path A
from the Stage 3c.1 plan: record fixtures from Python (with real
fastembed embeddings + the post-bug-fix code), drive the Rust port over
the same inputs, and assert byte-equal output on every recorded
scenario.

What's in:
- `tests/parity/record_smart_crusher.py`: standalone recorder for
  `SmartCrusher.crush(content, query, bias)`. The generic recorder
  framework only captures one positional, so this script writes its
  own JSON envelope `{input: {content, query, bias}, config, output}`.
  17 scenarios cover the planning paths exercised by ContentRouter
  in production: passthrough, smart_sample, top_n, time-series,
  duplicates, unicode (`ensure_ascii=False`), nested-3-deep, empties,
  bias above and below 1.0.
- `crates/headroom-parity/src/lib.rs`: `SmartCrusherComparator`
  reconstructs `SmartCrusherConfig` from the fixture's config block,
  runs Rust `SmartCrusher::crush()`, emits the same JSON shape Python
  serialized.
- `crates/headroom-parity/examples/diff_fixture.rs`: diagnostic CLI
  that prints expected-vs-actual for one fixture (used during the
  iteration that found the serializer bug below).

Serializer fix — found by the harness:
SmartCrusher uses `safe_json_dumps` (compact `(",", ":")` separators
+ `ensure_ascii=False`) for the wire bytes. The Rust port was using
`python_json_dumps` (default Python: `(", ", ": ")` + `ensure_ascii=
True`), which is the right choice for hashing but wrong for the
output. Refactored `anchor_selector.rs` to take a small
`JsonFmt { sort_keys, compact, ensure_ascii }` config so the three
flavors share one writer, added `python_safe_json_dumps`, and
switched `_smart_crush_content` to call it. All three flavors now
have byte-exact tests.

Two unit tests in `crusher.rs` were pinning the old (wrong) format
and have been re-pinned to the compact form.

Cross-language status:
- All 17 empty-query fixtures: byte-equal.
- Embedding-driven (non-empty-query) fixtures deferred until the
  ~0.0002 numeric drift between Python `onnxruntime` and Rust `ort`
  is resolved (or until we accept the drift via a tolerance — none of
  the 17 fixtures exercises a borderline relevance_threshold call,
  and downstream code only branches at the 0.3 threshold).

Tests: cargo test --workspace (388 + supporting) green.
2026-04-27 00:01:26 -07:00
chopratejas 48c13245c5 fix(diff): close ContentRouter routing gaps for merge diffs and long preambles
User audit caught three gaps that prevented DiffCompressor from being
invoked even when the input was a real diff. These complement the four
emit-time bugs fixed in the previous commit — those fixes only kick in
once DiffCompressor receives the input. Without these gap fixes, real
merge-commit diffs and `git log -p` outputs with long commit messages
were misrouted away from DiffCompressor entirely.

# The three gaps (each fixed in Python; gap 3 also fixed in Rust)

1. Detector scan window was hardcoded to first 50 lines.
   `_try_detect_diff` in content_detector.py only inspected
   `content.split("\n")[:50]`. `git log -p` outputs commonly have
   commit messages longer than 50 lines (releases, squashed commits,
   bots), pushing the `diff --git` header out of the detection window.
   Result: input was returned with `content_type=PLAIN_TEXT` and routed
   to the text compressor, never reaching DiffCompressor. Fix: window
   widened to 500 lines.

2. Detector regex didn't recognize merge-commit headers.
   `_DIFF_HEADER_PATTERN` matched `diff --git`, `--- a/`, and the
   regular `@@ -A,B +C,D @@` hunk header. Merge-commit diffs from
   `git log -p` use `diff --combined <path>`, `diff --cc <path>`, and
   combined-diff hunk headers `@@@+`. The shared `--- a/` line still
   triggered the detector with low confidence, but only barely. Fix:
   extended the regex to recognize all four merge-shaped header forms.

3. DiffCompressor parser only matched `^diff --git`.
   Even after fixing detection, the parser's `_DIFF_GIT_PATTERN`
   wouldn't match `diff --combined` or `diff --cc`, so merge diffs
   reached DiffCompressor and were treated as one giant pre-diff blob —
   passed through unchanged after the previous PR's pre-diff
   preservation fix. Fix: added `_DIFF_COMBINED_PATTERN` and
   `_DIFF_CC_PATTERN`; `_parse_diff` starts a new file section on any
   of the three header forms. Mirrored in Rust as `is_diff_header`
   helper that checks all three regexes.

# Why this matters end-to-end

DiffCompressor's value comes from being routed to. Detection +
parser-level coverage are upstream of the compressor — without them,
the compressor never sees the input. The previous PR's four bug fixes
(rename, combined-diff hunks, no-newline marker, pre-diff content) are
correct and necessary, but for merge commits and long-preamble diffs,
they were only firing on the rare cases where the detector misclicked
into DiffCompressor anyway. With these three gaps closed, the
ContentRouter→DiffCompressor pipeline actually engages on:
- `git log -p` outputs of any commit-message length
- Merge-commit diffs (`diff --combined`, `diff --cc`)
- Combined-diff snippets (`@@@`+ hunk-only inputs)

# New fixtures (3 added to the existing 24)

- `066bc82…` — `diff --combined` merge diff (3-way)
- `5d950a94…` — `diff --cc` merge diff (alternate form)
- `66c86f64…` — long pre-diff content (60-line commit message)
  followed by a rename diff (exercises detector scan widening +
  pre-diff preservation in tandem)

Parity: total=27 matched=27 skipped=0 diffed=0.

# Tests

- Python: 4 new tests across 2 new test classes —
  `TestRoutingGapMergeDiffs` (combined / cc parser) and
  `TestRoutingGapDetectorScanWindow` (long preamble detection +
  combined-diff regex recognition).
- Rust: 2 new tests covering combined / cc parser sections.

# Verification

- 27/27 parity fixtures byte-equal.
- Python: 41/41 tests pass (was 37).
- Rust: 18/18 transforms tests; 62/62 workspace; 5/5 proptests.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
  -- -D warnings` clean.
2026-04-26 09:13:08 -07:00
chopratejas 6d47a0cd00 fix(diff_compressor): four silent information-loss paths in Python AND Rust
Audit caught four bugs that the byte-equal parity harness can't catch on
its own — both Python and Rust were faithfully emitting the buggy output.
Fixed in lockstep so parity is maintained while the underlying behavior
is now correct on inputs the existing 20 fixtures didn't exercise.

# The four bugs (each fixed in both Python and Rust)

1. Renames silently dropped from output. Parser captured `is_renamed=True`
   but the emitter never emitted ANY rename markers. Output of a rename
   looked exactly like a plain modification of the old path. Fix: capture
   `rename from` / `rename to` / `similarity index N%` / `dissimilarity
   index N%` / `copy from` / `copy to` lines in a new `rename_lines` field
   on `DiffFile`; emit them after `diff --git` in canonical git ordering.

2. Combined diff hunks (`@@@`) silently dropped. Hunk-header regex only
   matched `@@`, so 3-way merge hunks had `current_hunk` never set and
   ALL their content fell through to the no-op branch. Fix in Python:
   regex switched to `^(@@+) ... \1` (backreferences match any number of
   `@`s on each side). Fix in Rust: alternation over `@@`, `@@@`, `@@@@`
   since `regex` is RE2-based and rejects backreferences. n>3 octopus
   merges still fall through; rare in practice.

3. `\ No newline at end of file` markers can be context-trimmed away.
   Treated as ordinary "other" lines — if more than `max_context_lines`
   from a `+`/`-` change, dropped. Round-trip-breaking for patches; can
   change whether the trailing line has a newline. Fix: in
   `_reduce_context`, force-add any line starting with `\` to the keep
   set regardless of distance.

4. Pre-diff content silently dropped. Anything before the first `diff
   --git` — commit messages from `git log -p`, email headers from `git
   format-patch`, fork-and-rebase metadata — was discarded. Fix:
   `_parse_diff` now returns `(pre_diff_lines, files)`; `format_output`
   prepends pre-diff content verbatim when present.

# Hidden parity bug found during the work

`_compress_files` constructed a fresh `DiffFile` from the parsed one but
only copied a subset of the fields by name. The new `rename_lines` and
`original_*_line` fields were silently dropped here, so the parser
populated them correctly but the emitter saw an empty `rename_lines`
list. Caught by writing a real test instead of a smoke test — the smoke
test passed because it hit the no-diff-found short-circuit, not the
parser/emitter pipeline. Constructor now copies all fields explicitly.

# Parity status

- Existing 20 fixtures: still byte-equal between fixed Python and fixed
  Rust. None of them exercised the buggy paths.
- 4 NEW fixtures recorded against fixed Python, exercising each bug-fix
  path: rename, 3-way combined diff, `\ No newline` marker far from
  changes, pre-diff commit headers. All 4 byte-equal between Python and
  Rust.
- Parity harness: total=24 matched=24 skipped=0 diffed=0.

# Observability

Some normalizations remain parity-bound (file mode `100644` hardcode,
`Binary files differ` simplification). Those are surfaced in
`DiffCompressorStats::file_mode_normalizations` /
`binary_files_simplified` (Rust) and via `logger.warning` (Python's new
`_log_loss_signals` helper, called once per compress).

# Tests

- Python: 4 new test classes (11 tests) covering rename markers,
  combined diffs, no-newline preservation, pre-diff content. Edge case:
  no pre-diff content must NOT add a leading blank line.
- Rust: 4 new `bugfix_*` unit tests with the same scenarios.
- Existing Python tests calling `_parse_diff` directly were updated for
  the new `(pre_diff, files)` tuple return.

# Verification

- Python: 37/37 tests pass (was 26).
- Rust: 16/16 transforms tests; 60/60 workspace unit tests; 5/5
  proptests; 1/1 doctest.
- Parity: 24/24 byte-equal.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
  -- -D warnings` clean.
2026-04-26 09:13:08 -07:00
chopratejasandClaude Opus 4.7 0414cb70e4 feat(rust): scaffold workspace + parity harness (phase-0)
Bootstrap the Rust port of Headroom. Additive only — no existing Python
code modified. Ships the workshop, not the widgets.

Layout
  Cargo.toml (workspace) + rust-toolchain.toml
  crates/headroom-core    — transform library, stub only
  crates/headroom-proxy   — axum binary, /healthz only
  crates/headroom-py      — PyO3 cdylib, exposes headroom._core.hello()
  crates/headroom-parity  — Rust-vs-Python oracle harness + parity-run CLI

Tooling
  Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint
  .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly
  deny.toml for cargo-deny

Parity corpus
  tests/parity/recorder.py + scripts/record_fixtures.py
  125 recorded fixtures across 5 leaf transforms (ccr, tokenizer,
  log_compressor, diff_compressor, cache_aligner)

Docs
  RUST_DEV.md — developer setup and workspace reference
  docs/spec/022-rust-migration.md — migration plan and stage breakdown

.gitignore: whitelist scripts/record_fixtures.py; ignore target/

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 13:39:48 -07:00