mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9355ec7019 |
fix(diff): renumber hunk headers after trimming context (#4319)
## Description `DiffCompressor` trims each hunk to `max_context_lines` (2) around its changes but keeps the hunk's original `@@ -a,b +c,d @@` header. The header no longer describes the lines under it: - Leading context is dropped first, so every line number read off a hunk is off by the lines cut (one with git's default `-U3`). - Context dropped between two changes more than 4 lines apart leaves them next to each other. Nothing marks the gap, and every line after it is numbered wrong. A 400-line file with lines 10 and 17 changed, `git diff` (73 lines) through `DiffCompressor().compress()` on 0.41.0: ```text @@ -7,14 +7,14 @@ line 6 line 8 line 9 -line 10 +line 10 changed line 11 line 12 line 15 line 16 -line 17 +line 17 changed ``` By the header, `line 8` is line 7 and `-line 17` is line 14. Anything that maps these hunks to file lines (a model writing an edit, `git apply`) gets the wrong place. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `reduce_context` splits a trimmed hunk at each gap and recomputes every header from the original line numbers, one range per parent for combined diffs, formatted the way git writes them. The section heading is carried over unchanged. - A hunk that loses no lines keeps its header byte-for-byte. - Lines after the extent a header declares (the empty string from a trailing newline, a format-patch `-- ` signature) are not numbered and no longer count as changes. - A format-patch signature and what follows it stay whatever the context limit. - A hunk with edits past the extent its header declares is malformed input and passes through as it came. - `hunks_kept` / `hunks_removed` still count the original hunks, so a split can't underflow `hunks_removed`. - Five `diff_compressor` parity fixtures recorded the old headers. In their outputs 33 hunks had header counts that disagree with their own lines; with this change there are none. Only `output.compressed` changes in those files. ## Testing - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo test -p headroom-core --lib test result: ok. 979 passed; 0 failed; 1 ignored # new tests against the previous commit: left: ["@@ -7,14 +7,14 @@ l6", "@@ -23,6 +23,7 @@ l22"] right: ["@@ -8,5 +8,5 @@ l6", "@@ -15,5 +15,5 @@ l6", "@@ -24,4 +24,5 @@ l22"] test result: FAILED. 0 passed; 2 failed $ cargo run -p headroom-parity -- run --fixtures tests/parity/fixtures [diff_compressor ] total=27 matched=27 skipped=0 diffed=0 $ pytest <the 12 test files that contain hunk headers> tests/test_compress*.py tests/test_transforms/test_diff_compressor_rust_parity.py tests/parity 754 passed, 3 skipped $ cargo clippy -p headroom-core --lib --tests -- -D warnings # clean ``` ## Real Behavior Proof - Environment: Ubuntu, Python 3.12, headroom-ai 0.41.0 from PyPI vs this branch built with `pip install -e .` - Exact command / steps: for each commit, `git diff c~1 c` through `DiffCompressor(DiffCompressorConfig(max_hunks_per_file=10**6, max_files=10**6)).compress(...)`. Then compare every hunk (header ranges and lines) with `git diff -U2 c~1 c`. Last 300 non-merge commits of this repo, plus 200 of another Python repo. Merge commits the same way with `git show -c` / `-c -U2`. - Observed result: ```text 0.41.0, this repo: compressed=276 match_git_U2=3 branch, this repo: compressed=276 match_git_U2=276 branch, second repo: compressed=163 match_git_U2=163 branch, git show -c: compressed=36 match=36 ``` Same example after the change: ```text @@ -8,5 +8,5 @@ line 6 line 8 line 9 -line 10 +line 10 changed line 11 line 12 @@ -15,5 +15,5 @@ line 6 line 15 ``` - Not fully matched: `git show --cc`. 8 of 11 merges match `--cc -U2`. In the other 3, git hides more single-parent hunks when the context shrinks, which a context trim doesn't try to copy. The compressor keeps every change line of the `--cc` input there. ## Runtime Rollout Safety N/A: no rollout-managed feature, no config change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes The section heading after the second `@@` is kept from the original hunk. git recomputes it for each hunk, but that depends on the file's language driver, and the original heading still names the enclosing context. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4319" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
eef44f60cc |
fix(kompress): read the pinned snapshot from Python's one HF cache in the native proxy (#4438)
## Description **Stacked on #4190.** This branch contains #4190's commits (it reuses #4190's Rust HF-cache resolver). Please review only the five commits listed under Changes Made. Once #4190 merges, I'll rebase onto main and the diff will shrink to those commits. **Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now linear, with no merge commits: - #4190's first three commits: `e173e100`, `344cf365`, `41918cf0`. - This PR's five commits (SHAs updated under Changes Made). - `d9f16d39` fix(relevance): keep an empty HF cache variable, as huggingface_hub does. This is #4190's last commit, applied to `crate::hf_cache` since this PR moves the resolver there. It replaces the earlier merge `a2e89118`. What the rebase changed: - The #4152 commit (`cb1c3a33`) is gone from the branch: main already has it as `f380692b`. - The main-sync merges (`7e17a2d3`, `6904fd19`, `8be25204`, `79e3a5cb`, `19b1adc5`, and #4190's merges) are linearized away. - The Rust embedder conflicts with main's batch-size cap (#4507) were re-resolved to the same content as before: every file this PR touches is byte-identical between the previous head `19b1adc5` and the new head `d9f16d39` (`git diff |
||
|
|
93279f3208 |
fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description Closes #3697. An unchanged SmartCrusher result now keeps the original bytes and reports no structural compression. Formatting-only savings come from the existing Rust JsonMinifier, not from pretending the JSON value changed. The minifier preserves duplicate keys, order, escapes and number literals. A mixed result cannot use one section's CCR marker to hide unmarked loss in another. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace savings through JsonMinifier in Python and the eligible native live zone. - Keep all records and JSON literals; do not store originals or inject CCR markers for lossless minification. - Preserve mixed routing and reject uncovered lossy sections in both tool-result formats. Explicit no-CCR behavior is unchanged. - Replace incidental forwarding tests with native minification and complete-record regressions; document the distinction. ## Testing - [ ] Entire Python suite — not claimed. - [x] Four consumer/native modules: `python -m pytest -q tests/test_recursive_json.py tests/test_transforms_content_router.py tests/test_transforms/test_smart_crusher_rust_parity.py tests/test_transforms/test_smart_crusher_lossless_default.py` - [x] Changed-file Ruff checks; normal commit hooks, including mypy and commitlint. - [x] Manual public-API runs, with and without the HTML extra. ### Test Output Consumer/native modules: ```text collected 136 items ============================= 136 passed in 36.43s ============================= ``` Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native build, Python checks, commitlint): ```text ======================== 180 passed, 1 warning in 8.35s ======================== ✅ ci-precheck PASSED — safe to push. To https://github.com/oxura/headroom.git 64688217..0002af3e HEAD -> fix/3697-unchanged-json-passthrough ``` The first push stopped on an inherited commit footer because `origin/main` was stale. Fetching the actual upstream ref fixed the lint range; the normal push then passed. No source merge, lint-rule change or hook bypass. Hosted checks on `0002af3e` now pass: all four Python shards, Windows permissions, smoke tests, native wrappers, parity and Docker e2e. [Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks). ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual extension built from this checkout. No provider billing calls. - Exact command / steps: the earlier localhost `/v1/compress` and TransformPipeline runs exercised the original fix. For this follow-up, call `ContentRouter.compress` on a data-only HTML page containing 60 complete records, with `smart_crusher_lossless_only=True`, `smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run it once without trafilatura, then with the locked 2.0.0 extra. The new public-API test contains the full input/config. Separately call `route_embedded_json` on a compact array followed by a padded array, with Unicode and CRLF surroundings. - Observed result: both HTML modes keep all 60 records and the surrounding markup, with no unrecoverable section. The unchanged first JSON span does not stop the second span's compaction; surrounding text stays exact. Final Codecov is 96%; the two remaining lines are the deferred HTML fallback return. The real no-HTML-extra API run executed both, and local coverage also exercises the unchanged-span branch. - Not tested: live provider acceptance, invoices or production cache hit rates. No native eligibility or admission gates changed. ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing JSON routing. - Minimum rollout channel: unchanged. - Stable/default behavior changed: eligible formatting savings return as json_minifier; standalone SmartCrusher remains byte-exact on no-ops. - Kill switch / disable path: existing routing configuration and HEADROOM_LIVE_ZONE_DISABLE_ARMS. - Unsafe override required: no. - Qualification impact: no-op, formatting, structural and CCR paths exercised separately. - Rollback path: revert the minifier integration; no persistence migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes Addresses the [minifier review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285). The maintainer’s main merge is retained. The latest follow-up only changes consumer tests. AI-assisted; no subagents. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
9bb4b130ff |
fix(relevance): load the Rust default embedder from Python's pinned snapshot (#4190)
## Description Parent status: #4152 has merged. The pinned-loader and batch-cap merge resolution was reviewed at |
||
|
|
b9049d703a |
deps: Bump the cargo-minor-patch group across 1 directory with 12 updates (#4553)
## Purpose Retain the complete selected 12-version Cargo group and qualify its compatibility with Headroom. The AWS release moves its config, credential, Smithy runtime/schema and SDK dependencies together to a new coordinated compatibility line; upstream explicitly warns against upgrading only part of that line. Existing proxy credential resolution and SigV4 signing must continue to work with that coherent set. This is a compatibility-qualified grouped update. Companion updates are not individually claimed to fix a Headroom bug where upstream only added unused APIs. ## Selected versions and project relevance | Package | Selected update | Relevance and qualification | | --- | --- | --- | | tokio | 1.53.3 → 1.54.1 | Proxy HTTP/SSE/WebSocket execution and timers; includes upstream timer-handle regression revert. Local native proxy/SSE/WSS tests exercise runtime. | | aws-config | 1.12.0 → 1.13.0 | Required companion for new AWS schema/runtime compatibility line; production environment credential chain exercised with explicit fake credentials. | | aws-credential-types | 1.3.0 → 1.3.1 | Same coordinated AWS line, production credential conversion and SigV4 signing exercised. | | aws-smithy-runtime-api | 1.19.0 → 1.19.2 | Same coordinated AWS line; timeout finalizer behavior changed upstream. Local signing/wire tests do not establish SSO/STS timeout behavior. | | sha2 | 0.11.0 → 0.11.1 | Existing cache/SigV4-related payload hashing; upstream unsafe-op lint fix. Local signed payload SHA256 verified; no claimed cryptographic vulnerability fixed. | | redis | 1.7.1 → 1.8.0 | Optional multiworker CCR backend; upstream TLS error/secret redaction and subscription correctness fixes. Production backend roundtrip exercised against local RESP fixture; cluster/pubsub are not Headroom backend usage. | | tree-sitter | 0.27.0 → 0.27.1 | Code compressor parser/error recovery fixes. Reconstructed 9 samples × 8 pinned languages compare old/new node kinds, complete row/column spans, ERROR/MISSING: 72/72 identical. | | uuid | 1.28.0 → 1.29.0 | Full-group companion target; new NonNilUuid constructors are unused. Existing request-ID generation exercised through real proxy calls and both response x-request-id values parsed as UUIDv4; no invented bug/security rationale. | | crc32fast | 1.5.2 → 1.5.3 | Full-group companion target; upstream total-length API is unused. Existing Bedrock EventStream checksum/framing path exercised by streaming integration suite. | | rustls-native-certs | 0.8.4 → 0.8.5 | Native trust-store handling; avoids duplicate Unix certificate-file reads. Windows build and WSS fail-closed behavior exercised; Unix store traversal not locally exercised. | | webpki-roots | 1.0.10 → 1.0.11 | Updated bundled CA trust set (CCADB CA removals); WSS verification tests exercise existing explicit TLS configuration. No public CA endpoint contacted. | | cc | 1.8.0 → 1.10.0 | Native C/C++ dependency builds; upstream fixes MSVC archive object paths and cl.exe wrapper handling. Actual clean Windows MSVC native dependency compilation performed. | ## Maintenance and install surface The packages remain maintained by their existing upstream projects: Tokio, AWS/Smithy, RustCrypto, redis-rs, tree-sitter, uuid-rs, srijs/rust-crc32fast, rustls and rust-lang/cc-rs. Release/compare links in the original Dependabot body identify the reviewed releases; AWS authoritative release notes: https://github.com/awslabs/aws-sdk-rust/blob/main/CHANGELOG.md (October 20–30, 2026). UUID comparison confirms its change only adds NonNilUuid constructors; it is not presented as a security fix. No new package source/registry is introduced. Cargo resolves registry downloads at build time; cc executes the native compiler for existing C/C++ dependencies (tree-sitter grammars and other existing native crates). The lock update changes the transitive AWS SDK/schema/runtime family together. Existing ONNX runtime loading/model behavior is not altered by this PR. Redis is feature gated and adds runtime network only when explicitly configured. AWS credentials can normally resolve through environment, profiles, SSO/STS or metadata; qualification uses only explicit fake environment credentials, NUL config files and disabled IMDS. Normal configured proxy upstream traffic and native trust-store reads remain existing runtime surfaces. Pinned qualification compiler: Rust 1.95.0 on Windows/MSVC. Resolved proxy AWS floor is 1.94.1; core/Python/parity graph includes tree-sitter-language floor 1.90. Both already existed in the base lock, so no global rust-version change is made; inherited workspace rust-version=1.80 is not a valid full-workspace source-build promise. Fresh base/head Windows-filtered metadata comparison confirms unchanged floors: core/Python/parity1.90,proxy1.94.1,simulators1.86(ICU). These are dependency-declared effective floors, not measured lower-toolchain builds. No lower-toolchain build was attempted. ## Real behavior proof Windows, Rust1.95.0 x86_64-pc-windows-msvc, isolated worktree, dedicated CARGO_TARGET_DIR, CARGO_BUILD_JOBS=2. No actual AWS/provider credentials, profiles, cloud inference or model downloads used. - `cargo test --locked -p headroom-core -p headroom-proxy` with explicit existing x64 ONNX Runtime DLL (`ORT_DYLIB_PATH`) and `HEADROOM_OFFLINE=1`, `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`: **1,659 passed, 0 failed, 3 ignored; exit 0**, across 61 test/doc-test executables. Initial run without the Windows loader path was diagnosed loader-blocked and terminated; its separate failed log is retained. - `cargo test --locked -p headroom-proxy --test integration_bedrock_invoke --test integration_bedrock_streaming --test integration_ws_tls --test integration_sse`: 23passed,0failed (8invoke,11streaming,2SSE,2WSS). - `python G:/git/headroom/.codex/dep4553-runtime.py`: built headroom-proxy.exe on ephemeral ports with compression disabled; passthrough and native Bedrock routes each return local200, both bodies byte equal; AWS environment provider loaded fake credentials and SigV4 Credential prefix/payload SHA256 checked; both generated response x-request-id headers validated UUIDv4. Synthetic wire fixture, not real AWS service emulation. - `python G:/git/headroom/.codex/dep4553-redis-wire.py`: temporary integration test calls production RedisCcrStore using redis1.8.0; local RESP fixture validates PING,miss,UTF8 payload write/read,TTL refresh and replacement; 1passed. Synthetic protocol server; supplemented by `python G:/git/headroom/.codex/dep4553-redis-real.py`: real Redis7Alpine ephemeral loopback Docker service with persistence disabled; production backend2testsPASS,including real TTLrefresh andexpiry. Owned --rmcontainer stopped automatically. TLS/cluster/failover/pubsub not tested. - Standalone canary under `.codex/dep4553-canary`: exact engine versions0.27.0 and0.27.1, same eight pinned grammars,72/72 AST snapshots identical. Historical canary source was absent; reconstructed inputs are recorded, including second distinct valid fixture per language (explicit Cfunctionpointer sample), CRLF,truncated,error,concatenated andUnicode cases. - Isolated Python3.12 env: recorder regression9tests passed; Ruff checks/format passed. Original official fixture generator reproduced KeyError(bash). Minimal3line helper fix raises ValueError, matching existing detector unsupported-language contract; all8pinned grammars remain covered. Official generator then recorded30fixtures. - Generator revealed12preexisting Python-reference fixture differences. Exact main74277d47 source extracted independently, same Python environment +same helper fix:30/30 regenerated semantic fixture objects match PR source, excluding recorded_at only. Preserve original repo fixture bytes; do not mask unrelated Python-reference drift by replacing goldens. Baseline/regenerated files and comparisons retained as artifacts. ## Remaining gates / limitations Initial broad Cargo suite was loader-blocked: source live_zone.rs documents indefinite Windows initialization without ORT_DYLIB_PATH. Controlled termination of exact owned testPID717412 produced authoritative exit1 for original session37711. Distinct corrected configuration uses existing verified x64ORT1.26.0 DLL(API24), explicit ORT_DYLIB_PATH and offline guards; corrected full core+proxy qualification completed exit0,1659passed,0failed,3ignored. This is an environment diagnosis, not a demonstrated dependency regression. Hosted checks must be rerun on the exact final remote SHA; prior shard4 red is not claimed fixed. Canonical fixture rerecord was executed and its baseline-identical semantic drift documented, not silently normalized away. No Linux/macOS build, lower MSRV build, real AWS SSO/STS/IMDS, public CA endpoint, Redis TLS/cluster/failover, production provider inference or model download was tested. No remote push, PR edit/review or merge performed by this qualification task. ## Public reproduction The committed standard-library harness reproduces the full 72-case parser comparison and launches the actual native proxy against credentialless loopback fixtures. It stores both complete AST snapshots and inputs when `--artifacts-dir` is provided. No private `.codex` artifacts are needed. Windows, from a clean checkout with Rust 1.95/MSVC and Python 3.11+: ```powershell cargo build --locked -p headroom-proxy --target-dir target/qualification python scripts/qualify_cargo_dependencies.py all --binary target/qualification/debug/headroom-proxy.exe --artifacts-dir target/qualification-proof --canary-target-dir target/qualification-canary ``` Linux/macOS use the same commands with the binary path `target/qualification/debug/headroom-proxy` (omit `.exe`). Those platforms are reproduction instructions, not platforms locally qualified here. The canary compiles standalone exact engine versions0.27.0/0.27.1 against all eight production grammar pins, so Cargo registry downloads/native compilers may be needed. Runtime checks use fake AWS credentials, platform null config files, disabled IMDS, offline model guards, compression off, and local HTTP endpoints only. The source-comment-required Python fixture rerecord is independently reproducible: ```text python -m pip install tree-sitter==0.25.2 tree-sitter-python==0.25.0 tree-sitter-javascript==0.25.0 tree-sitter-typescript==0.23.2 tree-sitter-go==0.25.0 tree-sitter-rust==0.24.2 tree-sitter-java==0.23.5 tree-sitter-c==0.24.2 tree-sitter-cpp==0.23.4 python scripts/record_code_compressor_fixtures.py ``` Use an isolated development Python environment with the project development dependencies installed. The recorder helper fix in this PR preserves its existing eight-grammar scope; unsupported detector candidates are skipped through the existing ValueError contract. Regeneration currently exposes baseline Python-reference drift as documented above; inspect semantic differences rather than replacing goldens blindly. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
023dd1bc54 |
fix(logs): count omitted levels from dropped lines (#4141)
## Description Closes #4070. The omission marker previously counted severities in the whole input, including retained errors. It now counts ERROR/FAIL/WARN/INFO only on dropped line identities. Selection and whole-input result.stats are unchanged. If no dropped line has a recognized severity, the marker is simply `[N lines omitted]`, not a severity breakdown. CCR retains the exact original. ## Type of Change - [x] Bug fix (non-breaking) - [x] Documentation update ## Changes Made - Keep the existing retained-line identity set and a single-pass fixed stack histogram; no omitted-line collection or repeated full scans. - Restore complete structured-marker assertions in Rust and Python, including bounded failure names, repeated ids, whole-input totals, and non-severity omissions. - Restore parity fixture `3bc015edc0a36387.json` using its original input, explicit configuration and input digest. Output and compression ratio were re-recorded through the freshly built native extension rather than manually copied. New runtime-frame defaults remain implicit in the legacy configuration. - Remove the constructor-only result-copy test; retain consumer behavior regressions. - Document omitted-versus-input counts and the exact bare-marker form in the existing text/log guide. No manual changelog change. ## Testing - [x] Scoped Rust and Python tests passed - [x] Normal touched-file Ruff and repository mypy hooks passed - [x] Actual native recorder and parity CLI exercised Head `6c472f6b` includes upstream main `f1bc2609`. - Focused Rust log-compressor tests: **41 passed**. - Python public-transform tests: **11 passed**, using the freshly built native extension. - Actual parity CLI: **20 matched, 0 skipped, 0 differences**, including the restored original-input fixture: ```text cargo run --locked -p headroom-parity --bin parity-run -- run --fixtures tests/parity/fixtures --only log_compressor [log_compressor ] total=20 matched=20 skipped=0 diffed=0 ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.14, native extension freshly built from this worktree. - Exact command / steps: Re-record the original log fixture through LogCompressor.compress with its original explicit configuration, then run the actual parity CLI shown above. - Observed result: The real native recorder produced `[297 lines omitted: 297 INFO]` while whole-input statistics remained 300 INFO / one ERROR / 305 total / eight selected. The exact original was unchanged and the measured compression ratio is `0.024777201057682892`. The original fixture digest remains `3bc015edc0a36387194d075139845c120a308afb314bae436ab454906b431d84`. Normal commit and full pre-push hooks passed without bypass: Rust formatting, warnings-as-errors Clippy, workspace tests, native editable rebuild/import verification, **181 critical Python tests**, and branch-range commitlint. The critical Python run reported five dependency/provider warnings, no failures. These local results do not claim that hosted checks have finished. - Not tested: New Windows/macOS runs, live-provider sessions and the full Python suite. Earlier before/after actual-native CCR qualification showed the retained ERROR was not counted as omitted, whole-input totals stayed unchanged, and retrieval recovered the exact original. That historical implementation evidence is in the review discussion; the explicit results above qualify this follow-up head. No new Windows/macOS or live-provider qualification is claimed. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing LogCompressor and CCR; no new feature or gate. - Minimum rollout channel: Unchanged (stable); no channel promotion. - Stable/default behavior changed: Yes, omitted-severity accounting is corrected; selection and stored originals remain unchanged. - Kill switch / disable path: ContentRouterConfig(enable_log_compressor=False) disables the log route; existing CCR settings disable retrieval separately. - Unsafe override required: No. - Qualification impact: The scoped native/Python/parity proof and required normal push gate above cover this marker path. No new platform/live-provider qualification is claimed. - Rollback path: Revert the fix; no schema or stored-original migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review - [x] Existing style and documentation conventions followed - [x] Consumer-visible regressions and actual native/CLI paths exercised - [x] Normal hooks passed; no bypass - [x] No manual CHANGELOG edit AI-assisted: omp / GPT-6.1 Sol. This review follow-up was completed without subagents. No manual human-validation or full Python-suite claim; scoped tests and the required push gate are reported separately. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c1fc84ade6 |
fix(smart-crusher): disclose dropped scalar-array items with a retrievable ccr sentinel (#3692)
## Description
On default config, `SmartCrusher` silently truncated long
string/number/mixed arrays down to `max_items_after_crush` and emitted
no disclosure at all — valid JSON with no marker, no sentinel, and
untouched sibling count fields. This contradicts the repo's own "Safety
first: never drop user/assistant content" principle.
The dict-array path already disclosed drops via a `{"_ccr_dropped":
...}` sentinel; scalar arrays had no equivalent. This PR gives them one:
when the `StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, it appends `"… N more items <<ccr:HASH
N_items_offloaded>>"` and stores the full original array in the CCR
store (same SHA256-12 hashing the Python bridge and retrieval use), so
the marker round-trips through `headroom_retrieve`. No-op when nothing
is dropped or the marker gate is off.
Closes #3650
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
| File | What changed |
|---|---|
| `crates/headroom-core/src/transforms/smart_crusher/crusher.rs` | New
`append_scalar_drop_sentinel()`: when
`StringArray`/`NumberArray`/`MixedArray` arms drop items and
`enable_ccr_marker` is on, append the sentinel and store the full
original array in the CCR store. No-op when nothing is dropped or the
marker gate is off. |
| `headroom/transforms/smart_crusher.py` | `is_ccr_sentinel()` now
recognizes the string sentinel shape (previously only the dict shape),
so downstream code can filter it like the dict sentinel; docstring
updates. |
| `tests/test_transforms/test_smart_crusher_issue_3650.py` | New
regression tests: sentinel is emitted with the right drop count, carries
a retrievable CCR marker whose hash resolves to the full original array,
and is suppressed when the marker gate is off. |
| `crates/.../smart_crusher/crusher.rs` (tests module) | Rust regression
tests mirroring the above (sentinel emission, marker retrieval of the
full 120-item array, gate-off suppression). |
| `tests/parity/fixtures/smart_crusher/*.json` (3 files) | Recorded
outputs updated — strategy strings now reflect the trailing sentinel. |
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
cargo test --lib transforms::smart_crusher → 337 passed, 0 failed
(includes the new #3650 Rust regression tests)
pytest tests/test_transforms/test_smart_crusher_issue_3650.py \
tests/test_transforms/test_smart_crusher_bugs.py \
tests/test_transforms/test_smart_crusher_ccr_roundtrip.py \
tests/test_transforms/test_smart_crusher_rust_parity.py \
tests/test_transforms/test_content_router.py → 106 passed
(includes 18 Rust-parity fixture tests)
ruff check and ruff format --check on the touched Python files → clean
```
## Real Behavior Proof
- Environment: Linux container, Python 3.12, editable install from
`main` with the maturin-built native `_core` module (no ONNX model
involved — this path is pure Rust crusher).
- Exact command / steps: the repro snippet from #3650, verbatim
(`ContentRouter(ContentRouterConfig())`, 120 slugs + `"total": 120`).
- Observed result: **before** → 16 of 120 kept, `"<<ccr:" in out`
False; **after** → 16 kept + `'… 104 more items <<ccr:4f03caae30bb
104_items_offloaded>>'`, marker present, and the marker's hash
retrieves all 120 originals from the CCR store. The `total` sibling
field is left untouched — the truncation is now disclosed in-band next
to it rather than silently contradicting it.
- Not tested: the ONNX/Kompress model path from #3673 (a separate issue;
this fix is model-independent), Windows/macOS install flows, and the
`lossless_only=True` path (unchanged: nothing dropped there, so no
sentinel is emitted).
## Runtime Rollout Safety
- Rollout-managed feature(s): CCR marker gate (`enable_ccr_marker`)
- Minimum rollout channel: n/a (open-source library; applies to default
config where the marker gate is on)
- Stable/default behavior changed: Yes — truncated scalar arrays now
carry a trailing sentinel string instead of being silently shortened;
sibling count fields untouched
- Kill switch / disable path: set `enable_ccr_marker=False` — no
sentinel emitted, gate-off path unchanged
- Unsafe override required: No
- Qualification impact: None expected
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — library behavior; observed output captured under Real Behavior
Proof.
## Additional Notes
Raphael Malikian — rtmalikian@gmail.com
If you'd like similar help with your own project or codebase, I'm open
to collaborate — feel free to reach out.
*This PR was prepared with AI assistance and reviewed by the author
before submission.*
---------
Signed-off-by: Raphael Malikian <rtmalikian@gmail.com>
|
||
|
|
96dfc7005f |
chore: finish the org and docs-domain migration (#3389)
Two stale identities left over from moves that were only half applied. ## GitHub org `install/state.py` already migrated Docker images to `ghcr.io/headroomlabs-ai/headroom` and carries a `_DEPRECATED_IMAGE_REPO` constant that rewrites the old path on load — so this move was decided long ago. Everything else still said `chopratejas/headroom`. Rewritten across packaging (`pyproject.toml`, `Cargo.toml`), both plugin marketplaces, the agent-hooks manifests, install scripts, CI templates, crate docs and tests. **Two deliberate exclusions:** - **`ghcr.io/chopratejas/headroom` is untouched everywhere.** It is the *input* to `_migrate_deprecated_image()` and to the tests covering it. Rewriting it would silently disable the migration. - **`CHANGELOG.md` keeps its 238 historical references.** GitHub's rename redirect resolves them, and rewriting would bury this diff in noise. **HuggingFace model ids are NOT affected and were never in scope.** `chopratejas/kompress-base`, `kompress-v2-base`, `kompress-finance`, `technique-router`, `technique-router-onnx` and `siglip-image-encoder-onnx` are account-scoped identifiers on a different service. All six were validated against the HF API before this change: | Model | HTTP | |---|---| | `kompress-base` | 200 | | `kompress-v2-base` | 200 | | `technique-router` | 200 | | `technique-router-onnx` | 200 | | `siglip-image-encoder-onnx` | 200 | | `kompress-finance` | **401 (private)** | The 401 is unrelated to this migration, but worth knowing: `kompress_compressor.py:1255` references it, so that path is not reachable by an OSS user without access. **One functional fix, not cosmetic:** `_marketplace_source()` (`headroom/cli/init.py`) returned the old org as its fallback, so plugin installs were pointed at the retired repo. ## Docs domain The site moved to `docs.headroomlabs.ai`, but eleven references still named the vercel.app host — including **`llms.txt` (27 links)**, which is served at `docs.headroomlabs.ai/llms.txt` and read by LLM crawlers, the **dashboard's own Documentation link**, and a `headroom wrap` help message. The docs app's canonical-URL fallbacks were fixed separately in #3388. ## Also folded in `RUST_DEV.md` corrections found while verifying the above: - `make build-wheel` points at `Cargo.toml`, not `pyproject.toml` - the Rust toolchain is pinned, not tracking `stable` - `headroom-simulators` was missing from the workspace listing - the Python `CompressionStore` defaults to **SQLite**, not in-memory, when `HEADROOM_CCR_BACKEND` is unset (verified by execution) — this one matters for multi-worker reasoning And the install scripts pointed at `docs/docker-install.md`, a path that no longer exists; they now point at the live docs page. ## Verification 297 tests pass, including the plugin-manifest, install-state and release-workflow suites that assert on these exact strings. `ruff check` clean. Unrelated lockfile churn (`uv.lock`, both `package-lock.json`) is deliberately excluded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
e530de5ad2 |
feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154)
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for Python, JavaScript, TypeScript, Go, Rust, Java, C and C++. Parity-only, like #1153. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs, and live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched and no Python source changes, so the engine is unreachable from the shipped package. #1155 wires it into live-zone dispatch. Every grammar is pinned with '=' to the exact version of the corresponding Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means the same grammar.js, hence the same generated parser.c, hence node-for-node identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8 languages confirmed identical node-type and line-span trees at these pins; bumping any pin requires re-running it and re-recording the fixtures. Ships 30 recorded parity fixtures, a CodeCompressorComparator in headroom-parity, and scripts/record_code_compressor_fixtures.py. Verified byte-identical to the recorded Python output: [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0 Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped (cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under ONNX Runtime 1.24.4 (see #2591). Also verified cargo check -p headroom-core --no-default-features passes, so the static-musl path stays intact. |
||
|
|
83e27e5036 |
feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the kompress-v2-base ONNX model through ort with a cache-only loader that never touches the network. Parity-only. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve prose compression. The pyo3 bridge is untouched and no Python source changes, so the new engine is unreachable from the shipped package. #1155 wires it up. Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and scripts/record_kompress_fixtures.py. Verified byte-identical to the recorded Python output: [kompress] total=21 matched=21 skipped=0 diffed=0 That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead of erroring, which is why these fixtures had never been run. In CI the model is absent from the HF cache, so the comparator errors and the fixtures report Skipped rather than hanging. Also gates the module behind the ml feature, matching magika_detector: kompress.rs uses ort, which is optional = true, so an unconditional pub mod broke cargo check --no-default-features (the static-musl path). CI does not catch that class of break because cargo test --workspace only builds default features. |
||
|
|
4035c04187 |
feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)
## Description `TextCrusher` (the native extractive prose compressor added in #1171) only handled ASCII: `split_segments` split on `.!?`+whitespace and `tokens` split on whitespace/alphanumeric runs. CJK (Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a whole CJK paragraph collapsed into **one segment / one token** — it passed through at ~0% compression, and BM25 relevance + salience scored zero terms. This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU (`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path, with a length fallback for terminator-sparse runs, a local BM25 over the ICU word tokens, and ICU-token salience. Dispatch is on **content only**, so pure-ASCII text is byte-identical to before — the shared `BM25Scorer` and the ASCII path are untouched. It also adds a committed, reproducible answer-retention eval (`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI regression gate, so the improvement below is permanently verifiable rather than a one-off measurement. Extends #1171. ## Type of Change - [x] Bug fix (CJK passed through near-uncompressed) - [x] New feature (CJK segmentation / relevance support) - [x] Performance improvement (CJK now compresses; ICU segmenters cached, not rebuilt per call) ## Changes Made - `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK punctuation, full/half-width forms). - `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory length fallback (whitespace / CJK punctuation / hard cap) for terminator-sparse runs; ASCII path unchanged. - `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path unchanged. - `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII `BM25Scorer` scores zero terms for CJK and is parity-locked, so this is an intentional separate scorer (documented in code). - CJK salience uses ICU tokens (whitespace-split gave one giant "word" → zero salience). - `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for space-free text. - ICU segmenters resolved once in `static LazyLock` (compiled_data is static) instead of rebuilt per call. - New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency below). - `benchmarks/i18n_compression_eval.py` + `tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko answer-retention eval — a deterministic needle CI gate (always-runs, no external data), real-transcript fidelity with CJK-aware salient, and optional `multi-wiki-qa` natural-data retention (loaded via the `[evals]` `datasets` extra, skipped if absent; data never vendored — CC-BY-NC-SA). ## Testing - [x] Unit tests pass (`pytest` + `cargo test`) - [x] Linting passes (`ruff check`/`format` on the new eval + test — clean) - [ ] Type checking passes (`mypy headroom`) — N/A, the only Python added is a benchmark + test, not `headroom/` source - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher running 12 tests test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py 15 passed $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py 6 passed # deterministic zh/ja/ko needle CI gate $ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py # both clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv, `headroom-core` built via `uv pip install -e .` (maturin), branch `feat/cjk-text-compression`. - Exact command / steps: built `_core`, then ran a mixed Chinese+Japanese doc (no spaces, `。` terminators) through `TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated answer-retention on the public CMRC2018 Chinese QA dev set (bury the gold-answer paragraph among 25 distractors, query = the question, compress to 30%, check the gold answer survives), and end-to-end through `ContentRouter`. - Observed result: a mixed Chinese+Japanese doc compressed 189 → 78 tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence surviving — before this change the same doc was a single segment → 100% passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention under 30% compression rose 34% → 93% (multiple seeds). End-to-end through `ContentRouter` on real CJK content, aggregate savings rose 16% → 40%. Pure-ASCII (English) output stayed byte-identical (the English parity fixtures did not move). Demo terminal output: ```text ORIGINAL tokens= 189 chars=189 COMPRESS tokens= 78 ratio=0.41 segments kept 3/8 QUERY-RELEVANT sentence survived: True --- compressed output (verbatim kept CJK sentences) --- 认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。 请求重试使用指数退避并设置最大次数上限。 数据备份每天凌晨执行并保留最近三十天的快照。 ``` The committed eval now demonstrates this across all three CJK languages. The deterministic needle gate (in CI via `tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has TextCrusher keep the query-relevant needle while truncate/random drop it in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang), query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs **25–41%** for the truncate/random baselines: ```text === Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3) === lang text_crusher truncate random zh-cn 74% 25% 38% ja 70% 31% 39% ko 50% 26% 41% ``` Korean is measurably weaker (ICU has no Korean dictionary and falls back to UAX#29 word-breaking) — still well above baselines, and scoped as a follow-up. - Not tested: the live proxy HTTP path (validated at the `ContentRouter` / `TextCrusher` layer, not via a running proxy); no-space Korean (standard Korean is space-delimited and is covered); non-CJK SE-Asian scripts (out of scope). ## Dependency (per CONTRIBUTING supply-chain policy) `icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`: - **Why this package (vs. ourselves / existing deps):** CJK needs dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91% bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X covers zh/ja/ko in one crate. The existing `unicode-segmentation` does UAX#29 only (no CJK dictionary), so it can't word-segment space-free CJK. - **Who maintains it:** the official `unicode-org` ICU4X project; active release cadence (2.2 in 2025); no known CVEs. - **Install surface:** ~13 new pure-Rust crates, no build scripts, no native code, no build/runtime network. `compiled_data` bundles locale data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled — LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in `libm` for nothing. - **Why this version:** 2.x is the stabilized ICU4X API (1.x used a different data-provider model); floored at 2.2 (Cargo.lock pins the patch) since segmenter boundaries are observable in output and bumps should be deliberate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings (clippy + fmt clean) - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes - **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with `headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a separate local scorer because the shared one's tokenizer is ASCII-only. The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper over `_core`), so there is no Python mirror to keep in sync; the parity fixtures stay green (only the CJK `unicode` fixture was re-recorded, intentionally; English fixtures unchanged). - **Known by-design gap (not a bug):** CJK content + a pure-ASCII query yields no token overlap, so relevance falls back to recency + salience (cross-script query matching is unsupported). - The Python added is a benchmark (`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/` runtime source — both are `ruff`-clean; `mypy headroom` is unaffected. - **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa` (CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if absent — the dataset is never vendored into the repo, and the always-run CI gate (Part C) uses only our own deterministic data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
6c68ff4e9f |
perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description On a cold-start large context, kompress (ModernBERT ONNX) runs **synchronously on the request thread** — ~200–300s for ~1M tokens. It blows the 30s compression budget, leaks a non-preemptible worker, and cascades (executor saturation → queue timeouts on healthy requests); on timeout the request is forwarded **uncompressed** after eating 30s. This adds four layered, **default-off, fail-open** mitigations so the request path is never blocked on ML compression. Closes #1171 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000): route oversized text away from ModernBERT (→ LogCompressor / TextCrusher / passthrough) at the single `_try_ml_compressor` boundary. - **Phase 1 — cooperative deadline** (`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run self-terminates at the next chunk boundary past the budget, keeping the unprocessed tail verbatim. - **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native Rust** extractive prose compressor in `crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as `headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25, and ships record/replay parity fixtures (mirroring the SmartCrusher Rust-core + Python-shim pattern). - **Phase 3 — off-path compression** (`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately and compress in a per-process background drain; a byte-identical cache hit on a later turn means the request never blocks on ML. - Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG entry, and docstrings documenting the fail-open limits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`, new modules) - [x] New tests added for new functionality - [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed on real traffic in earlier iterations; Phase 3 off-path is unit- + byte-identity-tested, not yet live-validated) ### Test Output ```text $ pytest tests/test_transforms/ tests/test_cache/ \ tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q 501 passed, 37 skipped in 40.33s $ cargo test -p headroom-core --lib text_crusher test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out $ ruff check <changed files> All checks passed! $ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py Success: no issues found in 2 source files ``` New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS + TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3 byte-identity round-trip; TextCrusher unit + parity. ## Real Behavior Proof - Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv pip install -e .`. - Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy` commands shown under Test Output; quality eval `python benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`. - Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on changed/new modules. Quality eval: TextCrusher keeps ~94% of buried SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT takes minutes (fast-vs-slow contrast, not a same-input run). - Not tested: Phase 3 off-path on live traffic; multi-worker (per-process by design — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - **All four features are off by default and fail-open** — with the env flags unset the paths are no-ops for realistic inputs; on any error the request is forwarded (compressed if possible, else verbatim), never dropped. A full background queue / duplicate key surfaces as `deferred:dropped`. - **Known limits (documented in `background_compression.py`):** Phase 3 is per-process, in-memory, and token-mode-only — these are **lost-savings, never lost-correctness**, and consistent with the project's existing per-process compression cache + sticky-session multi-worker model. The startup multi-worker warning now names off-path background compression. - Phase 2 reuses the existing BM25 scorer; reuse did not improve answer-retention over a Python prototype (query-awareness dominates) — its value is the Rust speed + repo-conventional Rust-core/Python-shim shape. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
0ce68dedd7 |
fix(proxy): restore Codex usage headers on WS and streaming SSE transports (#577) (#794)
## Description Codex's subscription/rate-limit window (the `x-codex-*` headers) was being **stripped on every transport Codex actually uses**, so session/weekly usage never reached the Codex CLI's own `/status` display, Headroom `/stats`/dashboard, or any consumer that sniffs the client-facing handshake. This PR restores it on **both** the WebSocket and streaming-SSE paths — the two halves of #577 — in one place. Fixes #577 **Supersedes #582 and #590.** This PR incorporates #582's SSE fix (carried verbatim with a `Co-authored-by` trailer) and additionally forwards the window onto the client `101` on the WS path, which #582/#590's capture-only WS code cannot do. Both can be closed as superseded once this merges — GitHub closing keywords only auto-close issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual close. ### WebSocket (`gpt-5.4+`) OpenAI delivers `x-codex-*` **only** on the upstream WS handshake response, never in data frames. `handle_openai_responses_ws` accepted the client WS *before* it connected upstream and never read `upstream.response.headers`, so the window was dropped. This reorders the handler to **connect upstream first**, extract the `x-codex-*` subset, then **accept the client WS with those headers attached** to the `101`, and refresh the Python state for `/stats` parity. ### Streaming SSE (incorporated from #582, @m16khb) Codex CLI almost always streams. `streaming.py` neither captured `x-codex-*` into `CodexRateLimitState` nor forwarded it — the forwarded-header filter matched only the substring `"ratelimit"`, which `x-codex-*` does not contain. This calls `update_from_headers()` **before** the `>=400` early-return (so a streaming 429/5xx still refreshes the window, matching the non-streaming handlers) and widens the forward filter to pass `x-codex-*`. > Credit: the SSE fix is @m16khb's work from #582, carried here verbatim with a > `Co-authored-by` trailer so the maintainer gets a single PR covering both > transports. This supersedes #582/#590's **WS** capture (which only writes > `/stats`); the connect-before-accept reorder additionally forwards the window to > the client `101`, which capture-only cannot do. #590's optional snapshot > persistence is intentionally left out (separable; hot-path sync write; doesn't > help the `101`-sniff consumers). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `openai.py`: add `_extract_codex_handshake_headers()` (strictly `x-codex-*`, via `raw_items()` to avoid `MultipleValuesError`; never `set-cookie`/`authorization`). - `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry loop runs before `accept()`; `accept(headers=...)` carries the forwarded window; first client frame read afterward. HTTP fallback preserved; it now also refreshes `/stats` from the HTTP response headers. - `streaming.py`: capture `x-codex-*` on all statuses + widen the forwarded-header filter (from #582). ### Diff-size note The bulk of the `openai.py` line count is **whitespace-only relocation**: the relay block dedents one level out of the old per-attempt `async with`. Logical change is ~290 lines. **Review with `?w=1`.** In API-key mode the handshake carries no `x-codex-*`, so the accept-header list is empty and the path behaves exactly as before — the fix only activates for ChatGPT-subscription auth. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed - WS: `test_ws_connect_happens_before_accept`, `test_ws_forwards_codex_headers_to_client_accept` (only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded; `/stats` refreshed), `test_ws_connect_failure_falls_back_to_http`, `test_ws_first_frame_timeout_after_connect_closes_upstream`. - Fallback: `test_fallback_refreshes_codex_rate_limit_state`. - SSE: `test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`, `test_codex_rate_limit_captured_on_streaming_429` (from #582). - Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real proxy + fake upstream + real `websockets` client and reads the client `101` — closes the gap the unit tests stub (that uvicorn/starlette actually write `accept(headers=...)`). ## Test Output ``` $ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \ tests/test_ws_http_fallback.py \ tests/test_openai_codex_ws_lifecycle.py \ tests/test_openai_codex_ws_timings.py \ tests/test_codex_rate_limits.py -q 63 passed in 0.83s $ .venv/bin/python tests/e2e_ws_codex_usage_headers.py [codex-hdr-e2e] client 101 headers: x-codex-primary-used-percent: 42 x-codex-primary-window-minutes: 300 x-codex-secondary-used-percent: 7 x-codex-secondary-window-minutes: 10080 [codex-hdr-e2e] /stats reflects codex window (primary-used=42) === CODEX-HDR E2E ALL GREEN === $ uv run ruff check . && uv run ruff format --check <touched files> All checks passed! ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - **Why connect-before-accept (not capture-only).** Once `accept()` sends the `101`, headers can no longer be added; the `x-codex-*` window only exists after we connect upstream. Capturing into Python state (as #582/#590's WS code does) fixes `/stats` but not the Codex CLI's native display or any `101`-sniffing consumer — those need the headers *on the client handshake*, which requires the reorder. - **Security.** Forwarding is filtered strictly to `x-codex-*`; `set-cookie`, `authorization`, and all other upstream headers are never forwarded to the client (asserted by both the unit test and the e2e). 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Contract Schemas Per maintainer request: a JSON Schema (draft 2020-12) artifact enshrining the OpenAI interaction expectations this changeset relies on, so drift is detectable later. Committed following the repo's parity convention: - schema: `tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json` - test: `tests/test_codex_openai_contract_parity.py` binds the schema to the **live code** in both directions, so drift fails CI rather than living only in this description - every declared `x-codex-*` header must be consumed by `parse_codex_rate_limits`, and `_extract_codex_handshake_headers` must forward exactly the declared subset and never `set-cookie`/`authorization`. No new dependency (does not pull in `jsonschema`). It covers, as `$defs`: - `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` - the upstream `x-codex-*` header family (full superset, with per-header wire pattern + the parsed semantic type) the WS and SSE captures read. Source of truth: `parse_codex_rate_limits`. - `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny** contract: only `x-codex-*` may be forwarded; `set-cookie`/`authorization` are explicitly forbidden (`propertyNames` + `not`). - `ClientForwardedStreamingHeaders` - the wider SSE forward set (`*ratelimit*` OR `x-codex*`). - `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` - the WS frame envelopes and the unwrapped HTTP-fallback POST body. - `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity tests assert. Validated with `jsonschema` (Draft202012 `check_schema` passes; positive instances from the e2e validate; negative instances - a leaked `set-cookie`, a fallback body still carrying a top-level `type` - are correctly rejected). <details> <summary><code>codex-openai-interaction.schema.json</code> (draft 2020-12)</summary> ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json", "title": "Codex <-> OpenAI interaction contracts (PR #794)", "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).", "$defs": { "OpenAICodexWindowHeaders": { "title": "x-codex-*-{primary,secondary} window headers", "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.", "type": "object", "properties": { "x-codex-primary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the primary window consumed. Gates creation of the primary window." }, "x-codex-primary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int", "description": "Primary window size in minutes." }, "x-codex-primary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)", "description": "Absolute reset time of the primary window." }, "x-codex-secondary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the secondary window consumed. Gates creation of the secondary window." }, "x-codex-secondary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int" }, "x-codex-secondary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)" } }, "additionalProperties": true }, "OpenAICodexCreditsHeaders": { "title": "x-codex-credits-* headers", "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.", "type": "object", "properties": { "x-codex-credits-has-credits": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)", "description": "Gates creation of the credits snapshot." }, "x-codex-credits-unlimited": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)" }, "x-codex-credits-balance": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Free-form server string, e.g. \"$5.00\"." } }, "additionalProperties": true }, "OpenAICodexMetaHeaders": { "title": "x-codex meta headers", "type": "object", "properties": { "x-codex-limit-name": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"." }, "x-codex-promo-message": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Server announcement. Also gates snapshot creation when present." } }, "additionalProperties": true }, "OpenAICodexRateLimitHeaders": { "title": "Full x-codex-* header family OpenAI may emit", "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.", "type": "object", "allOf": [ { "$ref": "#/$defs/OpenAICodexWindowHeaders" }, { "$ref": "#/$defs/OpenAICodexCreditsHeaders" }, { "$ref": "#/$defs/OpenAICodexMetaHeaders" } ], "additionalProperties": true }, "WSUpstreamHandshakeResponse": { "title": "OpenAI WS handshake (101) response headers consumed by the WS fix", "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "StreamingUpstreamResponseHeaders": { "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix", "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "ClientForwardedHandshakeHeaders": { "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)", "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.", "type": "object", "propertyNames": { "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-" }, "not": { "anyOf": [ { "required": ["set-cookie"] }, { "required": ["Set-Cookie"] }, { "required": ["authorization"] }, { "required": ["Authorization"] } ] }, "additionalProperties": { "type": "string" } }, "ClientForwardedStreamingHeaders": { "title": "Headers forwarded to the client on the streaming SSE path", "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.", "type": "object", "propertyNames": { "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]" }, "additionalProperties": { "type": "string" } }, "WSClientRequestFrame": { "title": "Client -> proxy WS data frame (Responses API over WS)", "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.", "type": "object", "properties": { "type": { "const": "response.create" }, "response": { "type": "object", "properties": { "model": { "type": "string", "description": "e.g. gpt-5.4" }, "input": { "description": "String prompt or Responses-API structured input array.", "type": ["string", "array"] }, "stream": { "type": "boolean" } }, "required": ["model"], "additionalProperties": true } }, "required": ["type", "response"], "additionalProperties": true }, "WSRelayEvent": { "title": "proxy -> client WS data frame (relayed Responses API event)", "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.", "anyOf": [ { "title": "lifecycle event", "type": "object", "properties": { "type": { "type": "string", "examples": [ "response.created", "response.output_item.added", "response.completed" ] }, "response": { "type": "object", "additionalProperties": true } }, "required": ["type"], "additionalProperties": true }, { "title": "error event", "type": "object", "properties": { "type": { "const": "error" }, "error": { "type": "object", "properties": { "message": { "type": "string" } }, "required": ["message"], "additionalProperties": true } }, "required": ["type", "error"], "additionalProperties": true } ] }, "HTTPFallbackRequestBody": { "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback", "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.", "type": "object", "properties": { "model": { "type": "string" }, "stream": { "const": true }, "input": { "type": ["string", "array"] } }, "required": ["model", "stream"], "not": { "required": ["type"] }, "additionalProperties": true }, "CodexRateLimitStatsOutput": { "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)", "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.", "type": "object", "properties": { "limit_id": { "const": "codex" }, "limit_name": { "type": ["string", "null"] }, "primary": { "$ref": "#/$defs/CodexWindowDict" }, "secondary": { "$ref": "#/$defs/CodexWindowDict" }, "credits": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "has_credits": { "type": "boolean" }, "unlimited": { "type": "boolean" }, "balance": { "type": ["string", "null"] } }, "required": ["has_credits", "unlimited", "balance"], "additionalProperties": false } ] }, "promo_message": { "type": ["string", "null"] }, "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." } }, "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"], "additionalProperties": false }, "CodexWindowDict": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "used_percent": { "type": "number" }, "window_minutes": { "type": ["integer", "null"] }, "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." }, "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." }, "seconds_until_reset": { "type": ["integer", "null"] } }, "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"], "additionalProperties": false } ] } } } ``` </details> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: m16khb <m16khb@gmail.com> |
||
|
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
|
21989e3640 |
fix(rust): port MessageScorer to Rust + parity harness (PR-A)
Direct port of `headroom.transforms.scoring.MessageScorer` (459 LOC). Foundation piece for the IntelligentContext port (PR-B onward). What's wired: - Deterministic factors fully ported: recency (exp-decay), forward references (tool_call_id graph), token density (unique/total). - External-dep factors gated behind traits: `EmbeddingProvider` and `ToinProvider`. No concrete impls yet — both default to neutral values matching Python's `embedding_provider=None` / `toin=None`. PR-A1 wires fastembed; PR-A2 plugs in a PyO3 ToinProvider. - ScoringWeights + MessageScore with serde + BTreeMap-ordered breakdown for stable JSON. Parity: - 13 fixtures recorded from Python, byte-equal under the comparator. - Floats rounded to 5 decimals on both sides — absorbs f32-vs-f64 drift in the weighted sum without masking real bugs. Drive-by: re-fix three pre-existing clippy errors in smart_crusher/crusher.rs that re-emerged with new test additions (field_reassign_with_default + dead hash_array_for_ccr). |
||
|
|
625290cfbc |
chore(rust): port ContentDetector to Rust + parity harness + PyO3 bridge
Faithful port of `headroom/transforms/content_detector.py` into `headroom-core`. Same regex patterns, dispatch order, confidence formulas, and line-count caps; lockstep with Python via 21 recorded parity fixtures (every dispatch branch exercised). - crates/headroom-core/src/transforms/content_detector.rs: regex-only detector (no ML); ContentType/DetectionResult mirror Python's enum + dataclass surface; metadata uses serde_json::Map for clean PyO3 bridging. - Tie-break in code detection: track scores in first-match insertion order (matches Python dict iteration semantics on `max()` ties). - TypeScript second pattern is start-anchored — Python's `pattern.match(line)` is start-anchored, but the regex crate's `is_match` is unanchored, so the literal `:` prefix is required for parity. - crates/headroom-parity: ContentDetectorComparator + universal f64 normalization in `compare_fixture` (serde_json's lossy parse vs full-precision serialize creates a 1-ULP asymmetry that broke comparison; round-trip the actual through to_string/from_str). - crates/headroom-py: detect_content_type/is_json_array_of_dicts and PyDetectionResult exposed via PyO3 with GIL released during scan. - tests/parity/recorder.py: new `_wrap_function` for free-function recording; content_detector hook + 21 varied inputs covering JSON arrays, diffs, HTML, search, build/log, six languages, and fallbacks. Sets up PR2 (ContentRouter scaffold) to call into Rust ContentDetector in-process via the bridge. |
||
|
|
da7716a95a |
chore(rust): SmartCrusher CCR marker injection + walker unification
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
|
||
|
|
43d1aa0329 |
parity(smart_crusher): byte-equal harness — 17 fixtures green
Adds the SmartCrusher half of the Rust-vs-Python parity harness. Path A
from the Stage 3c.1 plan: record fixtures from Python (with real
fastembed embeddings + the post-bug-fix code), drive the Rust port over
the same inputs, and assert byte-equal output on every recorded
scenario.
What's in:
- `tests/parity/record_smart_crusher.py`: standalone recorder for
`SmartCrusher.crush(content, query, bias)`. The generic recorder
framework only captures one positional, so this script writes its
own JSON envelope `{input: {content, query, bias}, config, output}`.
17 scenarios cover the planning paths exercised by ContentRouter
in production: passthrough, smart_sample, top_n, time-series,
duplicates, unicode (`ensure_ascii=False`), nested-3-deep, empties,
bias above and below 1.0.
- `crates/headroom-parity/src/lib.rs`: `SmartCrusherComparator`
reconstructs `SmartCrusherConfig` from the fixture's config block,
runs Rust `SmartCrusher::crush()`, emits the same JSON shape Python
serialized.
- `crates/headroom-parity/examples/diff_fixture.rs`: diagnostic CLI
that prints expected-vs-actual for one fixture (used during the
iteration that found the serializer bug below).
Serializer fix — found by the harness:
SmartCrusher uses `safe_json_dumps` (compact `(",", ":")` separators
+ `ensure_ascii=False`) for the wire bytes. The Rust port was using
`python_json_dumps` (default Python: `(", ", ": ")` + `ensure_ascii=
True`), which is the right choice for hashing but wrong for the
output. Refactored `anchor_selector.rs` to take a small
`JsonFmt { sort_keys, compact, ensure_ascii }` config so the three
flavors share one writer, added `python_safe_json_dumps`, and
switched `_smart_crush_content` to call it. All three flavors now
have byte-exact tests.
Two unit tests in `crusher.rs` were pinning the old (wrong) format
and have been re-pinned to the compact form.
Cross-language status:
- All 17 empty-query fixtures: byte-equal.
- Embedding-driven (non-empty-query) fixtures deferred until the
~0.0002 numeric drift between Python `onnxruntime` and Rust `ort`
is resolved (or until we accept the drift via a tolerance — none of
the 17 fixtures exercises a borderline relevance_threshold call,
and downstream code only branches at the 0.3 threshold).
Tests: cargo test --workspace (388 + supporting) green.
|
||
|
|
48c13245c5 |
fix(diff): close ContentRouter routing gaps for merge diffs and long preambles
User audit caught three gaps that prevented DiffCompressor from being
invoked even when the input was a real diff. These complement the four
emit-time bugs fixed in the previous commit — those fixes only kick in
once DiffCompressor receives the input. Without these gap fixes, real
merge-commit diffs and `git log -p` outputs with long commit messages
were misrouted away from DiffCompressor entirely.
# The three gaps (each fixed in Python; gap 3 also fixed in Rust)
1. Detector scan window was hardcoded to first 50 lines.
`_try_detect_diff` in content_detector.py only inspected
`content.split("\n")[:50]`. `git log -p` outputs commonly have
commit messages longer than 50 lines (releases, squashed commits,
bots), pushing the `diff --git` header out of the detection window.
Result: input was returned with `content_type=PLAIN_TEXT` and routed
to the text compressor, never reaching DiffCompressor. Fix: window
widened to 500 lines.
2. Detector regex didn't recognize merge-commit headers.
`_DIFF_HEADER_PATTERN` matched `diff --git`, `--- a/`, and the
regular `@@ -A,B +C,D @@` hunk header. Merge-commit diffs from
`git log -p` use `diff --combined <path>`, `diff --cc <path>`, and
combined-diff hunk headers `@@@+`. The shared `--- a/` line still
triggered the detector with low confidence, but only barely. Fix:
extended the regex to recognize all four merge-shaped header forms.
3. DiffCompressor parser only matched `^diff --git`.
Even after fixing detection, the parser's `_DIFF_GIT_PATTERN`
wouldn't match `diff --combined` or `diff --cc`, so merge diffs
reached DiffCompressor and were treated as one giant pre-diff blob —
passed through unchanged after the previous PR's pre-diff
preservation fix. Fix: added `_DIFF_COMBINED_PATTERN` and
`_DIFF_CC_PATTERN`; `_parse_diff` starts a new file section on any
of the three header forms. Mirrored in Rust as `is_diff_header`
helper that checks all three regexes.
# Why this matters end-to-end
DiffCompressor's value comes from being routed to. Detection +
parser-level coverage are upstream of the compressor — without them,
the compressor never sees the input. The previous PR's four bug fixes
(rename, combined-diff hunks, no-newline marker, pre-diff content) are
correct and necessary, but for merge commits and long-preamble diffs,
they were only firing on the rare cases where the detector misclicked
into DiffCompressor anyway. With these three gaps closed, the
ContentRouter→DiffCompressor pipeline actually engages on:
- `git log -p` outputs of any commit-message length
- Merge-commit diffs (`diff --combined`, `diff --cc`)
- Combined-diff snippets (`@@@`+ hunk-only inputs)
# New fixtures (3 added to the existing 24)
- `066bc82…` — `diff --combined` merge diff (3-way)
- `5d950a94…` — `diff --cc` merge diff (alternate form)
- `66c86f64…` — long pre-diff content (60-line commit message)
followed by a rename diff (exercises detector scan widening +
pre-diff preservation in tandem)
Parity: total=27 matched=27 skipped=0 diffed=0.
# Tests
- Python: 4 new tests across 2 new test classes —
`TestRoutingGapMergeDiffs` (combined / cc parser) and
`TestRoutingGapDetectorScanWindow` (long preamble detection +
combined-diff regex recognition).
- Rust: 2 new tests covering combined / cc parser sections.
# Verification
- 27/27 parity fixtures byte-equal.
- Python: 41/41 tests pass (was 37).
- Rust: 18/18 transforms tests; 62/62 workspace; 5/5 proptests.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
-- -D warnings` clean.
|
||
|
|
6d47a0cd00 |
fix(diff_compressor): four silent information-loss paths in Python AND Rust
Audit caught four bugs that the byte-equal parity harness can't catch on its own — both Python and Rust were faithfully emitting the buggy output. Fixed in lockstep so parity is maintained while the underlying behavior is now correct on inputs the existing 20 fixtures didn't exercise. # The four bugs (each fixed in both Python and Rust) 1. Renames silently dropped from output. Parser captured `is_renamed=True` but the emitter never emitted ANY rename markers. Output of a rename looked exactly like a plain modification of the old path. Fix: capture `rename from` / `rename to` / `similarity index N%` / `dissimilarity index N%` / `copy from` / `copy to` lines in a new `rename_lines` field on `DiffFile`; emit them after `diff --git` in canonical git ordering. 2. Combined diff hunks (`@@@`) silently dropped. Hunk-header regex only matched `@@`, so 3-way merge hunks had `current_hunk` never set and ALL their content fell through to the no-op branch. Fix in Python: regex switched to `^(@@+) ... \1` (backreferences match any number of `@`s on each side). Fix in Rust: alternation over `@@`, `@@@`, `@@@@` since `regex` is RE2-based and rejects backreferences. n>3 octopus merges still fall through; rare in practice. 3. `\ No newline at end of file` markers can be context-trimmed away. Treated as ordinary "other" lines — if more than `max_context_lines` from a `+`/`-` change, dropped. Round-trip-breaking for patches; can change whether the trailing line has a newline. Fix: in `_reduce_context`, force-add any line starting with `\` to the keep set regardless of distance. 4. Pre-diff content silently dropped. Anything before the first `diff --git` — commit messages from `git log -p`, email headers from `git format-patch`, fork-and-rebase metadata — was discarded. Fix: `_parse_diff` now returns `(pre_diff_lines, files)`; `format_output` prepends pre-diff content verbatim when present. # Hidden parity bug found during the work `_compress_files` constructed a fresh `DiffFile` from the parsed one but only copied a subset of the fields by name. The new `rename_lines` and `original_*_line` fields were silently dropped here, so the parser populated them correctly but the emitter saw an empty `rename_lines` list. Caught by writing a real test instead of a smoke test — the smoke test passed because it hit the no-diff-found short-circuit, not the parser/emitter pipeline. Constructor now copies all fields explicitly. # Parity status - Existing 20 fixtures: still byte-equal between fixed Python and fixed Rust. None of them exercised the buggy paths. - 4 NEW fixtures recorded against fixed Python, exercising each bug-fix path: rename, 3-way combined diff, `\ No newline` marker far from changes, pre-diff commit headers. All 4 byte-equal between Python and Rust. - Parity harness: total=24 matched=24 skipped=0 diffed=0. # Observability Some normalizations remain parity-bound (file mode `100644` hardcode, `Binary files differ` simplification). Those are surfaced in `DiffCompressorStats::file_mode_normalizations` / `binary_files_simplified` (Rust) and via `logger.warning` (Python's new `_log_loss_signals` helper, called once per compress). # Tests - Python: 4 new test classes (11 tests) covering rename markers, combined diffs, no-newline preservation, pre-diff content. Edge case: no pre-diff content must NOT add a leading blank line. - Rust: 4 new `bugfix_*` unit tests with the same scenarios. - Existing Python tests calling `_parse_diff` directly were updated for the new `(pre_diff, files)` tuple return. # Verification - Python: 37/37 tests pass (was 26). - Rust: 16/16 transforms tests; 60/60 workspace unit tests; 5/5 proptests; 1/1 doctest. - Parity: 24/24 byte-equal. - `cargo fmt --check` clean; `cargo clippy --workspace --all-targets -- -D warnings` clean. |
||
|
|
0414cb70e4 |
feat(rust): scaffold workspace + parity harness (phase-0)
Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |