mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
93279f3208 |
fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description Closes #3697. An unchanged SmartCrusher result now keeps the original bytes and reports no structural compression. Formatting-only savings come from the existing Rust JsonMinifier, not from pretending the JSON value changed. The minifier preserves duplicate keys, order, escapes and number literals. A mixed result cannot use one section's CCR marker to hide unmarked loss in another. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace savings through JsonMinifier in Python and the eligible native live zone. - Keep all records and JSON literals; do not store originals or inject CCR markers for lossless minification. - Preserve mixed routing and reject uncovered lossy sections in both tool-result formats. Explicit no-CCR behavior is unchanged. - Replace incidental forwarding tests with native minification and complete-record regressions; document the distinction. ## Testing - [ ] Entire Python suite — not claimed. - [x] Four consumer/native modules: `python -m pytest -q tests/test_recursive_json.py tests/test_transforms_content_router.py tests/test_transforms/test_smart_crusher_rust_parity.py tests/test_transforms/test_smart_crusher_lossless_default.py` - [x] Changed-file Ruff checks; normal commit hooks, including mypy and commitlint. - [x] Manual public-API runs, with and without the HTML extra. ### Test Output Consumer/native modules: ```text collected 136 items ============================= 136 passed in 36.43s ============================= ``` Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native build, Python checks, commitlint): ```text ======================== 180 passed, 1 warning in 8.35s ======================== ✅ ci-precheck PASSED — safe to push. To https://github.com/oxura/headroom.git 64688217..0002af3e HEAD -> fix/3697-unchanged-json-passthrough ``` The first push stopped on an inherited commit footer because `origin/main` was stale. Fetching the actual upstream ref fixed the lint range; the normal push then passed. No source merge, lint-rule change or hook bypass. Hosted checks on `0002af3e` now pass: all four Python shards, Windows permissions, smoke tests, native wrappers, parity and Docker e2e. [Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks). ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual extension built from this checkout. No provider billing calls. - Exact command / steps: the earlier localhost `/v1/compress` and TransformPipeline runs exercised the original fix. For this follow-up, call `ContentRouter.compress` on a data-only HTML page containing 60 complete records, with `smart_crusher_lossless_only=True`, `smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run it once without trafilatura, then with the locked 2.0.0 extra. The new public-API test contains the full input/config. Separately call `route_embedded_json` on a compact array followed by a padded array, with Unicode and CRLF surroundings. - Observed result: both HTML modes keep all 60 records and the surrounding markup, with no unrecoverable section. The unchanged first JSON span does not stop the second span's compaction; surrounding text stays exact. Final Codecov is 96%; the two remaining lines are the deferred HTML fallback return. The real no-HTML-extra API run executed both, and local coverage also exercises the unchanged-span branch. - Not tested: live provider acceptance, invoices or production cache hit rates. No native eligibility or admission gates changed. ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing JSON routing. - Minimum rollout channel: unchanged. - Stable/default behavior changed: eligible formatting savings return as json_minifier; standalone SmartCrusher remains byte-exact on no-ops. - Kill switch / disable path: existing routing configuration and HEADROOM_LIVE_ZONE_DISABLE_ARMS. - Unsafe override required: no. - Qualification impact: no-op, formatting, structural and CCR paths exercised separately. - Rollback path: revert the minifier integration; no persistence migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes Addresses the [minifier review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285). The maintainer’s main merge is retained. The latest follow-up only changes consumer tests. AI-assisted; no subagents. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
8dbbd1d896 |
fix(transforms): keep record-bearing JSON out of Kompress (#3673) (#3880)
> **Severity: silent data loss.** Kompress deletes whole records from single-line JSON tool output and leaves *valid* JSON behind, so no parser, log line or model can tell. This hits the default `coding` profile with `accuracy_guard=strict`, on every release up to and including v0.39.1. In our deployment an agent repeatedly and confidently told a user that a data collection did not exist; it did. The only mitigation today is `HEADROOM_DISABLE_KOMPRESS=1`, which also turns off prose compression. ## Description Keeps JSON whose record structure a prose model can destroy out of Kompress, on every route that reaches it, while prose around that JSON still compresses. This carries #3680 to a mergeable state. It is the same four commits, authorship preserved: - `113b6451` by @iuiu-py: the whole-document guard. - `2ebf6b54`: the prefixed-line and relevance-split routes, which were the P1 review. - `15b04665` by @JerrettDavis: formatting. - `9386c994` by @Ayushraj06-bit (from iuiu-py/headroom#1, open since 22 Sep and not picked up): narrows the guard, as the last review on #3680 asked. Everything is rebased on current `main` (`ffc6edb4`). #3680 has been waiting on the author since 24 Sep (`needs author action`). If @iuiu-py would rather land it there, I will close this one; nothing here is meant to take the credit. Closes #3673 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `recursive_json.json_document_spans()` lists every top-level object or array span that parses. `carries_record_array()` replaces `_has_routable_json()` under the same semantics. - `content_router._protected_json_spans()` protects two shapes: - a span carrying an array of objects, because deleting `},{` there leaves valid JSON; - a block that is one JSON document end to end. - Separate top-level values (JSONL, space-joined search hits) are **not** protected. Welding two of them yields unparseable output, not a silently shorter document. That is what the last review asked for, and it is why `test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress` and the three `test_kompress_deadline` tests pass again. - `_try_ml_compressor` skips Kompress when protected JSON is present. - `_relevance_split_compress` declines only when a protected span straddles a `segment()` cut, so line-contained JSONL still splits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check <4 changed files> && ruff format --check <4 changed files> All checks passed! / 4 files already formatted $ mypy headroom/transforms/content_router.py headroom/transforms/recursive_json.py Success: no issues found $ pytest tests/test_transforms tests/test_transforms_content_router.py tests/test_recursive_json.py \ tests/test_relevance*.py tests/test_*smart_crusher*.py tests/test_*kompress*.py 668 passed, 84 skipped, 1 failed ``` The one failure is `test_smart_crusher_issue_3650::test_issue_3650_router_emits_visible_retrievable_sentinel`. It fails identically on `origin/main` in my environment, because I cannot build `_core` locally and ran against the `_core` from the 0.39.1 wheel. For the same reason I skipped the Rust-parity tests. CI will be the real check for both. The 4 tests that fail on #3680's head (`fec63a13`, CI shard `test (4)`) pass here. ## Real Behavior Proof - Environment: real Kompress ONNX model (`chopratejas/kompress-v2-base`, int8-wo), `coding` profile env applied via `get_agent_savings_profile("coding").apply_proxy_env_defaults`, `HEADROOM_OFFLINE=1`; the `payload_synthetic.json` attached to #3673 (5 records, 3721 bytes, one line) - Exact command / steps: `ContentRouter().compress()` over three routes (bare document, `Tool result: {…}` on the same line, document embedded in prose), 3 runs each, on `origin/main` (`ffc6edb4`) and on this branch, plus a pure-prose control - Observed result: `main` loses 1 of 5 records on all three routes in 3/3 runs (3721 → 3126 bytes on the bare document); this branch keeps 5/5 byte-exact on all three, the prose around the document still compresses (9651 → 9129) and the pure-prose control compresses identically on both (2964 → 2702). Table below - Not tested: a full proxy run end to end on this branch. The original report comes from a proxy deployment, where the same record loss was observed on the wire | route | `main` | this branch | |---|---|---| | bare document | 4/5 records, 3721 → 3126 bytes | 5/5, byte-exact | | `Tool result: {…}` on the same line | 4/5 | 5/5, byte-exact | | document inside prose | 4/5 | 5/5, byte-exact; prose around it still compresses (9651 → 9129) | | control: pure prose | 2964 → 2702 | 2964 → 2702 (unchanged) | ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a (stable) - Stable/default behavior changed: yes, narrowly: record-bearing JSON is no longer handed to Kompress and is forwarded byte-exact instead of lossy-compressed; structural compressors are unaffected - Kill switch / disable path: none new; revert the PR - Unsafe override required: no - Qualification impact: less lossy savings on JSON tool outputs that previously went through Kompress (those savings were partly the data loss itself) - Rollback path: revert this PR; no state, schema or config migration involved ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation: no user-facing doc change; the behavior now matches what the JSON docs already promise ("keys, brackets … remain preserved"). - Local test caveat: `test_smart_crusher_issue_3650::test_issue_3650_router_emits_visible_retrievable_sentinel` fails identically on `origin/main` in my environment (I ran against the 0.39.1 wheel's `_core`, not a local build), and Rust-parity tests were skipped for the same reason; CI is the real check for both. - Workflows are awaiting maintainer approval (first PR from this fork). --------- Signed-off-by: wangzifei <wangzifei@cit.group.hk> Co-authored-by: iuiuiu <92971770+iuiu-py@users.noreply.github.com> Co-authored-by: wangzifei <wangzifei@cit.group.hk> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Ayushraj06-bit <ayush10d172390@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
57bf720d5c |
feat(router): route embedded & nested JSON through the compressor dispatch (#2623)
## Description
`ContentRouter` only compressed JSON when the **whole** `tool_result`
block was a single JSON value. JSON embedded inside larger output (`gh
api` dumps, MCP tool results, `curl | jq` tails, log lines ending in a
JSON blob) was invisible to the JSON compressors — and in practice that
embedded shape is the large majority of JSON an agent actually sees.
This adds a structural routing step: find balanced JSON spans at **any
offset** in a block and route each one through the router's **existing,
unchanged** `_apply_strategy_to_content`, splicing the result back with
the surrounding bytes kept exact.
Because each span takes the same dispatch path a whole-block JSON
already takes, SmartCrusher/CodeCompressor register their `<<ccr:…>>`
retrieval markers exactly as before — CCR is hash-keyed, so it is
location-agnostic and unaffected by nesting.
Closes #
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- New `headroom/transforms/recursive_json.py` — `route_embedded_json()`:
deterministic balanced-span scan + splice; skips spans already carrying
a `<<ccr:` marker (never re-compresses); token-gated.
- One guarded call at the top of `_apply_strategy_to_content` plus an
`_allow_embedded` **one-shot re-entrancy guard** (not a depth cap).
- **No size/min or depth thresholds** — the only gates are correctness
(round-trip) and benefit (token reduction). Strict no-op when a block
has no embedded JSON, so the 97%+ of non-JSON blocks are byte-identical
to today.
- Deterministic + per-block → prefix-cache- and CCR-store-stable.
- `tests/test_recursive_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_recursive_json.py tests/test_content_router_compact_json.py tests/test_content_router_tool_role_reversibility.py -q
19 passed in 5.90s
$ ruff check headroom/transforms/recursive_json.py headroom/transforms/content_router.py tests/test_recursive_json.py
All checks passed!
$ mypy headroom/transforms/recursive_json.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- **Environment:** local,
`ContentRouter(ContentRouterConfig(lossless=False))`, pure-Python
content detector.
- **Exact steps:** `router.compress(block)` where `block` = prose with a
120-row JSON array embedded mid-text (`"I queried the ECS API
...\n[{...}]\nAll services healthy."`).
- **Observed result:** `strategy_used=MIXED`; block **11,986 → 3,798
chars**; leading/trailing prose preserved byte-exact; the embedded JSON
folded to a columnar table. Previously this block's embedded array was
not routed to the JSON compressor at all.
- **CCR:** a span already containing `<<ccr:` is passed through
untouched (unit-tested); folded spans go through the unchanged dispatch,
so markers register and resolve identically to whole-block JSON.
- **Not tested:** live proxy end-to-end with markers force-enabled
(covered by the unchanged dispatch path + unit tests); non-CC transcript
shapes beyond the local corpus.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Scoped to the OSS structural-routing step only. The lossless *fold
kinds* it routes into (JSON columnar / log template) are maintained in
the `headroom-lossless-guard` extension. N/A: no docs/screenshots;
`Closes #` left blank (no tracking issue).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|