Commit Graph
3 Commits
Author SHA1 Message Date
Mansurshakh JaparovandJerrettDavis 93279f3208 fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description

Closes #3697.

An unchanged SmartCrusher result now keeps the original bytes and
reports no structural compression. Formatting-only savings come from the
existing Rust JsonMinifier, not from pretending the JSON value changed.

The minifier preserves duplicate keys, order, escapes and number
literals. A mixed result cannot use one section's CCR marker to hide
unmarked loss in another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace
savings through JsonMinifier in Python and the eligible native live
zone.
- Keep all records and JSON literals; do not store originals or inject
CCR markers for lossless minification.
- Preserve mixed routing and reject uncovered lossy sections in both
tool-result formats. Explicit no-CCR behavior is unchanged.
- Replace incidental forwarding tests with native minification and
complete-record regressions; document the distinction.

## Testing

- [ ] Entire Python suite — not claimed.
- [x] Four consumer/native modules: `python -m pytest -q
tests/test_recursive_json.py tests/test_transforms_content_router.py
tests/test_transforms/test_smart_crusher_rust_parity.py
tests/test_transforms/test_smart_crusher_lossless_default.py`
- [x] Changed-file Ruff checks; normal commit hooks, including mypy and
commitlint.
- [x] Manual public-API runs, with and without the HTML extra.

### Test Output

Consumer/native modules:

```text
collected 136 items
============================= 136 passed in 36.43s =============================
```

Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native
build, Python checks, commitlint):

```text
======================== 180 passed, 1 warning in 8.35s ========================
✅ ci-precheck PASSED — safe to push.
To https://github.com/oxura/headroom.git
   64688217..0002af3e  HEAD -> fix/3697-unchanged-json-passthrough
```

The first push stopped on an inherited commit footer because
`origin/main` was stale. Fetching the actual upstream ref fixed the lint
range; the normal push then passed. No source merge, lint-rule change or
hook bypass.

Hosted checks on `0002af3e` now pass: all four Python shards, Windows
permissions, smoke tests, native wrappers, parity and Docker e2e.
[Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks).

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual
extension built from this checkout. No provider billing calls.
- Exact command / steps: the earlier localhost `/v1/compress` and
TransformPipeline runs exercised the original fix. For this follow-up,
call `ContentRouter.compress` on a data-only HTML page containing 60
complete records, with `smart_crusher_lossless_only=True`,
`smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run
it once without trafilatura, then with the locked 2.0.0 extra. The new
public-API test contains the full input/config. Separately call
`route_embedded_json` on a compact array followed by a padded array,
with Unicode and CRLF surroundings.
- Observed result: both HTML modes keep all 60 records and the
surrounding markup, with no unrecoverable section. The unchanged first
JSON span does not stop the second span's compaction; surrounding text
stays exact. Final Codecov is 96%; the two remaining lines are the
deferred HTML fallback return. The real no-HTML-extra API run executed
both, and local coverage also exercises the unchanged-span branch.
- Not tested: live provider acceptance, invoices or production cache hit
rates. No native eligibility or admission gates changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing JSON routing.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: eligible formatting savings return as
json_minifier; standalone SmartCrusher remains byte-exact on no-ops.
- Kill switch / disable path: existing routing configuration and
HEADROOM_LIVE_ZONE_DISABLE_ARMS.
- Unsafe override required: no.
- Qualification impact: no-op, formatting, structural and CCR paths
exercised separately.
- Rollback path: revert the minifier integration; no persistence
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

Addresses the [minifier
review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285).
The maintainer’s main merge is retained. The latest follow-up only
changes consumer tests.

AI-assisted; no subagents.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 21:32:42 -06:00
8dbbd1d896 fix(transforms): keep record-bearing JSON out of Kompress (#3673) (#3880)
> **Severity: silent data loss.** Kompress deletes whole records from
single-line JSON tool output and leaves *valid* JSON behind, so no
parser, log line or model can tell. This hits the default `coding`
profile with `accuracy_guard=strict`, on every release up to and
including v0.39.1. In our deployment an agent repeatedly and confidently
told a user that a data collection did not exist; it did. The only
mitigation today is `HEADROOM_DISABLE_KOMPRESS=1`, which also turns off
prose compression.

## Description

Keeps JSON whose record structure a prose model can destroy out of
Kompress, on every route that reaches it, while prose around that JSON
still compresses.

This carries #3680 to a mergeable state. It is the same four commits,
authorship preserved:

- `113b6451` by @iuiu-py: the whole-document guard.
- `2ebf6b54`: the prefixed-line and relevance-split routes, which were
the P1 review.
- `15b04665` by @JerrettDavis: formatting.
- `9386c994` by @Ayushraj06-bit (from iuiu-py/headroom#1, open since 22
Sep and not picked up): narrows the guard, as the last review on #3680
asked.

Everything is rebased on current `main` (`ffc6edb4`). #3680 has been
waiting on the author since 24 Sep (`needs author action`). If @iuiu-py
would rather land it there, I will close this one; nothing here is meant
to take the credit.

Closes #3673

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `recursive_json.json_document_spans()` lists every top-level object or
array span that parses. `carries_record_array()` replaces
`_has_routable_json()` under the same semantics.
- `content_router._protected_json_spans()` protects two shapes:
- a span carrying an array of objects, because deleting `},{` there
leaves valid JSON;
  - a block that is one JSON document end to end.
- Separate top-level values (JSONL, space-joined search hits) are
**not** protected. Welding two of them yields unparseable output, not a
silently shorter document. That is what the last review asked for, and
it is why
`test_force_kompress_routes_anthropic_tool_result_to_targeted_kompress`
and the three `test_kompress_deadline` tests pass again.
- `_try_ml_compressor` skips Kompress when protected JSON is present.
- `_relevance_split_compress` declines only when a protected span
straddles a `segment()` cut, so line-contained JSONL still splits.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check <4 changed files> && ruff format --check <4 changed files>
All checks passed! / 4 files already formatted
$ mypy headroom/transforms/content_router.py headroom/transforms/recursive_json.py
Success: no issues found
$ pytest tests/test_transforms tests/test_transforms_content_router.py tests/test_recursive_json.py \
         tests/test_relevance*.py tests/test_*smart_crusher*.py tests/test_*kompress*.py
668 passed, 84 skipped, 1 failed
```

The one failure is
`test_smart_crusher_issue_3650::test_issue_3650_router_emits_visible_retrievable_sentinel`.
It fails identically on `origin/main` in my environment, because I
cannot build `_core` locally and ran against the `_core` from the 0.39.1
wheel. For the same reason I skipped the Rust-parity tests. CI will be
the real check for both.

The 4 tests that fail on #3680's head (`fec63a13`, CI shard `test (4)`)
pass here.

## Real Behavior Proof

- Environment: real Kompress ONNX model (`chopratejas/kompress-v2-base`,
int8-wo), `coding` profile env applied via
`get_agent_savings_profile("coding").apply_proxy_env_defaults`,
`HEADROOM_OFFLINE=1`; the `payload_synthetic.json` attached to #3673 (5
records, 3721 bytes, one line)
- Exact command / steps: `ContentRouter().compress()` over three routes
(bare document, `Tool result: {…}` on the same line, document embedded
in prose), 3 runs each, on `origin/main` (`ffc6edb4`) and on this
branch, plus a pure-prose control
- Observed result: `main` loses 1 of 5 records on all three routes in
3/3 runs (3721 → 3126 bytes on the bare document); this branch keeps 5/5
byte-exact on all three, the prose around the document still compresses
(9651 → 9129) and the pure-prose control compresses identically on both
(2964 → 2702). Table below
- Not tested: a full proxy run end to end on this branch. The original
report comes from a proxy deployment, where the same record loss was
observed on the wire

| route | `main` | this branch |
|---|---|---|
| bare document | 4/5 records, 3721 → 3126 bytes | 5/5, byte-exact |
| `Tool result: {…}` on the same line | 4/5 | 5/5, byte-exact |
| document inside prose | 4/5 | 5/5, byte-exact; prose around it still
compresses (9651 → 9129) |
| control: pure prose | 2964 → 2702 | 2964 → 2702 (unchanged) |

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a (stable)
- Stable/default behavior changed: yes, narrowly: record-bearing JSON is
no longer handed to Kompress and is forwarded byte-exact instead of
lossy-compressed; structural compressors are unaffected
- Kill switch / disable path: none new; revert the PR
- Unsafe override required: no
- Qualification impact: less lossy savings on JSON tool outputs that
previously went through Kompress (those savings were partly the data
loss itself)
- Rollback path: revert this PR; no state, schema or config migration
involved

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no user-facing doc change; the behavior now matches
what the JSON docs already promise ("keys, brackets … remain
preserved").
- Local test caveat:
`test_smart_crusher_issue_3650::test_issue_3650_router_emits_visible_retrievable_sentinel`
fails identically on `origin/main` in my environment (I ran against the
0.39.1 wheel's `_core`, not a local build), and Rust-parity tests were
skipped for the same reason; CI is the real check for both.
- Workflows are awaiting maintainer approval (first PR from this fork).

---------

Signed-off-by: wangzifei <wangzifei@cit.group.hk>
Co-authored-by: iuiuiu <92971770+iuiu-py@users.noreply.github.com>
Co-authored-by: wangzifei <wangzifei@cit.group.hk>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Ayushraj06-bit <ayush10d172390@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:40:14 -05:00
Tejas Chopra 57bf720d5c feat(router): route embedded & nested JSON through the compressor dispatch (#2623)
## Description

`ContentRouter` only compressed JSON when the **whole** `tool_result`
block was a single JSON value. JSON embedded inside larger output (`gh
api` dumps, MCP tool results, `curl | jq` tails, log lines ending in a
JSON blob) was invisible to the JSON compressors — and in practice that
embedded shape is the large majority of JSON an agent actually sees.

This adds a structural routing step: find balanced JSON spans at **any
offset** in a block and route each one through the router's **existing,
unchanged** `_apply_strategy_to_content`, splicing the result back with
the surrounding bytes kept exact.

Because each span takes the same dispatch path a whole-block JSON
already takes, SmartCrusher/CodeCompressor register their `<<ccr:…>>`
retrieval markers exactly as before — CCR is hash-keyed, so it is
location-agnostic and unaffected by nesting.

Closes #

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- New `headroom/transforms/recursive_json.py` — `route_embedded_json()`:
deterministic balanced-span scan + splice; skips spans already carrying
a `<<ccr:` marker (never re-compresses); token-gated.
- One guarded call at the top of `_apply_strategy_to_content` plus an
`_allow_embedded` **one-shot re-entrancy guard** (not a depth cap).
- **No size/min or depth thresholds** — the only gates are correctness
(round-trip) and benefit (token reduction). Strict no-op when a block
has no embedded JSON, so the 97%+ of non-JSON blocks are byte-identical
to today.
- Deterministic + per-block → prefix-cache- and CCR-store-stable.
- `tests/test_recursive_json.py`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_recursive_json.py tests/test_content_router_compact_json.py tests/test_content_router_tool_role_reversibility.py -q
19 passed in 5.90s

$ ruff check headroom/transforms/recursive_json.py headroom/transforms/content_router.py tests/test_recursive_json.py
All checks passed!

$ mypy headroom/transforms/recursive_json.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- **Environment:** local,
`ContentRouter(ContentRouterConfig(lossless=False))`, pure-Python
content detector.
- **Exact steps:** `router.compress(block)` where `block` = prose with a
120-row JSON array embedded mid-text (`"I queried the ECS API
...\n[{...}]\nAll services healthy."`).
- **Observed result:** `strategy_used=MIXED`; block **11,986 → 3,798
chars**; leading/trailing prose preserved byte-exact; the embedded JSON
folded to a columnar table. Previously this block's embedded array was
not routed to the JSON compressor at all.
- **CCR:** a span already containing `<<ccr:` is passed through
untouched (unit-tested); folded spans go through the unchanged dispatch,
so markers register and resolve identically to whole-block JSON.
- **Not tested:** live proxy end-to-end with markers force-enabled
(covered by the unchanged dispatch path + unit tests); non-CC transcript
shapes beyond the local corpus.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Scoped to the OSS structural-routing step only. The lossless *fold
kinds* it routes into (JSON columnar / log template) are maintained in
the `headroom-lossless-guard` extension. N/A: no docs/screenshots;
`Closes #` left blank (no tracking issue).

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-27 20:52:18 -07:00