Commit Graph
2 Commits
Author SHA1 Message Date
Mansurshakh JaparovandJerrettDavis 93279f3208 fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description

Closes #3697.

An unchanged SmartCrusher result now keeps the original bytes and
reports no structural compression. Formatting-only savings come from the
existing Rust JsonMinifier, not from pretending the JSON value changed.

The minifier preserves duplicate keys, order, escapes and number
literals. A mixed result cannot use one section's CCR marker to hide
unmarked loss in another.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace
savings through JsonMinifier in Python and the eligible native live
zone.
- Keep all records and JSON literals; do not store originals or inject
CCR markers for lossless minification.
- Preserve mixed routing and reject uncovered lossy sections in both
tool-result formats. Explicit no-CCR behavior is unchanged.
- Replace incidental forwarding tests with native minification and
complete-record regressions; document the distinction.

## Testing

- [ ] Entire Python suite — not claimed.
- [x] Four consumer/native modules: `python -m pytest -q
tests/test_recursive_json.py tests/test_transforms_content_router.py
tests/test_transforms/test_smart_crusher_rust_parity.py
tests/test_transforms/test_smart_crusher_lossless_default.py`
- [x] Changed-file Ruff checks; normal commit hooks, including mypy and
commitlint.
- [x] Manual public-API runs, with and without the HTML extra.

### Test Output

Consumer/native modules:

```text
collected 136 items
============================= 136 passed in 36.43s =============================
```

Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native
build, Python checks, commitlint):

```text
======================== 180 passed, 1 warning in 8.35s ========================
✅ ci-precheck PASSED — safe to push.
To https://github.com/oxura/headroom.git
   64688217..0002af3e  HEAD -> fix/3697-unchanged-json-passthrough
```

The first push stopped on an inherited commit footer because
`origin/main` was stale. Fetching the actual upstream ref fixed the lint
range; the normal push then passed. No source merge, lint-rule change or
hook bypass.

Hosted checks on `0002af3e` now pass: all four Python shards, Windows
permissions, smoke tests, native wrappers, parity and Docker e2e.
[Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks).

## Real Behavior Proof

- Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual
extension built from this checkout. No provider billing calls.
- Exact command / steps: the earlier localhost `/v1/compress` and
TransformPipeline runs exercised the original fix. For this follow-up,
call `ContentRouter.compress` on a data-only HTML page containing 60
complete records, with `smart_crusher_lossless_only=True`,
`smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run
it once without trafilatura, then with the locked 2.0.0 extra. The new
public-API test contains the full input/config. Separately call
`route_embedded_json` on a compact array followed by a padded array,
with Unicode and CRLF surroundings.
- Observed result: both HTML modes keep all 60 records and the
surrounding markup, with no unrecoverable section. The unchanged first
JSON span does not stop the second span's compaction; surrounding text
stays exact. Final Codecov is 96%; the two remaining lines are the
deferred HTML fallback return. The real no-HTML-extra API run executed
both, and local coverage also exercises the unchanged-span branch.
- Not tested: live provider acceptance, invoices or production cache hit
rates. No native eligibility or admission gates changed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing JSON routing.
- Minimum rollout channel: unchanged.
- Stable/default behavior changed: eligible formatting savings return as
json_minifier; standalone SmartCrusher remains byte-exact on no-ops.
- Kill switch / disable path: existing routing configuration and
HEADROOM_LIVE_ZONE_DISABLE_ARMS.
- Unsafe override required: no.
- Qualification impact: no-op, formatting, structural and CCR paths
exercised separately.
- Rollback path: revert the minifier integration; no persistence
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it

## Additional Notes

Addresses the [minifier
review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285).
The maintainer’s main merge is retained. The latest follow-up only
changes consumer tests.

AI-assisted; no subagents.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 21:32:42 -06:00
AshishandClaude Fable 5 737b332129 feat(config): fold TOML array-of-tables to csv-schema via SmartCrusher (#1799)
## Description

Adds a schema-fold tier to the structured-config compressor (introduced
in #1784). TOML files containing an `[[array-of-tables]]` are parsed
with the stdlib `tomllib` reference parser and bridged to SmartCrusher's
lossless `csv-schema` renderer, which folds the repeated per-record keys
into a single schema over the rows. On lockfiles and override-lists —
where the repeated keys dominate the byte count — this is a large win.

**Stacked on #1784 — review only the top commit** (`feat(config): fold
TOML array-of-tables to csv-schema via SmartCrusher`). The base commit
is #1784's config-compressor PR; this PR will collapse to the single new
commit once #1784 merges.

Faithfulness is guaranteed by construction, not by a heuristic:
- `tomllib` is the reference TOML parser, so the extracted records are
ground-truth.
- `csv-schema` is a lossless JSON renderer (`smart_crusher.py` documents
it as such), so the model reads a faithful, reformatted view of the
exact parsed data.
- Byte-exact recovery rides the existing CCR path — the original is
persisted to the `CompressionStore` and a `Retrieve original: hash=…`
marker is emitted. The fold is only emitted when that store write
succeeds, so nothing is ever unrecoverable.

Scope is deliberately **TOML-only**: `tomllib` is stdlib, whereas PyYAML
is only a *transitive* dependency (not declared in `pyproject.toml`),
and INI record-sections would need a bespoke dict-of-dicts→records
transform. Those flavors can follow in a separate PR with an explicit
dependency decision.

## Type of Change

- [x] New feature (non-breaking change which adds functionality)

## Changes Made

- `headroom/transforms/config_compressor.py`: added Tier 3
(`_schema_fold`) — TOML `[[array-of-tables]]` → `tomllib` → JSON →
`SmartCrusher(csv-schema)`. New `enable_schema_fold` config flag
(default on; auto-off in lossless mode since it rides `enable_ccr`). The
fold competes with the reversible text tiers and is adopted only when
strictly smaller. Added `_load_toml` (stdlib parser with tomli backport)
and `_json_default` (TOML date/time → ISO; bail on any other
non-serializable value).
- Recovery reuses the existing `CompressionStore` + `Retrieve original:
hash=` marker; no new CCR plumbing.
- `tests/test_transforms_config_compressor.py`: 14 new tests covering
the fold, big-win assertion, byte-exact CCR round-trip, lossless-mode
disable, flag-off, non-TOML skip, no-array skip, small-array
`passthrough` decline, store-failure fallback, savings-floor rejection,
unparseable/non-serializable bails, `_load_toml`/`_json_default` units,
and a datetime-valued fold.

## Testing

- [x] New and existing unit tests pass locally
- [x] New tests added for the new behavior

### Test Output

```
tests/test_transforms_config_compressor.py ............................. [ 59%]
headroom/transforms/config_compressor.py     127      0     36      0   100%
============================== 49 passed in 0.61s ==============================
```

Must-stay-green suites (`test_lossless_mode`,
`test_lossless_excluded_compaction`,
`test_transforms_content_detection`,
`test_compression_fidelity_regression`) — 48 passed.
Router/tabular/smart_crusher regression — 73 + 82 passed. `mypy
--strict` clean on the changed module.

## Real Behavior Proof

- Environment: local, Python 3.11.0, macOS (darwin), `HF_HUB_OFFLINE=1`
- Exact command / steps: parsed a 25-record `[[tool.mypy.overrides]]`
TOML through
`ConfigCompressor(ConfigCompressorConfig(enable_ccr=True)).compress()`,
then retrieved the CCR hash from the `CompressionStore`.
- Observed result: `strategy=config_schema_fold`, 2765 → 840 chars (30%
of original); the marker hash resolved to the byte-exact original
(`recovered == original` True); with `enable_ccr=False` (lossless mode)
the fold did not run and no marker was emitted; a 3-record long-valued
`[[package]]` array correctly declined (SmartCrusher `passthrough`).
- Not tested: the live proxy end-to-end path and non-TOML flavors
(YAML/INI schema folding is intentionally out of scope for this PR).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 19:58:27 +00:00