mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
93279f3208 |
fix(smart-crusher): preserve unchanged JSON bytes (#4044)
## Description Closes #3697. An unchanged SmartCrusher result now keeps the original bytes and reports no structural compression. Formatting-only savings come from the existing Rust JsonMinifier, not from pretending the JSON value changed. The minifier preserves duplicate keys, order, escapes and number literals. A mixed result cannot use one section's CCR marker to hide unmarked loss in another. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update ## Changes Made - Keep SmartCrusher's exact-byte no-op behavior; route actual whitespace savings through JsonMinifier in Python and the eligible native live zone. - Keep all records and JSON literals; do not store originals or inject CCR markers for lossless minification. - Preserve mixed routing and reject uncovered lossy sections in both tool-result formats. Explicit no-CCR behavior is unchanged. - Replace incidental forwarding tests with native minification and complete-record regressions; document the distinction. ## Testing - [ ] Entire Python suite — not claimed. - [x] Four consumer/native modules: `python -m pytest -q tests/test_recursive_json.py tests/test_transforms_content_router.py tests/test_transforms/test_smart_crusher_rust_parity.py tests/test_transforms/test_smart_crusher_lossless_default.py` - [x] Changed-file Ruff checks; normal commit hooks, including mypy and commitlint. - [x] Manual public-API runs, with and without the HTML extra. ### Test Output Consumer/native modules: ```text collected 136 items ============================= 136 passed in 36.43s ============================= ``` Normal pre-push (fmt, clippy, 1,668 Rust tests with 3 ignored, native build, Python checks, commitlint): ```text ======================== 180 passed, 1 warning in 8.35s ======================== ✅ ci-precheck PASSED — safe to push. To https://github.com/oxura/headroom.git 64688217..0002af3e HEAD -> fix/3697-unchanged-json-passthrough ``` The first push stopped on an inherited commit footer because `origin/main` was stale. Fetching the actual upstream ref fixed the lint range; the normal push then passed. No source merge, lint-rule change or hook bypass. Hosted checks on `0002af3e` now pass: all four Python shards, Windows permissions, smoke tests, native wrappers, parity and Docker e2e. [Checks](https://github.com/headroomlabs-ai/headroom/pull/4044/checks). ## Real Behavior Proof - Environment: Linux x86-64, Python 3.13.14, Rust 1.95 and the actual extension built from this checkout. No provider billing calls. - Exact command / steps: the earlier localhost `/v1/compress` and TransformPipeline runs exercised the original fix. For this follow-up, call `ContentRouter.compress` on a data-only HTML page containing 60 complete records, with `smart_crusher_lossless_only=True`, `smart_crusher_with_compaction=False`, and `enable_kompress=False`. Run it once without trafilatura, then with the locked 2.0.0 extra. The new public-API test contains the full input/config. Separately call `route_embedded_json` on a compact array followed by a padded array, with Unicode and CRLF surroundings. - Observed result: both HTML modes keep all 60 records and the surrounding markup, with no unrecoverable section. The unchanged first JSON span does not stop the second span's compaction; surrounding text stays exact. Final Codecov is 96%; the two remaining lines are the deferred HTML fallback return. The real no-HTML-extra API run executed both, and local coverage also exercises the unchanged-span branch. - Not tested: live provider acceptance, invoices or production cache hit rates. No native eligibility or admission gates changed. ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing JSON routing. - Minimum rollout channel: unchanged. - Stable/default behavior changed: eligible formatting savings return as json_minifier; standalone SmartCrusher remains byte-exact on no-ops. - Kill switch / disable path: existing routing configuration and HEADROOM_LIVE_ZONE_DISABLE_ARMS. - Unsafe override required: no. - Qualification impact: no-op, formatting, structural and CCR paths exercised separately. - Rollback path: revert the minifier integration; no persistence migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I did not edit CHANGELOG.md — release-please owns it ## Additional Notes Addresses the [minifier review](https://github.com/headroomlabs-ai/headroom/pull/4044#issuecomment-6157583285). The maintainer’s main merge is retained. The latest follow-up only changes consumer tests. AI-assisted; no subagents. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
737b332129 |
feat(config): fold TOML array-of-tables to csv-schema via SmartCrusher (#1799)
## Description Adds a schema-fold tier to the structured-config compressor (introduced in #1784). TOML files containing an `[[array-of-tables]]` are parsed with the stdlib `tomllib` reference parser and bridged to SmartCrusher's lossless `csv-schema` renderer, which folds the repeated per-record keys into a single schema over the rows. On lockfiles and override-lists — where the repeated keys dominate the byte count — this is a large win. **Stacked on #1784 — review only the top commit** (`feat(config): fold TOML array-of-tables to csv-schema via SmartCrusher`). The base commit is #1784's config-compressor PR; this PR will collapse to the single new commit once #1784 merges. Faithfulness is guaranteed by construction, not by a heuristic: - `tomllib` is the reference TOML parser, so the extracted records are ground-truth. - `csv-schema` is a lossless JSON renderer (`smart_crusher.py` documents it as such), so the model reads a faithful, reformatted view of the exact parsed data. - Byte-exact recovery rides the existing CCR path — the original is persisted to the `CompressionStore` and a `Retrieve original: hash=…` marker is emitted. The fold is only emitted when that store write succeeds, so nothing is ever unrecoverable. Scope is deliberately **TOML-only**: `tomllib` is stdlib, whereas PyYAML is only a *transitive* dependency (not declared in `pyproject.toml`), and INI record-sections would need a bespoke dict-of-dicts→records transform. Those flavors can follow in a separate PR with an explicit dependency decision. ## Type of Change - [x] New feature (non-breaking change which adds functionality) ## Changes Made - `headroom/transforms/config_compressor.py`: added Tier 3 (`_schema_fold`) — TOML `[[array-of-tables]]` → `tomllib` → JSON → `SmartCrusher(csv-schema)`. New `enable_schema_fold` config flag (default on; auto-off in lossless mode since it rides `enable_ccr`). The fold competes with the reversible text tiers and is adopted only when strictly smaller. Added `_load_toml` (stdlib parser with tomli backport) and `_json_default` (TOML date/time → ISO; bail on any other non-serializable value). - Recovery reuses the existing `CompressionStore` + `Retrieve original: hash=` marker; no new CCR plumbing. - `tests/test_transforms_config_compressor.py`: 14 new tests covering the fold, big-win assertion, byte-exact CCR round-trip, lossless-mode disable, flag-off, non-TOML skip, no-array skip, small-array `passthrough` decline, store-failure fallback, savings-floor rejection, unparseable/non-serializable bails, `_load_toml`/`_json_default` units, and a datetime-valued fold. ## Testing - [x] New and existing unit tests pass locally - [x] New tests added for the new behavior ### Test Output ``` tests/test_transforms_config_compressor.py ............................. [ 59%] headroom/transforms/config_compressor.py 127 0 36 0 100% ============================== 49 passed in 0.61s ============================== ``` Must-stay-green suites (`test_lossless_mode`, `test_lossless_excluded_compaction`, `test_transforms_content_detection`, `test_compression_fidelity_regression`) — 48 passed. Router/tabular/smart_crusher regression — 73 + 82 passed. `mypy --strict` clean on the changed module. ## Real Behavior Proof - Environment: local, Python 3.11.0, macOS (darwin), `HF_HUB_OFFLINE=1` - Exact command / steps: parsed a 25-record `[[tool.mypy.overrides]]` TOML through `ConfigCompressor(ConfigCompressorConfig(enable_ccr=True)).compress()`, then retrieved the CCR hash from the `CompressionStore`. - Observed result: `strategy=config_schema_fold`, 2765 → 840 chars (30% of original); the marker hash resolved to the byte-exact original (`recovered == original` True); with `enable_ccr=False` (lossless mode) the fold did not run and no marker was emitted; a 3-record long-valued `[[package]]` array correctly declined (SmartCrusher `passthrough`). - Not tested: the live proxy end-to-end path and non-TOML flavors (YAML/INI schema folding is intentionally out of scope for this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |