fix(transforms): stop embedded JSON pre-pass short-circuiting HTML extraction (#3620)

## Description

The embedded-JSON pre-pass in `_apply_strategy_to_content` runs before
any strategy dispatch and returns immediately on a successful splice.
For a whole HTML document containing a JSON fragment, a small local
minify pre-empts whole-document extraction: the report measured a 2%
splice win blocking a 95.5% extraction (my reproduction: 12.4% blocking
45.9%). The parent format's stronger compressor never runs.

This defers the embedded-JSON route when the block's strategy is HTML:
extraction gets first refusal, and the splice runs in the passthrough
fallback only when extraction found nothing. This is option 3 from the
issue ("only allow embedded JSON compression to short-circuit routing
when there is no stronger parent-level strategy available"), scoped to
the HTML case the report measured. Non-HTML strategies keep the
immediate pre-pass untouched. Lossless mode never reaches the strategy
dispatch, so it keeps the immediate-return behavior (JSON minification
is itself byte-lossless).

While tracing the fallback I also fixed an adjacent hole in the same
branch: `trafilatura` returns `''` (not `None`) when nothing extracts,
and the HTML branch treated `''` as a successful compression, so an
unextractable page blanked the whole block and only the empty-output
guard in `compress()` caught it. Empty extraction now collapses to the
passthrough path like `None` does, which is also what makes the deferred
splice reachable for shell-only documents.

Closes #3609

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/content_router.py`: the embedded-JSON pre-pass
body moved unchanged into `_embedded_json_route()`; HTML blocks skip the
immediate splice and retry it in the passthrough fallback, appending
`embedded_json` to the existing chain (`["html", "embedded_json"]`) so
log readers can see the extraction miss. Empty HTML extraction (`''`)
now falls through to passthrough instead of being accepted as a
compression result.
- `tests/test_transforms_content_router.py`: two regression tests
(extractor runs on HTML with embedded JSON; deferred splice still fires
when extraction misses).
- `route_embedded_json` itself is unchanged; its unit tests pass
untouched.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_transforms_content_router.py tests/test_recursive_json.py -q
71 passed in 5.78s

$ uv run ruff check .
All checks passed!

$ uv run mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```

Both new tests fail on base `35b1156` (chain `["embedded_json"]`,
`<html>` retained) and pass with the fix. I also ran every test file
that imports `ContentRouter` (86 files, `grep -rl`): `1462 passed, 41
skipped, 0 failed`.

## Real Behavior Proof

- Environment: WSL2 Ubuntu, Python 3.11 via uv, headroom 0.37.1-dev from
source (`uv sync --frozen --extra dev --extra html`, trafilatura 2.0.0),
pure-Python detection (no ONNX runtime installed).
- Exact command / steps: a real HTML document (30 varied prose
paragraphs x2 + one 60-row embedded JSON array in a single-line
`<script>`, 19,297 chars) through `ContentRouter().compress()`, run with
the patch and then with the patched file stashed (same interpreter):

```text
# WITH FIX
detection : ContentType.HTML
doc tokens: 4824 -> compressed tokens: 2610
strategy  : html  chain=['html']
reduction : 45.9%
<html> tag still present: False

# BASE 35b1156
detection : ContentType.HTML
doc tokens: 4824 -> compressed tokens: 4226
strategy  : html  chain=['embedded_json']
reduction : 12.4%
<html> tag still present: True
```

- Observed result: with the fix the extraction runs and the markup is
gone; on base the JSON splice short-circuits and the full HTML document
(markup included) survives.
- Not tested: proxy mode end-to-end (`/v1/compress` through a live LLM
provider); Windows/macOS; ONNX native detection (the pure-Python
fallback classifies these documents as HTML identically).

## Runtime Rollout Safety

- Rollout-managed feature(s): none — no flag/rollout system gates this
path.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, for HTML-classified blocks
containing routable embedded JSON: the strategy chain changes from
`["embedded_json"]` to `["html"]` (extraction wins) or `["html",
"embedded_json"]` (extraction misses). Everything else is
byte-identical.
- Kill switch / disable path: `enable_html_extractor=false` routes HTML
blocks to passthrough; the deferred splice then fires exactly as the
pre-fix pre-pass did.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a — no
user-facing docs cover route ordering)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

A tiny local JSON win should never block the parent document's real
compressor. Extraction is lossy and the splice is lossless, so deferring
is a behavior trade (extraction output replaces markup+JSON entirely);
the fallback keeps the splice whenever extraction has nothing to offer.

AI use: an AI agent located the short-circuit, ran the before/after
reproductions and the test suite, and drafted this description. The fix
approach follows option 3 suggested in the issue. I reviewed every line
of the diff and the reproduced numbers.
This commit is contained in:
Xmon Dai
2026-09-24 11:09:48 -05:00
committed by GitHub
parent 62d1cc08b1
commit 195910b218
2 changed files with 156 additions and 17 deletions
+57 -17
View File
@@ -3515,22 +3515,20 @@ class ContentRouter(Transform):
# call is a one-shot re-entrancy guard (NOT a depth cap). Deterministic +
# benefit-gated (no size/min thresholds) → prefix-cache- and CCR-store-
# stable, and a strict no-op when the block has no embedded JSON.
if _allow_embedded:
from headroom.transforms.recursive_json import route_embedded_json
def _dispatch_span(span: str) -> str | None:
strat = self._strategy_from_detection_type(_detect_content(span).content_type)
text, _t, _c = self._apply_strategy_to_content(
span,
strat,
context,
question=question,
bias=bias,
_allow_embedded=False,
)
return text if text != span else None
routed = route_embedded_json(content, _dispatch_span, tok=_estimate_tokens)
#
# HTML blocks are the one exception (#3609): whole-document extraction
# routinely beats a local JSON minify by orders of magnitude (the report
# measured a 2% splice win pre-empting a 95% extraction), so the splice
# must not short-circuit the parent strategy. Defer it: run the HTML
# dispatch below first and compute the splice in the passthrough
# fallback only when extraction found nothing. Lossless mode never
# reaches that dispatch, so it keeps the immediate-return behavior
# (JSON minification is itself byte-lossless).
_defer_embedded_for_html = (
_allow_embedded and strategy is CompressionStrategy.HTML and not self.config.lossless
)
if _allow_embedded and not _defer_embedded_for_html:
routed = self._embedded_json_route(content, context, question, bias)
if routed is not None:
return routed, _estimate_tokens(routed), ["embedded_json"]
@@ -3830,9 +3828,15 @@ class ContentRouter(Transform):
# ``result.extracted is None`` path (chain
# ``[html, passthrough]``). A real extraction is
# byte-identical to the historical ``result.extracted``.
# An EMPTY extraction (trafilatura found no article text)
# collapses to None too: a blank page is "nothing
# extracts", and returning "" here would discard the
# whole block instead of falling through (#3609).
output = self._registry_compress("html", strategy, content, context, bias)
compressed = (
output.content if output is not None and output.compressed else None
output.content
if output is not None and output.compressed and output.content.strip()
else None
)
# A page that is all <script>/<style> extracts to "".
# That is "nothing extracted" (fall through to the
@@ -4067,6 +4071,16 @@ class ContentRouter(Transform):
)
return compressed, compressed_tokens, strategy_chain
# #3609: deferred embedded-JSON splice. The HTML strategy ran above and
# extracted nothing (``compressed`` is still None), so the local JSON
# win — held back at the top to give whole-document extraction first
# refusal — is now the best available outcome.
if _defer_embedded_for_html:
routed = self._embedded_json_route(content, context, question, bias)
if routed is not None:
strategy_chain.append("embedded_json")
return routed, _estimate_tokens(routed), strategy_chain
# Fallback: return unchanged
if self.config.enable_dense_line_elision:
dense = self._elide_dense(content, context)
@@ -4140,6 +4154,32 @@ class ContentRouter(Transform):
return None
return elided, _estimate_tokens(elided)
def _embedded_json_route(
self, content: str, context: str, question: str | None, bias: float
) -> str | None:
"""Route embedded JSON spans inside ``content`` through the JSON compressors.
Extracted from the ``_apply_strategy_to_content`` pre-pass so the HTML
strategy can defer the splice (#3609) and retry it in the passthrough
fallback. Returns the spliced block, or ``None`` when nothing
safe/smaller applied.
"""
from headroom.transforms.recursive_json import route_embedded_json
def _dispatch_span(span: str) -> str | None:
strat = self._strategy_from_detection_type(_detect_content(span).content_type)
text, _t, _c = self._apply_strategy_to_content(
span,
strat,
context,
question=question,
bias=bias,
_allow_embedded=False,
)
return text if text != span else None
return route_embedded_json(content, _dispatch_span, tok=_estimate_tokens)
def _try_ml_compressor(
self,
content: str,
+99
View File
@@ -2151,3 +2151,102 @@ def test_prompt_text_compresses_without_replay_guarantee(monkeypatch: pytest.Mon
min_tokens_to_compress=1,
)
assert _TASK in calls
def _html_doc_with_embedded_json(*, padded: bool = True) -> str:
"""A whole HTML document with one embedded JSON array-of-objects.
The JSON sits in a single-line ``<script>`` so the mixed-content regex
heuristics do not fire: the block detects as pure HTML and takes the
``_compress_pure`` path. ``padded`` keeps the default ``json.dumps``
separators so the array is minifiable — a compact array gives the JSON
route no benefit and both passes no-op.
"""
rows = [
{
"id": i,
"name": f"item-{i}",
"tags": ["alpha", "beta"],
"score": i * 3,
"description": "a routine record with repeated descriptive fields",
}
for i in range(60)
]
embedded = json.dumps(rows) if padded else json.dumps(rows, separators=(",", ":"))
paras = "".join(
f"<p>Paragraph {i}: distributed systems rely on consensus protocols to "
f"agree on state. Raft separates leader election from log replication, "
f"which makes it easier to reason about.</p>\n"
for i in range(30)
)
return (
"<!DOCTYPE html>\n<html><head><title>Raft Notes</title></head>\n<body>\n"
"<article><h1>Raft consensus notes</h1>\n"
+ paras
+ '<script type="application/json">'
+ embedded
+ "</script>\n"
+ paras
+ "</article>\n</body></html>\n"
)
def test_html_with_embedded_json_runs_extractor_not_json_shortcut() -> None:
"""#3609: the embedded-JSON pre-pass must not short-circuit HTML extraction.
Before the fix, a small local JSON minify (~10% win) returned immediately
from ``_apply_strategy_to_content`` and whole-document extraction (~36%
win on this document) never ran; the issue reported the same shape at
2% vs 95.5%. The splice is now deferred for HTML: extraction gets first
refusal and the JSON route only fires when extraction finds nothing.
"""
pytest.importorskip("trafilatura")
doc = _html_doc_with_embedded_json()
assert _detect_content(doc).content_type is ContentType.HTML
assert not is_mixed_content(doc)
result = ContentRouter().compress(doc)
assert result.strategy_used is CompressionStrategy.HTML
assert result.strategy_chain == ["html"]
# extraction ran: markup is gone and the document collapsed hard
# (the pre-fix JSON shortcut left <html> in place at ~10% reduction)
assert "<html>" not in result.compressed
assert len(result.compressed) < len(doc) * 3 // 5
def test_html_extraction_miss_still_gets_deferred_json_splice() -> None:
"""Deferring must not LOSE the embedded-JSON win when extraction finds nothing.
A shell-only HTML document makes the extractor come back empty; the
deferred splice then still compresses the embedded array. Before the fix
this shape short-circuited at the top (chain ``["embedded_json"]``); now
the block first proves the HTML strategy is a miss (chain
``["html", "embedded_json"]``) with the same spliced output.
"""
pytest.importorskip("trafilatura")
embedded = json.dumps(
[
{"id": i, "name": f"item-{i}", "tags": ["alpha", "beta"], "score": i * 3}
for i in range(40)
]
)
shell = (
"<!DOCTYPE html>\n<html><head><title>t</title></head>\n<body>\n"
'<script type="application/json">' + embedded + "</script>\n"
"</body></html>\n"
)
router = ContentRouter()
# sanity: extraction finds nothing on this shell (empty or None)
extractor = router._get_html_extractor()
assert extractor is not None
assert not (extractor.extract(shell).extracted or "").strip()
text, _tokens, chain = router._apply_strategy_to_content(
shell, CompressionStrategy.HTML, "", None, 1.0
)
assert chain == ["html", "embedded_json"]
assert len(text) < len(shell)
json_marker = '"id":'
assert json_marker not in text # the array was compressed away