fix(content-router): trust confident HTML detection over mixed-content heuristics (#3627)

## Description

A complete HTML document (clear DOCTYPE/`<html>` root) that also
contains embedded JSON (e.g. a `<script type="application/ld+json">`
block) or enough body prose trips `is_mixed_content`'s regex heuristics
(`has_json_blocks`, `has_prose`), so it gets routed through the generic
MIXED path (split section-by-section, each section compressed
independently) instead of the dedicated HTML extractor. The split
sections lose the parent HTML type, so the document never reaches
whole-document extraction. On the 1244-char product page used in the
tests, `compress()` keeps 1211 chars through MIXED (ratio 0.97) and 571
chars through the HTML strategy (ratio 0.46).

`_determine_strategy` already has this exact override for `SOURCE_CODE`
(a Python file with dict literals and docstrings trips the same
heuristics, and the code trusts the detector's SOURCE_CODE verdict over
them). HTML never got the same override.

Closes #3608

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/content_router.py`: `_determine_strategy` now
trusts a `ContentType.HTML` detection over the `is_mixed_content` regex
heuristics, same pattern as the `SOURCE_CODE` override immediately above
it. Only a complete top-level document qualifies: the payload (after
envelope stripping) must start with a DOCTYPE or an `<html>` root. The
detector scores tags anywhere in the sample, so a fenced HTML example
inside a prose answer also reads as HTML, and extracting the whole
answer would drop the surrounding instructions and the fence lines.
There is deliberately no confidence check at this call site.
`detect_content_type` only returns HTML above 0.7 and the native backend
hardcodes 1.0, so the previous `>= 0.7` re-check could never be false
and only read as a guard that was not one.
- `tests/test_transforms_content_router.py`: six tests. A complete
product page with inline JSON-LD is shown to trip the real
`is_mixed_content` and the real pure-Python detector, then route to
HTML. The override is pinned as a single gate. A prose answer with a
fenced `<!DOCTYPE html>` example is shown to be mixed and detected HTML,
and is pinned to stay MIXED. An `apply()`-level test drives a
`role=tool` message through the real router with a deterministic
extractor that drops the JSON-LD, and asserts the SKU, price and rating
survive. Two supporting tests keep the guard scoped to tool output and
cover the no-`trafilatura` passthrough.

Tool output is protected by `CompressionStrategy.HTML` being in
`LOSSY_UNMARKED_STRATEGIES`. That entry landed independently on `main`
in #3778, so this PR does not add it; the `apply()` test above pins the
interaction between the routing change and that guard.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom/transforms/content_router.py`
is clean on Linux; the local Windows run reports only pre-existing
POSIX-only attribute errors, see below)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# Windows 11, Python 3.13.3
$ python -m pytest tests/test_transforms_content_router.py tests/test_html_tool_reversibility.py \
    tests/test_content_router_tool_role_reversibility.py tests/test_compression_units.py \
    tests/test_compression_batches.py tests/test_no_ccr_lossy.py tests/test_kompress_marker_gate.py \
    tests/test_router_registry_dispatch.py tests/test_builtin_compressor_adapters.py \
    tests/test_canonical_pipeline.py tests/test_cross_turn_dedup.py -q
193 passed, 1 skipped in 26.05s

# Ubuntu (WSL2), Python 3.13.16, trafilatura 2.3.1, onnxruntime 1.31.1, same 11 files
193 passed, 1 skipped in 26.77s

$ python -m pytest tests/test_transforms_content_router.py -k html -q
8 passed, 85 deselected in 3.71s

$ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py
All checks passed!

$ ruff format --check headroom/transforms/content_router.py tests/test_transforms_content_router.py
2 files already formatted

$ python -m mypy headroom/transforms/content_router.py
headroom\fileperms.py:92: error: Module has no attribute "fchmod"; maybe "chmod"?  [attr-defined]
headroom\proxy\interceptors\astgrep.py:220,229,242: error: Module has no attribute "O_DIRECTORY" / "O_NOFOLLOW" / "O_NONBLOCK"  [attr-defined]
Found 7 errors in 2 files (checked 1 source file)

# Ubuntu (WSL2), same command
$ python -m mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```

The mypy errors are all POSIX-only `os` attributes missing on Windows,
in files this PR does not touch.

Fail-without/pass-with: removing `CompressionStrategy.HTML` from
`LOSSY_UNMARKED_STRATEGIES` makes
`test_tool_html_with_inline_jsonld_kept_verbatim` fail with the 571-char
extraction replacing the page. Reverting just the routing branch makes
`test_html_override_has_no_second_confidence_gate` fail (`MIXED`
returned instead of `HTML`).

## Real Behavior Proof

- Environment: two. (1) Windows 11, Python 3.13.3, `trafilatura==2.3.1`,
source checkout at `main` `17d38e56` plus this patch, `PYTHONPATH`
pointed at the checkout. Windows selects the pure-Python detection
backend by default. (2) Ubuntu on WSL2 (kernel 6.18), Python 3.13.16,
`trafilatura==2.3.1`, `onnxruntime==1.31.1`, same checkout and patch,
run with both `HEADROOM_DETECT_BACKEND=rust` and `=python`. On Linux the
default is the native Magika/ONNX detector, which is the backend CI
uses. The fixture is a synthetic product page: DOCTYPE,
`<html>`/`<head>`/`<body>`, an `application/ld+json` block with SKU,
price and `aggregateRating`, six paragraphs of article copy, a nav and a
footer.
- Exact command / steps: a script builds that page, prints
`is_mixed_content`, `detect_content_type`, what the real extractor
keeps, then runs `ContentRouter().compress(doc)` and
`ContentRouter().apply([{"role": "tool", "tool_call_id": "call_bash_1",
"content": doc}], tokenizer, protect_recent=0,
protect_analysis_context=False)`. It was run once with
`content_router.py` taken from `main` and once with this patch.
- Observed result, `compress()`: before, `is_mixed_content` is `True`,
detection is `ContentType.HTML` at confidence 1.0, strategy is `mixed`,
1244 -> 1211 chars (ratio 0.967); after, same detection, strategy is
`html`, 1244 -> 571 chars (ratio 0.457).
- Observed result, raw extractor and `apply()`: the real extractor
returns 571 chars and drops all three facts. With this patch, `apply()`
keeps the tool message verbatim at 1244 chars and the SKU, price and
rating are all present. The unchanged result comes from the
`LOSSY_UNMARKED_STRATEGIES` guard (#3778) now that routing reaches HTML.
Removing that one entry reproduces the reviewer's replay on this branch:
`apply()` replaces the message with the 571-char extraction and the SKU,
price and rating are gone.
- Observed result, native detector on Linux:
`HEADROOM_DETECT_BACKEND=rust` detects the same page as
`ContentType.HTML` at confidence 1.0, routes it to `strategy=html` (1244
-> 571), `apply()` keeps all 1244 chars and all three facts, and the
mutation above still makes the regression test fail. The `=python` run
matches. This is the backend Linux CI takes by default, so the routing
branch is not relying on the Windows regex detector's verdict.
- Observed result, fenced example: a prose answer with a fenced complete
HTML page is `is_mixed_content: True` and detected `ContentType.HTML` at
0.71. With the unconditional override it routed to `strategy=html` and
shrank 842 -> 450 chars, dropping the closing instructions and the fence
lines. With the top-level check it stays `mixed` and the surrounding
text survives. `test_fenced_html_example_stays_mixed` fails when the
check is removed (it sees `HTML` instead of `MIXED`).
- Not tested: the reporters' actual pages, a live provider request, or
the full pytest suite with the `html` extra installed. The targeted set
above is what ran locally. The native Magika chain in the WSL run came
from the `headroom-ai==0.41.0` manylinux abi3 wheel's `_core.abi3.so`,
not from a local Rust build of this checkout; only the Python routing
code under test came from the patch. The `_core` ABI is unchanged by
this PR.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a routing decision inside
`ContentRouter`, not behind a new flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes, in two ways. A complete HTML
document that trips the mixed heuristics now routes to the HTML
strategy, which compresses it much further (0.97 -> 0.46 on the fixture)
and, for non-tool content, keeps only extracted prose. For `role=tool`
content whose extraction is unmarked, the guard keeps the original
verbatim, so mixed HTML tool pages that used to shrink slightly through
MIXED (about 3% on the fixture) now keep all their bytes. Same trade-off
#3778 already made for HTML pages that route to HTML directly.
- Kill switch / disable path: `enable_html_extractor=False` stops
extraction and the HTML branch falls through to passthrough.
`ccr_inject_marker=False` remains the existing intentional opt-out of
the tool-output guard.
- Unsafe override required: none.
- Qualification impact: none. CI's `test-extras` job already installs
the `html` extra and runs the HTML reversibility tests (#3778).
- Rollback path: revert this commit. One additive branch plus tests, no
state, schema or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: no
public API or config change)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective and that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The branch was resynchronized with current `main` (`17d38e56`, 578
commits since the original base) and squashed into a single commit on
top of it, so the merge conflict with #3778's guard changes is resolved
by taking `main`'s version of the strategy set as-is. The delta is now
the routing branch plus the tests, with no changes to
`LOSSY_UNMARKED_STRATEGIES`.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3627"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->


### Maintainer current-main requalification

Exact head `82e1ef5ab82c9553b0104ba934f7231213c480b8`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head 89da6eb0f and author discussion,
including the earlier JSON-LD/tool-output preservation and
fenced-example fixes. Clean integration with upstream/main
2d3ee51846. The router trusts HTML
detection only for a top-level DOCTYPE/html document; prose containing
fenced HTML remains mixed. Existing tool-output preservation guard is
retained. No new PR-specific source blocker found in reviewed scope.

Combined router/release/version/MCP run: 181 passed, three failed. All
three failing cases were rerun on a separate clean current-main worktree
with the same existing native extension and Python detection backend,
and failed identically: JSON-in-HTML unrecoverable_sections and two
minification token-shrink assertions. They are
inherited/local-environment qualification failures, not PR regressions
and not grounds for withholding approval under the user directive. Ruff
check/format and whitespace pass. No fresh native build/live provider
claim. Maintainers handle fresh exact-head qualification; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

Co-authored-by: Ayush <ayush@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Ayush
2026-11-05 20:20:59 -06:00
committed by GitHub
co-authored by Ayush JD Davis
parent bd164a2c3b
commit f50db67006
2 changed files with 265 additions and 0 deletions
+30
View File
@@ -1214,6 +1214,11 @@ def _resolve_detect_backend() -> str:
_DETECT_TIMEOUT_ENV = "HEADROOM_DETECT_TIMEOUT_SECS"
_DEFAULT_DETECT_TIMEOUT_SECS = 5.0
# A complete document starts with a DOCTYPE or the root element. The HTML
# detector scores tags anywhere in the sample, so this is what separates a real
# page from a fenced example embedded in prose.
_TOP_LEVEL_HTML_DOCUMENT = re.compile(r"^\s*(?:<!doctype\s+html|<html[\s>])", re.IGNORECASE)
def _detect_timeout_secs() -> float:
"""Watchdog budget (seconds) for one native detect call.
@@ -3507,6 +3512,31 @@ class ContentRouter(Transform):
# trust it over the regex heuristics.
if detection.content_type == ContentType.SOURCE_CODE and detection.confidence >= 0.8:
return self._strategy_from_detection(detection)
# Same false-positive shape on a complete HTML document: embedded
# JSON (``<script type="application/ld+json">``) or ordinary body
# prose trips ``has_json_blocks``/``has_prose``, so a page with a
# clear DOCTYPE/``<html>`` root still gets routed through MIXED —
# split section-by-section instead of reaching the HTML extractor.
# The split JSON section can compress well in isolation while the
# surrounding markup never gets HTML-specific extraction, and the
# split sections lose the parent HTML type entirely.
#
# No confidence gate here on purpose. ``detect_content_type`` only
# returns HTML when its own score clears 0.7 (content_detector
# ``_try_detect_html``), and the native backend hardcodes
# ``confidence=1.0`` for every result, so a threshold at this call
# site would read as "high-confidence only" while never filtering
# anything. The detector's gate is the real one.
if detection.content_type == ContentType.HTML:
# Only a complete top-level document gets the override. The
# detector scores tags anywhere in the sample, so a fenced HTML
# example inside a prose answer also reads as HTML; extracting
# the whole response would drop the surrounding instructions and
# the fence lines. Keep that shape on MIXED.
document = _strip_detection_envelope(content)
if _TOP_LEVEL_HTML_DOCUMENT.match(document):
return self._strategy_from_detection(detection)
return CompressionStrategy.MIXED
return CompressionStrategy.MIXED
# 2. Not mixed — map the detected type straight to a strategy.
+235
View File
@@ -1116,6 +1116,241 @@ def test_source_code_code_aware_enabled_uses_code_aware(
assert strategy is CompressionStrategy.CODE_AWARE
# ---------------------------------------------------------------------------
# MIXED false-positive on a complete HTML document: embedded JSON or ordinary
# body prose triggers ``has_json_blocks``/``has_prose`` in ``is_mixed_content``,
# misclassifying a page with a clear DOCTYPE/``<html>`` root as MIXED. The
# detector says HTML. ``_determine_strategy`` must trust it over the regex
# heuristics, same as the SOURCE_CODE case above. See #3608, #3609.
# ---------------------------------------------------------------------------
# Product page with inline JSON-LD: the shape from #3608. The ``ld+json`` block
# trips ``has_json_blocks``, the article copy trips ``has_prose``, so
# ``is_mixed_content`` says True even though the document is unmistakably HTML.
# trafilatura keeps the prose and drops the ``<script>`` payload, which is why
# the tool-role guard below has to treat HTML as lossy-unmarked.
MIXED_HTML_PRODUCT_PAGE = """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Field Jacket | Example Outfitters</title>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Product",
"name": "Field Jacket",
"sku": "SKU-48213",
"offers": {"@type": "Offer", "price": "189.00", "priceCurrency": "USD"},
"aggregateRating": {"@type": "AggregateRating", "ratingValue": "4.7", "reviewCount": "132"}
}
</script>
</head>
<body>
<header><nav><a href="/">Home</a> <a href="/jackets">Jackets</a></nav></header>
<article>
<h1>Field Jacket</h1>
<p>The Field Jacket is cut from a waxed cotton shell that sheds light rain without feeling stiff.</p>
<p>Two chest pockets and two hand pockets close with brass snaps and the collar stands up on its own.</p>
<p>We recommend sizing down if you plan to layer it over a sweater during colder months.</p>
<p>Every jacket is finished by hand in our workshop and ships within two business days.</p>
<p>Returns are accepted for thirty days as long as the tags are still attached to the garment.</p>
<p>Care instructions are printed on the inner label so sponge clean only and never machine wash.</p>
</article>
<footer><p>Example Outfitters, 42 Market Street</p></footer>
</body>
</html>"""
# What an HTML extractor keeps from that page: prose only, no JSON-LD facts.
HTML_PROSE_ONLY = "\n\n".join(
[
"Field Jacket",
"The Field Jacket is cut from a waxed cotton shell that sheds light rain without "
"feeling stiff.",
"Two chest pockets and two hand pockets close with brass snaps and the collar stands "
"up on its own.",
"We recommend sizing down if you plan to layer it over a sweater during colder months.",
"Every jacket is finished by hand in our workshop and ships within two business days.",
"Returns are accepted for thirty days as long as the tags are still attached to the "
"garment.",
"Care instructions are printed on the inner label so sponge clean only and never "
"machine wash.",
"Example Outfitters, 42 Market Street",
]
)
# A prose answer that embeds a fenced complete HTML example. Seven sentences
# trip ``has_prose`` and the fence trips ``has_code_fences``, so
# ``is_mixed_content`` says True; the tags inside the fence also make the
# detector report HTML. The override must not treat the whole answer as a page.
FENCED_HTML_EXAMPLE = """Here is how the page should look. Keep the header and the footer.
The layout uses a centered container and a sticky header.
Colours come from the theme variables defined at the top of the stylesheet.
Do not rename the ids, the script queries them directly.
The footer should stay at the bottom on short pages.
Test it in both light and dark mode before shipping it.
If the cards wrap badly, reduce the gap between them.
```html
<!DOCTYPE html>
<html lang="en">
<head><meta charset="utf-8"><title>Demo</title></head>
<body>
<div id="header">Header</div>
<div id="nav">Nav</div>
<div id="main">
<div class="card">Card one</div>
<div class="card">Card two</div>
</div>
<div id="footer">Footer</div>
</body>
</html>
```
Save that as index.html and open it in a browser.
The `div` ids are what the JavaScript expects, so do not rename them.
"""
def _stub_html_extractor(extracted: str) -> tuple[SimpleNamespace, list[str]]:
"""Deterministic stand-in for HTMLExtractor (no trafilatura needed)."""
seen: list[str] = []
def _extract(html: str) -> SimpleNamespace:
seen.append(html)
return SimpleNamespace(extracted=extracted)
return SimpleNamespace(extract=_extract), seen
def test_mixed_html_product_page_routes_to_html() -> None:
"""A complete HTML page with inline JSON-LD must reach the HTML strategy.
Both signals come from the real code: ``is_mixed_content`` and the
pure-Python detector (the same one ``_detect_content`` falls back to on
Windows). No monkeypatching, so the heuristic shapes that caused #3608 are
the ones under test.
"""
from headroom.transforms.content_detector import detect_content_type
router = ContentRouter(ContentRouterConfig())
assert is_mixed_content(MIXED_HTML_PRODUCT_PAGE) is True
detection = detect_content_type(MIXED_HTML_PRODUCT_PAGE)
assert detection.content_type is ContentType.HTML
strategy = router._determine_strategy(MIXED_HTML_PRODUCT_PAGE, mixed=True, detection=detection)
assert strategy is CompressionStrategy.HTML
def test_html_override_has_no_second_confidence_gate() -> None:
"""The router trusts the detector's HTML type without re-checking confidence.
``detect_content_type`` only returns HTML when its own score clears 0.7, and
the native backend hardcodes confidence 1.0, so a threshold here would never
filter anything — it only read as a real guard. This pins the single gate so
it isn't re-added as dead code.
"""
router = ContentRouter(ContentRouterConfig())
strategy = router._determine_strategy(
MIXED_HTML_PRODUCT_PAGE,
mixed=True,
detection=DetectionResult(ContentType.HTML, 0.5, {}),
)
assert strategy is CompressionStrategy.HTML
def test_fenced_html_example_stays_mixed() -> None:
"""A fenced HTML example inside prose must not be extracted as a page.
The detector reads the tags inside the fence and reports HTML, and the
answer is mixed, so without the top-level check the whole response goes to
the HTML extractor and the surrounding instructions and fence lines are
lost. Both signals come from the real detector and the real heuristic.
"""
from headroom.transforms.content_detector import detect_content_type
router = ContentRouter(ContentRouterConfig())
assert is_mixed_content(FENCED_HTML_EXAMPLE) is True
detection = detect_content_type(FENCED_HTML_EXAMPLE)
assert detection.content_type is ContentType.HTML
strategy = router._determine_strategy(FENCED_HTML_EXAMPLE, mixed=True, detection=detection)
assert strategy is CompressionStrategy.MIXED
def test_tool_html_with_inline_jsonld_kept_verbatim(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""apply() must not replace tool HTML with prose that lost its JSON-LD (#3627).
Routing now sends this page to the HTML extractor, which keeps the article
and drops SKU, price and rating without storing the original or emitting a
CCR marker. The tool-ground-truth guard has to keep the page verbatim.
"""
router = ContentRouter(ContentRouterConfig())
monkeypatch.setattr(
content_router_module,
"_detect_content",
lambda content: DetectionResult(ContentType.HTML, 1.0, {}),
)
extractor, seen = _stub_html_extractor(HTML_PROSE_ONLY)
monkeypatch.setattr(router, "_get_html_extractor", lambda: extractor)
result = router.apply(
[{"role": "tool", "tool_call_id": "call_bash_1", "content": MIXED_HTML_PRODUCT_PAGE}],
_ChurnTokenizer(),
protect_recent=0,
protect_analysis_context=False,
)
assert seen, "extractor never ran: routing did not reach the HTML strategy"
out = result.messages[0]["content"]
assert out == MIXED_HTML_PRODUCT_PAGE
for fact in ("SKU-48213", "189.00", "4.7"):
assert fact in out
def test_assistant_html_still_compressed(monkeypatch: pytest.MonkeyPatch) -> None:
"""The HTML guard is scoped to tool ground truth, like the other lossy paths."""
router = ContentRouter(ContentRouterConfig())
monkeypatch.setattr(
content_router_module,
"_detect_content",
lambda content: DetectionResult(ContentType.HTML, 1.0, {}),
)
extractor, seen = _stub_html_extractor(HTML_PROSE_ONLY)
monkeypatch.setattr(router, "_get_html_extractor", lambda: extractor)
result = router.apply(
[{"role": "assistant", "content": MIXED_HTML_PRODUCT_PAGE}],
_ChurnTokenizer(),
protect_recent=0,
protect_analysis_context=False,
)
assert seen, "extractor never ran: routing did not reach the HTML strategy"
assert result.messages[0]["content"] == HTML_PROSE_ONLY
def test_html_strategy_without_extractor_passes_through(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Without the ``[html]`` extra (no trafilatura) HTML stays untouched."""
router = ContentRouter(ContentRouterConfig())
monkeypatch.setattr(
content_router_module,
"_detect_content",
lambda content: DetectionResult(ContentType.HTML, 1.0, {}),
)
monkeypatch.setattr(router, "_get_html_extractor", lambda: None)
result = router.compress(MIXED_HTML_PRODUCT_PAGE)
assert result.compressed == MIXED_HTML_PRODUCT_PAGE
def test_source_code_passthrough_does_not_invoke_kompress(
monkeypatch: pytest.MonkeyPatch,
) -> None: