mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(content-router): trust confident HTML detection over mixed-content heuristics (#3627)
## Description A complete HTML document (clear DOCTYPE/`<html>` root) that also contains embedded JSON (e.g. a `<script type="application/ld+json">` block) or enough body prose trips `is_mixed_content`'s regex heuristics (`has_json_blocks`, `has_prose`), so it gets routed through the generic MIXED path (split section-by-section, each section compressed independently) instead of the dedicated HTML extractor. The split sections lose the parent HTML type, so the document never reaches whole-document extraction. On the 1244-char product page used in the tests, `compress()` keeps 1211 chars through MIXED (ratio 0.97) and 571 chars through the HTML strategy (ratio 0.46). `_determine_strategy` already has this exact override for `SOURCE_CODE` (a Python file with dict literals and docstrings trips the same heuristics, and the code trusts the detector's SOURCE_CODE verdict over them). HTML never got the same override. Closes #3608 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_router.py`: `_determine_strategy` now trusts a `ContentType.HTML` detection over the `is_mixed_content` regex heuristics, same pattern as the `SOURCE_CODE` override immediately above it. Only a complete top-level document qualifies: the payload (after envelope stripping) must start with a DOCTYPE or an `<html>` root. The detector scores tags anywhere in the sample, so a fenced HTML example inside a prose answer also reads as HTML, and extracting the whole answer would drop the surrounding instructions and the fence lines. There is deliberately no confidence check at this call site. `detect_content_type` only returns HTML above 0.7 and the native backend hardcodes 1.0, so the previous `>= 0.7` re-check could never be false and only read as a guard that was not one. - `tests/test_transforms_content_router.py`: six tests. A complete product page with inline JSON-LD is shown to trip the real `is_mixed_content` and the real pure-Python detector, then route to HTML. The override is pinned as a single gate. A prose answer with a fenced `<!DOCTYPE html>` example is shown to be mixed and detected HTML, and is pinned to stay MIXED. An `apply()`-level test drives a `role=tool` message through the real router with a deterministic extractor that drops the JSON-LD, and asserts the SKU, price and rating survive. Two supporting tests keep the guard scoped to tool output and cover the no-`trafilatura` passthrough. Tool output is protected by `CompressionStrategy.HTML` being in `LOSSY_UNMARKED_STRATEGIES`. That entry landed independently on `main` in #3778, so this PR does not add it; the `apply()` test above pins the interaction between the routing change and that guard. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py` is clean on Linux; the local Windows run reports only pre-existing POSIX-only attribute errors, see below) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Windows 11, Python 3.13.3 $ python -m pytest tests/test_transforms_content_router.py tests/test_html_tool_reversibility.py \ tests/test_content_router_tool_role_reversibility.py tests/test_compression_units.py \ tests/test_compression_batches.py tests/test_no_ccr_lossy.py tests/test_kompress_marker_gate.py \ tests/test_router_registry_dispatch.py tests/test_builtin_compressor_adapters.py \ tests/test_canonical_pipeline.py tests/test_cross_turn_dedup.py -q 193 passed, 1 skipped in 26.05s # Ubuntu (WSL2), Python 3.13.16, trafilatura 2.3.1, onnxruntime 1.31.1, same 11 files 193 passed, 1 skipped in 26.77s $ python -m pytest tests/test_transforms_content_router.py -k html -q 8 passed, 85 deselected in 3.71s $ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py All checks passed! $ ruff format --check headroom/transforms/content_router.py tests/test_transforms_content_router.py 2 files already formatted $ python -m mypy headroom/transforms/content_router.py headroom\fileperms.py:92: error: Module has no attribute "fchmod"; maybe "chmod"? [attr-defined] headroom\proxy\interceptors\astgrep.py:220,229,242: error: Module has no attribute "O_DIRECTORY" / "O_NOFOLLOW" / "O_NONBLOCK" [attr-defined] Found 7 errors in 2 files (checked 1 source file) # Ubuntu (WSL2), same command $ python -m mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` The mypy errors are all POSIX-only `os` attributes missing on Windows, in files this PR does not touch. Fail-without/pass-with: removing `CompressionStrategy.HTML` from `LOSSY_UNMARKED_STRATEGIES` makes `test_tool_html_with_inline_jsonld_kept_verbatim` fail with the 571-char extraction replacing the page. Reverting just the routing branch makes `test_html_override_has_no_second_confidence_gate` fail (`MIXED` returned instead of `HTML`). ## Real Behavior Proof - Environment: two. (1) Windows 11, Python 3.13.3, `trafilatura==2.3.1`, source checkout at `main` `17d38e56` plus this patch, `PYTHONPATH` pointed at the checkout. Windows selects the pure-Python detection backend by default. (2) Ubuntu on WSL2 (kernel 6.18), Python 3.13.16, `trafilatura==2.3.1`, `onnxruntime==1.31.1`, same checkout and patch, run with both `HEADROOM_DETECT_BACKEND=rust` and `=python`. On Linux the default is the native Magika/ONNX detector, which is the backend CI uses. The fixture is a synthetic product page: DOCTYPE, `<html>`/`<head>`/`<body>`, an `application/ld+json` block with SKU, price and `aggregateRating`, six paragraphs of article copy, a nav and a footer. - Exact command / steps: a script builds that page, prints `is_mixed_content`, `detect_content_type`, what the real extractor keeps, then runs `ContentRouter().compress(doc)` and `ContentRouter().apply([{"role": "tool", "tool_call_id": "call_bash_1", "content": doc}], tokenizer, protect_recent=0, protect_analysis_context=False)`. It was run once with `content_router.py` taken from `main` and once with this patch. - Observed result, `compress()`: before, `is_mixed_content` is `True`, detection is `ContentType.HTML` at confidence 1.0, strategy is `mixed`, 1244 -> 1211 chars (ratio 0.967); after, same detection, strategy is `html`, 1244 -> 571 chars (ratio 0.457). - Observed result, raw extractor and `apply()`: the real extractor returns 571 chars and drops all three facts. With this patch, `apply()` keeps the tool message verbatim at 1244 chars and the SKU, price and rating are all present. The unchanged result comes from the `LOSSY_UNMARKED_STRATEGIES` guard (#3778) now that routing reaches HTML. Removing that one entry reproduces the reviewer's replay on this branch: `apply()` replaces the message with the 571-char extraction and the SKU, price and rating are gone. - Observed result, native detector on Linux: `HEADROOM_DETECT_BACKEND=rust` detects the same page as `ContentType.HTML` at confidence 1.0, routes it to `strategy=html` (1244 -> 571), `apply()` keeps all 1244 chars and all three facts, and the mutation above still makes the regression test fail. The `=python` run matches. This is the backend Linux CI takes by default, so the routing branch is not relying on the Windows regex detector's verdict. - Observed result, fenced example: a prose answer with a fenced complete HTML page is `is_mixed_content: True` and detected `ContentType.HTML` at 0.71. With the unconditional override it routed to `strategy=html` and shrank 842 -> 450 chars, dropping the closing instructions and the fence lines. With the top-level check it stays `mixed` and the surrounding text survives. `test_fenced_html_example_stays_mixed` fails when the check is removed (it sees `HTML` instead of `MIXED`). - Not tested: the reporters' actual pages, a live provider request, or the full pytest suite with the `html` extra installed. The targeted set above is what ran locally. The native Magika chain in the WSL run came from the `headroom-ai==0.41.0` manylinux abi3 wheel's `_core.abi3.so`, not from a local Rust build of this checkout; only the Python routing code under test came from the patch. The `_core` ABI is unchanged by this PR. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is a routing decision inside `ContentRouter`, not behind a new flag. - Minimum rollout channel: N/A - Stable/default behavior changed: yes, in two ways. A complete HTML document that trips the mixed heuristics now routes to the HTML strategy, which compresses it much further (0.97 -> 0.46 on the fixture) and, for non-tool content, keeps only extracted prose. For `role=tool` content whose extraction is unmarked, the guard keeps the original verbatim, so mixed HTML tool pages that used to shrink slightly through MIXED (about 3% on the fixture) now keep all their bytes. Same trade-off #3778 already made for HTML pages that route to HTML directly. - Kill switch / disable path: `enable_html_extractor=False` stops extraction and the HTML branch falls through to passthrough. `ccr_inject_marker=False` remains the existing intentional opt-out of the tool-output guard. - Unsafe override required: none. - Qualification impact: none. CI's `test-extras` job already installs the `html` extra and runs the HTML reversibility tests (#3778). - Rollback path: revert this commit. One additive branch plus tests, no state, schema or migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: no public API or config change) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective and that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The branch was resynchronized with current `main` (`17d38e56`, 578 commits since the original base) and squashed into a single commit on top of it, so the merge conflict with #3778's guard changes is resolved by taking `main`'s version of the strategy set as-is. The delta is now the routing branch plus the tests, with no changes to `LOSSY_UNMARKED_STRATEGIES`. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3627" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `82e1ef5ab82c9553b0104ba934f7231213c480b8`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head89da6eb0fand author discussion, including the earlier JSON-LD/tool-output preservation and fenced-example fixes. Clean integration with upstream/main2d3ee51846. The router trusts HTML detection only for a top-level DOCTYPE/html document; prose containing fenced HTML remains mixed. Existing tool-output preservation guard is retained. No new PR-specific source blocker found in reviewed scope. Combined router/release/version/MCP run: 181 passed, three failed. All three failing cases were rerun on a separate clean current-main worktree with the same existing native extension and Python detection backend, and failed identically: JSON-in-HTML unrecoverable_sections and two minification token-shrink assertions. They are inherited/local-environment qualification failures, not PR regressions and not grounds for withholding approval under the user directive. Ruff check/format and whitespace pass. No fresh native build/live provider claim. Maintainers handle fresh exact-head qualification; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Co-authored-by: Ayush <ayush@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -1214,6 +1214,11 @@ def _resolve_detect_backend() -> str:
|
||||
_DETECT_TIMEOUT_ENV = "HEADROOM_DETECT_TIMEOUT_SECS"
|
||||
_DEFAULT_DETECT_TIMEOUT_SECS = 5.0
|
||||
|
||||
# A complete document starts with a DOCTYPE or the root element. The HTML
|
||||
# detector scores tags anywhere in the sample, so this is what separates a real
|
||||
# page from a fenced example embedded in prose.
|
||||
_TOP_LEVEL_HTML_DOCUMENT = re.compile(r"^\s*(?:<!doctype\s+html|<html[\s>])", re.IGNORECASE)
|
||||
|
||||
|
||||
def _detect_timeout_secs() -> float:
|
||||
"""Watchdog budget (seconds) for one native detect call.
|
||||
@@ -3507,6 +3512,31 @@ class ContentRouter(Transform):
|
||||
# trust it over the regex heuristics.
|
||||
if detection.content_type == ContentType.SOURCE_CODE and detection.confidence >= 0.8:
|
||||
return self._strategy_from_detection(detection)
|
||||
# Same false-positive shape on a complete HTML document: embedded
|
||||
# JSON (``<script type="application/ld+json">``) or ordinary body
|
||||
# prose trips ``has_json_blocks``/``has_prose``, so a page with a
|
||||
# clear DOCTYPE/``<html>`` root still gets routed through MIXED —
|
||||
# split section-by-section instead of reaching the HTML extractor.
|
||||
# The split JSON section can compress well in isolation while the
|
||||
# surrounding markup never gets HTML-specific extraction, and the
|
||||
# split sections lose the parent HTML type entirely.
|
||||
#
|
||||
# No confidence gate here on purpose. ``detect_content_type`` only
|
||||
# returns HTML when its own score clears 0.7 (content_detector
|
||||
# ``_try_detect_html``), and the native backend hardcodes
|
||||
# ``confidence=1.0`` for every result, so a threshold at this call
|
||||
# site would read as "high-confidence only" while never filtering
|
||||
# anything. The detector's gate is the real one.
|
||||
if detection.content_type == ContentType.HTML:
|
||||
# Only a complete top-level document gets the override. The
|
||||
# detector scores tags anywhere in the sample, so a fenced HTML
|
||||
# example inside a prose answer also reads as HTML; extracting
|
||||
# the whole response would drop the surrounding instructions and
|
||||
# the fence lines. Keep that shape on MIXED.
|
||||
document = _strip_detection_envelope(content)
|
||||
if _TOP_LEVEL_HTML_DOCUMENT.match(document):
|
||||
return self._strategy_from_detection(detection)
|
||||
return CompressionStrategy.MIXED
|
||||
return CompressionStrategy.MIXED
|
||||
|
||||
# 2. Not mixed — map the detected type straight to a strategy.
|
||||
|
||||
@@ -1116,6 +1116,241 @@ def test_source_code_code_aware_enabled_uses_code_aware(
|
||||
assert strategy is CompressionStrategy.CODE_AWARE
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MIXED false-positive on a complete HTML document: embedded JSON or ordinary
|
||||
# body prose triggers ``has_json_blocks``/``has_prose`` in ``is_mixed_content``,
|
||||
# misclassifying a page with a clear DOCTYPE/``<html>`` root as MIXED. The
|
||||
# detector says HTML. ``_determine_strategy`` must trust it over the regex
|
||||
# heuristics, same as the SOURCE_CODE case above. See #3608, #3609.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Product page with inline JSON-LD: the shape from #3608. The ``ld+json`` block
|
||||
# trips ``has_json_blocks``, the article copy trips ``has_prose``, so
|
||||
# ``is_mixed_content`` says True even though the document is unmistakably HTML.
|
||||
# trafilatura keeps the prose and drops the ``<script>`` payload, which is why
|
||||
# the tool-role guard below has to treat HTML as lossy-unmarked.
|
||||
MIXED_HTML_PRODUCT_PAGE = """<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>Field Jacket | Example Outfitters</title>
|
||||
<script type="application/ld+json">
|
||||
{
|
||||
"@context": "https://schema.org",
|
||||
"@type": "Product",
|
||||
"name": "Field Jacket",
|
||||
"sku": "SKU-48213",
|
||||
"offers": {"@type": "Offer", "price": "189.00", "priceCurrency": "USD"},
|
||||
"aggregateRating": {"@type": "AggregateRating", "ratingValue": "4.7", "reviewCount": "132"}
|
||||
}
|
||||
</script>
|
||||
</head>
|
||||
<body>
|
||||
<header><nav><a href="/">Home</a> <a href="/jackets">Jackets</a></nav></header>
|
||||
<article>
|
||||
<h1>Field Jacket</h1>
|
||||
<p>The Field Jacket is cut from a waxed cotton shell that sheds light rain without feeling stiff.</p>
|
||||
<p>Two chest pockets and two hand pockets close with brass snaps and the collar stands up on its own.</p>
|
||||
<p>We recommend sizing down if you plan to layer it over a sweater during colder months.</p>
|
||||
<p>Every jacket is finished by hand in our workshop and ships within two business days.</p>
|
||||
<p>Returns are accepted for thirty days as long as the tags are still attached to the garment.</p>
|
||||
<p>Care instructions are printed on the inner label so sponge clean only and never machine wash.</p>
|
||||
</article>
|
||||
<footer><p>Example Outfitters, 42 Market Street</p></footer>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
# What an HTML extractor keeps from that page: prose only, no JSON-LD facts.
|
||||
HTML_PROSE_ONLY = "\n\n".join(
|
||||
[
|
||||
"Field Jacket",
|
||||
"The Field Jacket is cut from a waxed cotton shell that sheds light rain without "
|
||||
"feeling stiff.",
|
||||
"Two chest pockets and two hand pockets close with brass snaps and the collar stands "
|
||||
"up on its own.",
|
||||
"We recommend sizing down if you plan to layer it over a sweater during colder months.",
|
||||
"Every jacket is finished by hand in our workshop and ships within two business days.",
|
||||
"Returns are accepted for thirty days as long as the tags are still attached to the "
|
||||
"garment.",
|
||||
"Care instructions are printed on the inner label so sponge clean only and never "
|
||||
"machine wash.",
|
||||
"Example Outfitters, 42 Market Street",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
# A prose answer that embeds a fenced complete HTML example. Seven sentences
|
||||
# trip ``has_prose`` and the fence trips ``has_code_fences``, so
|
||||
# ``is_mixed_content`` says True; the tags inside the fence also make the
|
||||
# detector report HTML. The override must not treat the whole answer as a page.
|
||||
FENCED_HTML_EXAMPLE = """Here is how the page should look. Keep the header and the footer.
|
||||
The layout uses a centered container and a sticky header.
|
||||
Colours come from the theme variables defined at the top of the stylesheet.
|
||||
Do not rename the ids, the script queries them directly.
|
||||
The footer should stay at the bottom on short pages.
|
||||
Test it in both light and dark mode before shipping it.
|
||||
If the cards wrap badly, reduce the gap between them.
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head><meta charset="utf-8"><title>Demo</title></head>
|
||||
<body>
|
||||
<div id="header">Header</div>
|
||||
<div id="nav">Nav</div>
|
||||
<div id="main">
|
||||
<div class="card">Card one</div>
|
||||
<div class="card">Card two</div>
|
||||
</div>
|
||||
<div id="footer">Footer</div>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
Save that as index.html and open it in a browser.
|
||||
The `div` ids are what the JavaScript expects, so do not rename them.
|
||||
"""
|
||||
|
||||
|
||||
def _stub_html_extractor(extracted: str) -> tuple[SimpleNamespace, list[str]]:
|
||||
"""Deterministic stand-in for HTMLExtractor (no trafilatura needed)."""
|
||||
seen: list[str] = []
|
||||
|
||||
def _extract(html: str) -> SimpleNamespace:
|
||||
seen.append(html)
|
||||
return SimpleNamespace(extracted=extracted)
|
||||
|
||||
return SimpleNamespace(extract=_extract), seen
|
||||
|
||||
|
||||
def test_mixed_html_product_page_routes_to_html() -> None:
|
||||
"""A complete HTML page with inline JSON-LD must reach the HTML strategy.
|
||||
|
||||
Both signals come from the real code: ``is_mixed_content`` and the
|
||||
pure-Python detector (the same one ``_detect_content`` falls back to on
|
||||
Windows). No monkeypatching, so the heuristic shapes that caused #3608 are
|
||||
the ones under test.
|
||||
"""
|
||||
from headroom.transforms.content_detector import detect_content_type
|
||||
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
|
||||
assert is_mixed_content(MIXED_HTML_PRODUCT_PAGE) is True
|
||||
detection = detect_content_type(MIXED_HTML_PRODUCT_PAGE)
|
||||
assert detection.content_type is ContentType.HTML
|
||||
|
||||
strategy = router._determine_strategy(MIXED_HTML_PRODUCT_PAGE, mixed=True, detection=detection)
|
||||
assert strategy is CompressionStrategy.HTML
|
||||
|
||||
|
||||
def test_html_override_has_no_second_confidence_gate() -> None:
|
||||
"""The router trusts the detector's HTML type without re-checking confidence.
|
||||
|
||||
``detect_content_type`` only returns HTML when its own score clears 0.7, and
|
||||
the native backend hardcodes confidence 1.0, so a threshold here would never
|
||||
filter anything — it only read as a real guard. This pins the single gate so
|
||||
it isn't re-added as dead code.
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
|
||||
strategy = router._determine_strategy(
|
||||
MIXED_HTML_PRODUCT_PAGE,
|
||||
mixed=True,
|
||||
detection=DetectionResult(ContentType.HTML, 0.5, {}),
|
||||
)
|
||||
assert strategy is CompressionStrategy.HTML
|
||||
|
||||
|
||||
def test_fenced_html_example_stays_mixed() -> None:
|
||||
"""A fenced HTML example inside prose must not be extracted as a page.
|
||||
|
||||
The detector reads the tags inside the fence and reports HTML, and the
|
||||
answer is mixed, so without the top-level check the whole response goes to
|
||||
the HTML extractor and the surrounding instructions and fence lines are
|
||||
lost. Both signals come from the real detector and the real heuristic.
|
||||
"""
|
||||
from headroom.transforms.content_detector import detect_content_type
|
||||
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
|
||||
assert is_mixed_content(FENCED_HTML_EXAMPLE) is True
|
||||
detection = detect_content_type(FENCED_HTML_EXAMPLE)
|
||||
assert detection.content_type is ContentType.HTML
|
||||
|
||||
strategy = router._determine_strategy(FENCED_HTML_EXAMPLE, mixed=True, detection=detection)
|
||||
assert strategy is CompressionStrategy.MIXED
|
||||
|
||||
|
||||
def test_tool_html_with_inline_jsonld_kept_verbatim(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""apply() must not replace tool HTML with prose that lost its JSON-LD (#3627).
|
||||
|
||||
Routing now sends this page to the HTML extractor, which keeps the article
|
||||
and drops SKU, price and rating without storing the original or emitting a
|
||||
CCR marker. The tool-ground-truth guard has to keep the page verbatim.
|
||||
"""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
monkeypatch.setattr(
|
||||
content_router_module,
|
||||
"_detect_content",
|
||||
lambda content: DetectionResult(ContentType.HTML, 1.0, {}),
|
||||
)
|
||||
extractor, seen = _stub_html_extractor(HTML_PROSE_ONLY)
|
||||
monkeypatch.setattr(router, "_get_html_extractor", lambda: extractor)
|
||||
|
||||
result = router.apply(
|
||||
[{"role": "tool", "tool_call_id": "call_bash_1", "content": MIXED_HTML_PRODUCT_PAGE}],
|
||||
_ChurnTokenizer(),
|
||||
protect_recent=0,
|
||||
protect_analysis_context=False,
|
||||
)
|
||||
|
||||
assert seen, "extractor never ran: routing did not reach the HTML strategy"
|
||||
out = result.messages[0]["content"]
|
||||
assert out == MIXED_HTML_PRODUCT_PAGE
|
||||
for fact in ("SKU-48213", "189.00", "4.7"):
|
||||
assert fact in out
|
||||
|
||||
|
||||
def test_assistant_html_still_compressed(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The HTML guard is scoped to tool ground truth, like the other lossy paths."""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
monkeypatch.setattr(
|
||||
content_router_module,
|
||||
"_detect_content",
|
||||
lambda content: DetectionResult(ContentType.HTML, 1.0, {}),
|
||||
)
|
||||
extractor, seen = _stub_html_extractor(HTML_PROSE_ONLY)
|
||||
monkeypatch.setattr(router, "_get_html_extractor", lambda: extractor)
|
||||
|
||||
result = router.apply(
|
||||
[{"role": "assistant", "content": MIXED_HTML_PRODUCT_PAGE}],
|
||||
_ChurnTokenizer(),
|
||||
protect_recent=0,
|
||||
protect_analysis_context=False,
|
||||
)
|
||||
|
||||
assert seen, "extractor never ran: routing did not reach the HTML strategy"
|
||||
assert result.messages[0]["content"] == HTML_PROSE_ONLY
|
||||
|
||||
|
||||
def test_html_strategy_without_extractor_passes_through(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Without the ``[html]`` extra (no trafilatura) HTML stays untouched."""
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
monkeypatch.setattr(
|
||||
content_router_module,
|
||||
"_detect_content",
|
||||
lambda content: DetectionResult(ContentType.HTML, 1.0, {}),
|
||||
)
|
||||
monkeypatch.setattr(router, "_get_html_extractor", lambda: None)
|
||||
|
||||
result = router.compress(MIXED_HTML_PRODUCT_PAGE)
|
||||
assert result.compressed == MIXED_HTML_PRODUCT_PAGE
|
||||
|
||||
|
||||
def test_source_code_passthrough_does_not_invoke_kompress(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
|
||||
Reference in New Issue
Block a user