mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description `HTMLExtractor` can reduce a tool result to article prose without a CCR retrieval marker. The existing tool ground-truth guard omitted `CompressionStrategy.HTML`, so the deleted script or structured data could not be recovered. This is independent of #3627's HTML-vs-MIXED routing change and #3774's test isolation change. Closes #3775. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add `HTML` to the lossy/unmarked strategy sets used by `ContentRouter.apply` and the provider-neutral shell-output unit path. Unmarked HTML extraction now leaves tool output unchanged; a future extraction with a real CCR marker can still be accepted. - Add three real-`trafilatura` regression cases for Anthropic `tool_result`, OpenAI `role=tool`, and a structured `local_shell_call_output`. The tests failed on the original code and passed with the two set entries. - Install the existing `html` extra in CI's `test-extras` job and run the HTML extractor and new reversibility tests there. The main pytest shards currently install `[dev]` without `html`; the `[sandbox]` smoke job installs it but does not run pytest. ## Testing - [ ] Unit tests pass (`pytest`) — the full suite was not run; the focused run below passed. - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — local Windows run has the unrelated errors listed below. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Before the source change: python -m pytest tests/test_html_tool_reversibility.py -q --tb=short 3 failed (unmarked HTML replaced both message shapes and the provider unit). After the source change, Windows Python 3.12.13 with trafilatura 2.0.0: python -m pytest tests/test_transforms/test_html_extractor.py tests/test_html_tool_reversibility.py tests/test_compression_units.py tests/test_compression_batches.py tests/test_content_router_tool_role_reversibility.py tests/test_transforms_content_router.py tests/test_no_ccr_lossy.py -q --tb=short 137 passed in 9.55s ruff check . All checks passed! ruff format --check . 1642 files already formatted git diff --check (no output) python -m pytest tests/test_transforms/test_dense_line_elider.py -q --tb=short 9 passed on current head after #3774's test isolation change merged into main. mypy headroom 2 errors in unmodified files on Windows: headroom/fileperms.py:71 (os.fchmod), headroom/integrations/langchain/chat_model.py:394 (truthy-function). Hosted Linux CI runs a separately pinned mypy version. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.13, source checkout at `df8ebdf97` plus this patch, `uv sync --frozen --extra dev --extra html` using that interpreter, `trafilatura==2.0.0`. `ContentRouterConfig(enable_kompress=False, min_section_tokens=10)`, default CCR marker setting, local `gpt-4o` token counter. Windows selected the pure-Python content detector; no live provider request was made. - Exact command / steps: run this from the repository root after installing those extras: ```powershell @' from headroom.providers import OpenAIProvider from headroom.tokenizer import Tokenizer from headroom.transforms.content_router import ContentRouter, ContentRouterConfig js = "var a=1;function f(x){return x*2};" * 300 html = "<!doctype html><html><head><script>" + js + "</script></head><body><p>the answer is 42</p></body></html>" messages = [ {"role": "user", "content": "check the site"}, {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "Bash", "input": {"command": "curl -s x"}}]}, {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": html}]}, {"role": "assistant", "content": "ok"}, {"role": "user", "content": "and now?"}, ] tokenizer = Tokenizer(OpenAIProvider().get_token_counter("gpt-4o"), "gpt-4o") router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10)) direct = router.compress(html, context="tool_result") block = router.apply(messages, tokenizer).messages[2]["content"][0]["content"] out = block if isinstance(block, str) else block[0]["text"] print("direct:", direct.strategy_used.value, len(html), "->", len(direct.compressed), repr(direct.compressed)) print("tool_result:", len(html), "->", len(out), "same:", out == html, "marker:", "Retrieve original: hash=" in out) '@ | .\.venv\Scripts\python.exe - ``` - Observed result: the unfixed router replaced 10,294 HTML characters with 16 unmarked characters; after this patch the tool result retained all 10,294 original characters. ```text Before: direct: html 10294 -> 16 'the answer is 42' tool_result: 10294 -> 16, same: False, marker: False After: direct: html 10294 -> 16 'the answer is 42' tool_result: 10294 -> 10294, same: True, marker: False ``` A second unmocked probe with the same fixture yielded `role=tool: 10294 -> 10294, same=True` and `local_shell_call_output: html, modified=False, same=True, reason=lossy_unrecoverable_tool_output`. - Not tested: the reporter's actual page, a live LLM/provider request, Linux/native detection, or a full local pytest run with `html` installed. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: yes; tool outputs whose HTML extraction is lossy and unmarked are retained verbatim, so those outputs use more tokens. Direct `ContentRouter.compress()` extraction is unchanged. - Kill switch / disable path: `enable_html_extractor=False` prevents HTML extraction. `ccr_inject_marker=False` remains the existing intentional opt-out for the `ContentRouter.apply` guard. - Unsafe override required: none. - Qualification impact: CI's existing `test-extras` job now installs `[html]` and runs real HTML tests; main shards are untouched. The #3774 test isolation change is in current main, outside this PR diff. - Rollback path: revert this commit; no state migration or new dependency is introduced. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Current head `d361388e` merges latest `main`, including [#3790](https://github.com/headroomlabs-ai/headroom/pull/3790)'s fix for the unrelated SQLite test that failed in the previous CI run. On this head, that test, the 137 focused HTML/router/unit tests, and all 9 dense-line-elider tests passed locally; Ruff and `git diff --check` passed. Exact-head hosted checks are pending. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: no new public API or setting) - [ ] My changes generate no new warnings (full suite not verified locally) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes (focused tests pass; full local suite was not run) - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from my Conventional Commit PR title ## Additional Notes This keeps the existing reversible-tool-output contract: an unmarked lossy transform cannot replace tool ground truth. It deliberately does not add a retrieval store or change HTML routing. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
99 lines
3.4 KiB
Python
99 lines
3.4 KiB
Python
"""Real HTML extraction must not erase unrecoverable tool ground truth (#3775)."""
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("trafilatura")
|
|
|
|
from headroom.parser import CCR_RETRIEVAL_MARKER_RE
|
|
from headroom.providers import OpenAIProvider
|
|
from headroom.tokenizer import Tokenizer
|
|
from headroom.transforms.compression_units import CompressionUnit, compress_unit_with_router
|
|
from headroom.transforms.content_router import (
|
|
CompressionStrategy,
|
|
ContentRouter,
|
|
ContentRouterConfig,
|
|
)
|
|
|
|
|
|
def _script_heavy_html() -> str:
|
|
js = "var a=1;function f(x){return x*2};" * 300
|
|
return (
|
|
"<!doctype html><html><head><script>"
|
|
+ js
|
|
+ "</script></head><body><p>the answer is 42</p></body></html>"
|
|
)
|
|
|
|
|
|
def _tokenizer() -> Tokenizer:
|
|
return Tokenizer(OpenAIProvider().get_token_counter("gpt-4o"), "gpt-4o")
|
|
|
|
|
|
@pytest.mark.parametrize("shape", ["tool_result", "role_tool"])
|
|
def test_html_tool_ground_truth_is_recoverable(shape: str) -> None:
|
|
html = _script_heavy_html()
|
|
router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10))
|
|
extracted = router.compress(html, context="tool_result")
|
|
assert extracted.strategy_used is CompressionStrategy.HTML
|
|
assert extracted.compressed != html
|
|
assert router._frozen_verdict_recoverable(
|
|
CompressionStrategy.HTML, extracted.compressed
|
|
) == bool(CCR_RETRIEVAL_MARKER_RE.search(extracted.compressed))
|
|
|
|
if shape == "tool_result":
|
|
messages = [
|
|
{"role": "user", "content": "check the site"},
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "t1",
|
|
"name": "Bash",
|
|
"input": {"command": "curl -s x"},
|
|
}
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": html}],
|
|
},
|
|
{"role": "assistant", "content": "ok"},
|
|
{"role": "user", "content": "and now?"},
|
|
]
|
|
result = router.apply(messages, _tokenizer())
|
|
block = result.messages[2]["content"][0]["content"]
|
|
output = block if isinstance(block, str) else block[0]["text"]
|
|
else:
|
|
result = router.apply(
|
|
[{"role": "tool", "tool_call_id": "call_bash_1", "content": html}],
|
|
_tokenizer(),
|
|
protect_recent=0,
|
|
protect_analysis_context=False,
|
|
)
|
|
output = result.messages[0]["content"]
|
|
|
|
assert output == html or CCR_RETRIEVAL_MARKER_RE.search(output)
|
|
|
|
|
|
def test_html_provider_shell_unit_is_recoverable() -> None:
|
|
html = _script_heavy_html().replace("><", ">\n<")
|
|
router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10))
|
|
extracted = router.compress(html)
|
|
assert extracted.strategy_used is CompressionStrategy.HTML
|
|
assert extracted.compressed != html
|
|
|
|
result = compress_unit_with_router(
|
|
CompressionUnit(
|
|
text=html,
|
|
provider="openai",
|
|
endpoint="responses",
|
|
role="tool",
|
|
item_type="local_shell_call_output",
|
|
min_bytes=1,
|
|
),
|
|
router=router,
|
|
tokenizer=_tokenizer(),
|
|
)
|
|
|
|
assert result.compressed == html or CCR_RETRIEVAL_MARKER_RE.search(result.compressed)
|