Commit Graph
1 Commits
Author SHA1 Message Date
EasternandJerrettDavis 7c9fbed61a fix(router): preserve unmarked HTML tool output (#3778)
## Description

`HTMLExtractor` can reduce a tool result to article prose without a CCR
retrieval marker. The existing tool ground-truth guard omitted
`CompressionStrategy.HTML`, so the deleted script or structured data
could not be recovered.

This is independent of #3627's HTML-vs-MIXED routing change and #3774's
test isolation change. Closes #3775.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add `HTML` to the lossy/unmarked strategy sets used by
`ContentRouter.apply` and the provider-neutral shell-output unit path.
Unmarked HTML extraction now leaves tool output unchanged; a future
extraction with a real CCR marker can still be accepted.
- Add three real-`trafilatura` regression cases for Anthropic
`tool_result`, OpenAI `role=tool`, and a structured
`local_shell_call_output`. The tests failed on the original code and
passed with the two set entries.
- Install the existing `html` extra in CI's `test-extras` job and run
the HTML extractor and new reversibility tests there. The main pytest
shards currently install `[dev]` without `html`; the `[sandbox]` smoke
job installs it but does not run pytest.

## Testing

- [ ] Unit tests pass (`pytest`) — the full suite was not run; the
focused run below passed.
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — local Windows run has the
unrelated errors listed below.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Before the source change:
python -m pytest tests/test_html_tool_reversibility.py -q --tb=short
3 failed (unmarked HTML replaced both message shapes and the provider unit).

After the source change, Windows Python 3.12.13 with trafilatura 2.0.0:
python -m pytest tests/test_transforms/test_html_extractor.py tests/test_html_tool_reversibility.py tests/test_compression_units.py tests/test_compression_batches.py tests/test_content_router_tool_role_reversibility.py tests/test_transforms_content_router.py tests/test_no_ccr_lossy.py -q --tb=short
137 passed in 9.55s

ruff check .
All checks passed!
ruff format --check .
1642 files already formatted
git diff --check
(no output)

python -m pytest tests/test_transforms/test_dense_line_elider.py -q --tb=short
9 passed on current head after #3774's test isolation change merged into main.

mypy headroom
2 errors in unmodified files on Windows: headroom/fileperms.py:71 (os.fchmod), headroom/integrations/langchain/chat_model.py:394 (truthy-function). Hosted Linux CI runs a separately pinned mypy version.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.13, source checkout at
`df8ebdf97` plus this patch, `uv sync --frozen --extra dev --extra html`
using that interpreter, `trafilatura==2.0.0`.
`ContentRouterConfig(enable_kompress=False, min_section_tokens=10)`,
default CCR marker setting, local `gpt-4o` token counter. Windows
selected the pure-Python content detector; no live provider request was
made.
- Exact command / steps: run this from the repository root after
installing those extras:

```powershell
@'
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig

js = "var a=1;function f(x){return x*2};" * 300
html = "<!doctype html><html><head><script>" + js + "</script></head><body><p>the answer is 42</p></body></html>"
messages = [
    {"role": "user", "content": "check the site"},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "Bash", "input": {"command": "curl -s x"}}]},
    {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": html}]},
    {"role": "assistant", "content": "ok"},
    {"role": "user", "content": "and now?"},
]
tokenizer = Tokenizer(OpenAIProvider().get_token_counter("gpt-4o"), "gpt-4o")
router = ContentRouter(ContentRouterConfig(enable_kompress=False, min_section_tokens=10))
direct = router.compress(html, context="tool_result")
block = router.apply(messages, tokenizer).messages[2]["content"][0]["content"]
out = block if isinstance(block, str) else block[0]["text"]
print("direct:", direct.strategy_used.value, len(html), "->", len(direct.compressed), repr(direct.compressed))
print("tool_result:", len(html), "->", len(out), "same:", out == html, "marker:", "Retrieve original: hash=" in out)
'@ | .\.venv\Scripts\python.exe -
```

- Observed result: the unfixed router replaced 10,294 HTML characters
with 16 unmarked characters; after this patch the tool result retained
all 10,294 original characters.

```text
Before: direct: html 10294 -> 16 'the answer is 42'
        tool_result: 10294 -> 16, same: False, marker: False
After:  direct: html 10294 -> 16 'the answer is 42'
        tool_result: 10294 -> 10294, same: True, marker: False
```

A second unmocked probe with the same fixture yielded `role=tool: 10294
-> 10294, same=True` and `local_shell_call_output: html, modified=False,
same=True, reason=lossy_unrecoverable_tool_output`.
- Not tested: the reporter's actual page, a live LLM/provider request,
Linux/native detection, or a full local pytest run with `html`
installed.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: yes; tool outputs whose HTML
extraction is lossy and unmarked are retained verbatim, so those outputs
use more tokens. Direct `ContentRouter.compress()` extraction is
unchanged.
- Kill switch / disable path: `enable_html_extractor=False` prevents
HTML extraction. `ccr_inject_marker=False` remains the existing
intentional opt-out for the `ContentRouter.apply` guard.
- Unsafe override required: none.
- Qualification impact: CI's existing `test-extras` job now installs
`[html]` and runs real HTML tests; main shards are untouched. The #3774
test isolation change is in current main, outside this PR diff.
- Rollback path: revert this commit; no state migration or new
dependency is introduced.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Current head `d361388e` merges latest `main`, including
[#3790](https://github.com/headroomlabs-ai/headroom/pull/3790)'s fix for
the unrelated SQLite test that failed in the previous CI run. On this
head, that test, the 137 focused HTML/router/unit tests, and all 9
dense-line-elider tests passed locally; Ruff and `git diff --check`
passed. Exact-head hosted checks are pending.

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: no
new public API or setting)
- [ ] My changes generate no new warnings (full suite not verified
locally)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (focused
tests pass; full local suite was not run)
- [x] I did **not** edit `CHANGELOG.md`, it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

This keeps the existing reversible-tool-output contract: an unmarked
lossy transform cannot replace tool ground truth. It deliberately does
not add a retrieval store or change HTML routing.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-29 11:16:39 -05:00