fix(router): pass caveman_retrieve recovery output through verbatim (#4010) (#4014)

## Description

When Headroom sits behind caveman-proxy, Caveman's `caveman_retrieve`
tool returns the exact original of something Caveman compressed earlier.
Headroom then compressed that result again, so the model asked for "the
complete original", got a lossy copy with a new `hash=` marker, and had
to call `headroom_retrieve` to undo it.

This PR treats `caveman_retrieve` the same way Headroom already treats
its own `headroom_retrieve`: its output is passed through byte-for-byte.

Closes #4010

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/config.py`: add `caveman_retrieve` to
`DEFAULT_EXCLUDE_TOOLS` (no lossy compression) and to
`DEFAULT_VERBATIM_EXCLUDE_TOOLS` (no lossless fold such as JSON minify,
no cross-turn dedup). Both are needed: with only the first, indented
JSON still comes back minified. The verbatim set is already read by the
Chat Completions, Anthropic and Responses API paths, so one entry covers
all of them. MCP-prefixed names like `mcp__caveman__caveman_retrieve`
match through the existing `is_tool_excluded()` alias handling.
- `tests/test_content_router_recovery_tool_exclusion.py`: new regression
tests (Anthropic and OpenAI message shapes, bare and MCP-prefixed name).
They use indented JSON on purpose, so they fail if only one of the two
lists is updated.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New tests, with the fix
tests\test_content_router_recovery_tool_exclusion.py ....                [100%]
============================== 4 passed in 6.62s ==============================

# Same tests on upstream/main without the config change
FAILED tests/test_content_router_recovery_tool_exclusion.py::test_anthropic_recovery_result_is_verbatim[caveman_retrieve]
FAILED tests/test_content_router_recovery_tool_exclusion.py::test_anthropic_recovery_result_is_verbatim[mcp__caveman__caveman_retrieve]
FAILED tests/test_content_router_recovery_tool_exclusion.py::test_openai_recovery_result_is_verbatim
FAILED tests/test_content_router_recovery_tool_exclusion.py::test_recovery_tool_in_verbatim_set
============================== 4 failed in 4.74s ==============================

# Every test file that references the exclude-tool sets (13 files)
====================== 318 passed, 2 warnings in 10.34s =======================

# Content router / retrieve exemption / Responses / Strands suites
====================== 368 passed, 2 warnings in 13.78s =======================

$ ruff check headroom/config.py tests/test_content_router_recovery_tool_exclusion.py
All checks passed!
$ ruff format --check headroom/config.py tests/test_content_router_recovery_tool_exclusion.py
2 files already formatted

$ mypy headroom --ignore-missing-imports --no-site-packages   (mypy 1.20.2)
headroom\fileperms.py:92: error: Module has no attribute "fchmod"  [attr-defined]
Found 1 error in 1 file (checked 562 source files)
# Windows-only false positive in an untouched file (os.fchmod does not exist on Windows); clean on Linux CI.
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12 (uv tool env with the headroom
editable install and built `headroom._core`), branch rebased on
upstream/main `dea9092e`
- Exact command / steps: ran
`ContentRouter(ContentRouterConfig()).apply(...)` on an Anthropic
`tool_use` + `tool_result` pair holding a 60-row JSON payload (compact
and indented), once per tool name: `headroom_retrieve`,
`caveman_retrieve`, `mcp__caveman__caveman_retrieve`, `some_other_tool`
- Observed result: before the fix, `caveman_retrieve` 3680 -> 1861 chars
via `router:tool_result:smart_crusher` (same for the MCP-prefixed name).
After the fix, both stay 3680 -> 3680 and 4762 -> 4762 (indented) via
`router:excluded:tool`, byte-identical. `some_other_tool` still
compresses (3680 -> 1861, 4762 -> 1798), so ordinary tools are
unaffected.
- Not tested: a live client -> caveman-proxy -> Headroom chain with a
real upstream model; the Responses API path was covered by the existing
Responses suites, not by a new caveman-specific test

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable (default config data change, no new
flag)
- Stable/default behavior changed: yes, narrowly: tool results from a
tool named `caveman_retrieve` (or its MCP-prefixed alias) are no longer
compressed; nothing else changes
- Kill switch / disable path: no runtime switch, same as
`headroom_retrieve` (in the proxy, `HEADROOM_EXCLUDE_TOOLS` adds to the
built-in list and cannot remove entries); disable by reverting this
commit
- Unsafe override required: no
- Qualification impact: none expected; only affects requests that
contain `caveman_retrieve` results, which are usually small and need to
be exact
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`; it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no docs page lists the built-in exclude-tool defaults,
so there was nothing to update.
- Why not a blanket `*_retrieve` pattern, as the issue suggests: names
like `memory_retrieve` or `docs_retrieve` are usually search tools whose
large output Headroom should keep compressing. Users with other recovery
tools can already add them with `HEADROOM_EXCLUDE_TOOLS` (glob patterns
are supported). That path stops lossy compression but still applies the
lossless fold (for example JSON minify). If maintainers want user-named
tools to be fully byte-exact too, that would be a separate setting and a
follow-up.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4014"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Parideboy
2026-10-07 21:06:05 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent b4dda22b6d
commit 3c418f6f0b
2 changed files with 101 additions and 0 deletions
+7
View File
@@ -229,6 +229,10 @@ DEFAULT_EXCLUDE_TOOLS: frozenset[str] = frozenset(
"WebSearch",
"WebFetch",
"headroom_retrieve",
# Caveman's recovery tool: like headroom_retrieve, it returns the stored
# original of something Caveman compressed upstream of us. Compressing it
# again hands the model a lossy copy of the "complete original" (#4010).
"caveman_retrieve",
# Copilot CLI's file-read tool (its `Read` equivalent): raw file bytes
# the model byte-patches against.
"view",
@@ -282,6 +286,9 @@ DEFAULT_VERBATIM_EXCLUDE_TOOLS: frozenset[str] = frozenset(
"web_search",
"web_fetch",
"headroom_retrieve",
# A recovery tool's output must reach the model exactly as stored; even
# the lossless fold (e.g. JSON minify) changes it (#4010).
"caveman_retrieve",
# `view` (Copilot CLI file read) must stay BYTE-EXACT: the model produces
# line/byte-precise edits against it, and even "lossless" JSON rewrites
# or cross-turn dedup folds break old_str matching and force re-reads.
@@ -0,0 +1,94 @@
"""Regression tests: other tools' recovery output passes through byte-for-byte (#4010).
``caveman_retrieve`` returns the stored original of something Caveman compressed
upstream of Headroom. Compressing that result again hands the model a lossy copy
of the "complete original" plus a fresh retrieve marker, so recovery never
finishes in one call.
The payload is indented JSON on purpose: a name that is only in
``DEFAULT_EXCLUDE_TOOLS`` still gets the excluded-tool lossless fold, which
minifies it. Only ``DEFAULT_VERBATIM_EXCLUDE_TOOLS`` keeps it byte-exact.
"""
from __future__ import annotations
import json
import pytest
from headroom.config import DEFAULT_VERBATIM_EXCLUDE_TOOLS, is_tool_excluded
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
def _get_tokenizer():
from headroom.providers import OpenAIProvider
from headroom.tokenizer import Tokenizer
provider = OpenAIProvider()
token_counter = provider.get_token_counter("gpt-4o")
return Tokenizer(token_counter, "gpt-4o")
def _big_indented_json() -> str:
return json.dumps(
[{"id": i, "value": "x" * 20, "active": i % 2 == 0} for i in range(60)], indent=2
)
@pytest.mark.parametrize("tool_name", ["caveman_retrieve", "mcp__caveman__caveman_retrieve"])
def test_anthropic_recovery_result_is_verbatim(tool_name: str) -> None:
content = _big_indented_json()
messages = [
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": "toolu_rec_1",
"name": tool_name,
"input": {"handle": "ccr://abc123"},
}
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "toolu_rec_1", "content": content}],
},
]
result = ContentRouter(ContentRouterConfig()).apply(messages, _get_tokenizer())
out = result.messages[1]["content"][0]["content"]
assert out == content
assert "hash=" not in out and "<<ccr:" not in out
def test_openai_recovery_result_is_verbatim() -> None:
content = _big_indented_json()
messages = [
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_rec_1",
"type": "function",
"function": {
"name": "caveman_retrieve",
"arguments": '{"handle":"ccr://abc123"}',
},
}
],
},
{"role": "tool", "tool_call_id": "call_rec_1", "content": content},
]
result = ContentRouter(ContentRouterConfig()).apply(messages, _get_tokenizer())
assert result.messages[1]["content"] == content
def test_recovery_tool_in_verbatim_set() -> None:
# The Responses API guard and cross-turn dedup key off this set directly.
assert is_tool_excluded("caveman_retrieve", DEFAULT_VERBATIM_EXCLUDE_TOOLS)
assert is_tool_excluded("mcp__caveman__caveman_retrieve", DEFAULT_VERBATIM_EXCLUDE_TOOLS)