mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description When Headroom sits behind caveman-proxy, Caveman's `caveman_retrieve` tool returns the exact original of something Caveman compressed earlier. Headroom then compressed that result again, so the model asked for "the complete original", got a lossy copy with a new `hash=` marker, and had to call `headroom_retrieve` to undo it. This PR treats `caveman_retrieve` the same way Headroom already treats its own `headroom_retrieve`: its output is passed through byte-for-byte. Closes #4010 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/config.py`: add `caveman_retrieve` to `DEFAULT_EXCLUDE_TOOLS` (no lossy compression) and to `DEFAULT_VERBATIM_EXCLUDE_TOOLS` (no lossless fold such as JSON minify, no cross-turn dedup). Both are needed: with only the first, indented JSON still comes back minified. The verbatim set is already read by the Chat Completions, Anthropic and Responses API paths, so one entry covers all of them. MCP-prefixed names like `mcp__caveman__caveman_retrieve` match through the existing `is_tool_excluded()` alias handling. - `tests/test_content_router_recovery_tool_exclusion.py`: new regression tests (Anthropic and OpenAI message shapes, bare and MCP-prefixed name). They use indented JSON on purpose, so they fail if only one of the two lists is updated. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New tests, with the fix tests\test_content_router_recovery_tool_exclusion.py .... [100%] ============================== 4 passed in 6.62s ============================== # Same tests on upstream/main without the config change FAILED tests/test_content_router_recovery_tool_exclusion.py::test_anthropic_recovery_result_is_verbatim[caveman_retrieve] FAILED tests/test_content_router_recovery_tool_exclusion.py::test_anthropic_recovery_result_is_verbatim[mcp__caveman__caveman_retrieve] FAILED tests/test_content_router_recovery_tool_exclusion.py::test_openai_recovery_result_is_verbatim FAILED tests/test_content_router_recovery_tool_exclusion.py::test_recovery_tool_in_verbatim_set ============================== 4 failed in 4.74s ============================== # Every test file that references the exclude-tool sets (13 files) ====================== 318 passed, 2 warnings in 10.34s ======================= # Content router / retrieve exemption / Responses / Strands suites ====================== 368 passed, 2 warnings in 13.78s ======================= $ ruff check headroom/config.py tests/test_content_router_recovery_tool_exclusion.py All checks passed! $ ruff format --check headroom/config.py tests/test_content_router_recovery_tool_exclusion.py 2 files already formatted $ mypy headroom --ignore-missing-imports --no-site-packages (mypy 1.20.2) headroom\fileperms.py:92: error: Module has no attribute "fchmod" [attr-defined] Found 1 error in 1 file (checked 562 source files) # Windows-only false positive in an untouched file (os.fchmod does not exist on Windows); clean on Linux CI. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12 (uv tool env with the headroom editable install and built `headroom._core`), branch rebased on upstream/main `dea9092e` - Exact command / steps: ran `ContentRouter(ContentRouterConfig()).apply(...)` on an Anthropic `tool_use` + `tool_result` pair holding a 60-row JSON payload (compact and indented), once per tool name: `headroom_retrieve`, `caveman_retrieve`, `mcp__caveman__caveman_retrieve`, `some_other_tool` - Observed result: before the fix, `caveman_retrieve` 3680 -> 1861 chars via `router:tool_result:smart_crusher` (same for the MCP-prefixed name). After the fix, both stay 3680 -> 3680 and 4762 -> 4762 (indented) via `router:excluded:tool`, byte-identical. `some_other_tool` still compresses (3680 -> 1861, 4762 -> 1798), so ordinary tools are unaffected. - Not tested: a live client -> caveman-proxy -> Headroom chain with a real upstream model; the Responses API path was covered by the existing Responses suites, not by a new caveman-specific test ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: stable (default config data change, no new flag) - Stable/default behavior changed: yes, narrowly: tool results from a tool named `caveman_retrieve` (or its MCP-prefixed alias) are no longer compressed; nothing else changes - Kill switch / disable path: no runtime switch, same as `headroom_retrieve` (in the proxy, `HEADROOM_EXCLUDE_TOOLS` adds to the built-in list and cannot remove entries); disable by reverting this commit - Unsafe override required: no - Qualification impact: none expected; only affects requests that contain `caveman_retrieve` results, which are usually small and need to be exact - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`; it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation: no docs page lists the built-in exclude-tool defaults, so there was nothing to update. - Why not a blanket `*_retrieve` pattern, as the issue suggests: names like `memory_retrieve` or `docs_retrieve` are usually search tools whose large output Headroom should keep compressing. Users with other recovery tools can already add them with `HEADROOM_EXCLUDE_TOOLS` (glob patterns are supported). That path stops lossy compression but still applies the lossless fold (for example JSON minify). If maintainers want user-named tools to be fully byte-exact too, that would be a separate setting and a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4014" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
b4dda22b6d
commit
3c418f6f0b
@@ -229,6 +229,10 @@ DEFAULT_EXCLUDE_TOOLS: frozenset[str] = frozenset(
|
||||
"WebSearch",
|
||||
"WebFetch",
|
||||
"headroom_retrieve",
|
||||
# Caveman's recovery tool: like headroom_retrieve, it returns the stored
|
||||
# original of something Caveman compressed upstream of us. Compressing it
|
||||
# again hands the model a lossy copy of the "complete original" (#4010).
|
||||
"caveman_retrieve",
|
||||
# Copilot CLI's file-read tool (its `Read` equivalent): raw file bytes
|
||||
# the model byte-patches against.
|
||||
"view",
|
||||
@@ -282,6 +286,9 @@ DEFAULT_VERBATIM_EXCLUDE_TOOLS: frozenset[str] = frozenset(
|
||||
"web_search",
|
||||
"web_fetch",
|
||||
"headroom_retrieve",
|
||||
# A recovery tool's output must reach the model exactly as stored; even
|
||||
# the lossless fold (e.g. JSON minify) changes it (#4010).
|
||||
"caveman_retrieve",
|
||||
# `view` (Copilot CLI file read) must stay BYTE-EXACT: the model produces
|
||||
# line/byte-precise edits against it, and even "lossless" JSON rewrites
|
||||
# or cross-turn dedup folds break old_str matching and force re-reads.
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Regression tests: other tools' recovery output passes through byte-for-byte (#4010).
|
||||
|
||||
``caveman_retrieve`` returns the stored original of something Caveman compressed
|
||||
upstream of Headroom. Compressing that result again hands the model a lossy copy
|
||||
of the "complete original" plus a fresh retrieve marker, so recovery never
|
||||
finishes in one call.
|
||||
|
||||
The payload is indented JSON on purpose: a name that is only in
|
||||
``DEFAULT_EXCLUDE_TOOLS`` still gets the excluded-tool lossless fold, which
|
||||
minifies it. Only ``DEFAULT_VERBATIM_EXCLUDE_TOOLS`` keeps it byte-exact.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.config import DEFAULT_VERBATIM_EXCLUDE_TOOLS, is_tool_excluded
|
||||
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
||||
|
||||
|
||||
def _get_tokenizer():
|
||||
from headroom.providers import OpenAIProvider
|
||||
from headroom.tokenizer import Tokenizer
|
||||
|
||||
provider = OpenAIProvider()
|
||||
token_counter = provider.get_token_counter("gpt-4o")
|
||||
return Tokenizer(token_counter, "gpt-4o")
|
||||
|
||||
|
||||
def _big_indented_json() -> str:
|
||||
return json.dumps(
|
||||
[{"id": i, "value": "x" * 20, "active": i % 2 == 0} for i in range(60)], indent=2
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool_name", ["caveman_retrieve", "mcp__caveman__caveman_retrieve"])
|
||||
def test_anthropic_recovery_result_is_verbatim(tool_name: str) -> None:
|
||||
content = _big_indented_json()
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "toolu_rec_1",
|
||||
"name": tool_name,
|
||||
"input": {"handle": "ccr://abc123"},
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": "toolu_rec_1", "content": content}],
|
||||
},
|
||||
]
|
||||
|
||||
result = ContentRouter(ContentRouterConfig()).apply(messages, _get_tokenizer())
|
||||
|
||||
out = result.messages[1]["content"][0]["content"]
|
||||
assert out == content
|
||||
assert "hash=" not in out and "<<ccr:" not in out
|
||||
|
||||
|
||||
def test_openai_recovery_result_is_verbatim() -> None:
|
||||
content = _big_indented_json()
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_rec_1",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "caveman_retrieve",
|
||||
"arguments": '{"handle":"ccr://abc123"}',
|
||||
},
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "call_rec_1", "content": content},
|
||||
]
|
||||
|
||||
result = ContentRouter(ContentRouterConfig()).apply(messages, _get_tokenizer())
|
||||
|
||||
assert result.messages[1]["content"] == content
|
||||
|
||||
|
||||
def test_recovery_tool_in_verbatim_set() -> None:
|
||||
# The Responses API guard and cross-turn dedup key off this set directly.
|
||||
assert is_tool_excluded("caveman_retrieve", DEFAULT_VERBATIM_EXCLUDE_TOOLS)
|
||||
assert is_tool_excluded("mcp__caveman__caveman_retrieve", DEFAULT_VERBATIM_EXCLUDE_TOOLS)
|
||||
Reference in New Issue
Block a user