mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description Fixes #3837. `headroom_retrieve` results issued through Hermes' deferred-tool bridge were re-compressed into `<<ccr:…>>` markers, making retrieved content unreachable. **Root cause.** Hermes' `tool_call` bridge advertises a **batch** shape, `{"calls": [{"name", "arguments"}, …]}`, and only tolerates the legacy `{"name", "arguments"}` form (Hermes `tools/tool_search_validation.py::normalize_tool_call_entries`). `unwrap_tool_call_name` read only the legacy form, so every batched call resolved to the wrapper name `tool_call`: - The `ccr_retrieve` guard never fired. SmartCrusher re-compressed the retrieved rows into a marker whose hash matches the original. - `--protect-tool-results` and exclude lists silently no-op'd for **every** deferred tool. Since late August, Hermes also defers its core tools (terminal, file reads) behind the same bridge. Reproduced end to end through a real proxy (token mode, `--target-ratio 0.7`) against a fake Anthropic upstream. A Hermes batch-shaped retrieve of a compressed JSON output: | | retrieve result forwarded upstream | PERF transforms | |---|---|---| | main | `{"hash":…,"original_content":"<<ccr:3c73cd7e951b,string,14.4KB>>",…}` | `router:tool_result:smart_crusher` | | this PR | verbatim, on every later turn too | `router:excluded:ccr_retrieve` | The reporter's "0.38.0 works, 0.39.1 doesn't" is most likely a Hermes-side change to the advertised batch shape, not a Headroom regression. v0.38.0's unwrap is identical and fails the same way on the batch shape. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/config.py`: new `unwrap_tool_call(name, arguments) -> (real_name, real_arguments)`, mirroring Hermes' normalizer: - `calls` may be a list, a bare object or a JSON string; the legacy shape is still handled. - A batch resolves only when every entry names the same tool. Malformed or mixed-name batches fail open to the wrapper name, as before. - `unwrap_tool_call_name` now delegates to it. - `content_router._build_tool_name_map`: uses the unwrapped **arguments** too, so read protection (`_tool_call_commands`) and the relevance query see the real call, not the wrapper envelope. - `handlers/openai.py` (Responses units path): `headroom_retrieve` detection uses the shared `is_tool_excluded` matcher instead of `== / endswith("__headroom_retrieve")`. That also covers Hermes' `mcp_srv_x` and OpenCode's `headroom_headroom_retrieve`. The import is hoisted to module level. ## Testing - [x] Unit tests pass. 270 tests across `test_hermes_tool_call_unwrap`, `test_openai_responses_*`, `test_content_router*`, `test_tool_name_aliases` and the exclude-tools suites. - [x] Linting passes: CI's `ruff 0.16.8` check plus `format --check`. - [x] Type checking passes: `mypy` on the three changed modules. - [x] New tests added. 8 of them fail on the unfixed code, including an end-to-end `ContentRouter.apply()` case with a non-retrieve control. The existing Responses retrieve-protection test is parametrized over the alias and bridge forms. - [x] Manual testing performed (proxy + fake upstream, table above). ## Follow-up (not in this PR) A separate hole turned up while reproducing: in token mode the session compression cache (`CompressionCache.apply_cached`, via `session_engine.prepare_turn`) swaps a tool result's content by **content hash** before the router runs, without looking at the tool name. So a retrieve result whose bytes equal an earlier compressed output gets the cached compressed form back, even though the router logs `excluded:ccr_retrieve`. The MCP retrieve returns a JSON envelope, so this doesn't trigger for it in practice today. It will for any client that returns the raw original. It is present in 0.38.0 too. It will be filed separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
358 lines
13 KiB
Python
358 lines
13 KiB
Python
"""Tests for Hermes deferred-tool (`tool_call` wrapper) unwrapping.
|
|
|
|
Hermes Agent loads on-demand tools via a `tool_search`/`tool_describe`/
|
|
`tool_call` indirection: on the wire the emitted tool call is named
|
|
`tool_call` and the REAL tool name lives in the arguments payload
|
|
(`{"name": "...", "arguments": {...}}`). Tool exclusion / protect lists
|
|
match on the real name, so `_build_tool_name_map` must unwrap the bridge
|
|
or whitelists silently no-op for all deferred tools.
|
|
|
|
These tests pin the `unwrap_tool_call_name` helper and its integration
|
|
into `ContentRouter._build_tool_name_map` (OpenAI + Anthropic paths).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
from headroom.config import (
|
|
DEFAULT_EXCLUDE_TOOLS,
|
|
is_tool_excluded,
|
|
unwrap_tool_call,
|
|
unwrap_tool_call_name,
|
|
)
|
|
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helper unit tests
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_unwrap_passthrough_plain_name() -> None:
|
|
assert unwrap_tool_call_name("read_file", '{"path": "/x"}') == "read_file"
|
|
|
|
|
|
def test_unwrap_passthrough_none_arguments() -> None:
|
|
assert unwrap_tool_call_name("tool_call", None) == "tool_call"
|
|
|
|
|
|
def test_unwrap_passthrough_bad_json() -> None:
|
|
assert unwrap_tool_call_name("tool_call", "bad json") == "tool_call"
|
|
|
|
|
|
def test_unwrap_passthrough_missing_name_key() -> None:
|
|
assert unwrap_tool_call_name("tool_call", '{"no_name": true}') == "tool_call"
|
|
|
|
|
|
def test_unwrap_passthrough_empty_name() -> None:
|
|
assert unwrap_tool_call_name("", None) == ""
|
|
|
|
|
|
def test_unwrap_web_search() -> None:
|
|
assert (
|
|
unwrap_tool_call_name("tool_call", '{"name": "web_search", "arguments": {}}')
|
|
== "web_search"
|
|
)
|
|
|
|
|
|
def test_unwrap_read_file() -> None:
|
|
assert (
|
|
unwrap_tool_call_name("tool_call", '{"name": "read_file", "arguments": {"path": "/x"}}')
|
|
== "read_file"
|
|
)
|
|
|
|
|
|
def test_unwrap_mcp_tool() -> None:
|
|
assert (
|
|
unwrap_tool_call_name(
|
|
"tool_call", '{"name": "mcp__codebase_memory__search", "arguments": {}}'
|
|
)
|
|
== "mcp__codebase_memory__search"
|
|
)
|
|
|
|
|
|
def test_unwrap_dict_arguments_form() -> None:
|
|
"""Arguments may arrive as a dict (not JSON string) on some paths."""
|
|
assert (
|
|
unwrap_tool_call_name("tool_call", {"name": "search_files", "arguments": {"pattern": "x"}})
|
|
== "search_files"
|
|
)
|
|
|
|
|
|
def test_unwrap_whitelist_activation() -> None:
|
|
"""Unwrapped names must activate the DEFAULT_EXCLUDE_TOOLS whitelist."""
|
|
assert is_tool_excluded("web_search", DEFAULT_EXCLUDE_TOOLS) is True
|
|
unwrapped = unwrap_tool_call_name("tool_call", '{"name": "web_search", "arguments": {}}')
|
|
assert is_tool_excluded(unwrapped, DEFAULT_EXCLUDE_TOOLS) is True
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# _build_tool_name_map integration tests
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _router(exclude_tools: set[str] | None = None) -> ContentRouter:
|
|
config = ContentRouterConfig(
|
|
min_section_tokens=10,
|
|
enable_kompress=False,
|
|
exclude_tools=exclude_tools,
|
|
)
|
|
return ContentRouter(config)
|
|
|
|
|
|
def test_build_tool_name_map_openai_wrapped() -> None:
|
|
"""OpenAI-format assistant tool_calls with Hermes tool_call wrapper."""
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_wrapped_1",
|
|
"type": "function",
|
|
"function": {
|
|
"name": "tool_call",
|
|
"arguments": '{"name": "read_file", "arguments": {"path": "/x"}}',
|
|
},
|
|
},
|
|
{
|
|
"id": "call_plain_2",
|
|
"type": "function",
|
|
"function": {"name": "web_search", "arguments": '{"query": "q"}'},
|
|
},
|
|
],
|
|
}
|
|
]
|
|
router = _router()
|
|
mapping = router._build_tool_name_map(messages)
|
|
assert mapping["call_wrapped_1"] == "read_file", (
|
|
"wrapped tool_call must map to the real tool name"
|
|
)
|
|
assert mapping["call_plain_2"] == "web_search", "plain tool names must pass through unchanged"
|
|
|
|
|
|
def test_build_tool_name_map_anthropic_wrapped() -> None:
|
|
"""Anthropic-format tool_use blocks with Hermes tool_call wrapper."""
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "toolu_wrapped_1",
|
|
"name": "tool_call",
|
|
"input": {"name": "headroom_retrieve", "arguments": {"hash": "abc"}},
|
|
},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "toolu_plain_2",
|
|
"name": "Read",
|
|
"input": {"file_path": "/x"},
|
|
},
|
|
],
|
|
}
|
|
]
|
|
router = _router()
|
|
mapping = router._build_tool_name_map(messages)
|
|
assert mapping["toolu_wrapped_1"] == "headroom_retrieve", (
|
|
"wrapped tool_call must map to the real tool name"
|
|
)
|
|
assert mapping["toolu_plain_2"] == "Read", "plain tool names must pass through unchanged"
|
|
|
|
|
|
def test_build_tool_name_map_wrapped_not_excluded_before_unwrap() -> None:
|
|
"""Sanity: without unwrapping, a wrapped tool is NOT excluded.
|
|
|
|
This documents the failure mode the fix addresses: `tool_call` is not in
|
|
DEFAULT_EXCLUDE_TOOLS, so a whitelist match would never fire.
|
|
|
|
The inner name is `codebase_search` rather than `read_file` purely because
|
|
the second assertion needs a tool that is genuinely absent from the
|
|
defaults, and `read_file` (Cursor's `Read`) has since been added to them on
|
|
purpose. The property under test is the WRAPPER's name not matching, which
|
|
is the first assertion; the inner name is only an example.
|
|
"""
|
|
assert is_tool_excluded("tool_call", DEFAULT_EXCLUDE_TOOLS) is False
|
|
assert is_tool_excluded("codebase_search", DEFAULT_EXCLUDE_TOOLS) is False
|
|
|
|
|
|
def test_build_tool_name_map_exclusion_after_unwrap() -> None:
|
|
"""Unwrapped names feed is_tool_excluded for whitelist decisions."""
|
|
router = _router(exclude_tools=set(DEFAULT_EXCLUDE_TOOLS) | {"read_file"})
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_rf_1",
|
|
"type": "function",
|
|
"function": {
|
|
"name": "tool_call",
|
|
"arguments": '{"name": "read_file", "arguments": {"path": "/x"}}',
|
|
},
|
|
}
|
|
],
|
|
}
|
|
]
|
|
mapping = router._build_tool_name_map(messages)
|
|
assert mapping["call_rf_1"] == "read_file"
|
|
assert is_tool_excluded(mapping["call_rf_1"], router.config.exclude_tools or set()) is True
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Batch shape: {"calls": [{"name", "arguments"}, ...]} (#3837)
|
|
#
|
|
# Hermes advertises `tool_call(calls)`; the legacy top-level {"name", ...} form
|
|
# is only tolerated. Mirrors Hermes' normalize_tool_call_entries.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_unwrap_batch_of_one_dict() -> None:
|
|
args = {"calls": [{"name": "headroom_retrieve", "arguments": {"hash": "abc"}}]}
|
|
assert unwrap_tool_call("tool_call", args) == ("headroom_retrieve", {"hash": "abc"})
|
|
|
|
|
|
def test_unwrap_batch_of_one_json_string() -> None:
|
|
args = '{"calls": [{"name": "read_file", "arguments": {"path": "/x"}}]}'
|
|
assert unwrap_tool_call("tool_call", args) == ("read_file", {"path": "/x"})
|
|
|
|
|
|
def test_unwrap_batch_calls_as_json_string() -> None:
|
|
args = {"calls": '[{"name": "web_search", "arguments": {"query": "q"}}]'}
|
|
assert unwrap_tool_call_name("tool_call", args) == "web_search"
|
|
|
|
|
|
def test_unwrap_batch_calls_as_bare_object() -> None:
|
|
args = {"calls": {"name": "web_search", "arguments": {}}}
|
|
assert unwrap_tool_call_name("tool_call", args) == "web_search"
|
|
|
|
|
|
def test_unwrap_batch_same_name_keeps_wrapper_arguments() -> None:
|
|
args = {
|
|
"calls": [
|
|
{"name": "connectors__gh__search", "arguments": {"q": "a"}},
|
|
{"name": "connectors__gh__search", "arguments": {"q": "b"}},
|
|
]
|
|
}
|
|
assert unwrap_tool_call("tool_call", args) == ("connectors__gh__search", args)
|
|
|
|
|
|
def test_unwrap_batch_mixed_names_fails_open() -> None:
|
|
args = {
|
|
"calls": [
|
|
{"name": "connectors__gh__search", "arguments": {}},
|
|
{"name": "connectors__jira__search", "arguments": {}},
|
|
]
|
|
}
|
|
assert unwrap_tool_call("tool_call", args) == ("tool_call", args)
|
|
|
|
|
|
def test_unwrap_batch_malformed_fails_open() -> None:
|
|
for args in (
|
|
{"calls": []},
|
|
{"calls": "not json"},
|
|
{"calls": [{"arguments": {}}]},
|
|
{"calls": ["headroom_retrieve"]},
|
|
):
|
|
assert unwrap_tool_call("tool_call", args) == ("tool_call", args)
|
|
|
|
|
|
def test_unwrap_legacy_shape_returns_inner_arguments() -> None:
|
|
args = {"name": "read_file", "arguments": {"path": "/x"}}
|
|
assert unwrap_tool_call("tool_call", args) == ("read_file", {"path": "/x"})
|
|
|
|
|
|
def test_unwrap_plain_name_keeps_arguments() -> None:
|
|
assert unwrap_tool_call("Read", {"file_path": "/x"}) == ("Read", {"file_path": "/x"})
|
|
|
|
|
|
def test_build_tool_name_map_batch_shape_both_formats() -> None:
|
|
batch = {"calls": [{"name": "terminal", "arguments": {"command": "cat a.py"}}]}
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_b1",
|
|
"type": "function",
|
|
"function": {"name": "tool_call", "arguments": json.dumps(batch)},
|
|
}
|
|
],
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "tool_use", "id": "toolu_b2", "name": "tool_call", "input": batch}
|
|
],
|
|
},
|
|
]
|
|
router = _router()
|
|
mapping = router._build_tool_name_map(messages)
|
|
assert mapping == {"call_b1": "terminal", "toolu_b2": "terminal"}
|
|
# The per-call arguments, not the wrapper envelope, feed read protection
|
|
# and the relevance query.
|
|
assert router._tool_call_commands == {"call_b1": "cat a.py", "toolu_b2": "cat a.py"}
|
|
assert router._tool_call_args["toolu_b2"] == "cat a.py"
|
|
|
|
|
|
def _retrieve_rows_payload() -> str:
|
|
rows = [
|
|
{"id": i, "name": f"item-{i}", "status": "ok" if i % 3 else "fail", "value": i * 7}
|
|
for i in range(300)
|
|
]
|
|
return json.dumps(rows)
|
|
|
|
|
|
def _anthropic_retrieve_conversation(tool_use: dict, payload: str) -> list[dict]:
|
|
return [
|
|
{"role": "user", "content": "get the data"},
|
|
{"role": "assistant", "content": [tool_use]},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": tool_use["id"], "content": payload}],
|
|
},
|
|
{"role": "assistant", "content": "ok"},
|
|
{"role": "user", "content": "continue"},
|
|
]
|
|
|
|
|
|
def test_batch_wrapped_headroom_retrieve_result_is_not_recompressed() -> None:
|
|
"""#3837: a retrieve issued through Hermes' batch bridge must pass through.
|
|
|
|
Before the fix the name map resolved to `tool_call`, the ccr_retrieve guard
|
|
never fired, and SmartCrusher re-compressed the retrieved rows back into a
|
|
marker, leaving the original unreachable.
|
|
"""
|
|
from headroom.providers import AnthropicProvider
|
|
from headroom.tokenizer import Tokenizer
|
|
|
|
tokenizer = Tokenizer(AnthropicProvider().get_token_counter("claude-sonnet-4-5"))
|
|
payload = _retrieve_rows_payload()
|
|
for retrieve_name in ("headroom_retrieve", "mcp_headroom_headroom_retrieve"):
|
|
tool_use = {
|
|
"type": "tool_use",
|
|
"id": "toolu_retrieve",
|
|
"name": "tool_call",
|
|
"input": {"calls": [{"name": retrieve_name, "arguments": {"hash": "abc123"}}]},
|
|
}
|
|
result = ContentRouter(ContentRouterConfig()).apply(
|
|
_anthropic_retrieve_conversation(tool_use, payload), tokenizer
|
|
)
|
|
assert result.messages[2]["content"][0]["content"] == payload, retrieve_name
|
|
assert "router:excluded:ccr_retrieve" in result.transforms_applied, retrieve_name
|
|
|
|
# Control: the same payload behind a non-retrieve deferred tool does
|
|
# compress, so the assertion above is not vacuous.
|
|
control = {
|
|
"type": "tool_use",
|
|
"id": "toolu_retrieve",
|
|
"name": "tool_call",
|
|
"input": {"calls": [{"name": "mcp_db_query", "arguments": {"q": "all"}}]},
|
|
}
|
|
result = ContentRouter(ContentRouterConfig()).apply(
|
|
_anthropic_retrieve_conversation(control, payload), tokenizer
|
|
)
|
|
assert result.messages[2]["content"][0]["content"] != payload
|