mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): drop a tool_reference naming the search tool itself (#3172)
## Description Anthropic can answer a match-all `tool_search_tool_regex` call (empty `input`) with a hit on **its own server tool**. Claude Code adds every hit to the session's loaded-tool set and replays it as a `tool_reference` in the `tools` array, so every later turn 400s with: ``` Tool reference 'tool_search_tool_regex' not found in available tools ``` A typed search tool is the search mechanism, never a reference target. The history repair from #2805/#2971 cannot reach this: it walks `messages` only, and the poison is in `tools`. That is why `/compact` does not clear it — the session is dead from the first poisoned turn until the transcript is edited by hand. This is consistent with a principle the codebase already holds. `strip_unsupported_tool_search_blocks` excludes typed search tools when building its `available` set, with the comment: > Typed search tools (type starts with `tool_search_tool_`) are the search mechanism itself — they are never the target of a tool_reference lookup. That reasoning is applied to `messages`. This PR extends the same rule to the `tools` array, which the existing function only reads and never filters. ## Evidence Observed on a real Claude Code session (`claude-cli/2.1.237`), poisoned at the one search whose result self-referenced — 1 of 6 searches in that session, both of the others with empty `input` came back clean, so it is nondeterministic upstream behavior. The failure splits by request class: | | headroom session | outcome | |---|---|---| | Main-thread turns | `inject_sticky_replay` | 200, every time | | Auto-compact side-requests | `inject_first_time`, `CCR: skipping proactive tracking for Claude Code compact summary` | 400, every time | The repair fired on every failing request (`Tool search: dropped 2 unsupportable history block(s)`) and the request still 400'd. That the rejected content is the client's own is visible on the wire: on all four 400s the outbound `body_bytes` equalled the inbound `content_length` exactly (658154, 523845, 525028, 511860), while across the same log only 10 of 701 `source=canonical` rows match any inbound size at all. Switching model gave exactly one clean turn — the side-request state reset, its two fresh searches came back clean — then the next auto-compact side-request replayed the poisoned array and 400'd again. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py` — new `strip_unsupported_tool_search_references(tools)` drops `tool_reference` entries whose name starts with `tool_search_tool_`. Returns the **original** object when nothing was removed, so the caller skips the write-back and the prefix cache is undisturbed on ~all requests (same identity contract as `strip_unsupported_tool_search_blocks`). - `headroom/proxy/handlers/anthropic.py` — wired in immediately **before** the block repair. Ordering matters: stripping the tools array shrinks what a history `tool_reference` can resolve against, so the block repair must validate against the final view — the same constraint as #2888. - `tests/test_issue_746_tool_search.py` — two cases: a poisoned array is cleaned, and a clean array is an identity no-op. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Run against **v0.36.2** with this change cherry-picked (clean apply, no conflicts). ### Test Output ```text $ python -m pytest tests/test_issue_746_tool_search.py -q 60 passed in 0.52s $ python -m pytest tests/test_proxy/test_tool_search_repair_after_turn_hooks.py -q 4 passed in 1.34s # broader tool-search regression $ python -m pytest tests/test_issue_746_tool_search.py \ tests/test_proxy/test_tool_search_repair_after_turn_hooks.py \ tests/test_openai_tool_search_deferral.py \ tests/test_savings_tool_search_aggregation.py -q 103 passed in 1.84s $ python -m ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py All checks passed! $ python -m ruff format --check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py 3 files already formatted $ python -m mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py Success: no issues found in 2 source files ``` `tests/test_proxy/test_tool_search_repair_after_turn_hooks.py` is the file most adjacent to this change — it asserts what the repair sees in the tools array after the turn hooks run. An earlier revision of this description reported it as un-runnable. That was an environment gap, now resolved: the blocker is the optional **`h2`** dependency (`ImportError: Using http2=True, but the 'h2' package is not installed`), not the Rust extension. With `pip install 'httpx[http2]'` the file collects and all 4 tests pass. Those 4 tests were also confirmed to fail identically on **clean v0.36.2 without this patch**, so the earlier failure was environmental and not a regression introduced here. ## Real Behavior Proof - Environment: live Claude Code session (`claude-cli/2.1.237`) through a self-hosted headroom proxy; plus the unit-level runs above against v0.36.2 on Linux / Python 3.11. - Exact command / steps: normal Claude Code usage until a match-all `tool_search_tool_regex` returned a hit naming `tool_search_tool_regex` itself; thereafter every auto-compact side-request replayed the poisoned `tools` array. Proxy logs inspected for `body_bytes` vs inbound `content_length` and for the repair's own log lines. - Observed result: persistent `400 Tool reference 'tool_search_tool_regex' not found in available tools` on auto-compact side-requests while main-thread turns returned 200; the existing block repair logged `dropped 2 unsupportable history block(s)` and the request still 400'd, confirming the poison is in `tools` and out of that repair's reach. `/compact` did not clear it. - Not tested: no live end-to-end rerun of a poisoned session against a proxy carrying this patch — the trigger is nondeterministic upstream behavior (1 of 6 searches) and has not been reproduced on demand. The fix is covered at unit level by the added cases plus the 4 after-turn-hook tests above. ## Runtime Rollout Safety - Rollout-managed feature(s): none — unconditional repair on an already-broken request shape, no flag. - Minimum rollout channel: n/a — ships with the change. - Stable/default behavior changed: only for requests that are currently rejected by Anthropic. A `tool_reference` naming a `tool_search_tool_*` entry is never resolvable, so removing it cannot change the outcome of a request that works today. - Kill switch / disable path: none. The function is a no-op (returns the original object, identity-preserved) whenever no such entry is present, which is ~all traffic. - Unsafe override required: no. - Qualification impact: none. Sessions that were 400ing continue past the poisoned turn; unaffected sessions see a byte-identical `tools` array, so prefix caching is undisturbed. - Rollback path: revert the commit; single and self-contained. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — terminal and log output above. ## Additional Notes - **Documentation:** marked N/A. No user-facing flag, endpoint, or configuration surface changes; the repair is internal to the Anthropic request path. - **Ordering constraint:** the call site must stay ahead of the block repair. If a later change reorders them, `strip_unsupported_tool_search_blocks` will validate history against a stale tools array and the original 400 can return in a different shape. - **Identity contract:** returning the original object on a no-op is load-bearing, not a micro-optimization — the caller keys its write-back on identity, and a fresh object on every request would perturb the prefix cache. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5
JD Davis
parent
117ff72ec9
commit
b73adaa0ce
@@ -3251,7 +3251,27 @@ class AnthropicHandlerMixin:
|
||||
# and the prefix is untouched) AND the turn hooks (a hook may rewrite the
|
||||
# tools array, and repairing before it validated against a stale view).
|
||||
# Nothing past this point mutates `body["tools"]` on the outbound path.
|
||||
from headroom.proxy.helpers import strip_unsupported_tool_search_blocks
|
||||
from headroom.proxy.helpers import (
|
||||
strip_unsupported_tool_search_blocks,
|
||||
strip_unsupported_tool_search_references,
|
||||
)
|
||||
|
||||
# The tools array is repaired first: it shrinks what a history
|
||||
# tool_reference can resolve against, so the block repair below has to
|
||||
# validate against the final view (same reason as the ORDERING note).
|
||||
_ts_tools, _ts_refs_dropped = strip_unsupported_tool_search_references(
|
||||
body.get("tools")
|
||||
)
|
||||
if _ts_refs_dropped:
|
||||
body["tools"] = tools = _ts_tools
|
||||
body_mutation_tracker.mark_mutated("tool_search_reference_repair")
|
||||
transforms_applied.append(f"router:tool_search_ref_repair:{_ts_refs_dropped}refs")
|
||||
logger.info(
|
||||
"[%s] Tool search: dropped %d unresolvable tool_reference "
|
||||
"entr(y/ies) naming a typed search tool",
|
||||
request_id,
|
||||
_ts_refs_dropped,
|
||||
)
|
||||
|
||||
_ts_repaired, _ts_stripped = strip_unsupported_tool_search_blocks(
|
||||
body.get("messages"), body.get("tools")
|
||||
|
||||
@@ -4016,6 +4016,17 @@ _TOOL_SEARCH_RESULT_TYPE = "tool_search_tool_result"
|
||||
_CLIENT_TOOL_REF_PLACEHOLDER = "[tool reference no longer available]"
|
||||
|
||||
|
||||
def _tool_entry_name(entry: dict[str, Any]) -> str | None:
|
||||
"""Return the name a tool-search entry carries, or ``None``.
|
||||
|
||||
Server-side blocks use ``tool_name``; be liberal about ``name``. The one
|
||||
precedence rule for this file, so every reader agrees on an entry that
|
||||
carries both keys.
|
||||
"""
|
||||
name = entry.get("tool_name") or entry.get("name")
|
||||
return str(name) if name else None
|
||||
|
||||
|
||||
def _tool_search_reference_names(content: Any) -> list[str]:
|
||||
"""Return the ``tool_reference`` names carried by a tool-search result block.
|
||||
|
||||
@@ -4028,13 +4039,64 @@ def _tool_search_reference_names(content: Any) -> list[str]:
|
||||
names = []
|
||||
for entry in entries:
|
||||
if isinstance(entry, dict) and entry.get("type") == "tool_reference":
|
||||
# Server-side blocks use ``tool_name``; be liberal about ``name``.
|
||||
name = entry.get("tool_name") or entry.get("name")
|
||||
name = _tool_entry_name(entry)
|
||||
if name:
|
||||
names.append(str(name))
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
|
||||
def strip_unsupported_tool_search_references(tools: Any) -> tuple[Any, int]:
|
||||
"""Drop ``tool_reference`` entries in ``tools`` that name a typed search tool.
|
||||
|
||||
Anthropic occasionally returns ``tool_search_tool_regex`` as a hit inside its
|
||||
own match-all result (empty ``input``). Claude Code adds every hit to the
|
||||
session's loaded-tool set and replays it as a ``tool_reference`` in ``tools``
|
||||
on later turns, so upstream then 400s with "Tool reference
|
||||
'tool_search_tool_regex' not found in available tools" — a typed search tool
|
||||
is the search mechanism, never a reference target. The block repair below
|
||||
cannot reach this: the poison is in the tools array, not the history, which
|
||||
is why ``/compact`` does not clear it and the session stays dead.
|
||||
|
||||
Scoped to the search mechanisms this request actually carries: the names are
|
||||
derived from entries whose ``type`` starts with the typed-search prefix (the
|
||||
same signal ``strip_unsupported_tool_search_blocks`` keys on), and a
|
||||
``tool_reference`` is dropped only when its name matches one of them exactly.
|
||||
Matching on the name prefix alone would also remove a legitimate client tool
|
||||
that merely happens to be called ``tool_search_tool_*`` — a typeless deferred
|
||||
tool with such a name is a normal reference target, not a mechanism.
|
||||
|
||||
Returns ``(tools, entries_removed)``, and the ORIGINAL ``tools`` object when
|
||||
nothing was removed — callers rely on identity to skip the write-back.
|
||||
"""
|
||||
if not isinstance(tools, list):
|
||||
return tools, 0
|
||||
|
||||
# The search mechanisms present on THIS request, identified by type. Names
|
||||
# on both sides go through _tool_entry_name, so a mechanism shaped like a
|
||||
# server-side block (``tool_name``) registers too.
|
||||
mechanism_names = {
|
||||
name
|
||||
for t in tools
|
||||
if isinstance(t, dict)
|
||||
and str(t.get("type") or "").startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
and (name := _tool_entry_name(t))
|
||||
}
|
||||
if not mechanism_names:
|
||||
return tools, 0
|
||||
|
||||
kept = [
|
||||
t
|
||||
for t in tools
|
||||
if not (
|
||||
isinstance(t, dict)
|
||||
and t.get("type") == "tool_reference"
|
||||
and _tool_entry_name(t) in mechanism_names
|
||||
)
|
||||
]
|
||||
removed = len(tools) - len(kept)
|
||||
return (kept, removed) if removed else (tools, 0)
|
||||
|
||||
|
||||
# Stand-in for a tool-search block the outbound tools array cannot support. Text
|
||||
# so it is inert to every validator, short so it costs ~10 tokens, and constant so
|
||||
# the repaired prefix stays byte-stable across turns (the provider cache needs the
|
||||
|
||||
@@ -398,7 +398,9 @@ def test_core_tools_match_leading_underscore_namespace() -> None:
|
||||
|
||||
from headroom.proxy.helpers import ( # noqa: E402
|
||||
_CLIENT_TOOL_REF_PLACEHOLDER,
|
||||
_tool_search_reference_names,
|
||||
strip_unsupported_tool_search_blocks,
|
||||
strip_unsupported_tool_search_references,
|
||||
)
|
||||
|
||||
_SEARCH_TOOL = {"type": _TOOL_SEARCH_DEFAULT_TYPE, "name": _TOOL_SEARCH_DEFAULT_NAME}
|
||||
@@ -811,3 +813,115 @@ def test_repair_does_not_move_signed_thinking_blocks() -> None:
|
||||
for block in message["content"]
|
||||
if block["type"] in ("tool_search_tool_result", "server_tool_use")
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# tools-array repair: a tool_reference naming the search tool itself
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_reference_repair_drops_typed_search_tool_reference() -> None:
|
||||
# Anthropic answered a match-all search (empty input) with a hit on its own
|
||||
# server tool. Claude Code stored it in the session's loaded-tool set and
|
||||
# replays it as a tool_reference, so every later turn 400s with "Tool
|
||||
# reference 'tool_search_tool_regex' not found in available tools". The
|
||||
# block repair cannot see this: the poison is in tools, not the history.
|
||||
tools = [
|
||||
_SEARCH_TOOL,
|
||||
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
{"type": "tool_reference", "name": "Bash"},
|
||||
{"name": "Read", "input_schema": {}},
|
||||
]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 1
|
||||
assert {"type": "tool_reference", "name": "Bash"} in repaired
|
||||
assert _SEARCH_TOOL in repaired # the search mechanism itself must survive
|
||||
assert all(
|
||||
t.get("name") != _TOOL_SEARCH_DEFAULT_NAME
|
||||
for t in repaired
|
||||
if t.get("type") == "tool_reference"
|
||||
)
|
||||
|
||||
|
||||
def test_reference_repair_is_a_noop_on_a_clean_tools_array() -> None:
|
||||
# Identity return, so the caller skips the write-back and the prefix cache
|
||||
# is not disturbed on the overwhelming majority of requests.
|
||||
tools = [_SEARCH_TOOL, {"type": "tool_reference", "name": "Bash"}]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 0
|
||||
assert repaired is tools
|
||||
|
||||
|
||||
def test_reference_repair_keeps_a_deferred_tool_named_like_a_search_tool() -> None:
|
||||
# A typeless client tool may legitimately be called "tool_search_tool_*".
|
||||
# It is a normal reference target, not a search mechanism, so a reference to
|
||||
# it must survive: the request carries no typed mechanism under that name.
|
||||
# Matching on the name prefix alone would wrongly strip it.
|
||||
deferred = {"name": "tool_search_tool_custom", "input_schema": {}}
|
||||
tools = [
|
||||
_SEARCH_TOOL,
|
||||
deferred,
|
||||
{"type": "tool_reference", "name": "tool_search_tool_custom"},
|
||||
]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 0
|
||||
assert repaired is tools
|
||||
|
||||
|
||||
def test_reference_repair_keeps_prefixed_reference_with_no_matching_mechanism() -> None:
|
||||
# Prefix-shaped reference, but this request carries no typed search tool at
|
||||
# all — there is nothing for it to be a self-reference to, so it is not ours
|
||||
# to remove. Dropping it here would delete a reference the request needs.
|
||||
tools = [
|
||||
{"name": "Bash", "input_schema": {}},
|
||||
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 0
|
||||
assert repaired is tools
|
||||
|
||||
|
||||
def test_reference_repair_matches_mechanism_name_exactly() -> None:
|
||||
# Scoped to the mechanism names actually present: the exact name goes, a
|
||||
# merely prefix-sharing sibling stays.
|
||||
tools = [
|
||||
_SEARCH_TOOL,
|
||||
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME + "_other"},
|
||||
]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 1
|
||||
kept_names = [t.get("name") for t in repaired if t.get("type") == "tool_reference"]
|
||||
assert kept_names == [_TOOL_SEARCH_DEFAULT_NAME + "_other"]
|
||||
|
||||
|
||||
def test_reference_repair_uses_the_file_wide_name_precedence() -> None:
|
||||
# One precedence rule for the file: ``tool_name`` wins over ``name``, as in
|
||||
# _tool_search_reference_names. An entry carrying both keys is judged by
|
||||
# ``tool_name``, so these two readers can never disagree about it.
|
||||
tools = [
|
||||
_SEARCH_TOOL,
|
||||
{"type": "tool_reference", "tool_name": _TOOL_SEARCH_DEFAULT_NAME, "name": "Bash"},
|
||||
{"type": "tool_reference", "tool_name": "Bash", "name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 1
|
||||
assert repaired == [
|
||||
_SEARCH_TOOL,
|
||||
{"type": "tool_reference", "tool_name": "Bash", "name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
]
|
||||
# Same answer from the history-side reader for the same entries.
|
||||
assert _tool_search_reference_names(tools[1:]) == [_TOOL_SEARCH_DEFAULT_NAME, "Bash"]
|
||||
|
||||
|
||||
def test_reference_repair_registers_a_mechanism_named_by_tool_name() -> None:
|
||||
# A mechanism shaped like a server-side block (``tool_name``, no ``name``)
|
||||
# must still register, symmetric with the liberal reference matching.
|
||||
tools = [
|
||||
{"type": _TOOL_SEARCH_DEFAULT_TYPE, "tool_name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
|
||||
{"type": "tool_reference", "name": "Bash"},
|
||||
]
|
||||
repaired, removed = strip_unsupported_tool_search_references(tools)
|
||||
assert removed == 1
|
||||
assert {"type": "tool_reference", "name": "Bash"} in repaired
|
||||
|
||||
Reference in New Issue
Block a user