fix(proxy): drop a tool_reference naming the search tool itself (#3172)

## Description

Anthropic can answer a match-all `tool_search_tool_regex` call (empty
`input`) with a hit on **its own server tool**. Claude Code adds every
hit to the session's loaded-tool set and replays it as a
`tool_reference` in the `tools` array, so every later turn 400s with:

```
Tool reference 'tool_search_tool_regex' not found in available tools
```

A typed search tool is the search mechanism, never a reference target.

The history repair from #2805/#2971 cannot reach this: it walks
`messages` only, and the poison is in `tools`. That is why `/compact`
does not clear it — the session is dead from the first poisoned turn
until the transcript is edited by hand.

This is consistent with a principle the codebase already holds.
`strip_unsupported_tool_search_blocks` excludes typed search tools when
building its `available` set, with the comment:

> Typed search tools (type starts with `tool_search_tool_`) are the
search mechanism itself — they are never the target of a tool_reference
lookup.

That reasoning is applied to `messages`. This PR extends the same rule
to the `tools` array, which the existing function only reads and never
filters.

## Evidence

Observed on a real Claude Code session (`claude-cli/2.1.237`), poisoned
at the one search whose result self-referenced — 1 of 6 searches in that
session, both of the others with empty `input` came back clean, so it is
nondeterministic upstream behavior.

The failure splits by request class:

| | headroom session | outcome |
|---|---|---|
| Main-thread turns | `inject_sticky_replay` | 200, every time |
| Auto-compact side-requests | `inject_first_time`, `CCR: skipping
proactive tracking for Claude Code compact summary` | 400, every time |

The repair fired on every failing request (`Tool search: dropped 2
unsupportable history block(s)`) and the request still 400'd.

That the rejected content is the client's own is visible on the wire: on
all four 400s the outbound `body_bytes` equalled the inbound
`content_length` exactly (658154, 523845, 525028, 511860), while across
the same log only 10 of 701 `source=canonical` rows match any inbound
size at all.

Switching model gave exactly one clean turn — the side-request state
reset, its two fresh searches came back clean — then the next
auto-compact side-request replayed the poisoned array and 400'd again.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py` — new
`strip_unsupported_tool_search_references(tools)` drops `tool_reference`
entries whose name starts with `tool_search_tool_`. Returns the
**original** object when nothing was removed, so the caller skips the
write-back and the prefix cache is undisturbed on ~all requests (same
identity contract as `strip_unsupported_tool_search_blocks`).
- `headroom/proxy/handlers/anthropic.py` — wired in immediately
**before** the block repair. Ordering matters: stripping the tools array
shrinks what a history `tool_reference` can resolve against, so the
block repair must validate against the final view — the same constraint
as #2888.
- `tests/test_issue_746_tool_search.py` — two cases: a poisoned array is
cleaned, and a clean array is an identity no-op.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Run against **v0.36.2** with this change cherry-picked (clean apply, no
conflicts).

### Test Output

```text
$ python -m pytest tests/test_issue_746_tool_search.py -q
60 passed in 0.52s

$ python -m pytest tests/test_proxy/test_tool_search_repair_after_turn_hooks.py -q
4 passed in 1.34s

# broader tool-search regression
$ python -m pytest tests/test_issue_746_tool_search.py \
    tests/test_proxy/test_tool_search_repair_after_turn_hooks.py \
    tests/test_openai_tool_search_deferral.py \
    tests/test_savings_tool_search_aggregation.py -q
103 passed in 1.84s

$ python -m ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py
All checks passed!

$ python -m ruff format --check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_issue_746_tool_search.py
3 files already formatted

$ python -m mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py
Success: no issues found in 2 source files
```

`tests/test_proxy/test_tool_search_repair_after_turn_hooks.py` is the
file most adjacent to this change — it asserts what the repair sees in
the tools array after the turn hooks run. An earlier revision of this
description reported it as un-runnable. That was an environment gap, now
resolved: the blocker is the optional **`h2`** dependency (`ImportError:
Using http2=True, but the 'h2' package is not installed`), not the Rust
extension. With `pip install 'httpx[http2]'` the file collects and all 4
tests pass.

Those 4 tests were also confirmed to fail identically on **clean v0.36.2
without this patch**, so the earlier failure was environmental and not a
regression introduced here.

## Real Behavior Proof

- Environment: live Claude Code session (`claude-cli/2.1.237`) through a
self-hosted headroom proxy; plus the unit-level runs above against
v0.36.2 on Linux / Python 3.11.
- Exact command / steps: normal Claude Code usage until a match-all
`tool_search_tool_regex` returned a hit naming `tool_search_tool_regex`
itself; thereafter every auto-compact side-request replayed the poisoned
`tools` array. Proxy logs inspected for `body_bytes` vs inbound
`content_length` and for the repair's own log lines.
- Observed result: persistent `400 Tool reference
'tool_search_tool_regex' not found in available tools` on auto-compact
side-requests while main-thread turns returned 200; the existing block
repair logged `dropped 2 unsupportable history block(s)` and the request
still 400'd, confirming the poison is in `tools` and out of that
repair's reach. `/compact` did not clear it.
- Not tested: no live end-to-end rerun of a poisoned session against a
proxy carrying this patch — the trigger is nondeterministic upstream
behavior (1 of 6 searches) and has not been reproduced on demand. The
fix is covered at unit level by the added cases plus the 4
after-turn-hook tests above.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — unconditional repair on an
already-broken request shape, no flag.
- Minimum rollout channel: n/a — ships with the change.
- Stable/default behavior changed: only for requests that are currently
rejected by Anthropic. A `tool_reference` naming a `tool_search_tool_*`
entry is never resolvable, so removing it cannot change the outcome of a
request that works today.
- Kill switch / disable path: none. The function is a no-op (returns the
original object, identity-preserved) whenever no such entry is present,
which is ~all traffic.
- Unsafe override required: no.
- Qualification impact: none. Sessions that were 400ing continue past
the poisoned turn; unaffected sessions see a byte-identical `tools`
array, so prefix caching is undisturbed.
- Rollback path: revert the commit; single and self-contained.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — terminal and log output above.

## Additional Notes

- **Documentation:** marked N/A. No user-facing flag, endpoint, or
configuration surface changes; the repair is internal to the Anthropic
request path.
- **Ordering constraint:** the call site must stay ahead of the block
repair. If a later change reorders them,
`strip_unsupported_tool_search_blocks` will validate history against a
stale tools array and the original 400 can return in a different shape.
- **Identity contract:** returning the original object on a no-op is
load-bearing, not a micro-optimization — the caller keys its write-back
on identity, and a fresh object on every request would perturb the
prefix cache.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Serge ARADJ
2026-09-30 11:22:20 -05:00
committed by GitHub
co-authored by Claude Opus 5 JD Davis
parent 117ff72ec9
commit b73adaa0ce
3 changed files with 200 additions and 4 deletions
+21 -1
View File
@@ -3251,7 +3251,27 @@ class AnthropicHandlerMixin:
# and the prefix is untouched) AND the turn hooks (a hook may rewrite the
# tools array, and repairing before it validated against a stale view).
# Nothing past this point mutates `body["tools"]` on the outbound path.
from headroom.proxy.helpers import strip_unsupported_tool_search_blocks
from headroom.proxy.helpers import (
strip_unsupported_tool_search_blocks,
strip_unsupported_tool_search_references,
)
# The tools array is repaired first: it shrinks what a history
# tool_reference can resolve against, so the block repair below has to
# validate against the final view (same reason as the ORDERING note).
_ts_tools, _ts_refs_dropped = strip_unsupported_tool_search_references(
body.get("tools")
)
if _ts_refs_dropped:
body["tools"] = tools = _ts_tools
body_mutation_tracker.mark_mutated("tool_search_reference_repair")
transforms_applied.append(f"router:tool_search_ref_repair:{_ts_refs_dropped}refs")
logger.info(
"[%s] Tool search: dropped %d unresolvable tool_reference "
"entr(y/ies) naming a typed search tool",
request_id,
_ts_refs_dropped,
)
_ts_repaired, _ts_stripped = strip_unsupported_tool_search_blocks(
body.get("messages"), body.get("tools")
+65 -3
View File
@@ -4016,6 +4016,17 @@ _TOOL_SEARCH_RESULT_TYPE = "tool_search_tool_result"
_CLIENT_TOOL_REF_PLACEHOLDER = "[tool reference no longer available]"
def _tool_entry_name(entry: dict[str, Any]) -> str | None:
"""Return the name a tool-search entry carries, or ``None``.
Server-side blocks use ``tool_name``; be liberal about ``name``. The one
precedence rule for this file, so every reader agrees on an entry that
carries both keys.
"""
name = entry.get("tool_name") or entry.get("name")
return str(name) if name else None
def _tool_search_reference_names(content: Any) -> list[str]:
"""Return the ``tool_reference`` names carried by a tool-search result block.
@@ -4028,13 +4039,64 @@ def _tool_search_reference_names(content: Any) -> list[str]:
names = []
for entry in entries:
if isinstance(entry, dict) and entry.get("type") == "tool_reference":
# Server-side blocks use ``tool_name``; be liberal about ``name``.
name = entry.get("tool_name") or entry.get("name")
name = _tool_entry_name(entry)
if name:
names.append(str(name))
names.append(name)
return names
def strip_unsupported_tool_search_references(tools: Any) -> tuple[Any, int]:
"""Drop ``tool_reference`` entries in ``tools`` that name a typed search tool.
Anthropic occasionally returns ``tool_search_tool_regex`` as a hit inside its
own match-all result (empty ``input``). Claude Code adds every hit to the
session's loaded-tool set and replays it as a ``tool_reference`` in ``tools``
on later turns, so upstream then 400s with "Tool reference
'tool_search_tool_regex' not found in available tools" — a typed search tool
is the search mechanism, never a reference target. The block repair below
cannot reach this: the poison is in the tools array, not the history, which
is why ``/compact`` does not clear it and the session stays dead.
Scoped to the search mechanisms this request actually carries: the names are
derived from entries whose ``type`` starts with the typed-search prefix (the
same signal ``strip_unsupported_tool_search_blocks`` keys on), and a
``tool_reference`` is dropped only when its name matches one of them exactly.
Matching on the name prefix alone would also remove a legitimate client tool
that merely happens to be called ``tool_search_tool_*`` — a typeless deferred
tool with such a name is a normal reference target, not a mechanism.
Returns ``(tools, entries_removed)``, and the ORIGINAL ``tools`` object when
nothing was removed — callers rely on identity to skip the write-back.
"""
if not isinstance(tools, list):
return tools, 0
# The search mechanisms present on THIS request, identified by type. Names
# on both sides go through _tool_entry_name, so a mechanism shaped like a
# server-side block (``tool_name``) registers too.
mechanism_names = {
name
for t in tools
if isinstance(t, dict)
and str(t.get("type") or "").startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
and (name := _tool_entry_name(t))
}
if not mechanism_names:
return tools, 0
kept = [
t
for t in tools
if not (
isinstance(t, dict)
and t.get("type") == "tool_reference"
and _tool_entry_name(t) in mechanism_names
)
]
removed = len(tools) - len(kept)
return (kept, removed) if removed else (tools, 0)
# Stand-in for a tool-search block the outbound tools array cannot support. Text
# so it is inert to every validator, short so it costs ~10 tokens, and constant so
# the repaired prefix stays byte-stable across turns (the provider cache needs the
+114
View File
@@ -398,7 +398,9 @@ def test_core_tools_match_leading_underscore_namespace() -> None:
from headroom.proxy.helpers import ( # noqa: E402
_CLIENT_TOOL_REF_PLACEHOLDER,
_tool_search_reference_names,
strip_unsupported_tool_search_blocks,
strip_unsupported_tool_search_references,
)
_SEARCH_TOOL = {"type": _TOOL_SEARCH_DEFAULT_TYPE, "name": _TOOL_SEARCH_DEFAULT_NAME}
@@ -811,3 +813,115 @@ def test_repair_does_not_move_signed_thinking_blocks() -> None:
for block in message["content"]
if block["type"] in ("tool_search_tool_result", "server_tool_use")
]
# ---------------------------------------------------------------------------
# tools-array repair: a tool_reference naming the search tool itself
# ---------------------------------------------------------------------------
def test_reference_repair_drops_typed_search_tool_reference() -> None:
# Anthropic answered a match-all search (empty input) with a hit on its own
# server tool. Claude Code stored it in the session's loaded-tool set and
# replays it as a tool_reference, so every later turn 400s with "Tool
# reference 'tool_search_tool_regex' not found in available tools". The
# block repair cannot see this: the poison is in tools, not the history.
tools = [
_SEARCH_TOOL,
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
{"type": "tool_reference", "name": "Bash"},
{"name": "Read", "input_schema": {}},
]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 1
assert {"type": "tool_reference", "name": "Bash"} in repaired
assert _SEARCH_TOOL in repaired # the search mechanism itself must survive
assert all(
t.get("name") != _TOOL_SEARCH_DEFAULT_NAME
for t in repaired
if t.get("type") == "tool_reference"
)
def test_reference_repair_is_a_noop_on_a_clean_tools_array() -> None:
# Identity return, so the caller skips the write-back and the prefix cache
# is not disturbed on the overwhelming majority of requests.
tools = [_SEARCH_TOOL, {"type": "tool_reference", "name": "Bash"}]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 0
assert repaired is tools
def test_reference_repair_keeps_a_deferred_tool_named_like_a_search_tool() -> None:
# A typeless client tool may legitimately be called "tool_search_tool_*".
# It is a normal reference target, not a search mechanism, so a reference to
# it must survive: the request carries no typed mechanism under that name.
# Matching on the name prefix alone would wrongly strip it.
deferred = {"name": "tool_search_tool_custom", "input_schema": {}}
tools = [
_SEARCH_TOOL,
deferred,
{"type": "tool_reference", "name": "tool_search_tool_custom"},
]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 0
assert repaired is tools
def test_reference_repair_keeps_prefixed_reference_with_no_matching_mechanism() -> None:
# Prefix-shaped reference, but this request carries no typed search tool at
# all — there is nothing for it to be a self-reference to, so it is not ours
# to remove. Dropping it here would delete a reference the request needs.
tools = [
{"name": "Bash", "input_schema": {}},
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 0
assert repaired is tools
def test_reference_repair_matches_mechanism_name_exactly() -> None:
# Scoped to the mechanism names actually present: the exact name goes, a
# merely prefix-sharing sibling stays.
tools = [
_SEARCH_TOOL,
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME + "_other"},
]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 1
kept_names = [t.get("name") for t in repaired if t.get("type") == "tool_reference"]
assert kept_names == [_TOOL_SEARCH_DEFAULT_NAME + "_other"]
def test_reference_repair_uses_the_file_wide_name_precedence() -> None:
# One precedence rule for the file: ``tool_name`` wins over ``name``, as in
# _tool_search_reference_names. An entry carrying both keys is judged by
# ``tool_name``, so these two readers can never disagree about it.
tools = [
_SEARCH_TOOL,
{"type": "tool_reference", "tool_name": _TOOL_SEARCH_DEFAULT_NAME, "name": "Bash"},
{"type": "tool_reference", "tool_name": "Bash", "name": _TOOL_SEARCH_DEFAULT_NAME},
]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 1
assert repaired == [
_SEARCH_TOOL,
{"type": "tool_reference", "tool_name": "Bash", "name": _TOOL_SEARCH_DEFAULT_NAME},
]
# Same answer from the history-side reader for the same entries.
assert _tool_search_reference_names(tools[1:]) == [_TOOL_SEARCH_DEFAULT_NAME, "Bash"]
def test_reference_repair_registers_a_mechanism_named_by_tool_name() -> None:
# A mechanism shaped like a server-side block (``tool_name``, no ``name``)
# must still register, symmetric with the liberal reference matching.
tools = [
{"type": _TOOL_SEARCH_DEFAULT_TYPE, "tool_name": _TOOL_SEARCH_DEFAULT_NAME},
{"type": "tool_reference", "name": _TOOL_SEARCH_DEFAULT_NAME},
{"type": "tool_reference", "name": "Bash"},
]
repaired, removed = strip_unsupported_tool_search_references(tools)
assert removed == 1
assert {"type": "tool_reference", "name": "Bash"} in repaired