fix(tool-search): don't orphan deferred tools, and stop overstating the saving (#3739)

Investigating a customer regression — **+108% cost through the proxy on
Claude Code** — turned up four defects around tool-schema deferral.

**None of them is the regression's root cause.** That is Claude Code
disabling its *own* tool search when `ANTHROPIC_BASE_URL` points at a
non-first-party host — [documented upstream
behaviour](https://code.claude.com/docs/en/agent-sdk/tool-search) ("most
proxies don't forward `tool_reference` blocks"), fixed by setting
`ENABLE_TOOL_SEARCH=true`. But all four are real, and three are on the
gateway path.

## 1. Stripping the search tool orphaned every deferred tool

`strip_first_party_tool_search_tools_for_third_party_upstream` drops
first-party `tool_search_tool_*` entries before forwarding to a custom
upstream. That part is right — Bedrock needs a different beta token
(`tool-search-tool-2025-10-19`, in the body's `anthropic_beta` array)
and InvokeModel rather than Converse; Vertex and gateways reject the
type outright.

But `defer_loading` was left on every other tool:

```
search tools left: []
still deferred   : ['mcp__srv__a', 'mcp__srv__b']
```

Those tools arrive with nothing able to resolve them. The model has no
mechanism to load them, a `tool_reference` naming one is a documented
400, and a closed tool schema rejects the unknown field anyway. Now both
halves are removed and the downgrade is logged.

Reachable through the fix for the root cause: forcing
`ENABLE_TOOL_SEARCH=true` makes the client send the deferred shape,
which this function then half-removed.

## 2. Tool-search savings were booked as realized, on both providers

The figure is our own serialization of what we *asked* the provider to
defer, not a measurement of what it excluded — and it **cannot be
verified**: `usage` carries no deferral field, and `count_tokens`
rejects any request containing a tool-search tool.

An intermediary that rebuilds the tools array (Bedrock Converse
converters, gateway transforms) drops `defer_loading` silently and
returns 200, so on those routes the number can be pure fiction. Now
`realized=False, estimated=True`.

The OpenAI path's own comment already said *"a transform tag but no
tokens_saved claim"* while the code five lines below claimed it.

## 3. Detection was Anthropic-only and inconsistent between call sites

`request_already_defers_tools()` is now shared by
`inject_tool_search_deferral`, `claude_code_tool_search_inactive` and
the OpenAI Responses injector, matching the server-side shape by `type`
**or** `name` (including undated aliases). It keys on what the *client*
is doing rather than the upstream, so Kong/Bedrock keep our deferral
when nobody else defers.

**Deliberately not keyed on the bare name `ToolSearch`.** Claude Code
carries that tool to resolve its local registry (TaskCreate, WebFetch,
…), and real traffic shows it alongside a fully inline catalog — so
standing down on the name would disable us exactly when the client is
eager. Two existing fixtures encode this and keep passing.

## 4. `tool_search_mode` tag

Tagged `client` / `headroom` / `none`, so a stand-down is
distinguishable in `/stats` from the feature being switched off.

## Also: `HEADROOM_TOOL_SEARCH_CORE_TOOLS`

Claude Code has deferred its built-ins since v2.1.69 (~14–16K tokens of
schema down to under 1K). Behind a proxy the client stops deferring and
we keep those resident — so the direct baseline sheds them and we don't.

Deferring them here is **not obviously right**: our deferral is
server-side, so first use of a deferred tool costs a search round trip,
and that accuracy/latency trade is unmeasured. The default is therefore
unchanged; this makes it an experiment a deployment can run against real
traffic rather than a guess shipped as a default.

## Testing

32 new tests in `tests/test_tool_search_client_deferral.py`; 244 pass
across the tool-search, savings-ledger and cache suites.

## Not in this PR

- Bedrock tool-search support (needs InvokeModel + the body-array beta
token) — real work, not a flag.
- Negative-savings reporting — separate branch, different subsystem.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-09-23 15:24:14 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent 1cb779e1bc
commit 85f9e01d71
4 changed files with 528 additions and 27 deletions
+33 -1
View File
@@ -3047,7 +3047,24 @@ class AnthropicHandlerMixin:
from headroom.proxy.helpers import inject_tool_search_deferral
_ts_before = body.get("tools")
# Report WHO deferred. A stand-down because the client already
# sent the server-side tool_search shape used to look identical
# in /stats to the feature being switched off. The deferral is
# still happening and still saving tokens — it is just not ours
# to book, so name the mode and book nothing against it.
from headroom.proxy.helpers import request_already_defers_tools
_ts_client_defers = request_already_defers_tools(_ts_before)
_ts_after = inject_tool_search_deferral(_ts_before)
# "headroom" only when we actually deferred something. Injection
# also declines on a small tool surface or when nothing is
# deferrable, and calling that "headroom" would overstate our
# role on exactly the requests where we did nothing.
tags["tool_search_mode"] = (
"client"
if _ts_client_defers
else ("headroom" if _ts_after is not _ts_before else "none")
)
if _ts_after is not _ts_before:
_ts_deferred = [
t for t in _ts_after if isinstance(t, dict) and t.get("defer_loading")
@@ -3064,7 +3081,22 @@ class AnthropicHandlerMixin:
tags["tool_search_deferred_tokens"] = _ts_saved_tokens
from headroom.proxy.savings_attribution import record_savings
record_savings(tags, "tool_search", tokens=_ts_saved_tokens)
# estimated, NOT realized: this is our own serialization of
# what we asked the provider to defer, not a measurement of
# what it actually excluded. There is no way to verify it —
# ``usage`` carries no deferral field, and ``count_tokens``
# rejects any request containing a tool-search tool. An
# intermediary that rebuilds the tools array (Bedrock
# Converse converters, gateway transforms) drops
# ``defer_loading`` silently and returns 200, so a confident
# number here can be a pure fiction on those routes.
record_savings(
tags,
"tool_search",
tokens=_ts_saved_tokens,
realized=False,
estimated=True,
)
transforms_applied.append(
f"router:tool_search_deferral:{len(_ts_deferred)}tools:"
f"{_ts_saved_tokens}tok"
+7
View File
@@ -2938,12 +2938,19 @@ class OpenAIHandlerMixin:
for tool in _deferred_tools
if isinstance(tool, dict) and tool.get("defer_loading")
]
# estimated, NOT realized — and the comment above this block
# already says so ("a transform tag but no tokens_saved claim").
# The deferred definitions still ride in the request body; the
# saving is provider-side context exclusion we cannot observe in
# any response field, so claiming it as realized overstates it.
record_savings(
savings_tags if savings_tags is not None else {},
"tool_search",
tokens=self.openai_provider.get_token_counter(model).count_text(
_json_debug_dumps(deferred)
),
realized=False,
estimated=True,
)
except Exception:
logger.debug("tool-search savings attribution skipped", exc_info=True)
+151 -26
View File
@@ -3174,11 +3174,8 @@ def claude_code_tool_search_inactive(
return False
if not isinstance(tools, list) or not tools:
return False
for tool in tools:
if isinstance(tool, dict) and str(tool.get("type", "")).startswith(
_TOOL_SEARCH_TOOL_TYPE_PREFIX
):
return False
if request_already_defers_tools(tools):
return False
beta = (anthropic_beta or "").lower()
return not any(marker in beta for marker in _TOOL_SEARCH_BETA_MARKERS)
@@ -3282,6 +3279,32 @@ _TOOL_SEARCH_CORE_TOOLS = frozenset(
"toolsearch",
}
)
def resolved_core_tools() -> frozenset[str]:
"""Tools that stay resident, with a deployment override.
The default keeps the coding loop resident so routine edit/read/run never
pays a search round-trip. That default is now out of step with Claude Code,
which since v2.1.69 defers its built-ins too and thereby cuts built-in
schema context from roughly 14-16K tokens to under 1K. Behind a proxy the
client stops deferring, so Headroom carries those schemas in every request
while the direct-to-Anthropic baseline does not.
Deferring them here is NOT obviously right: our deferral is server-side, so
each first use of a deferred tool costs a search round trip, and the
accuracy and latency of that trade is unmeasured. The default therefore does
not change. ``HEADROOM_TOOL_SEARCH_CORE_TOOLS`` (comma-separated, or empty
to defer everything non-core) makes it an experiment a deployment can run
against real traffic instead of a guess shipped as a default.
"""
raw = os.environ.get("HEADROOM_TOOL_SEARCH_CORE_TOOLS")
if raw is None:
return _TOOL_SEARCH_CORE_TOOLS
return frozenset(_tool_search_resident_key(part) for part in raw.split(",") if part.strip())
_TOOL_SEARCH_DEFAULT_TYPE = "tool_search_tool_regex_20251119"
_TOOL_SEARCH_DEFAULT_NAME = "tool_search_tool_regex"
# Below this many tools the ~search round-trip isn't worth it (Anthropic's own
@@ -3358,6 +3381,65 @@ def _tool_search_resident_key(name: Any) -> str:
return str(name or "").lower().lstrip("_")
# Deliberately EMPTY by default. A client-side tool named ``ToolSearch`` does
# NOT imply the client is deferring: Claude Code carries it to resolve tools it
# keeps in a local registry (TaskCreate, WebFetch, ...), and real traffic shows
# it riding alongside a fully inline MCP catalog. Standing down on that name
# would disable Headroom precisely when the client is sending everything
# eagerly, which is the case we exist for.
#
# When Claude Code genuinely defers, it sends the SERVER-SIDE shape
# (``tool_search_tool_regex_20251119``), which is unambiguous and is what
# ``request_already_defers_tools`` keys on. ``HEADROOM_CLIENT_TOOL_SEARCH_NAMES``
# stays as an escape hatch for a future harness that signals deferral by name.
_CLIENT_TOOL_SEARCH_NAMES: frozenset[str] = frozenset()
def _client_tool_search_names() -> frozenset[str]:
"""Extra tool names that mean "the client is already deferring".
Empty by default; see ``_CLIENT_TOOL_SEARCH_NAMES``. Comma-separated in
``HEADROOM_CLIENT_TOOL_SEARCH_NAMES``, normalized the same way tool names
are, so a harness we have not seen can be handled without a release.
"""
extra = os.environ.get("HEADROOM_CLIENT_TOOL_SEARCH_NAMES", "")
if not extra.strip():
return _CLIENT_TOOL_SEARCH_NAMES
return _CLIENT_TOOL_SEARCH_NAMES | {
_tool_search_resident_key(part) for part in extra.split(",") if part.strip()
}
def request_already_defers_tools(tools: Any) -> bool:
"""Whether ``tools`` shows the CLIENT is already deferring tool schemas.
True for the Messages API server-side shape (``tool_search_tool_*`` as
either ``type`` or ``name``), which only appears when deferral is actually
on. NOT true for a bare client-side tool name such as ``ToolSearch``, which
a client may send whether or not it is deferring — see
``_CLIENT_TOOL_SEARCH_NAMES``.
Upstream-independent on purpose: the question is what the client is doing,
not where the request is forwarded, so the same answer holds for first-party
Anthropic, Bedrock, Vertex and gateways. That matters because through Kong
or Bedrock the client usually is NOT deferring, and there Headroom's
deferral is the only one available.
"""
if not isinstance(tools, list):
return False
for tool in tools:
if not isinstance(tool, dict):
continue
if str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX):
return True
name = _tool_search_resident_key(tool.get("name"))
if name.startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX) or name in _client_tool_search_names():
return True
return False
def anthropic_first_party_tool_search_supported(api_base_url: str | None) -> bool:
"""Return whether Anthropic server-side tool search is valid for this upstream."""
from headroom.providers.claude.runtime import is_custom_anthropic_base_url
@@ -3369,24 +3451,58 @@ def strip_first_party_tool_search_tools_for_third_party_upstream(
tools: Any,
api_base_url: str | None,
) -> Any:
"""Remove first-party Anthropic tool-search tools when forwarding to a custom upstream."""
"""Remove first-party Anthropic tool-search tools when forwarding to a custom upstream.
Custom upstreams reject the first-party shape: Bedrock needs a different
beta token (``tool-search-tool-2025-10-19``, in the body's ``anthropic_beta``
array) and a different API (InvokeModel, never Converse); Vertex and
gateways 400 on the tool type.
``defer_loading`` is cleared at the same time, and that half is
load-bearing. Removing only the search tool leaves every other tool marked
deferred with nothing left that can resolve it: the model has no mechanism
to load them, and a ``tool_reference`` naming one is a documented 400
("Tool reference 'X' not found in available tools"). A closed tool schema
also rejects the unknown field outright ("Extra inputs are not permitted").
Either way the half-stripped request is worse than both coherent options,
so strip both halves and forward a plain, valid tools array.
Reached in practice through the obvious workaround for Claude Code
disabling its own tool search behind a custom base URL: forcing
``ENABLE_TOOL_SEARCH=true`` makes the client send the deferred shape.
"""
if not isinstance(tools, list) or anthropic_first_party_tool_search_supported(api_base_url):
return tools
filtered = [
tool
for tool in tools
if not (
isinstance(tool, dict)
and str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
)
]
return filtered if len(filtered) != len(tools) else tools
out: list[Any] = []
changed = False
for tool in tools:
if not isinstance(tool, dict):
out.append(tool)
continue
if str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX):
changed = True
continue
if tool.get("defer_loading"):
stripped = {k: v for k, v in tool.items() if k != "defer_loading"}
out.append(stripped)
changed = True
continue
out.append(tool)
if not changed:
return tools
logger.info(
"event=tool_search_downgraded reason=third_party_upstream tools=%d hint=%s",
len(out),
"client-side tool deferral cannot be forwarded to this upstream; tools are sent eagerly",
)
return out
def inject_tool_search_deferral(
tools: Any,
*,
core_tools: frozenset[str] = _TOOL_SEARCH_CORE_TOOLS,
core_tools: frozenset[str] | None = None,
search_type: str = _TOOL_SEARCH_DEFAULT_TYPE,
search_name: str = _TOOL_SEARCH_DEFAULT_NAME,
) -> Any:
@@ -3402,14 +3518,15 @@ def inject_tool_search_deferral(
tool, it is moved to the last non-deferred real tool so the (smaller) tools
prefix still caches.
"""
if core_tools is None:
core_tools = resolved_core_tools()
if not isinstance(tools, list) or len(tools) < _TOOL_SEARCH_MIN_TOOLS:
return tools
for tool in tools:
if isinstance(tool, dict) and (
str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
or str(tool.get("name") or "").lower().startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
):
return tools # client already uses tool search — leave it alone
if request_already_defers_tools(tools):
# Client already defers (server-side tool_search_tool_*, or a client-side
# tool such as Claude Code's ToolSearch). Stand down entirely rather than
# deferring on top of it — see _CLIENT_TOOL_SEARCH_NAMES.
return tools
search_tool = {"type": search_type, "name": search_name}
out: list[Any] = [search_tool]
@@ -3799,7 +3916,7 @@ def inject_tool_search_deferral_openai(
model: str | None,
*,
client: str | None = None,
core_tools: frozenset[str] = _TOOL_SEARCH_CORE_TOOLS,
core_tools: frozenset[str] | None = None,
) -> Any:
"""Return a new Responses ``tools`` list with non-core function/MCP tools
deferred + a ``{"type": "tool_search"}`` tool injected, or the original list
@@ -3815,15 +3932,23 @@ def inject_tool_search_deferral_openai(
so routine edit/read/run loops never pay a search round-trip and the request
stays valid; the injected search tool is itself resident.
"""
if core_tools is None:
core_tools = resolved_core_tools()
if not openai_tool_search_client_supported(client):
return tools
if not _model_supports_openai_tool_search(model):
return tools
if not isinstance(tools, list) or len(tools) < _OPENAI_TOOL_SEARCH_MIN_TOOLS:
return tools
for tool in tools:
if isinstance(tool, dict) and tool.get("type") == _OPENAI_TOOL_SEARCH_TYPE:
return tools # client already uses tool search — leave it alone
if any(
isinstance(tool, dict) and tool.get("type") == _OPENAI_TOOL_SEARCH_TYPE for tool in tools
) or request_already_defers_tools(tools):
# Client already defers — its own Responses tool_search, or a
# client-side tool such as Claude Code's ToolSearch. Deferring on top of
# a client that is already deferring suppresses ITS mechanism and
# inlines the catalog we were trying to keep out. Shape-based, not a
# client allowlist, so it generalizes to harnesses we have not seen.
return tools
out: list[Any] = [{"type": _OPENAI_TOOL_SEARCH_TYPE}]
deferred = 0
+337
View File
@@ -0,0 +1,337 @@
"""Who defers tool schemas, and what Headroom must do about it.
Two mechanisms exist at two layers:
* **Server-side (Messages API / Responses).** ``tool_search_tool_regex`` /
``tool_search_tool_bm25`` types paired with ``defer_loading: true``, or
OpenAI's ``{"type": "tool_search"}``. This shape appears ONLY when deferral
is actually on, so it is an unambiguous signal.
* **Client-side.** Claude Code carries a tool named ``ToolSearch``. It resolves
tools the client keeps in a local registry (TaskCreate, WebFetch, ...) and
real traffic shows it riding alongside a fully inline MCP catalog. Its
presence therefore says nothing about whether MCP schemas were deferred.
The distinction is load-bearing in both directions:
* Deferring on top of a client that is already deferring suppresses the
client's mechanism and inlines the catalog we were trying to keep out.
* Standing down on the mere NAME ``ToolSearch`` would disable Headroom exactly
when the client is sending everything eagerly — which through Kong, Bedrock
or any custom base URL is the normal case, because Claude Code disables its
own tool search when ``ANTHROPIC_BASE_URL`` is non-first-party.
So: key on the server-side shape, never on the bare client-side name.
"""
from __future__ import annotations
from typing import Any
import pytest
from headroom.proxy.helpers import (
_TOOL_SEARCH_MIN_TOOLS,
claude_code_tool_search_inactive,
inject_tool_search_deferral,
inject_tool_search_deferral_openai,
request_already_defers_tools,
resolved_core_tools,
strip_first_party_tool_search_tools_for_third_party_upstream,
)
CLAUDE_CODE_TOOL_SEARCH: dict[str, Any] = {
"name": "ToolSearch",
"description": "Search for tools by name or description.",
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}}},
}
ANTHROPIC_SEARCH_REGEX: dict[str, Any] = {
"type": "tool_search_tool_regex_20251119",
"name": "tool_search_tool_regex",
}
ANTHROPIC_SEARCH_BM25: dict[str, Any] = {
"type": "tool_search_tool_bm25_20251119",
"name": "tool_search_tool_bm25",
}
THIRD_PARTY_URL = "https://kong.internal/anthropic"
def _mcp(n: int) -> list[dict[str, Any]]:
return [
{"name": f"mcp__srv{i}__do", "description": "x" * 400, "input_schema": {"type": "object"}}
for i in range(n)
]
def _core() -> list[dict[str, Any]]:
return [
{"name": n, "description": "core", "input_schema": {"type": "object"}}
for n in sorted(resolved_core_tools())
]
def _searches(tools: Any) -> list[Any]:
return [
t
for t in tools
if isinstance(t, dict) and str(t.get("type", "")).startswith("tool_search_tool_")
]
def _deferred(tools: Any) -> list[Any]:
return [t for t in tools if isinstance(t, dict) and t.get("defer_loading")]
# --------------------------------------------------------------------------
# detection
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"marker",
[
pytest.param(ANTHROPIC_SEARCH_REGEX, id="regex-type"),
pytest.param(ANTHROPIC_SEARCH_BM25, id="bm25-type"),
pytest.param({"name": "tool_search_tool_regex"}, id="by-name"),
pytest.param({"name": "_tool_search_tool_regex"}, id="namespaced-name"),
pytest.param({"type": "tool_search_tool_regex"}, id="undated-alias"),
],
)
def test_server_side_shape_means_the_client_defers(marker: dict[str, Any]) -> None:
assert request_already_defers_tools([marker, *_mcp(3)]) is True
@pytest.mark.parametrize(
"tools",
[
# The correction: a bare client-side name is NOT a deferral signal.
pytest.param([CLAUDE_CODE_TOOL_SEARCH, *_mcp(3)], id="claude-code-ToolSearch"),
pytest.param([{"name": "toolsearch"}], id="lowercase-toolsearch"),
pytest.param([], id="empty"),
pytest.param(None, id="not-a-list"),
pytest.param([{"name": "mcp__search__tools"}], id="mcp-tool-named-search"),
pytest.param([{"name": "SearchTool"}], id="reversed-words"),
pytest.param([{"name": "tool_search"}], id="prefix-but-incomplete"),
pytest.param([{"type": "web_search"}], id="unrelated-server-tool"),
pytest.param(["not-a-dict", 7], id="non-dict-entries"),
],
)
def test_not_a_deferral_signal(tools: Any) -> None:
assert request_already_defers_tools(tools) is False
def test_extra_names_can_be_configured(monkeypatch: pytest.MonkeyPatch) -> None:
"""Escape hatch for a harness that does signal deferral by name."""
tools = [{"name": "MyHarnessSearch"}, *_mcp(3)]
assert request_already_defers_tools(tools) is False
monkeypatch.setenv("HEADROOM_CLIENT_TOOL_SEARCH_NAMES", "myharnesssearch")
assert request_already_defers_tools(tools) is True
# --------------------------------------------------------------------------
# Anthropic injection
# --------------------------------------------------------------------------
def test_stands_down_when_the_client_already_defers() -> None:
tools = [ANTHROPIC_SEARCH_REGEX, *_core(), *_mcp(20)]
out = inject_tool_search_deferral(tools)
assert out is tools, "tools prefix must be byte-identical when we stand down"
assert _deferred(out) == []
def test_defers_when_the_client_is_eager() -> None:
"""The Kong / Bedrock / custom-base-URL case: nobody else is deferring."""
tools = [*_core(), *_mcp(20)]
out = inject_tool_search_deferral(tools)
assert out is not tools
assert len(_searches(out)) == 1
assert len(_deferred(out)) == 20
def test_toolsearch_alongside_an_inline_catalog_still_defers() -> None:
"""The case that made me get this wrong the first time.
``ToolSearch`` present AND the catalog inline means the client is NOT
deferring MCP schemas. Standing down here would leave every schema in
context, which is the opposite of what the feature is for.
"""
tools = [CLAUDE_CODE_TOOL_SEARCH, *_core(), *_mcp(20)]
out = inject_tool_search_deferral(tools)
assert len(_deferred(out)) == 20
by_name = {t.get("name"): t for t in out if isinstance(t, dict)}
assert by_name["ToolSearch"].get("defer_loading") is None, (
"the client's own search tool must stay resident or its local-registry "
"tools become permanently unreachable"
)
def test_core_tools_are_never_deferred() -> None:
tools = [*_core(), *_mcp(20)]
deferred_names = {t["name"] for t in _deferred(inject_tool_search_deferral(tools))}
assert deferred_names.isdisjoint({t["name"] for t in _core()})
def test_resident_set_is_overridable(monkeypatch: pytest.MonkeyPatch) -> None:
"""So deferring built-ins can be measured instead of guessed."""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "bash,read")
assert resolved_core_tools() == frozenset({"bash", "read"})
tools = [
{"name": "bash", "input_schema": {}},
{"name": "read", "input_schema": {}},
{"name": "grep", "input_schema": {}},
*_mcp(15),
]
by_name = {t.get("name"): t for t in inject_tool_search_deferral(tools) if isinstance(t, dict)}
assert by_name["bash"].get("defer_loading") is None
assert by_name["grep"].get("defer_loading") is True
def test_below_minimum_tool_count_is_left_alone() -> None:
tools = _mcp(_TOOL_SEARCH_MIN_TOOLS - 1)
assert inject_tool_search_deferral(tools) is tools
# --------------------------------------------------------------------------
# the issue-746 hint
# --------------------------------------------------------------------------
def test_hint_fires_for_an_eager_claude_code() -> None:
assert (
claude_code_tool_search_inactive(
client="claude-code", tools=[*_core(), *_mcp(20)], anthropic_beta=None
)
is True
)
def test_hint_is_silent_when_the_client_defers() -> None:
assert (
claude_code_tool_search_inactive(
client="claude-code", tools=[ANTHROPIC_SEARCH_REGEX, *_mcp(5)], anthropic_beta=None
)
is False
)
def test_hint_is_silent_with_a_tool_search_beta_marker() -> None:
assert (
claude_code_tool_search_inactive(
client="claude-code",
tools=[*_core(), *_mcp(20)],
anthropic_beta="advanced-tool-use-2025-11-20",
)
is False
)
# --------------------------------------------------------------------------
# third-party upstreams: Kong / Bedrock / Vertex
# --------------------------------------------------------------------------
def test_stripping_the_search_tool_also_clears_defer_loading() -> None:
"""Half-stripping leaves tools that nothing can resolve.
Bedrock needs a different beta token and InvokeModel rather than Converse;
Vertex and gateways reject the first-party type outright. Removing the
search tool is therefore right — but leaving ``defer_loading`` behind means
the model has no mechanism to load those tools, and a ``tool_reference``
naming one is a documented 400.
"""
tools = [
ANTHROPIC_SEARCH_REGEX,
{"name": "Bash", "input_schema": {}},
{"name": "mcp__srv__a", "input_schema": {}, "defer_loading": True},
{"name": "mcp__srv__b", "input_schema": {}, "defer_loading": True},
]
out = strip_first_party_tool_search_tools_for_third_party_upstream(tools, THIRD_PARTY_URL)
assert _searches(out) == []
assert _deferred(out) == []
# The tools themselves survive — only the deferral machinery is removed.
assert {t["name"] for t in out} == {"Bash", "mcp__srv__a", "mcp__srv__b"}
def test_stripping_preserves_everything_else_about_a_tool() -> None:
tools = [
ANTHROPIC_SEARCH_REGEX,
{"name": "mcp__srv__a", "input_schema": {"type": "object"}, "defer_loading": True},
]
out = strip_first_party_tool_search_tools_for_third_party_upstream(tools, THIRD_PARTY_URL)
assert out[0] == {"name": "mcp__srv__a", "input_schema": {"type": "object"}}
def test_first_party_upstream_is_untouched() -> None:
tools = [ANTHROPIC_SEARCH_REGEX, {"name": "a", "defer_loading": True}]
out = strip_first_party_tool_search_tools_for_third_party_upstream(
tools, "https://api.anthropic.com"
)
assert out is tools
def test_nothing_to_strip_returns_the_same_list() -> None:
tools = [{"name": "Bash", "input_schema": {}}, *_mcp(3)]
out = strip_first_party_tool_search_tools_for_third_party_upstream(tools, THIRD_PARTY_URL)
assert out is tools
# --------------------------------------------------------------------------
# OpenAI Responses path
# --------------------------------------------------------------------------
def _fn(name: str) -> dict[str, Any]:
return {"type": "function", "name": name, "parameters": {"type": "object", "properties": {}}}
def test_openai_stands_down_on_its_own_search_tool() -> None:
tools = [{"type": "tool_search"}, *[_fn(f"slack_{i}") for i in range(14)]]
assert inject_tool_search_deferral_openai(tools, "gpt-5.5") is tools
def test_openai_stands_down_on_an_anthropic_shaped_search_tool() -> None:
"""A mixed harness can carry the Messages API shape onto the OpenAI path."""
tools = [ANTHROPIC_SEARCH_REGEX, *[_fn(f"slack_{i}") for i in range(14)]]
assert inject_tool_search_deferral_openai(tools, "gpt-5.5") is tools
def test_openai_still_defers_for_an_eager_client() -> None:
tools = [_fn("bash"), *[_fn(f"slack_{i}") for i in range(14)]]
out = inject_tool_search_deferral_openai(tools, "gpt-5.5")
assert out is not tools
assert len(_deferred(out)) == 14
def test_openai_toolsearch_name_alone_does_not_stand_us_down() -> None:
tools = [_fn("ToolSearch"), *[_fn(f"slack_{i}") for i in range(14)]]
out = inject_tool_search_deferral_openai(tools, "gpt-5.5")
by_name = {t.get("name"): t for t in out if isinstance(t, dict)}
assert by_name["ToolSearch"].get("defer_loading") is None
assert by_name["slack_0"].get("defer_loading") is True