mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(tool-search): don't orphan deferred tools, and stop overstating the saving (#3739)
Investigating a customer regression — **+108% cost through the proxy on Claude Code** — turned up four defects around tool-schema deferral. **None of them is the regression's root cause.** That is Claude Code disabling its *own* tool search when `ANTHROPIC_BASE_URL` points at a non-first-party host — [documented upstream behaviour](https://code.claude.com/docs/en/agent-sdk/tool-search) ("most proxies don't forward `tool_reference` blocks"), fixed by setting `ENABLE_TOOL_SEARCH=true`. But all four are real, and three are on the gateway path. ## 1. Stripping the search tool orphaned every deferred tool `strip_first_party_tool_search_tools_for_third_party_upstream` drops first-party `tool_search_tool_*` entries before forwarding to a custom upstream. That part is right — Bedrock needs a different beta token (`tool-search-tool-2025-10-19`, in the body's `anthropic_beta` array) and InvokeModel rather than Converse; Vertex and gateways reject the type outright. But `defer_loading` was left on every other tool: ``` search tools left: [] still deferred : ['mcp__srv__a', 'mcp__srv__b'] ``` Those tools arrive with nothing able to resolve them. The model has no mechanism to load them, a `tool_reference` naming one is a documented 400, and a closed tool schema rejects the unknown field anyway. Now both halves are removed and the downgrade is logged. Reachable through the fix for the root cause: forcing `ENABLE_TOOL_SEARCH=true` makes the client send the deferred shape, which this function then half-removed. ## 2. Tool-search savings were booked as realized, on both providers The figure is our own serialization of what we *asked* the provider to defer, not a measurement of what it excluded — and it **cannot be verified**: `usage` carries no deferral field, and `count_tokens` rejects any request containing a tool-search tool. An intermediary that rebuilds the tools array (Bedrock Converse converters, gateway transforms) drops `defer_loading` silently and returns 200, so on those routes the number can be pure fiction. Now `realized=False, estimated=True`. The OpenAI path's own comment already said *"a transform tag but no tokens_saved claim"* while the code five lines below claimed it. ## 3. Detection was Anthropic-only and inconsistent between call sites `request_already_defers_tools()` is now shared by `inject_tool_search_deferral`, `claude_code_tool_search_inactive` and the OpenAI Responses injector, matching the server-side shape by `type` **or** `name` (including undated aliases). It keys on what the *client* is doing rather than the upstream, so Kong/Bedrock keep our deferral when nobody else defers. **Deliberately not keyed on the bare name `ToolSearch`.** Claude Code carries that tool to resolve its local registry (TaskCreate, WebFetch, …), and real traffic shows it alongside a fully inline catalog — so standing down on the name would disable us exactly when the client is eager. Two existing fixtures encode this and keep passing. ## 4. `tool_search_mode` tag Tagged `client` / `headroom` / `none`, so a stand-down is distinguishable in `/stats` from the feature being switched off. ## Also: `HEADROOM_TOOL_SEARCH_CORE_TOOLS` Claude Code has deferred its built-ins since v2.1.69 (~14–16K tokens of schema down to under 1K). Behind a proxy the client stops deferring and we keep those resident — so the direct baseline sheds them and we don't. Deferring them here is **not obviously right**: our deferral is server-side, so first use of a deferred tool costs a search round trip, and that accuracy/latency trade is unmeasured. The default is therefore unchanged; this makes it an experiment a deployment can run against real traffic rather than a guess shipped as a default. ## Testing 32 new tests in `tests/test_tool_search_client_deferral.py`; 244 pass across the tool-search, savings-ledger and cache suites. ## Not in this PR - Bedrock tool-search support (needs InvokeModel + the body-array beta token) — real work, not a flag. - Negative-savings reporting — separate branch, different subsystem. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1cb779e1bc
commit
85f9e01d71
@@ -3047,7 +3047,24 @@ class AnthropicHandlerMixin:
|
||||
from headroom.proxy.helpers import inject_tool_search_deferral
|
||||
|
||||
_ts_before = body.get("tools")
|
||||
# Report WHO deferred. A stand-down because the client already
|
||||
# sent the server-side tool_search shape used to look identical
|
||||
# in /stats to the feature being switched off. The deferral is
|
||||
# still happening and still saving tokens — it is just not ours
|
||||
# to book, so name the mode and book nothing against it.
|
||||
from headroom.proxy.helpers import request_already_defers_tools
|
||||
|
||||
_ts_client_defers = request_already_defers_tools(_ts_before)
|
||||
_ts_after = inject_tool_search_deferral(_ts_before)
|
||||
# "headroom" only when we actually deferred something. Injection
|
||||
# also declines on a small tool surface or when nothing is
|
||||
# deferrable, and calling that "headroom" would overstate our
|
||||
# role on exactly the requests where we did nothing.
|
||||
tags["tool_search_mode"] = (
|
||||
"client"
|
||||
if _ts_client_defers
|
||||
else ("headroom" if _ts_after is not _ts_before else "none")
|
||||
)
|
||||
if _ts_after is not _ts_before:
|
||||
_ts_deferred = [
|
||||
t for t in _ts_after if isinstance(t, dict) and t.get("defer_loading")
|
||||
@@ -3064,7 +3081,22 @@ class AnthropicHandlerMixin:
|
||||
tags["tool_search_deferred_tokens"] = _ts_saved_tokens
|
||||
from headroom.proxy.savings_attribution import record_savings
|
||||
|
||||
record_savings(tags, "tool_search", tokens=_ts_saved_tokens)
|
||||
# estimated, NOT realized: this is our own serialization of
|
||||
# what we asked the provider to defer, not a measurement of
|
||||
# what it actually excluded. There is no way to verify it —
|
||||
# ``usage`` carries no deferral field, and ``count_tokens``
|
||||
# rejects any request containing a tool-search tool. An
|
||||
# intermediary that rebuilds the tools array (Bedrock
|
||||
# Converse converters, gateway transforms) drops
|
||||
# ``defer_loading`` silently and returns 200, so a confident
|
||||
# number here can be a pure fiction on those routes.
|
||||
record_savings(
|
||||
tags,
|
||||
"tool_search",
|
||||
tokens=_ts_saved_tokens,
|
||||
realized=False,
|
||||
estimated=True,
|
||||
)
|
||||
transforms_applied.append(
|
||||
f"router:tool_search_deferral:{len(_ts_deferred)}tools:"
|
||||
f"{_ts_saved_tokens}tok"
|
||||
|
||||
@@ -2938,12 +2938,19 @@ class OpenAIHandlerMixin:
|
||||
for tool in _deferred_tools
|
||||
if isinstance(tool, dict) and tool.get("defer_loading")
|
||||
]
|
||||
# estimated, NOT realized — and the comment above this block
|
||||
# already says so ("a transform tag but no tokens_saved claim").
|
||||
# The deferred definitions still ride in the request body; the
|
||||
# saving is provider-side context exclusion we cannot observe in
|
||||
# any response field, so claiming it as realized overstates it.
|
||||
record_savings(
|
||||
savings_tags if savings_tags is not None else {},
|
||||
"tool_search",
|
||||
tokens=self.openai_provider.get_token_counter(model).count_text(
|
||||
_json_debug_dumps(deferred)
|
||||
),
|
||||
realized=False,
|
||||
estimated=True,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("tool-search savings attribution skipped", exc_info=True)
|
||||
|
||||
+151
-26
@@ -3174,11 +3174,8 @@ def claude_code_tool_search_inactive(
|
||||
return False
|
||||
if not isinstance(tools, list) or not tools:
|
||||
return False
|
||||
for tool in tools:
|
||||
if isinstance(tool, dict) and str(tool.get("type", "")).startswith(
|
||||
_TOOL_SEARCH_TOOL_TYPE_PREFIX
|
||||
):
|
||||
return False
|
||||
if request_already_defers_tools(tools):
|
||||
return False
|
||||
beta = (anthropic_beta or "").lower()
|
||||
return not any(marker in beta for marker in _TOOL_SEARCH_BETA_MARKERS)
|
||||
|
||||
@@ -3282,6 +3279,32 @@ _TOOL_SEARCH_CORE_TOOLS = frozenset(
|
||||
"toolsearch",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def resolved_core_tools() -> frozenset[str]:
|
||||
"""Tools that stay resident, with a deployment override.
|
||||
|
||||
The default keeps the coding loop resident so routine edit/read/run never
|
||||
pays a search round-trip. That default is now out of step with Claude Code,
|
||||
which since v2.1.69 defers its built-ins too and thereby cuts built-in
|
||||
schema context from roughly 14-16K tokens to under 1K. Behind a proxy the
|
||||
client stops deferring, so Headroom carries those schemas in every request
|
||||
while the direct-to-Anthropic baseline does not.
|
||||
|
||||
Deferring them here is NOT obviously right: our deferral is server-side, so
|
||||
each first use of a deferred tool costs a search round trip, and the
|
||||
accuracy and latency of that trade is unmeasured. The default therefore does
|
||||
not change. ``HEADROOM_TOOL_SEARCH_CORE_TOOLS`` (comma-separated, or empty
|
||||
to defer everything non-core) makes it an experiment a deployment can run
|
||||
against real traffic instead of a guess shipped as a default.
|
||||
"""
|
||||
|
||||
raw = os.environ.get("HEADROOM_TOOL_SEARCH_CORE_TOOLS")
|
||||
if raw is None:
|
||||
return _TOOL_SEARCH_CORE_TOOLS
|
||||
return frozenset(_tool_search_resident_key(part) for part in raw.split(",") if part.strip())
|
||||
|
||||
|
||||
_TOOL_SEARCH_DEFAULT_TYPE = "tool_search_tool_regex_20251119"
|
||||
_TOOL_SEARCH_DEFAULT_NAME = "tool_search_tool_regex"
|
||||
# Below this many tools the ~search round-trip isn't worth it (Anthropic's own
|
||||
@@ -3358,6 +3381,65 @@ def _tool_search_resident_key(name: Any) -> str:
|
||||
return str(name or "").lower().lstrip("_")
|
||||
|
||||
|
||||
# Deliberately EMPTY by default. A client-side tool named ``ToolSearch`` does
|
||||
# NOT imply the client is deferring: Claude Code carries it to resolve tools it
|
||||
# keeps in a local registry (TaskCreate, WebFetch, ...), and real traffic shows
|
||||
# it riding alongside a fully inline MCP catalog. Standing down on that name
|
||||
# would disable Headroom precisely when the client is sending everything
|
||||
# eagerly, which is the case we exist for.
|
||||
#
|
||||
# When Claude Code genuinely defers, it sends the SERVER-SIDE shape
|
||||
# (``tool_search_tool_regex_20251119``), which is unambiguous and is what
|
||||
# ``request_already_defers_tools`` keys on. ``HEADROOM_CLIENT_TOOL_SEARCH_NAMES``
|
||||
# stays as an escape hatch for a future harness that signals deferral by name.
|
||||
_CLIENT_TOOL_SEARCH_NAMES: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
def _client_tool_search_names() -> frozenset[str]:
|
||||
"""Extra tool names that mean "the client is already deferring".
|
||||
|
||||
Empty by default; see ``_CLIENT_TOOL_SEARCH_NAMES``. Comma-separated in
|
||||
``HEADROOM_CLIENT_TOOL_SEARCH_NAMES``, normalized the same way tool names
|
||||
are, so a harness we have not seen can be handled without a release.
|
||||
"""
|
||||
|
||||
extra = os.environ.get("HEADROOM_CLIENT_TOOL_SEARCH_NAMES", "")
|
||||
if not extra.strip():
|
||||
return _CLIENT_TOOL_SEARCH_NAMES
|
||||
return _CLIENT_TOOL_SEARCH_NAMES | {
|
||||
_tool_search_resident_key(part) for part in extra.split(",") if part.strip()
|
||||
}
|
||||
|
||||
|
||||
def request_already_defers_tools(tools: Any) -> bool:
|
||||
"""Whether ``tools`` shows the CLIENT is already deferring tool schemas.
|
||||
|
||||
True for the Messages API server-side shape (``tool_search_tool_*`` as
|
||||
either ``type`` or ``name``), which only appears when deferral is actually
|
||||
on. NOT true for a bare client-side tool name such as ``ToolSearch``, which
|
||||
a client may send whether or not it is deferring — see
|
||||
``_CLIENT_TOOL_SEARCH_NAMES``.
|
||||
|
||||
Upstream-independent on purpose: the question is what the client is doing,
|
||||
not where the request is forwarded, so the same answer holds for first-party
|
||||
Anthropic, Bedrock, Vertex and gateways. That matters because through Kong
|
||||
or Bedrock the client usually is NOT deferring, and there Headroom's
|
||||
deferral is the only one available.
|
||||
"""
|
||||
|
||||
if not isinstance(tools, list):
|
||||
return False
|
||||
for tool in tools:
|
||||
if not isinstance(tool, dict):
|
||||
continue
|
||||
if str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX):
|
||||
return True
|
||||
name = _tool_search_resident_key(tool.get("name"))
|
||||
if name.startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX) or name in _client_tool_search_names():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def anthropic_first_party_tool_search_supported(api_base_url: str | None) -> bool:
|
||||
"""Return whether Anthropic server-side tool search is valid for this upstream."""
|
||||
from headroom.providers.claude.runtime import is_custom_anthropic_base_url
|
||||
@@ -3369,24 +3451,58 @@ def strip_first_party_tool_search_tools_for_third_party_upstream(
|
||||
tools: Any,
|
||||
api_base_url: str | None,
|
||||
) -> Any:
|
||||
"""Remove first-party Anthropic tool-search tools when forwarding to a custom upstream."""
|
||||
"""Remove first-party Anthropic tool-search tools when forwarding to a custom upstream.
|
||||
|
||||
Custom upstreams reject the first-party shape: Bedrock needs a different
|
||||
beta token (``tool-search-tool-2025-10-19``, in the body's ``anthropic_beta``
|
||||
array) and a different API (InvokeModel, never Converse); Vertex and
|
||||
gateways 400 on the tool type.
|
||||
|
||||
``defer_loading`` is cleared at the same time, and that half is
|
||||
load-bearing. Removing only the search tool leaves every other tool marked
|
||||
deferred with nothing left that can resolve it: the model has no mechanism
|
||||
to load them, and a ``tool_reference`` naming one is a documented 400
|
||||
("Tool reference 'X' not found in available tools"). A closed tool schema
|
||||
also rejects the unknown field outright ("Extra inputs are not permitted").
|
||||
Either way the half-stripped request is worse than both coherent options,
|
||||
so strip both halves and forward a plain, valid tools array.
|
||||
|
||||
Reached in practice through the obvious workaround for Claude Code
|
||||
disabling its own tool search behind a custom base URL: forcing
|
||||
``ENABLE_TOOL_SEARCH=true`` makes the client send the deferred shape.
|
||||
"""
|
||||
|
||||
if not isinstance(tools, list) or anthropic_first_party_tool_search_supported(api_base_url):
|
||||
return tools
|
||||
filtered = [
|
||||
tool
|
||||
for tool in tools
|
||||
if not (
|
||||
isinstance(tool, dict)
|
||||
and str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
)
|
||||
]
|
||||
return filtered if len(filtered) != len(tools) else tools
|
||||
out: list[Any] = []
|
||||
changed = False
|
||||
for tool in tools:
|
||||
if not isinstance(tool, dict):
|
||||
out.append(tool)
|
||||
continue
|
||||
if str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX):
|
||||
changed = True
|
||||
continue
|
||||
if tool.get("defer_loading"):
|
||||
stripped = {k: v for k, v in tool.items() if k != "defer_loading"}
|
||||
out.append(stripped)
|
||||
changed = True
|
||||
continue
|
||||
out.append(tool)
|
||||
if not changed:
|
||||
return tools
|
||||
logger.info(
|
||||
"event=tool_search_downgraded reason=third_party_upstream tools=%d hint=%s",
|
||||
len(out),
|
||||
"client-side tool deferral cannot be forwarded to this upstream; tools are sent eagerly",
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def inject_tool_search_deferral(
|
||||
tools: Any,
|
||||
*,
|
||||
core_tools: frozenset[str] = _TOOL_SEARCH_CORE_TOOLS,
|
||||
core_tools: frozenset[str] | None = None,
|
||||
search_type: str = _TOOL_SEARCH_DEFAULT_TYPE,
|
||||
search_name: str = _TOOL_SEARCH_DEFAULT_NAME,
|
||||
) -> Any:
|
||||
@@ -3402,14 +3518,15 @@ def inject_tool_search_deferral(
|
||||
tool, it is moved to the last non-deferred real tool so the (smaller) tools
|
||||
prefix still caches.
|
||||
"""
|
||||
if core_tools is None:
|
||||
core_tools = resolved_core_tools()
|
||||
if not isinstance(tools, list) or len(tools) < _TOOL_SEARCH_MIN_TOOLS:
|
||||
return tools
|
||||
for tool in tools:
|
||||
if isinstance(tool, dict) and (
|
||||
str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
or str(tool.get("name") or "").lower().startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
|
||||
):
|
||||
return tools # client already uses tool search — leave it alone
|
||||
if request_already_defers_tools(tools):
|
||||
# Client already defers (server-side tool_search_tool_*, or a client-side
|
||||
# tool such as Claude Code's ToolSearch). Stand down entirely rather than
|
||||
# deferring on top of it — see _CLIENT_TOOL_SEARCH_NAMES.
|
||||
return tools
|
||||
|
||||
search_tool = {"type": search_type, "name": search_name}
|
||||
out: list[Any] = [search_tool]
|
||||
@@ -3799,7 +3916,7 @@ def inject_tool_search_deferral_openai(
|
||||
model: str | None,
|
||||
*,
|
||||
client: str | None = None,
|
||||
core_tools: frozenset[str] = _TOOL_SEARCH_CORE_TOOLS,
|
||||
core_tools: frozenset[str] | None = None,
|
||||
) -> Any:
|
||||
"""Return a new Responses ``tools`` list with non-core function/MCP tools
|
||||
deferred + a ``{"type": "tool_search"}`` tool injected, or the original list
|
||||
@@ -3815,15 +3932,23 @@ def inject_tool_search_deferral_openai(
|
||||
so routine edit/read/run loops never pay a search round-trip and the request
|
||||
stays valid; the injected search tool is itself resident.
|
||||
"""
|
||||
if core_tools is None:
|
||||
core_tools = resolved_core_tools()
|
||||
if not openai_tool_search_client_supported(client):
|
||||
return tools
|
||||
if not _model_supports_openai_tool_search(model):
|
||||
return tools
|
||||
if not isinstance(tools, list) or len(tools) < _OPENAI_TOOL_SEARCH_MIN_TOOLS:
|
||||
return tools
|
||||
for tool in tools:
|
||||
if isinstance(tool, dict) and tool.get("type") == _OPENAI_TOOL_SEARCH_TYPE:
|
||||
return tools # client already uses tool search — leave it alone
|
||||
if any(
|
||||
isinstance(tool, dict) and tool.get("type") == _OPENAI_TOOL_SEARCH_TYPE for tool in tools
|
||||
) or request_already_defers_tools(tools):
|
||||
# Client already defers — its own Responses tool_search, or a
|
||||
# client-side tool such as Claude Code's ToolSearch. Deferring on top of
|
||||
# a client that is already deferring suppresses ITS mechanism and
|
||||
# inlines the catalog we were trying to keep out. Shape-based, not a
|
||||
# client allowlist, so it generalizes to harnesses we have not seen.
|
||||
return tools
|
||||
|
||||
out: list[Any] = [{"type": _OPENAI_TOOL_SEARCH_TYPE}]
|
||||
deferred = 0
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
"""Who defers tool schemas, and what Headroom must do about it.
|
||||
|
||||
Two mechanisms exist at two layers:
|
||||
|
||||
* **Server-side (Messages API / Responses).** ``tool_search_tool_regex`` /
|
||||
``tool_search_tool_bm25`` types paired with ``defer_loading: true``, or
|
||||
OpenAI's ``{"type": "tool_search"}``. This shape appears ONLY when deferral
|
||||
is actually on, so it is an unambiguous signal.
|
||||
* **Client-side.** Claude Code carries a tool named ``ToolSearch``. It resolves
|
||||
tools the client keeps in a local registry (TaskCreate, WebFetch, ...) and
|
||||
real traffic shows it riding alongside a fully inline MCP catalog. Its
|
||||
presence therefore says nothing about whether MCP schemas were deferred.
|
||||
|
||||
The distinction is load-bearing in both directions:
|
||||
|
||||
* Deferring on top of a client that is already deferring suppresses the
|
||||
client's mechanism and inlines the catalog we were trying to keep out.
|
||||
* Standing down on the mere NAME ``ToolSearch`` would disable Headroom exactly
|
||||
when the client is sending everything eagerly — which through Kong, Bedrock
|
||||
or any custom base URL is the normal case, because Claude Code disables its
|
||||
own tool search when ``ANTHROPIC_BASE_URL`` is non-first-party.
|
||||
|
||||
So: key on the server-side shape, never on the bare client-side name.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.helpers import (
|
||||
_TOOL_SEARCH_MIN_TOOLS,
|
||||
claude_code_tool_search_inactive,
|
||||
inject_tool_search_deferral,
|
||||
inject_tool_search_deferral_openai,
|
||||
request_already_defers_tools,
|
||||
resolved_core_tools,
|
||||
strip_first_party_tool_search_tools_for_third_party_upstream,
|
||||
)
|
||||
|
||||
CLAUDE_CODE_TOOL_SEARCH: dict[str, Any] = {
|
||||
"name": "ToolSearch",
|
||||
"description": "Search for tools by name or description.",
|
||||
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}}},
|
||||
}
|
||||
ANTHROPIC_SEARCH_REGEX: dict[str, Any] = {
|
||||
"type": "tool_search_tool_regex_20251119",
|
||||
"name": "tool_search_tool_regex",
|
||||
}
|
||||
ANTHROPIC_SEARCH_BM25: dict[str, Any] = {
|
||||
"type": "tool_search_tool_bm25_20251119",
|
||||
"name": "tool_search_tool_bm25",
|
||||
}
|
||||
THIRD_PARTY_URL = "https://kong.internal/anthropic"
|
||||
|
||||
|
||||
def _mcp(n: int) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{"name": f"mcp__srv{i}__do", "description": "x" * 400, "input_schema": {"type": "object"}}
|
||||
for i in range(n)
|
||||
]
|
||||
|
||||
|
||||
def _core() -> list[dict[str, Any]]:
|
||||
return [
|
||||
{"name": n, "description": "core", "input_schema": {"type": "object"}}
|
||||
for n in sorted(resolved_core_tools())
|
||||
]
|
||||
|
||||
|
||||
def _searches(tools: Any) -> list[Any]:
|
||||
return [
|
||||
t
|
||||
for t in tools
|
||||
if isinstance(t, dict) and str(t.get("type", "")).startswith("tool_search_tool_")
|
||||
]
|
||||
|
||||
|
||||
def _deferred(tools: Any) -> list[Any]:
|
||||
return [t for t in tools if isinstance(t, dict) and t.get("defer_loading")]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# detection
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"marker",
|
||||
[
|
||||
pytest.param(ANTHROPIC_SEARCH_REGEX, id="regex-type"),
|
||||
pytest.param(ANTHROPIC_SEARCH_BM25, id="bm25-type"),
|
||||
pytest.param({"name": "tool_search_tool_regex"}, id="by-name"),
|
||||
pytest.param({"name": "_tool_search_tool_regex"}, id="namespaced-name"),
|
||||
pytest.param({"type": "tool_search_tool_regex"}, id="undated-alias"),
|
||||
],
|
||||
)
|
||||
def test_server_side_shape_means_the_client_defers(marker: dict[str, Any]) -> None:
|
||||
assert request_already_defers_tools([marker, *_mcp(3)]) is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"tools",
|
||||
[
|
||||
# The correction: a bare client-side name is NOT a deferral signal.
|
||||
pytest.param([CLAUDE_CODE_TOOL_SEARCH, *_mcp(3)], id="claude-code-ToolSearch"),
|
||||
pytest.param([{"name": "toolsearch"}], id="lowercase-toolsearch"),
|
||||
pytest.param([], id="empty"),
|
||||
pytest.param(None, id="not-a-list"),
|
||||
pytest.param([{"name": "mcp__search__tools"}], id="mcp-tool-named-search"),
|
||||
pytest.param([{"name": "SearchTool"}], id="reversed-words"),
|
||||
pytest.param([{"name": "tool_search"}], id="prefix-but-incomplete"),
|
||||
pytest.param([{"type": "web_search"}], id="unrelated-server-tool"),
|
||||
pytest.param(["not-a-dict", 7], id="non-dict-entries"),
|
||||
],
|
||||
)
|
||||
def test_not_a_deferral_signal(tools: Any) -> None:
|
||||
assert request_already_defers_tools(tools) is False
|
||||
|
||||
|
||||
def test_extra_names_can_be_configured(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Escape hatch for a harness that does signal deferral by name."""
|
||||
tools = [{"name": "MyHarnessSearch"}, *_mcp(3)]
|
||||
assert request_already_defers_tools(tools) is False
|
||||
|
||||
monkeypatch.setenv("HEADROOM_CLIENT_TOOL_SEARCH_NAMES", "myharnesssearch")
|
||||
assert request_already_defers_tools(tools) is True
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Anthropic injection
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_stands_down_when_the_client_already_defers() -> None:
|
||||
tools = [ANTHROPIC_SEARCH_REGEX, *_core(), *_mcp(20)]
|
||||
|
||||
out = inject_tool_search_deferral(tools)
|
||||
|
||||
assert out is tools, "tools prefix must be byte-identical when we stand down"
|
||||
assert _deferred(out) == []
|
||||
|
||||
|
||||
def test_defers_when_the_client_is_eager() -> None:
|
||||
"""The Kong / Bedrock / custom-base-URL case: nobody else is deferring."""
|
||||
tools = [*_core(), *_mcp(20)]
|
||||
|
||||
out = inject_tool_search_deferral(tools)
|
||||
|
||||
assert out is not tools
|
||||
assert len(_searches(out)) == 1
|
||||
assert len(_deferred(out)) == 20
|
||||
|
||||
|
||||
def test_toolsearch_alongside_an_inline_catalog_still_defers() -> None:
|
||||
"""The case that made me get this wrong the first time.
|
||||
|
||||
``ToolSearch`` present AND the catalog inline means the client is NOT
|
||||
deferring MCP schemas. Standing down here would leave every schema in
|
||||
context, which is the opposite of what the feature is for.
|
||||
"""
|
||||
tools = [CLAUDE_CODE_TOOL_SEARCH, *_core(), *_mcp(20)]
|
||||
|
||||
out = inject_tool_search_deferral(tools)
|
||||
|
||||
assert len(_deferred(out)) == 20
|
||||
by_name = {t.get("name"): t for t in out if isinstance(t, dict)}
|
||||
assert by_name["ToolSearch"].get("defer_loading") is None, (
|
||||
"the client's own search tool must stay resident or its local-registry "
|
||||
"tools become permanently unreachable"
|
||||
)
|
||||
|
||||
|
||||
def test_core_tools_are_never_deferred() -> None:
|
||||
tools = [*_core(), *_mcp(20)]
|
||||
|
||||
deferred_names = {t["name"] for t in _deferred(inject_tool_search_deferral(tools))}
|
||||
|
||||
assert deferred_names.isdisjoint({t["name"] for t in _core()})
|
||||
|
||||
|
||||
def test_resident_set_is_overridable(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""So deferring built-ins can be measured instead of guessed."""
|
||||
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "bash,read")
|
||||
|
||||
assert resolved_core_tools() == frozenset({"bash", "read"})
|
||||
|
||||
tools = [
|
||||
{"name": "bash", "input_schema": {}},
|
||||
{"name": "read", "input_schema": {}},
|
||||
{"name": "grep", "input_schema": {}},
|
||||
*_mcp(15),
|
||||
]
|
||||
by_name = {t.get("name"): t for t in inject_tool_search_deferral(tools) if isinstance(t, dict)}
|
||||
|
||||
assert by_name["bash"].get("defer_loading") is None
|
||||
assert by_name["grep"].get("defer_loading") is True
|
||||
|
||||
|
||||
def test_below_minimum_tool_count_is_left_alone() -> None:
|
||||
tools = _mcp(_TOOL_SEARCH_MIN_TOOLS - 1)
|
||||
|
||||
assert inject_tool_search_deferral(tools) is tools
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# the issue-746 hint
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_hint_fires_for_an_eager_claude_code() -> None:
|
||||
assert (
|
||||
claude_code_tool_search_inactive(
|
||||
client="claude-code", tools=[*_core(), *_mcp(20)], anthropic_beta=None
|
||||
)
|
||||
is True
|
||||
)
|
||||
|
||||
|
||||
def test_hint_is_silent_when_the_client_defers() -> None:
|
||||
assert (
|
||||
claude_code_tool_search_inactive(
|
||||
client="claude-code", tools=[ANTHROPIC_SEARCH_REGEX, *_mcp(5)], anthropic_beta=None
|
||||
)
|
||||
is False
|
||||
)
|
||||
|
||||
|
||||
def test_hint_is_silent_with_a_tool_search_beta_marker() -> None:
|
||||
assert (
|
||||
claude_code_tool_search_inactive(
|
||||
client="claude-code",
|
||||
tools=[*_core(), *_mcp(20)],
|
||||
anthropic_beta="advanced-tool-use-2025-11-20",
|
||||
)
|
||||
is False
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# third-party upstreams: Kong / Bedrock / Vertex
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_stripping_the_search_tool_also_clears_defer_loading() -> None:
|
||||
"""Half-stripping leaves tools that nothing can resolve.
|
||||
|
||||
Bedrock needs a different beta token and InvokeModel rather than Converse;
|
||||
Vertex and gateways reject the first-party type outright. Removing the
|
||||
search tool is therefore right — but leaving ``defer_loading`` behind means
|
||||
the model has no mechanism to load those tools, and a ``tool_reference``
|
||||
naming one is a documented 400.
|
||||
"""
|
||||
tools = [
|
||||
ANTHROPIC_SEARCH_REGEX,
|
||||
{"name": "Bash", "input_schema": {}},
|
||||
{"name": "mcp__srv__a", "input_schema": {}, "defer_loading": True},
|
||||
{"name": "mcp__srv__b", "input_schema": {}, "defer_loading": True},
|
||||
]
|
||||
|
||||
out = strip_first_party_tool_search_tools_for_third_party_upstream(tools, THIRD_PARTY_URL)
|
||||
|
||||
assert _searches(out) == []
|
||||
assert _deferred(out) == []
|
||||
# The tools themselves survive — only the deferral machinery is removed.
|
||||
assert {t["name"] for t in out} == {"Bash", "mcp__srv__a", "mcp__srv__b"}
|
||||
|
||||
|
||||
def test_stripping_preserves_everything_else_about_a_tool() -> None:
|
||||
tools = [
|
||||
ANTHROPIC_SEARCH_REGEX,
|
||||
{"name": "mcp__srv__a", "input_schema": {"type": "object"}, "defer_loading": True},
|
||||
]
|
||||
|
||||
out = strip_first_party_tool_search_tools_for_third_party_upstream(tools, THIRD_PARTY_URL)
|
||||
|
||||
assert out[0] == {"name": "mcp__srv__a", "input_schema": {"type": "object"}}
|
||||
|
||||
|
||||
def test_first_party_upstream_is_untouched() -> None:
|
||||
tools = [ANTHROPIC_SEARCH_REGEX, {"name": "a", "defer_loading": True}]
|
||||
|
||||
out = strip_first_party_tool_search_tools_for_third_party_upstream(
|
||||
tools, "https://api.anthropic.com"
|
||||
)
|
||||
|
||||
assert out is tools
|
||||
|
||||
|
||||
def test_nothing_to_strip_returns_the_same_list() -> None:
|
||||
tools = [{"name": "Bash", "input_schema": {}}, *_mcp(3)]
|
||||
|
||||
out = strip_first_party_tool_search_tools_for_third_party_upstream(tools, THIRD_PARTY_URL)
|
||||
|
||||
assert out is tools
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# OpenAI Responses path
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _fn(name: str) -> dict[str, Any]:
|
||||
return {"type": "function", "name": name, "parameters": {"type": "object", "properties": {}}}
|
||||
|
||||
|
||||
def test_openai_stands_down_on_its_own_search_tool() -> None:
|
||||
tools = [{"type": "tool_search"}, *[_fn(f"slack_{i}") for i in range(14)]]
|
||||
|
||||
assert inject_tool_search_deferral_openai(tools, "gpt-5.5") is tools
|
||||
|
||||
|
||||
def test_openai_stands_down_on_an_anthropic_shaped_search_tool() -> None:
|
||||
"""A mixed harness can carry the Messages API shape onto the OpenAI path."""
|
||||
tools = [ANTHROPIC_SEARCH_REGEX, *[_fn(f"slack_{i}") for i in range(14)]]
|
||||
|
||||
assert inject_tool_search_deferral_openai(tools, "gpt-5.5") is tools
|
||||
|
||||
|
||||
def test_openai_still_defers_for_an_eager_client() -> None:
|
||||
tools = [_fn("bash"), *[_fn(f"slack_{i}") for i in range(14)]]
|
||||
|
||||
out = inject_tool_search_deferral_openai(tools, "gpt-5.5")
|
||||
|
||||
assert out is not tools
|
||||
assert len(_deferred(out)) == 14
|
||||
|
||||
|
||||
def test_openai_toolsearch_name_alone_does_not_stand_us_down() -> None:
|
||||
tools = [_fn("ToolSearch"), *[_fn(f"slack_{i}") for i in range(14)]]
|
||||
|
||||
out = inject_tool_search_deferral_openai(tools, "gpt-5.5")
|
||||
|
||||
by_name = {t.get("name"): t for t in out if isinstance(t, dict)}
|
||||
assert by_name["ToolSearch"].get("defer_loading") is None
|
||||
assert by_name["slack_0"].get("defer_loading") is True
|
||||
Reference in New Issue
Block a user