fix(tool-search): make the core-tools override authoritative and the warning recurrent (#3745)

Three defects found while tracing why a customer benchmark showed
**+108% cost** with Headroom in front of Claude Code. Follow-on to
#3739.

## 1. The override was not authoritative on the OpenAI path

`helpers.py` unioned `terminal` into the resident set *after* applying
the operator's override. Setting `HEADROOM_TOOL_SEARCH_CORE_TOOLS` to a
list that excluded `terminal` got it back anyway, silently — the one
tool most likely to carry a large schema.

`resolved_core_tools` now takes provider additions as an `extra`
argument folded into the **default** only. An explicit override replaces
everything, provider additions included. That is what an override means.

## 2. The warning armed once per process

The "client tool search is disabled" warning was already
`logger.warning` — the defect was that it armed once per process. A
long-lived proxy reported the condition on the first request and went
quiet forever. **Twelve benchmark runs produced a single log line**,
which is why this took so long to find. It now re-arms hourly.

## 3. One knob, two spellings

Core read `HEADROOM_TOOL_SEARCH_CORE_TOOLS`; the plugin read
`HEADROOM_TOOL_SEARCH_CORE`. Both now resolve to the same setting —
canonical wins, legacy is honoured and warns once naming the
replacement. No existing configuration breaks.

## Testing

40 tests in `test_tool_search_client_deferral.py`, 1115 passing across
the related suites. ruff clean.

## Note on defaults

This PR does **not** change what is resident by default. Claude Code
defers all its built-ins as of v2.1.69, and matching that is the obvious
next step — but there are open client-side bugs against it (Grep/Glob
disappearing on macOS/Linux under `ENABLE_TOOL_SEARCH=true`, strict
endpoints 400ing the first request, signed-thinking-block errors on Opus
4.8). Those are upstream, not ours; they are the reason this PR makes
the knob a real off-switch first.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

### Update after self-review

Two live bugs, **one of them introduced by this PR**:

```
HEADROOM_TOOL_SEARCH_CORE_TOOLS="bash, read, terminal"
-> resident set: [' read', ' terminal', 'bash']
-> read: DEFERRED, terminal: DEFERRED
```

The override never stripped whitespace — `part.strip()` gated only the
emptiness test, while the key function lowercases and strips leading
underscores but not spaces. So the natural spelling of a list defers
exactly the tools you asked to pin. This was latent before: the OpenAI
injector re-unioned its resident names *after* the override and masked
it. **Removing that union in this PR is what made it bite.**

Separately, an override could defer the client's own `ToolSearch` —
Claude Code reaches tools held in a local registry through it and
nothing else can, so an override that omitted it orphaned that whole
category. A search-tool floor now survives every override.

Also: the hourly window was stamped only when the hint **fired**, so
once the operator fixed the condition the stamp stopped advancing and
every later request paid the full O(tools) scan for the life of the
process — a hot-path regression this PR added. It now throttles *scans*,
and claiming the slot before scanning collapses the old two-call
protocol. The legacy-var flag is now lock-guarded with a reset helper,
the gateway reference hook honours `resolved_core_tools()` instead of
the raw default (it ignored the override entirely), and the hint test no
longer monkeypatches stdlib `time.monotonic` process-wide.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-09-23 23:11:26 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent a2bf5ed1ba
commit a9757c9c7e
5 changed files with 318 additions and 43 deletions
+12 -10
View File
@@ -2379,8 +2379,8 @@ class AnthropicHandlerMixin:
# loads them all into local context. That is a client-side decision
# we cannot reverse from here, so emit a single actionable hint for
# users who launch `claude` manually (the wrap path sets the env var).
# Gate on the cheap one-time flag first so the detection scan stops
# running once the hint has fired; never let it break a request.
# Gate on the cheap throttle first so the detection scan runs at most
# once per interval; never let it break a request.
from headroom.proxy.helpers import tool_search_hint_pending
if tool_search_hint_pending():
@@ -2388,16 +2388,18 @@ class AnthropicHandlerMixin:
from headroom.proxy.helpers import (
claude_code_tool_search_inactive,
format_tool_search_disabled_hint,
take_tool_search_hint_slot,
take_tool_search_scan_slot,
)
if (
claude_code_tool_search_inactive(
client=client,
tools=tools,
anthropic_beta=request.headers.get("anthropic-beta"),
)
and take_tool_search_hint_slot()
# Claim the slot BEFORE scanning, so the window closes
# whatever the scan finds. Claiming it only on a positive
# result left the gate open forever once the operator fixed
# the condition, re-scanning the whole tool array on every
# request for the life of the process.
if take_tool_search_scan_slot() and claude_code_tool_search_inactive(
client=client,
tools=tools,
anthropic_beta=request.headers.get("anthropic-beta"),
):
logger.warning(
"[%s] %s", request_id, format_tool_search_disabled_hint(tools)
+124 -26
View File
@@ -3148,7 +3148,18 @@ _TOOL_SEARCH_TOOL_TYPE_PREFIX = "tool_search_tool_"
_TOOL_SEARCH_BETA_MARKERS = ("advanced-tool-use", "tool-search-tool")
_tool_search_hint_lock = threading.Lock()
_tool_search_hint_emitted = False
#: Re-arm interval for the tool-search-disabled warning. It used to fire exactly
#: once per process, which on a proxy that stays up for weeks means one line in
#: the log at startup and silence thereafter — the condition it reports persists
#: for the whole life of the deployment and costs tokens on every single request,
#: so a single line is not proportionate to it. Hourly is frequent enough that an
#: operator tailing logs will see it, rare enough that it is not noise.
_TOOL_SEARCH_HINT_INTERVAL_S = 3600.0
_tool_search_hint_last: float | None = None
# Module-level indirection over the clock so tests can drive it without
# monkeypatching stdlib ``time.monotonic``, which would freeze it for every
# other thread in the process. Same pattern as savings_tracker.
_monotonic = time.monotonic
def claude_code_tool_search_inactive(
@@ -3202,36 +3213,46 @@ def format_tool_search_disabled_hint(tools: list[Any]) -> str:
def tool_search_hint_pending() -> bool:
"""Cheap, lock-free check of whether the one-time hint may still fire.
"""Cheap, lock-free check of whether the detection scan may run now.
Lets the request hot path skip the (O(number-of-tools)) detection scan on
every request once the hint has already been emitted. A benign race here
only costs one extra detection scan, never a duplicate warning — the
actual one-shot guarantee lives in :func:`take_tool_search_hint_slot`.
Lets the request hot path skip the O(number-of-tools) scan while the hint
is throttled. A benign race here only costs one extra scan, never a
duplicate warning — the rate limit itself lives in
:func:`take_tool_search_scan_slot`.
"""
return not _tool_search_hint_emitted
last = _tool_search_hint_last
return last is None or (_monotonic() - last) >= _TOOL_SEARCH_HINT_INTERVAL_S
def take_tool_search_hint_slot() -> bool:
"""Return ``True`` exactly once per process, gating the one-time hint.
def take_tool_search_scan_slot() -> bool:
"""Claim the right to run the detection scan, at most once per interval.
Thread-safe so concurrent requests cannot each emit the warning.
The window governs SCANS, not emissions, and that distinction is
load-bearing. Stamping only when the hint actually fires means that once the
operator FIXES the condition the stamp stops advancing, ``pending`` stays
true forever, and every subsequent request pays the full scan over the
client's tool array for the life of the process — the opposite of the gate's
purpose, and worst on exactly the large tool surfaces this targets.
Claiming the slot before the scan also collapses the old two-call protocol
into one, so there is no window in which two threads both scan and both
emit.
"""
global _tool_search_hint_emitted
if _tool_search_hint_emitted:
global _tool_search_hint_last
if not tool_search_hint_pending():
return False
with _tool_search_hint_lock:
if _tool_search_hint_emitted:
if not tool_search_hint_pending():
return False
_tool_search_hint_emitted = True
_tool_search_hint_last = _monotonic()
return True
def reset_tool_search_hint_state() -> None:
"""Reset the one-time hint guard. Test helper only."""
global _tool_search_hint_emitted
"""Reset the hint rate limit. Test helper only."""
global _tool_search_hint_last
with _tool_search_hint_lock:
_tool_search_hint_emitted = False
_tool_search_hint_last = None
# ---------------------------------------------------------------------------
@@ -3248,6 +3269,14 @@ def reset_tool_search_hint_state() -> None:
# stays callable. Deterministic output → the tools prefix still prompt-caches.
# ---------------------------------------------------------------------------
# Resident no matter what the operator's override says. The client's own
# tool-search tool is the one thing that must never be deferred: it is what
# loads the tools it resolves, and Claude Code uses it to reach tools held in a
# local registry (TaskCreate, WebFetch, ...) that nothing else can reach. An
# override says which ORDINARY tools stay inline, so letting it drop this would
# silently orphan a whole category rather than defer it.
_CLIENT_SIDE_SEARCH_FLOOR = frozenset({"toolsearch"})
# Core coding tools kept non-deferred so routine edit/read/run loops never pay a
# search round-trip. Everything else (Slack/Linear/Sentry/Notion/Snowflake/…) is
# deferred and loaded on demand. Anthropic recommends keeping the 3–5 (here a few
@@ -3281,9 +3310,65 @@ _TOOL_SEARCH_CORE_TOOLS = frozenset(
)
def resolved_core_tools() -> frozenset[str]:
#: Canonical name for the resident-tool override, and the legacy name the
#: tool-search plugin shipped for the same idea. Two variables for one knob is a
#: trap: an operator sets the one they know, the other path silently keeps its
#: own list, and the two providers disagree about which tools are visible. Both
#: are read here so either spelling works everywhere; the canonical one wins.
CORE_TOOLS_ENV = "HEADROOM_TOOL_SEARCH_CORE_TOOLS"
CORE_TOOLS_ENV_LEGACY = "HEADROOM_TOOL_SEARCH_CORE"
_core_tools_legacy_lock = threading.Lock()
_core_tools_legacy_warned = False
def reset_core_tools_legacy_warn_state() -> None:
"""Re-arm the legacy-variable warning. Test helper only.
Without this the flag leaks between tests in a process, so a caplog
assertion on this warning passes alone and fails depending on ordering.
"""
global _core_tools_legacy_warned
with _core_tools_legacy_lock:
_core_tools_legacy_warned = False
def _core_tools_override() -> str | None:
"""Return the resident-tool override from either env var, canonical first."""
global _core_tools_legacy_warned
raw = os.environ.get(CORE_TOOLS_ENV)
if raw is not None:
return raw
legacy = os.environ.get(CORE_TOOLS_ENV_LEGACY)
should_warn = False
if legacy is not None:
# Check-and-set under the lock: unguarded, two threads racing on their
# first request could both warn.
with _core_tools_legacy_lock:
if not _core_tools_legacy_warned:
_core_tools_legacy_warned = True
should_warn = True
if should_warn:
logger.warning(
"event=tool_search_core_env_legacy old=%s new=%s "
"hint=honoring the legacy variable; rename it, both paths read the new one",
CORE_TOOLS_ENV_LEGACY,
CORE_TOOLS_ENV,
)
return legacy
def resolved_core_tools(extra: frozenset[str] = frozenset()) -> frozenset[str]:
"""Tools that stay resident, with a deployment override.
``extra`` carries a provider's own additions to the default resident set
(the OpenAI Responses path keeps ``terminal`` resident alongside the shared
coding loop). It is folded into the DEFAULT only. An explicit
``HEADROOM_TOOL_SEARCH_CORE_TOOLS`` replaces the whole set, additions
included -- otherwise the knob would silently fail to defer a tool the
operator had just asked to defer, and the two provider paths would honour
the same variable differently.
The default keeps the coding loop resident so routine edit/read/run never
pays a search round-trip. That default is now out of step with Claude Code,
which since v2.1.69 defers its built-ins too and thereby cuts built-in
@@ -3299,10 +3384,21 @@ def resolved_core_tools() -> frozenset[str]:
against real traffic instead of a guess shipped as a default.
"""
raw = os.environ.get("HEADROOM_TOOL_SEARCH_CORE_TOOLS")
raw = _core_tools_override()
if raw is None:
return _TOOL_SEARCH_CORE_TOOLS
return frozenset(_tool_search_resident_key(part) for part in raw.split(",") if part.strip())
if not extra:
return _TOOL_SEARCH_CORE_TOOLS
return _TOOL_SEARCH_CORE_TOOLS | {_tool_search_resident_key(name) for name in extra}
# ``part.strip()`` before keying, not just for the emptiness test: the key
# function lowercases and strips leading underscores but NOT spaces, so the
# natural spelling "bash, read, terminal" used to resolve to {" read",
# " terminal", "bash"} and defer the two tools the operator asked to pin.
named = {_tool_search_resident_key(part.strip()) for part in raw.split(",") if part.strip()}
# The client's own tool-search tool survives every override. Deferring it
# hides the only thing that can load the tools it resolves -- including
# tools the client keeps in a local registry, which nothing else can reach
# -- so an override that omits it would silently orphan them.
return frozenset(named | _CLIENT_SIDE_SEARCH_FLOOR)
_TOOL_SEARCH_DEFAULT_TYPE = "tool_search_tool_regex_20251119"
@@ -3407,7 +3503,7 @@ def _client_tool_search_names() -> frozenset[str]:
if not extra.strip():
return _CLIENT_TOOL_SEARCH_NAMES
return _CLIENT_TOOL_SEARCH_NAMES | {
_tool_search_resident_key(part) for part in extra.split(",") if part.strip()
_tool_search_resident_key(part.strip()) for part in extra.split(",") if part.strip()
}
@@ -3933,7 +4029,7 @@ def inject_tool_search_deferral_openai(
stays valid; the injected search tool is itself resident.
"""
if core_tools is None:
core_tools = resolved_core_tools()
core_tools = resolved_core_tools(_OPENAI_TOOL_SEARCH_RESIDENT_NAMES)
if not openai_tool_search_client_supported(client):
return tools
if not _model_supports_openai_tool_search(model):
@@ -3954,9 +4050,11 @@ def inject_tool_search_deferral_openai(
deferred = 0
# Normalize for the same reason as the Anthropic path above: clients may use
# different casing or a leading namespace marker for the same resident tool.
resident_keys = {_tool_search_resident_key(name) for name in core_tools} | {
_tool_search_resident_key(name) for name in _OPENAI_TOOL_SEARCH_RESIDENT_NAMES
}
# No unconditional union with _OPENAI_TOOL_SEARCH_RESIDENT_NAMES here: it is
# folded into the default by resolved_core_tools() above, so an explicit
# override drops it too. Unioning it in at this point would pin ``terminal``
# resident even when the operator set the core set to something without it.
resident_keys = {_tool_search_resident_key(name) for name in core_tools}
for tool in tools:
if not isinstance(tool, dict):
out.append(tool)
+7 -2
View File
@@ -25,9 +25,9 @@ from dataclasses import dataclass
from typing import Any
from headroom.proxy.helpers import (
_TOOL_SEARCH_CORE_TOOLS,
anthropic_model_is_first_party,
inject_tool_search_deferral,
resolved_core_tools,
tools_are_anthropic_shaped,
)
from headroom.proxy.turn_hooks import register_turn_hook
@@ -57,7 +57,12 @@ class DeferralHook:
return
if not tools_are_anthropic_shaped(ctx.tools):
return
after = inject_tool_search_deferral(ctx.tools, core_tools=_TOOL_SEARCH_CORE_TOOLS)
# resolved_core_tools(), not the raw default: passing the constant
# ignored HEADROOM_TOOL_SEARCH_CORE_TOOLS entirely, so an operator who
# set it got the built-in list on this path and their own on the proxy
# path -- the two-paths-disagree failure the one-knob change exists to
# prevent, reintroduced by the reference hook.
after = inject_tool_search_deferral(ctx.tools, core_tools=resolved_core_tools())
if after is ctx.tools:
return
ctx.tools = after
+4 -4
View File
@@ -22,7 +22,7 @@ from headroom.proxy.helpers import (
claude_code_tool_search_inactive,
format_tool_search_disabled_hint,
reset_tool_search_hint_state,
take_tool_search_hint_slot,
take_tool_search_scan_slot,
tool_search_hint_pending,
)
@@ -159,11 +159,11 @@ def test_hint_slot_fires_once() -> None:
reset_tool_search_hint_state()
try:
assert tool_search_hint_pending() is True
assert take_tool_search_hint_slot() is True
assert take_tool_search_scan_slot() is True
# Once consumed, the cheap gate flips so the hot path stops scanning.
assert tool_search_hint_pending() is False
assert take_tool_search_hint_slot() is False
assert take_tool_search_hint_slot() is False
assert take_tool_search_scan_slot() is False
assert take_tool_search_scan_slot() is False
finally:
reset_tool_search_hint_state()
+171 -1
View File
@@ -184,7 +184,7 @@ def test_resident_set_is_overridable(monkeypatch: pytest.MonkeyPatch) -> None:
"""So deferring built-ins can be measured instead of guessed."""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "bash,read")
assert resolved_core_tools() == frozenset({"bash", "read"})
assert resolved_core_tools() == frozenset({"bash", "read", "toolsearch"})
tools = [
{"name": "bash", "input_schema": {}},
@@ -335,3 +335,173 @@ def test_openai_toolsearch_name_alone_does_not_stand_us_down() -> None:
by_name = {t.get("name"): t for t in out if isinstance(t, dict)}
assert by_name["ToolSearch"].get("defer_loading") is None
assert by_name["slack_0"].get("defer_loading") is True
# --------------------------------------------------------------------------
# the resident override must mean the same thing on both paths
# --------------------------------------------------------------------------
def test_openai_keeps_terminal_resident_by_default() -> None:
tools = [_fn("terminal"), _fn("bash"), *[_fn(f"slack_{i}") for i in range(14)]]
by_name = {
t.get("name"): t
for t in inject_tool_search_deferral_openai(tools, "gpt-5.5")
if isinstance(t, dict)
}
assert by_name["terminal"].get("defer_loading") is None
assert by_name["bash"].get("defer_loading") is None
assert by_name["slack_0"].get("defer_loading") is True
def test_openai_override_can_drop_terminal(monkeypatch: pytest.MonkeyPatch) -> None:
"""An explicit core set is authoritative — provider additions included.
``terminal`` used to be unioned in unconditionally, so setting the knob to a
set without it still pinned it resident. The two provider paths then honoured
the same variable differently, and an operator asking to defer a tool was
silently refused.
"""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "bash,read")
tools = [_fn("terminal"), _fn("bash"), _fn("grep"), *[_fn(f"slack_{i}") for i in range(14)]]
by_name = {
t.get("name"): t
for t in inject_tool_search_deferral_openai(tools, "gpt-5.5")
if isinstance(t, dict)
}
assert by_name["bash"].get("defer_loading") is None
assert by_name["terminal"].get("defer_loading") is True
assert by_name["grep"].get("defer_loading") is True
def test_default_resident_set_is_unchanged_for_anthropic() -> None:
"""The OpenAI addition must not leak into the Anthropic default."""
assert "terminal" not in resolved_core_tools()
assert "terminal" in resolved_core_tools(frozenset({"terminal"}))
# --------------------------------------------------------------------------
# the tool-search-disabled warning must not go quiet forever
# --------------------------------------------------------------------------
def test_hint_rearms_after_the_interval(monkeypatch: pytest.MonkeyPatch) -> None:
"""One line per process is not proportionate to a permanent condition.
The warning reports a deployment that has its client's tool deferral turned
off — which costs tokens on every request for the life of the deployment.
Firing once and never again means a proxy up for weeks says it at startup
and is silent through everything after.
"""
from headroom.proxy import helpers as H
H.reset_tool_search_hint_state()
try:
clock = {"t": 1000.0}
monkeypatch.setattr(H, "_monotonic", lambda: clock["t"])
assert H.take_tool_search_scan_slot() is True
assert H.take_tool_search_scan_slot() is False
clock["t"] += H._TOOL_SEARCH_HINT_INTERVAL_S - 1
assert H.take_tool_search_scan_slot() is False, "must not re-arm early"
clock["t"] += 2
assert H.take_tool_search_scan_slot() is True, "must re-arm after the interval"
finally:
H.reset_tool_search_hint_state()
# --------------------------------------------------------------------------
# one knob, two spellings
# --------------------------------------------------------------------------
def test_legacy_env_var_is_honoured(monkeypatch: pytest.MonkeyPatch) -> None:
"""The plugin shipped HEADROOM_TOOL_SEARCH_CORE for the same idea.
Two variables for one knob means an operator sets the one they know and the
other path silently keeps its own list, so the two providers disagree about
which tools are visible. Either spelling now works everywhere.
"""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE", "bash,read")
assert resolved_core_tools() == frozenset({"bash", "read", "toolsearch"})
def test_canonical_env_var_wins_over_legacy(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE", "bash,read")
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "grep")
assert resolved_core_tools() == frozenset({"grep", "toolsearch"})
def test_neither_set_keeps_the_default(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("HEADROOM_TOOL_SEARCH_CORE", raising=False)
monkeypatch.delenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", raising=False)
assert "bash" in resolved_core_tools()
assert "read" in resolved_core_tools()
def test_empty_override_defers_everything_non_typed(monkeypatch: pytest.MonkeyPatch) -> None:
"""An explicit empty set is a real instruction, not an unset variable."""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "")
assert resolved_core_tools() == frozenset({"toolsearch"})
def test_the_override_tolerates_spaces_after_commas(monkeypatch: pytest.MonkeyPatch) -> None:
"""The natural spelling of a list must not defer the tools it names.
The key function lowercases and strips leading underscores but not spaces,
so ``"bash, read, terminal"`` resolved to ``{" read", " terminal", "bash"}``
and deferred exactly the two tools the operator asked to keep resident.
"""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "bash, read , terminal")
assert resolved_core_tools() == frozenset({"bash", "read", "terminal", "toolsearch"})
def test_an_override_cannot_defer_the_clients_search_tool(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Deferring ToolSearch hides the only thing that can load what it resolves.
Claude Code reaches tools held in a local registry through it, and nothing
else can. An override names which ORDINARY tools stay inline, so honouring
one that omits this would orphan a whole category rather than defer it.
"""
monkeypatch.setenv("HEADROOM_TOOL_SEARCH_CORE_TOOLS", "Bash,Read")
tools = [CLAUDE_CODE_TOOL_SEARCH, {"name": "Bash", "input_schema": {}}, *_mcp(14)]
by_name = {t.get("name"): t for t in inject_tool_search_deferral(tools) if isinstance(t, dict)}
assert by_name["ToolSearch"].get("defer_loading") is None
assert by_name["mcp__srv0__do"].get("defer_loading") is True
def test_the_scan_gate_closes_even_when_the_hint_does_not_fire(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Once the operator FIXES the condition, the scan must stop running.
Stamping the window only on emission meant the gate stayed open forever
after the fix, so every later request paid the full O(tools) scan for the
life of the process -- worst on the large tool surfaces this targets.
"""
from headroom.proxy import helpers as H
H.reset_tool_search_hint_state()
try:
clock = {"t": 1000.0}
monkeypatch.setattr(H, "_monotonic", lambda: clock["t"])
assert H.take_tool_search_scan_slot() is True # scanned; found nothing
assert H.tool_search_hint_pending() is False # ...and the gate closed anyway
finally:
H.reset_tool_search_hint_state()