fix(proxy): keep tool deferral for GitHub Copilot upstream (#4356)

## Description

`strip_first_party_tool_search_tools_for_third_party_upstream()` strips
`defer_loading`
from every tool when the upstream is not `api.anthropic.com`. GitHub
Copilot's endpoint
accepts the deferred shape and bills deferred schemas compactly, so
downgrading it
inlines the whole catalog. On an identical 295,634-byte body with 103
deferred tools,
billed prompt tokens went **48,016 → 109,233 (2.3×)**. Silent failure:
no error, and
`/stats` reports a net *saving* since it only counts the compression
delta.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_DEFERRAL_CAPABLE_HOSTS` frozenset listing non-Anthropic
upstreams that accept the deferred tool shape.
- `anthropic_first_party_tool_search_supported()` returns `True` for
those hosts, so `api.githubcopilot.com` keeps `defer_loading` and its
tool-search tool.
- Matching reuses `_gate_view_host()`, so it is exact —
`api.githubcopilot.com.evil.com` is still downgraded.
- Bedrock, Vertex and gateway upstreams are unchanged.
- Three tests: the fix, the exact-host guard, and a regression guard for
other third-party upstreams.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_tool_search_client_deferral.py -k "copilot or other_third_party" -q
# ---- source fix reverted, new tests present ----
tests/test_tool_search_client_deferral.py ...F..                         [100%]
_________________ test_github_copilot_upstream_keeps_deferral __________________
tests/test_tool_search_client_deferral.py:427: in test_github_copilot_upstream_keeps_deferral
    assert out is tools
E   AssertionError: assert [{'name': 'mcp__srv__a'}] is [{'name': 'tool_search_tool_regex',
    'type': 'tool_search_tool_regex_20251119'}, {'defer_loading': True, 'name': 'mcp__srv__a'}]
================== 1 failed, 5 passed, 60 deselected in 1.77s ==================

# ---- fix applied ----
tests/test_tool_search_client_deferral.py ......                         [100%]
======================= 6 passed, 60 deselected in 0.76s =======================

$ uv run pytest tests/test_tool_search_client_deferral.py tests/test_issue_746_tool_search.py -q
============================= 137 passed in 0.78s ==============================

$ uv run ruff check headroom/proxy/helpers.py tests/test_tool_search_client_deferral.py
All checks passed!

$ uv run mypy headroom/proxy/helpers.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

**Environment**

- macOS 15 (arm64), Python 3.13
- headroom-ai 0.40.0 installed via `uv tool`, with this change applied
to the installed package
- GitHub Copilot CLI 1.0.100, model `claude-haiku-5.5`
- Upstream: `https://api.githubcopilot.com`
- Launched via `headroom wrap copilot --native`

**Exact command / steps**

```bash
# one request, same prompt every run
headroom wrap copilot --native -- \
  -p "Reply with exactly: OK" --model claude-haiku-5.5 --allow-all-tools -s

# billed tokens, read from the session usage record
jq -r 'select(.type=="session.usage_record") | .data.usage
       | "body=\(.requestBodyBytes) billed=\(.inputTokens) cache_write=\(.cacheWriteTokens)"' \
   ~/.copilot/session-state/<id>/events.jsonl
```

Separately, a loopback relay was placed between Headroom and
`api.githubcopilot.com` to
record the forwarded body and read the provider's own `usage` object.

**Observed result — live request output**

Identical 295,634-byte request body in all three runs:

```text
ARM 1  no proxy (baseline)
  body=295,634  billed_prompt_tokens=48,909   defer_loading forwarded: 103/103

ARM 2  through Headroom, WITH this fix
  PATCHED+REAL-UPSTREAM    input=48,016   cW=48,012   cR=0        body=295,634
  tool_search_downgraded occurrences in proxy log: 0

ARM 3  through Headroom, WITHOUT this fix
  UNPATCHED+REAL-UPSTREAM  input=109,233  cW=22,060   cR=87,169   body=295,634
  tool_search_downgraded occurrences in proxy log: 1

ARM 2 re-run through the real entrypoint (headroom wrap copilot --native)
  wrap --native + PATCH:   input=48,016   cW=48,012   cR=0        body=295,634
```

| run | billed prompt tokens | vs baseline | `tool_search_downgraded` |
| --- | --- | --- | --- |
| no proxy | 48,909 | — | — |
| **with fix** | **48,016** | **0.98×** | **0** |
| without fix | 109,233 | 2.23× | 1 |

Log line emitted only by the unpatched run:

```text
event=tool_search_downgraded reason=third_party_upstream tools=252
hint=client-side tool deferral cannot be forwarded to this upstream; tools are sent eagerly
```

Provider-side confirmation from the relay — these are Anthropic's own
`usage` values, not
client estimates, captured on a separate pair of runs:

```text
direct to upstream (no Headroom)
  req_bytes=425,446  n_tools=252  defer_loading flags forwarded: 230
  usage = {'cache_creation_input_tokens': 47987, 'cache_read_input_tokens': 0,
           'input_tokens': 4, 'output_tokens': 4}

through Headroom, without this fix
  req_bytes=411,456  n_tools=253  defer_loading flags forwarded: 0
  usage = {'cache_creation_input_tokens': 157059, 'cache_read_input_tokens': 0,
           'input_tokens': 4, 'output_tokens': 4}
```

Headroom forwards a **smaller** body (411,456 vs 425,446 bytes — the
removed
`defer_loading` keys) and is billed **3.3× more**. Upstream returned
HTTP 200 in every
run and tools resolved normally with deferral preserved, so the endpoint
accepts the
deferred shape. Toggling only this predicate and re-running reproduced
109,233 exactly.

**Not tested**

- Enterprise / data-residency Copilot hosts reached via
`GITHUB_COPILOT_API_URL` — a
  different hostname, so they still take the existing downgrade path.
- Bedrock, Vertex and LiteLLM upstreams live; those are covered by unit
tests only.
- Windows and Linux.
- Non-Claude models.
- The OpenAI Responses deferral path.
- Evidence above was produced against the installed 0.40.0 package
carrying this same
one-line change, not a checkout of this branch. Logic is identical;
happy to rerun
  from the branch if you want that specific artifact.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** none. Neither
`anthropic_first_party_tool_search_supported()`
nor `strip_first_party_tool_search_tools_for_third_party_upstream()`
consults
`rollout.is_enabled()`; the predicate is evaluated per request from the
upstream URL.
- **Minimum rollout channel:** N/A — not gated.
- **Stable/default behavior changed:** yes, for one host. Requests to
`api.githubcopilot.com` now forward `defer_loading` and the tool-search
tool instead of
  having them stripped. Every other upstream is byte-identical.
- **Kill switch / disable path:** none added. The host set is a single
module-level
constant and can be made env-overridable if you want an explicit off
switch.
- **Unsafe override required:** no.
- **Qualification impact:** affects Copilot-routed Anthropic traffic
only. Reported
input and cache-write token counts drop for those sessions — a real
reduction in billed
tokens rather than a measurement change — so token baselines for Copilot
upstreams
  shift downward.
- **Rollback path:** revert the commit. No persisted state, cache
format, config or
  wire-schema change; the next request takes the old branch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — terminal output included above.

## Additional Notes

- Documentation checklist item is N/A: this restores intended behavior
on one upstream;
  no documented flag, config key or CLI surface changes.
- **Scope question for maintainers.** The downgrade keys on "not
`api.anthropic.com`", so
it also fires for any relay or gateway sitting in front of a
deferral-capable endpoint.
I hit this while building the harness: pointing `--anthropic-api-url` at
a loopback
relay triggered the strip even though the real upstream was Copilot. A
capability probe,
or making the host set configurable, would generalize better than a host
literal. Kept
this PR minimal rather than guessing at the broader design — happy to
take direction.
- Enterprise Copilot hosts via `GITHUB_COPILOT_API_URL` are deliberately
omitted since I
could not verify them; adding them is a one-line change to the
frozenset.

---------

Co-authored-by: divyamnpll <divyamnpll@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
divyamnpll
2026-10-24 08:43:58 -07:00
committed by GitHub
co-authored by divyamnpll JerrettDavis
parent a79def4dd1
commit 217b658845
2 changed files with 37 additions and 1 deletions
+13 -1
View File
@@ -4107,9 +4107,21 @@ def _warn_deferred_without_search_tool(tools: Any) -> None:
)
#: Non-Anthropic upstreams that accept the first-party deferred tool shape.
#: These bill deferred schemas compactly, so downgrading deferral for them
#: inlines the whole catalog and multiplies billed prompt tokens.
_DEFERRAL_CAPABLE_HOSTS: frozenset[str] = frozenset({"api.githubcopilot.com"})
def anthropic_first_party_tool_search_supported(api_base_url: str | None) -> bool:
"""Return whether Anthropic server-side tool search is valid for this upstream."""
from headroom.providers.claude.runtime import is_custom_anthropic_base_url
from headroom.providers.claude.runtime import (
_gate_view_host,
is_custom_anthropic_base_url,
)
if _gate_view_host((api_base_url or "").strip()) in _DEFERRAL_CAPABLE_HOSTS:
return True
return not is_custom_anthropic_base_url(api_base_url)
+24
View File
@@ -38,6 +38,7 @@ import pytest
from headroom.proxy.helpers import (
_TOOL_SEARCH_MIN_TOOLS,
anthropic_first_party_tool_search_supported,
claude_code_tool_search_inactive,
inject_tool_search_deferral,
inject_tool_search_deferral_openai,
@@ -410,6 +411,29 @@ def test_nothing_to_strip_returns_the_same_list() -> None:
assert out is tools
def test_github_copilot_upstream_keeps_deferral() -> None:
"""Copilot accepts the deferred shape and bills deferred schemas compactly."""
tools = [ANTHROPIC_SEARCH_REGEX, {"name": "mcp__srv__a", "defer_loading": True}]
out = strip_first_party_tool_search_tools_for_third_party_upstream(
tools, "https://api.githubcopilot.com"
)
assert out is tools
assert anthropic_first_party_tool_search_supported("https://api.githubcopilot.com")
def test_copilot_host_match_is_exact() -> None:
"""A lookalike host must not inherit the exemption."""
assert not anthropic_first_party_tool_search_supported("https://api.githubcopilot.com.evil.com")
def test_other_third_party_upstreams_still_downgrade() -> None:
"""The exemption is host-scoped: Bedrock/Vertex/gateways are unchanged."""
for url in (THIRD_PARTY_URL, "https://bedrock-runtime.us-east-1.amazonaws.com"):
assert not anthropic_first_party_tool_search_supported(url)
# --------------------------------------------------------------------------
# OpenAI Responses path
# --------------------------------------------------------------------------