Files
headroom/tests/gateway/test_hook_tool_growth.py
AlessandroSpallinaandClaude Opus 5.5 a90421083a feat(proxy): report the same tool savings on gateway /v1/compress as the proxy path (#4467)
## Description

Gateway-mode `/v1/compress` now reports the same savings as the proxy
path. The why is in #4466.

Closes #4466

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update

## Changes Made

Two commits, one per gap:

1. **`fix(proxy): count tool-schema compaction ...`**
- The gateway turn counts the tool definitions before and after
compaction.
- The handler adds both counts to `tokens_before` / `tokens_after`, with
the proxy path's guard (`0 < after < before`).
   - Tools the client already marked `defer_loading` are skipped.
2. **`feat(proxy): report tool_tokens_saved and tool_tokens_deferred
...`**
- Gateway-mode answers add `tool_tokens_saved` (from the tags `/stats`
sums) and `tool_tokens_deferred` (the estimated share).
   - Docs: `proxy.mdx`, `wiki/proxy.md`.

Non-gateway and fail-open answers do not change.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`, on the changed files)
- [x] Type checking passes (`mypy`, on the changed files)
- [x] New tests added for new functionality

```text
$ pytest -q tests/gateway/ tests/test_gateway_turn_unit.py tests/test_proxy_compress_endpoint.py
380 passed, 5 skipped
```

The tests also pass on the first commit alone (`tests/gateway/`: 280
passed). The new test for client-deferred tools fails without its fix.

## Real Behavior Proof

- Environment: macOS, Python 3.12. Before: image
`headroom:0.41.0-nonroot`. After: this branch, `HEADROOM_OFFLINE=1
headroom proxy`.
- Exact command / steps: start each build, then send it the same
gateway-mode `POST /v1/compress` — one message plus 5 tools whose
schemas carry `title` / `examples` / `default` (the annotation keys
schema compaction strips), with `"gateway": {"can_redrive": false,
"can_relay_response": false}` so the request is a single call. Read
`tokens_before`, `tokens_after`, `tokens_saved` and the `tool_*` fields
out of the JSON answer. No provider is contacted.
- Observed result: same request, same tools. On 0.41.0 the tool
definitions are invisible to the counts, so `tokens_saved` is 0 and
neither `tool_*` field is present. On this branch the compaction is
counted, so `tokens_before` / `tokens_after` carry the tool bytes and
`tokens_saved` is the 240 tokens compaction stripped
(`tool_tokens_saved` / `tool_tokens_deferred` stay 0 here: no
tool-search extension ran, and the extension is what books deferral).

```text
0.41.0:  tokens_before 13,  tokens_after 13,  tokens_saved 0    (no tool_* fields)
branch:  tokens_before 830, tokens_after 590, tokens_saved 240, tool_tokens_saved 0, tool_tokens_deferred 0
```

- Not tested: a live run with the headroom-tool-search extension
(deferral is covered by unit tests), and Responses-API bodies.

## Runtime Rollout Safety

- Rollout-managed feature(s): None gate this change — the gateway
`/v1/compress` accounting is not behind a registry feature. One
interaction: `proxy_output_shaper` (registry feature, available
`stable`, deliberately not default-enabled) consumes the augmented
count. `tokens_before` is folded at
`headroom/proxy/handlers/openai.py:11064` and then handed to the turn at
`:11083`, which passes it as `input_tokens` to `shape_gateway_body`
(`headroom/proxy/gateway_extension.py:213`) → `stratum_key` →
`input_bucket`. A turn whose tool definitions push it across one of the
2k / 8k / 32k / 128k boundaries therefore lands in a bigger stratum,
which picks a different output-savings baseline. The chat path already
has this coupling.
- Minimum rollout channel: n/a — no channel gate, so the change takes
effect wherever gateway mode is served, on every channel.
- Stable/default behavior changed: Yes, gateway-mode answers only.
`tokens_before` / `tokens_after` now include the tool definitions before
and after schema compaction, so `tokens_saved` covers the compaction
saving the same way the chat handlers already do, and two fields are
added. Non-gateway and fail-open answers are byte-identical.
- Kill switch / disable path: None. Without compaction there is nothing
to fold, and the two fields are additive on answers; a gateway that does
not want them ignores them.
- Unsafe override required: No. Nothing crosses a channel boundary.
- Qualification impact: None. No registry entry is added, so
`registry_digest` and `snapshot_digest` are unchanged. The gateway
`tokens_saved` figure itself changes, which a benchmark comparing arms
across this commit will see.
- Rollback path: Revert the two commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have made corresponding changes to the documentation
- [x] I have added tests that my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The fields are emitted for any claimed compress turn. Only the gateway
extension claims turns today.
- If a response-half hook rewrote the tool tags during a re-drive,
`/stats` would no longer match the answer. Core has no such hook.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 08:45:53 -07:00

134 lines
5.1 KiB
Python

"""/v1/compress nets tool definitions a turn hook adds against its saving.
A hook that folds messages but adds a tool (skill search's search tool) sends
those tool tokens too; ``tokens_after`` must include them, as on the chat path.
"""
from __future__ import annotations
import json
from typing import Any
import pytest
from headroom.proxy.savings_attribution import from_tags
from headroom.proxy.turn_hooks import register_turn_hook
from tests.gateway.conftest import compress
_LONG = "line of tool output that a hook can fold away entirely " * 60
_ADDED_TOOL = {
"name": "search_skills",
"description": "Find a skill by what it does. " * 40,
"input_schema": {"type": "object", "properties": {"q": {"type": "string"}}},
}
class _FoldAndAddTool:
name = savings_source = "skill_search"
stream_safe = True
def __init__(self, add_tool: bool) -> None:
self.add_tool = add_tool
def on_request(self, ctx: Any) -> None:
ctx.messages = [{**m, "content": "folded"} for m in ctx.messages]
if self.add_tool:
ctx.tools = [*(ctx.tools or []), dict(_ADDED_TOOL)]
def _run(make_headroom_client: Any, *, add_tool: bool) -> dict[str, Any]:
register_turn_hook(_FoldAndAddTool(add_tool))
client = make_headroom_client(optimize=False)
body = {
"model": "claude-sonnet-4-5",
"messages": [{"role": "user", "content": _LONG}],
"tools": [{"name": "noop", "description": "x", "input_schema": {"type": "object"}}],
"gateway": {"can_redrive": False, "can_relay_response": False},
}
resp = compress(client, body)
assert resp.status_code == 200, resp.text
return resp.json()
def test_tools_a_hook_adds_count_as_sent(make_headroom_client) -> None:
from headroom.proxy.turn_hooks import clear_turn_hooks
plain = _run(make_headroom_client, add_tool=False)
clear_turn_hooks()
grown = _run(make_headroom_client, add_tool=True)
assert plain["tokens_saved"] > 0
added = grown["tokens_after"] - plain["tokens_after"]
assert added > 100 # the search tool's definition
assert grown["tokens_saved"] == plain["tokens_saved"] - added
assert grown["tokens_before"] == plain["tokens_before"]
class _AddTextRemoveTools:
name = savings_source = "rewriter"
stream_safe = True
def __init__(self, repeat: int) -> None:
self.repeat = repeat
def on_request(self, ctx: Any) -> None:
ctx.messages = [*ctx.messages, {"role": "user", "content": "steer. " * self.repeat}]
ctx.tools = []
@pytest.mark.parametrize("session", [False, True], ids=["stateless", "session"])
@pytest.mark.parametrize("repeat", [5, 2_000], ids=["net-positive", "net-negative"])
def test_message_growth_offsets_tools_a_hook_removed(
make_headroom_client, outcome_spy, repeat, session
) -> None:
"""The opposite signed case on /v1/compress: a hook appends message text
and drops the tools. The headline is removed tools minus added text,
clamped once, not the full tool removal."""
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
from headroom.tokenizers import get_tokenizer
register_turn_hook(_AddTextRemoveTools(repeat))
client = make_headroom_client(optimize=False)
outcomes = outcome_spy(client)
before = [{"role": "user", "content": "hi"}]
body = {
"model": "claude-sonnet-4-5",
"messages": before,
"tools": [dict(_ADDED_TOOL)],
"gateway": {"can_redrive": False, "can_relay_response": False},
}
if session:
# Session turns recount the final (post-hook) messages: the added
# text must not be counted a second time on top of that.
body["config"] = {"session_id": f"growth-{repeat}"}
resp = compress(client, body)
assert resp.status_code == 200, resp.text
tok = get_tokenizer("claude-sonnet-4-5")
growth = tok.count_messages(
[*before, {"role": "user", "content": "steer. " * repeat}]
) - tok.count_messages(before)
removed = tok.count_text(json.dumps([_ADDED_TOOL], default=str))
net = removed - growth
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
headline = headline_tokens_saved(outcome.tokens_saved, outcome.tags)
# The answer reports the same headline, none of it estimated.
data = resp.json()
assert data["tokens_saved"] + data["tool_tokens_saved"] == headline
assert data["tool_tokens_deferred"] == 0
# The gateway compacts tool schemas before hooks run: the hook's own ledger
# covers the compacted schema, and the compaction delta is inside
# tokens_saved, so the headline is the two together.
from headroom.proxy.gateway_turn import compact_gateway_tools
compacted, _ = compact_gateway_tools([dict(_ADDED_TOOL)])
compaction = removed - tok.count_text(json.dumps(compacted, default=str))
entries = from_tags(outcome.tags)
if net > 0:
(entry,) = entries
hook_net = entry["details"]["tool_tokens_saved"] + entry["details"]["message_tokens_saved"]
assert entry["details"]["message_tokens_saved"] < 0
assert headline == hook_net + compaction == entry["tokens"] + compaction
else:
assert entries == []
assert headline == 0