feat(proxy): report the same tool savings on gateway /v1/compress as the proxy path (#4467)

## Description

Gateway-mode `/v1/compress` now reports the same savings as the proxy
path. The why is in #4466.

Closes #4466

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update

## Changes Made

Two commits, one per gap:

1. **`fix(proxy): count tool-schema compaction ...`**
- The gateway turn counts the tool definitions before and after
compaction.
- The handler adds both counts to `tokens_before` / `tokens_after`, with
the proxy path's guard (`0 < after < before`).
   - Tools the client already marked `defer_loading` are skipped.
2. **`feat(proxy): report tool_tokens_saved and tool_tokens_deferred
...`**
- Gateway-mode answers add `tool_tokens_saved` (from the tags `/stats`
sums) and `tool_tokens_deferred` (the estimated share).
   - Docs: `proxy.mdx`, `wiki/proxy.md`.

Non-gateway and fail-open answers do not change.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`, on the changed files)
- [x] Type checking passes (`mypy`, on the changed files)
- [x] New tests added for new functionality

```text
$ pytest -q tests/gateway/ tests/test_gateway_turn_unit.py tests/test_proxy_compress_endpoint.py
380 passed, 5 skipped
```

The tests also pass on the first commit alone (`tests/gateway/`: 280
passed). The new test for client-deferred tools fails without its fix.

## Real Behavior Proof

- Environment: macOS, Python 3.12. Before: image
`headroom:0.41.0-nonroot`. After: this branch, `HEADROOM_OFFLINE=1
headroom proxy`.
- Exact command / steps: start each build, then send it the same
gateway-mode `POST /v1/compress` — one message plus 5 tools whose
schemas carry `title` / `examples` / `default` (the annotation keys
schema compaction strips), with `"gateway": {"can_redrive": false,
"can_relay_response": false}` so the request is a single call. Read
`tokens_before`, `tokens_after`, `tokens_saved` and the `tool_*` fields
out of the JSON answer. No provider is contacted.
- Observed result: same request, same tools. On 0.41.0 the tool
definitions are invisible to the counts, so `tokens_saved` is 0 and
neither `tool_*` field is present. On this branch the compaction is
counted, so `tokens_before` / `tokens_after` carry the tool bytes and
`tokens_saved` is the 240 tokens compaction stripped
(`tool_tokens_saved` / `tool_tokens_deferred` stay 0 here: no
tool-search extension ran, and the extension is what books deferral).

```text
0.41.0:  tokens_before 13,  tokens_after 13,  tokens_saved 0    (no tool_* fields)
branch:  tokens_before 830, tokens_after 590, tokens_saved 240, tool_tokens_saved 0, tool_tokens_deferred 0
```

- Not tested: a live run with the headroom-tool-search extension
(deferral is covered by unit tests), and Responses-API bodies.

## Runtime Rollout Safety

- Rollout-managed feature(s): None gate this change — the gateway
`/v1/compress` accounting is not behind a registry feature. One
interaction: `proxy_output_shaper` (registry feature, available
`stable`, deliberately not default-enabled) consumes the augmented
count. `tokens_before` is folded at
`headroom/proxy/handlers/openai.py:11064` and then handed to the turn at
`:11083`, which passes it as `input_tokens` to `shape_gateway_body`
(`headroom/proxy/gateway_extension.py:213`) → `stratum_key` →
`input_bucket`. A turn whose tool definitions push it across one of the
2k / 8k / 32k / 128k boundaries therefore lands in a bigger stratum,
which picks a different output-savings baseline. The chat path already
has this coupling.
- Minimum rollout channel: n/a — no channel gate, so the change takes
effect wherever gateway mode is served, on every channel.
- Stable/default behavior changed: Yes, gateway-mode answers only.
`tokens_before` / `tokens_after` now include the tool definitions before
and after schema compaction, so `tokens_saved` covers the compaction
saving the same way the chat handlers already do, and two fields are
added. Non-gateway and fail-open answers are byte-identical.
- Kill switch / disable path: None. Without compaction there is nothing
to fold, and the two fields are additive on answers; a gateway that does
not want them ignores them.
- Unsafe override required: No. Nothing crosses a channel boundary.
- Qualification impact: None. No registry entry is added, so
`registry_digest` and `snapshot_digest` are unchanged. The gateway
`tokens_saved` figure itself changes, which a benchmark comparing arms
across this commit will see.
- Rollback path: Revert the two commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] I have made corresponding changes to the documentation
- [x] I have added tests that my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The fields are emitted for any claimed compress turn. Only the gateway
extension claims turns today.
- If a response-half hook rewrote the tool tags during a re-drive,
`/stats` would no longer match the answer. Core has no such hook.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
AlessandroSpallina
2026-10-26 08:45:53 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 3fef15d376
commit a90421083a
9 changed files with 184 additions and 4 deletions
+7
View File
@@ -824,6 +824,13 @@ The response keeps every field above and adds six:
Fail-open answers (bypass header, empty `messages`, stateless timeout) carry the same six keys with your original body, `obligations: []` and `headers: {}`.
**Savings on a gateway answer.** The token counts follow the proxy path, so a gateway can report the same figure `/stats` does. When compaction shortened `tools`, `tokens_before` and `tokens_after` also count the tool definitions before and after it, so `tokens_saved` includes the compaction saving. Two more fields carry the tool definitions kept out of the context, which no message count sees; both are absent on fail-open answers:
| Field | Description |
| --- | --- |
| `tool_tokens_saved` | Tool-search deferral plus tools a turn hook removed, net (`tool_search_deferred_tokens` after reconciliation + `turn_hook_tools_saved_tokens` after netting against message growth). `tokens_saved + tool_tokens_saved` is the `/stats` headline for the turn. |
| `tool_tokens_deferred` | The deferral share of `tool_tokens_saved`. `/stats` books it as `estimated`: a deferred tool the model loads later is billed after all. Subtract it for a figure of measured savings only. |
When `relay_usage` is owed, the turn's `/stats` record is deferred until the response half arrives with billed input and output, cache counts, provider status and latency; a turn that never comes back is recorded as-is when it expires. Pending turns live `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (default `120`, refreshed on every `redrive`), at most `HEADROOM_GATEWAY_MAX_PENDING_TURNS` at a time (default `10000`, oldest evicted first), and a turn may re-drive at most `HEADROOM_GATEWAY_MAX_REDRIVES` times (default `8`).
**`config.mode: "ccr"` on this contract.** When markers were inserted and re-driving is allowed, Headroom injects the `headroom_retrieve` tool into `body.tools` (`ccr_tool_injected` in `transforms_applied`), adds the `redrive` obligation, and the response half answers the model's retrieval calls itself — the gateway just keeps re-driving. Without `can_redrive`, `ccr` mode behaves as it does today: the markers are yours to resolve.
+3
View File
@@ -96,6 +96,9 @@ class CompressTurn(Protocol):
# Likewise optional: ``message_growth_tokens`` (message tokens it added)
# and ``hook_tool_saved_tokens`` (tool tokens it removed and booked), so
# the handler can net the two before clamping the saving.
# And ``compaction_tokens_before``/``compaction_tokens_after``, the tool
# definitions before and after the schema compaction ``transform`` ran,
# which the handler folds into both endpoints as the chat handlers do.
def finish(
self,
+8
View File
@@ -153,6 +153,14 @@ class GatewayTurn:
def hook_tool_saved_tokens(self) -> int:
return self._transformer.hook_tool_saved_tokens if self._transformer is not None else 0
@property
def compaction_tokens_before(self) -> int:
return self._transformer.compaction_tokens_before if self._transformer is not None else 0
@property
def compaction_tokens_after(self) -> int:
return self._transformer.compaction_tokens_after if self._transformer is not None else 0
# -- event-loop side ---------------------------------------------------
def finish(
self,
+21
View File
@@ -866,6 +866,11 @@ class RequestTransformer:
# (booked on turn_hook_tools_saved_tokens), for signed netting.
self.message_growth_tokens = 0
self.hook_tool_saved_tokens = 0
# Tool-definition tokens before and after schema compaction, both 0 when
# it changed nothing or could not be counted. The compress handler folds
# them into ``tokens_before``/``tokens_after``, as the chat handlers do.
self.compaction_tokens_before = 0
self.compaction_tokens_after = 0
self._tokenizer: Any = None
def _get_tokenizer(self) -> Any:
@@ -906,6 +911,15 @@ class RequestTransformer:
count_tools = _count_tools_with(tokenizer)
except Exception:
logger.debug("gateway turn: tokenizer unavailable; hook savings uncounted")
if tool_transforms:
# Counted before any hook runs: a hook may edit the compacted list.
# Tools the client already deferred are not billed, so their
# compaction saves nothing.
compacted_before = _safe_count(count_tools, _billed_tools(self.tools))
compacted_after = _safe_count(count_tools, _billed_tools(tools))
if compacted_before is not None and compacted_after is not None:
self.compaction_tokens_before = compacted_before
self.compaction_tokens_after = compacted_after
# Built even with no hook registered: the response half drives CCR
# retrieval through the same context (messages + live tools).
@@ -981,6 +995,13 @@ class RequestTransformer:
return messages
def _billed_tools(tools: Any) -> Any:
"""The tools that enter the billed context: all but those flagged ``defer_loading``."""
if not isinstance(tools, list):
return tools
return [t for t in tools if not (isinstance(t, dict) and t.get("defer_loading"))]
def _safe_count(counter: Callable[[Any], int] | None, value: Any) -> int | None:
if counter is None:
return None
+21
View File
@@ -54,6 +54,7 @@ from headroom.proxy.semantic_cache_key_policy import compute_request_cache_parti
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
from headroom.proxy.tool_schema_savings_policy import (
net_hook_tool_saving,
tool_schema_saved_from_tags,
without_deferral_flags,
)
from headroom.proxy.turn_hooks import run_responses_create_hooks
@@ -11055,6 +11056,13 @@ class OpenAIHandlerMixin:
# the saving, and offset tools the hook removed (signed, clamped once).
_hook_tool_saved = 0
if _turn is not None:
# Tool-schema compaction, folded into both endpoints as the chat
# handlers do: the message counts never see tool bytes.
_compacted_before = int(getattr(_turn, "compaction_tokens_before", 0) or 0)
_compacted_after = int(getattr(_turn, "compaction_tokens_after", 0) or 0)
if 0 < _compacted_after < _compacted_before:
tokens_before += _compacted_before
tokens_after += _compacted_after
tokens_after += max(0, int(getattr(_turn, "tool_growth_tokens", 0) or 0))
# Only a count of pre-hook messages lacks the text a hook added;
# a recount of the final messages (stateless fold, session) has it.
@@ -11134,6 +11142,19 @@ class OpenAIHandlerMixin:
}
if session_info is not None:
_payload["session"] = session_info
if _turn is not None:
# Tool definitions kept out of the context (deferral, a hook's
# tool shrink), which the message counts never see. The headline
# /stats reports is tokens_saved + tool_tokens_saved. The
# deferral share is an estimate (booked ``estimated``), so it is
# also reported on its own for a caller that bills on measured
# savings only.
_payload["tool_tokens_saved"] = tool_schema_saved_from_tags(tags)
try:
_deferred = int(tags.get("tool_search_deferred_tokens", 0) or 0)
except (TypeError, ValueError):
_deferred = 0
_payload["tool_tokens_deferred"] = max(0, _deferred)
if _finished is not None and _finished.fields:
_payload.update(_finished.fields)
return JSONResponse(_payload)
+12 -3
View File
@@ -111,14 +111,23 @@ def test_message_growth_offsets_tools_a_hook_removed(
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
headline = headline_tokens_saved(outcome.tokens_saved, outcome.tags)
# The gateway compacts tool schemas before hooks run, so compare with the
# hook's own measured deltas (the runner's ledger), not the raw schema.
# The answer reports the same headline, none of it estimated.
data = resp.json()
assert data["tokens_saved"] + data["tool_tokens_saved"] == headline
assert data["tool_tokens_deferred"] == 0
# The gateway compacts tool schemas before hooks run: the hook's own ledger
# covers the compacted schema, and the compaction delta is inside
# tokens_saved, so the headline is the two together.
from headroom.proxy.gateway_turn import compact_gateway_tools
compacted, _ = compact_gateway_tools([dict(_ADDED_TOOL)])
compaction = removed - tok.count_text(json.dumps(compacted, default=str))
entries = from_tags(outcome.tags)
if net > 0:
(entry,) = entries
hook_net = entry["details"]["tool_tokens_saved"] + entry["details"]["message_tokens_saved"]
assert entry["details"]["message_tokens_saved"] < 0
assert headline == hook_net == entry["tokens"]
assert headline == hook_net + compaction == entry["tokens"] + compaction
else:
assert entries == []
assert headline == 0
+26
View File
@@ -233,6 +233,32 @@ def test_hook_deferral_reaches_body_headers_and_outcome(
assert "turn_hook_tools_saved_tokens" not in tags
def test_deferred_tokens_are_reported_beside_tokens_saved(
headroom_client, outcome_spy, deferral_hook
) -> None:
"""The answer carries the tool tokens deferral kept out of the context, so a
gateway can report the same headline as /stats."""
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
outcomes = outcome_spy(headroom_client)
sent = _body(tools=_many_anthropic_tools(), gateway={})
data = compress(headroom_client, sent).json()
n_tok = _assert_deferred(data, sent["tools"])
assert data["tool_tokens_saved"] == n_tok
assert data["tool_tokens_deferred"] == n_tok
outcome = outcomes[-1]
assert data["tokens_saved"] + data["tool_tokens_saved"] == headline_tokens_saved(
outcome.tokens_saved, outcome.tags
)
def test_no_deferral_reports_no_tool_tokens_saved(headroom_client) -> None:
data = compress(headroom_client, _body(tools=_many_anthropic_tools(), gateway={})).json()
assert data["tool_tokens_saved"] == 0
assert data["tool_tokens_deferred"] == 0
def test_hook_header_without_client_value_is_just_the_hooks(headroom_client, deferral_hook) -> None:
data = compress(headroom_client, _body(tools=_many_anthropic_tools(), gateway={})).json()
assert data["headers"] == {"anthropic-beta": BETA_TOKEN}
@@ -0,0 +1,85 @@
"""/v1/compress counts the tool-schema compaction it ships.
The gateway compacts tool schemas (``compact_gateway_tools``) and returns the
compacted array in ``body``, but the message counts never see tool bytes. The
chat handlers fold the compaction delta into both endpoints, so
``tokens_saved`` covers it (``tool_schema_savings_policy``: compaction "is
ALREADY inside tokens_saved"); this path must do the same.
"""
from __future__ import annotations
import json
from typing import Any
from headroom.tokenizers import get_tokenizer
from tests.gateway.conftest import compress
MODEL = "claude-sonnet-4-5"
def _annotated_tool(i: int) -> dict[str, Any]:
# `title` and `examples` are annotation keys schema compaction strips.
return {
"name": f"tool_{i}",
"description": "Look something up.",
"input_schema": {
"type": "object",
"title": f"Tool {i} input, a long title compaction removes",
"properties": {
"q": {
"type": "string",
"title": "Query",
"examples": ["first example query", "second example query"],
}
},
},
}
def _compress(client: Any, tools: list[dict[str, Any]]) -> dict[str, Any]:
body = {
"model": MODEL,
"messages": [{"role": "user", "content": "hi"}],
"tools": tools,
"gateway": {"can_redrive": False, "can_relay_response": False},
}
resp = compress(client, body)
assert resp.status_code == 200, resp.text
return resp.json()
def test_compaction_is_inside_tokens_saved(make_headroom_client) -> None:
client = make_headroom_client(optimize=False)
sent = [_annotated_tool(i) for i in range(3)]
data = _compress(client, sent)
assert "tool_schema_compaction" in data["transforms_applied"]
tok = get_tokenizer(MODEL)
shipped = data["body"]["tools"]
removed = tok.count_text(json.dumps(sent, default=str)) - tok.count_text(
json.dumps(shipped, default=str)
)
assert removed > 0
# optimize=False leaves the messages alone: the whole saving is the tools.
assert data["tokens_saved"] == removed
assert data["tokens_before"] - data["tokens_after"] == data["tokens_saved"]
def test_tools_compaction_leaves_alone_count_nothing(make_headroom_client) -> None:
client = make_headroom_client(optimize=False)
plain = {"name": "plain", "description": "x", "input_schema": {"type": "object"}}
data = _compress(client, [plain])
assert "tool_schema_compaction" not in data["transforms_applied"]
assert data["tokens_saved"] == 0
def test_compacting_client_deferred_tools_counts_nothing(make_headroom_client) -> None:
# A tool the client marked defer_loading is not in the billed context.
client = make_headroom_client(optimize=False)
deferred = [{**_annotated_tool(i), "defer_loading": True} for i in range(3)]
data = _compress(client, deferred)
assert "tool_schema_compaction" in data["transforms_applied"]
assert data["tokens_saved"] == 0
+1 -1
View File
@@ -346,7 +346,7 @@ Note `protect_recent` is not a substitute — it guards the newest messages, whi
Optionally relay the provider's usage for attribution with `POST /v1/usage` `{"session_id": "...", "usage": {...}}`. Anthropic `cache_read_input_tokens` / `cache_creation_input_tokens` and OpenAI `prompt_tokens_details.cached_tokens` / flat `cached_tokens` are accepted; a block with none of them is a 400. Replies `{"session_id", "frozen_message_count", "applied"}` (`applied: false, reason: "no_cache_signal"` when the only present field is 0), 404 `unknown_session`, or 503 `session_busy` (retry). Telemetry only — it never raises the frozen count.
**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.
**Gateway mode (two-half turn contract).** Add a top-level `gateway` object (`{}` is enough) and one model turn becomes two calls, so a gateway that never lets Headroom see the provider response can still run transforms whose reload step needs it — the proxy's "no shrink without reload" rule at the API boundary. `gateway` fields: `can_redrive` (default `false`: the gateway can call the provider again with a request Headroom hands it; send `false` when streaming), `can_relay_response` (default `false`: the gateway will post status/usage after each turn), `session_affinity` (default `true`; `false` disables re-driving because pending turns are in-process), `plugin_version` (diagnostics). Every other top-level field (`system`, `tools`, `temperature`, …) passes through; `tools` is compacted deterministically (`tool_schema_compaction`; `tool_desc_compaction` with `HEADROOM_TOOL_DESC_MAX_CHARS`); extension turn hooks run, stream-safe-only unless `can_redrive` and `session_affinity` both hold. The response adds `body` (the complete provider request — forward it as-is; never contains `config`, `gateway`, `token_budget`), `turn_id`, `route` (`{model, provider, service_tier, reason}`, advisory; a routing extension's model is also written into `body.model`), `obligations` (`redrive` and/or `relay_usage`) and a `gateway` echo. Fail-open answers carry the same keys with originals and `obligations: []`. Savings follow the proxy path: when compaction shortened `tools`, `tokens_before`/`tokens_after` also count the tool definitions, and the answer adds `tool_tokens_saved` (tool-search deferral plus tools a turn hook removed, both net; `tokens_saved + tool_tokens_saved` is the `/stats` headline) and `tool_tokens_deferred` (its deferral share, booked `estimated`); both are absent on fail-open answers. A turn is registered only when `obligations` is non-empty; with `relay_usage` the `/stats` record waits for the response half. Knobs: `HEADROOM_GATEWAY_TURN_TTL_SECONDS` (120), `HEADROOM_GATEWAY_MAX_PENDING_TURNS` (10000), `HEADROOM_GATEWAY_MAX_REDRIVES` (8). With `config.mode: "ccr"` and re-drive allowed, `headroom_retrieve` is injected into `body.tools` (`ccr_tool_injected`) and the response half answers the model's retrieval calls itself.
`POST /v1/compress/response` (same exposure rules as `/v1/compress`) is the response half: `{"turn_id", "status": 200, "latency_ms", "usage": {...}, "response": {...}}`. `usage` accepts Anthropic, OpenAI chat (`prompt_tokens_details.cached_tokens`), OpenAI Responses (`input_tokens_details.cached_tokens`) and Kong's flat `cached_tokens` shapes, or the whole provider body (a nested `usage` key is descended into once); billed counters are summed across re-drive rounds. `response` is required when the turn carries `redrive` and the provider call succeeded; a failed call closes the turn with its `status` alone. Answers: `{"action": "done", "turn_id", "response": <replacement, the latest provider response after a re-drive, or null — forward what you hold>, "frozen_message_count", "usage_applied", "rounds", "billed_usage"}` or `{"action": "redrive", "turn_id", "request": <full provider body to send>, "round"}` — post the provider's JSON back under the same `turn_id`; `billed_usage` sums every counter (cache reads and writes included) across rounds in Anthropic keys, and a re-driven turn's `response` is never null and reports that total as its `usage`; past `HEADROOM_GATEWAY_MAX_REDRIVES` the turn ends with `done` and the latest provider response. Errors: 400 `invalid_request` / `missing_response`, 404 `unknown_turn` (unregistered, finished, or expired), 409 `turn_busy`.