fix(proxy): net tools a turn hook adds against the saving it books (#4057)

## Description

A turn hook that folds messages but **adds** a tool definition saved the
difference, not the fold alone. headroom-skill-search does exactly this:
it rewrites the skill list in the messages and appends a `search_skills`
tool. Two places over-credited it:

- **Hook runner (`run_request_hooks`):**
  - It clamped the message and tool halves separately at zero.
- So the added tool never reduced the hook's ledger entry (`/stats`
`savings.by_source.skill_search`).
- **Anthropic and OpenAI chat handlers:**
- Tool shrinkage from a hook goes into `turn_hook_tools_saved_tokens`,
but tool growth went nowhere.
- So the headline `tokens_saved` ignored a tool that is sent on every
request.

Found while auditing the enterprise plugins for the provider-unit
savings work (#4052 / #4055).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `turn_hooks.run_request_hooks`: signed message and tool deltas,
netted. An entry is recorded only when the net is a saving.
`details.tool_tokens_saved` is negative when tools were added.
- Anthropic handler: tool growth from turn hooks (`_th_tool_growth`) is
added to `optimized_tokens` at the recount. This is the same place
compaction deltas are folded, so `tok_before - tok_after == tok_saved`
stays coherent.
- OpenAI chat handler: the same, in its post-hook recount.
- `turn_hook_tools_saved_tokens` keeps its non-negative meaning
(shrinkage only). Every surface that sums it is unchanged.

## Testing

- [x] Unit tests pass (`pytest`): related suites
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy`) on changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed (plugin suites against this core)

### Test Output

```text
tests/test_hook_net_tool_additions.py: 6 passed
  - ledger entry is net of the added tool; no entry when the tool outweighs the fold
  - Anthropic /v1/messages and OpenAI /v1/chat/completions: tokens_saved == fold - added tool
  - a later hook that grows the request more than was saved cancels the earlier credit
  - a later partial loss caps the earlier credit at the sequence net, and the
    breakdown (message + tool) sums to the capped tokens (uncapped_* kept)
tests/gateway/test_hook_tool_growth.py: /v1/compress tokens_after includes the added tool
  -> each fails on the commit before its fix (e.g. 1499 == 1499 - 117, 0 > 100,
     ('fold', 200) == ('fold', 142), (200 + 0) == 142)
gateway + turn_hook / tool_search / tool_schema / savings_attribution / extension /
anthropic / openai / responses / compress suites: 2101 passed, 34 skipped
Plugin suites against this core (PYTHONPATH): skill-search 41, tool-search 144, lossless-guard 237 passed
ruff check / format: clean; mypy (changed modules): Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `LITELLM_LOCAL_MODEL_COST_MAP=True`;
the real proxy app (`create_app`) through FastAPI TestClient, upstream
mocked with respx.
- Exact command / steps: `pytest tests/test_hook_net_tool_additions.py
tests/gateway/test_hook_tool_growth.py` (a hook shaped like
SkillSearchHook folds messages and adds a search tool, on
`/v1/messages`, `/v1/chat/completions` and `/v1/compress`), then the
skill-search / tool-search / lossless-guard suites with `PYTHONPATH` set
to this branch.
- Observed result: on every path `tokens_saved` and the `skill_search`
ledger entry equal the message fold minus the added tool's tokens;
across several hooks the total credit never exceeds the sequence's net
saving and each entry's breakdown sums to its tokens.
- Live skill-search session (2026-10-15, branch head `7a81ec9f`):
- Setup: a real proxy run from this branch with headroom-skill-search
0.2.0 (`--proxy-extension skill_search`,
`HEADROOM_SKILL_SEARCH_HINT_MODE=first_sentence`,
`HEADROOM_BEACON=off`), real Anthropic upstream, `claude-haiku-4-5`.
Each request was a non-streaming curl to `/v1/messages` with 22 skills
listed the way Claude Code lists them.
  - Turn 1: HTTP 200, body 9116 B → 2744 B.
- The `skill_search` ledger booked `message_tokens_saved=1327` and
`tool_tokens_saved=-108`; the −108 is the added `search_skills` tool.
That gives `tokens=1219`, the signed net this PR books.
- Anthropic `count_tokens` on the original body: 2415. Input billed
through the proxy: 1153–1276.
  - Headline:
- On this branch alone it read `tok_before=424 tok_after=532
tok_saved=0`. The cause is a bug on main, not in this PR:
`snapshot_original_messages` aliases the live messages when only turn
hooks are registered, and skill search folds text in place, so the
recorded original was the folded one. The fix is #4241.
- With #4241 applied: `tok_before=1751 tok_after=532 tok_saved=1219` =
1327 − 108, the same as the ledger.
- Turn 2: the model called `search_skills` and the hook served
`image-resize`. After one reload round the final answer came back HTTP
200 `end_turn`. Headline `tok_saved=1219`; no new ledger entry was
booked.
  - `/stats` `savings.by_source.skill_search`: 1 event, 1219 tokens.
- Not tested: streaming (skill search acts only on buffered turns with
at least 18 skills), and the OpenAI path live.
- Merge order: the headline matches the ledger only once #4241 is on
main. Merge #4241 first.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, accounting only (requests are
byte-identical). Headline, `/v1/compress` `tokens_saved` and
`skill_search` attribution drop by the size of the tool the hook adds,
and a later hook's growth offsets earlier hook credits.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: one extra tool-array count per hooked OpenAI
chat request.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- **Docs:** none needed; this is an internal accounting fix.
- **Stacked on #4056** (base `fix/hook-tool-deferral-accounting`). Both
PRs change the same lines in `run_request_hooks`, so this branch carries
the resolution: the signed tool delta is measured with
`without_deferral_flags`, and the OpenAI tool-growth count strips the
flag too. Merge #4056 first. The combined branch passes 844 core tests
and the skill-search, tool-search and lossless-guard suites.

🤖 Generated with [Claude Code](https://claude.com/claude-code)



## Current maintainer verification (October 8)

Exact synchronized head: `de0208ae9dbabe7967d4b781e67364f6ee8b141d`,
containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d`. The main
merge preserves the contributor's signed sequence accounting and
net-saving helper, along with main's deferral-filtered tool counters.
Parent #4056 has merged, so the PR now targets main directly. No
contributor logic was changed.

25 focused hook/gateway/deferral tests pass. Two additional isolated
reviewer probes force only the session's final message recount to fail;
positive and negative net-savings cases both pass, reaching the fallback
without breaking the hook counter. This is local TestClient/native-core
evidence, not a live skill-search session. Current-head hosted
qualification is pending.

The previously acknowledged live skill-search runtime proof remains
outstanding; the new tests close the accounting finding but do not close
that evidence requirement.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-16 14:30:00 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent df11f27498
commit 85520c2941
9 changed files with 663 additions and 15 deletions
+8
View File
@@ -89,6 +89,14 @@ class CompressTurn(Protocol):
def count_messages(self, messages: list[dict[str, Any]], fallback: int) -> int:
"""Recount with the turn's own tokenizer; ``fallback`` on failure."""
# Optional, not part of the required protocol: a turn may expose
# ``tool_growth_tokens``, the tool-definition tokens ``transform`` ADDED to
# the request (a hook's search tool, say). The handler adds it to
# ``tokens_after``, since those tokens are sent too; absent reads as 0.
# Likewise optional: ``message_growth_tokens`` (message tokens it added)
# and ``hook_tool_saved_tokens`` (tool tokens it removed and booked), so
# the handler can net the two before clamping the saving.
def finish(
self,
*,
+12
View File
@@ -141,6 +141,18 @@ class GatewayTurn:
return fallback
return self._transformer.count_messages(messages, fallback)
@property
def tool_growth_tokens(self) -> int:
return self._transformer.tool_growth_tokens if self._transformer is not None else 0
@property
def message_growth_tokens(self) -> int:
return self._transformer.message_growth_tokens if self._transformer is not None else 0
@property
def hook_tool_saved_tokens(self) -> int:
return self._transformer.hook_tool_saved_tokens if self._transformer is not None else 0
# -- event-loop side ---------------------------------------------------
def finish(
self,
+12
View File
@@ -839,6 +839,14 @@ class RequestTransformer:
self.tags = tags
self.caps = caps
self.result: RequestTransformResult | None = None
# Tool-definition tokens the request hooks ADDED (e.g. a search tool).
# The compress handler adds it to ``tokens_after``: it is sent too.
self.tool_growth_tokens = 0
# Message tokens the request hooks ADDED (only counted by the handler
# when it did not recount the messages), and tool tokens they removed
# (booked on turn_hook_tools_saved_tokens), for signed netting.
self.message_growth_tokens = 0
self.hook_tool_saved_tokens = 0
self._tokenizer: Any = None
def _get_tokenizer(self) -> Any:
@@ -909,12 +917,16 @@ class RequestTransformer:
f"{int(self.tags.get('tool_search_deferred_tokens', 0) or 0)}tok"
)
msg_after = _safe_count(count_messages, messages)
if msg_before is not None and msg_after is not None:
self.message_growth_tokens = max(0, msg_after - msg_before)
if msg_before is not None and msg_after is not None and msg_after < msg_before:
transforms.append("turn_hook")
messages_rewritten = True
tools_after = _safe_count(count_tools, without_deferral_flags(tools))
if tools_before is not None and tools_after is not None:
self.tool_growth_tokens = max(0, tools_after - tools_before)
saved = max(0, tools_before - tools_after)
self.hook_tool_saved_tokens = saved
if saved > 0:
self.tags["turn_hook_tools_saved_tokens"] = (
int(self.tags.get("turn_hook_tools_saved_tokens", 0) or 0) + saved
+17 -2
View File
@@ -21,7 +21,10 @@ from urllib.parse import urlsplit
from headroom.proxy.session_affinity import session_affinity_headers
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
from headroom.proxy.tool_schema_savings_policy import without_deferral_flags
from headroom.proxy.tool_schema_savings_policy import (
net_hook_tool_saving,
without_deferral_flags,
)
if TYPE_CHECKING:
from fastapi import Request
@@ -3393,6 +3396,11 @@ class AnthropicHandlerMixin:
)
_pre_hook_tokens: int | None = None
# Tool-definition tokens a turn hook ADDED (e.g. a search tool). Folded
# into the forwarded count below so the headline nets them; shrinkage
# stays on the turn_hook_tools_saved_tokens tag.
_th_tool_growth = 0
_th_saved = 0
_req_ctx: TurnContext | None = None
if registered_turn_hooks():
_req_ctx = TurnContext(
@@ -3435,6 +3443,7 @@ class AnthropicHandlerMixin:
else 0
)
_th_saved = max(0, _th_tok_before - _th_tok_after)
_th_tool_growth = max(0, _th_tok_after - _th_tok_before)
if _th_saved > 0:
tags["turn_hook_tools_saved_tokens"] = (
int(tags.get("turn_hook_tools_saved_tokens", 0) or 0) + _th_saved
@@ -3560,6 +3569,8 @@ class AnthropicHandlerMixin:
if 0 < _tool_tokens_after < _tool_tokens_before:
original_tokens += _tool_tokens_before
optimized_tokens += _tool_tokens_after
# A tool a turn hook added is sent too: it reduces the saving.
optimized_tokens += _th_tool_growth
# First-appearance accounting for matured Reads. The client
# re-sends the raw conversation every turn, so this diff would
# otherwise re-book a matured Read's removal on every request
@@ -3580,7 +3591,11 @@ class AnthropicHandlerMixin:
# Advisory, like the maturation pass itself: a failure
# here must not skip the recount around it.
logger.debug("maturation replay debt skipped", exc_info=True)
tokens_saved = max(0, original_tokens - optimized_tokens - _replay_debt)
# Signed, then clamped once: message text a hook added offsets
# the tool definitions it removed (booked on the tag above).
tokens_saved = net_hook_tool_saving(
original_tokens - optimized_tokens - _replay_debt, tags, _th_saved
)
# Attribute the fold to the hook ONLY when the hook itself reduced
# tokens (same-tokenizer pre vs post) — not when the recount above
# merely normalized a cross-estimator scale difference.
+40 -3
View File
@@ -45,7 +45,10 @@ from headroom.proxy.provider_usage import (
from headroom.proxy.rate_limit_identity import rate_limit_identity
from headroom.proxy.semantic_cache_key_policy import compute_request_cache_partition
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
from headroom.proxy.tool_schema_savings_policy import without_deferral_flags
from headroom.proxy.tool_schema_savings_policy import (
net_hook_tool_saving,
without_deferral_flags,
)
from headroom.proxy.upstream_guard import is_safe_upstream_url
from headroom.proxy.ws_headers import WS_HOP_BY_HOP_HEADERS
from headroom.proxy.ws_session_registry import (
@@ -4346,6 +4349,7 @@ class OpenAIHandlerMixin:
)
_th_ctx: TurnContext | None = None
_th_raw_saved = None
if registered_turn_hooks():
_th_tools_before = body.get("tools")
_th_tok_before = (
@@ -4393,7 +4397,17 @@ class OpenAIHandlerMixin:
optimized_tokens = _th_msg_after
if 0 < tool_tokens_after_compaction < tool_tokens_before_compaction:
optimized_tokens += tool_tokens_after_compaction
tokens_saved = max(0, original_tokens - optimized_tokens)
# A tool the hook added is sent too: it reduces the saving.
_th_tools_after_count = (
tokenizer.count_text(
json.dumps(without_deferral_flags(_th_ctx.tools), default=str)
)
if _th_ctx.tools
else 0
)
optimized_tokens += max(0, _th_tools_after_count - _th_tok_before)
_th_raw_saved: int | None = original_tokens - optimized_tokens
tokens_saved = max(0, _th_raw_saved)
# Attribute to the hook ONLY when the hook itself reduced tokens.
if _th_msg_before is not None and _th_msg_after < _th_msg_before:
transforms_applied.append("turn_hook")
@@ -4410,6 +4424,10 @@ class OpenAIHandlerMixin:
int(tags.get("turn_hook_tools_saved_tokens", 0) or 0) + _th_saved
)
transforms_applied.append(f"turn_hook:tools:{_th_saved}tok")
# Signed, then clamped once: message text a hook added offsets the
# tool definitions it removed.
if _th_raw_saved is not None:
tokens_saved = net_hook_tool_saving(_th_raw_saved, tags, _th_saved)
# Compatibility shim: GPT-5 / o-series chat models REJECT the legacy
# `max_tokens` ("Unsupported parameter … Use 'max_completion_tokens'
@@ -10462,6 +10480,11 @@ class OpenAIHandlerMixin:
if _turn is not None:
_turn.prepare(model_name=model_name, tags=tags, config=self.config)
# Whether the returned ``tokens_after`` was counted over the FINAL,
# post-hook messages (so it already includes any text a hook added).
# Set by whichever closure runs; read after the executor.
_after_counts_hook = [False]
def _run_stateless():
result = pipeline.apply(messages=messages, model=model, **pipeline_kwargs)
final = result.messages
@@ -10470,6 +10493,7 @@ class OpenAIHandlerMixin:
final = _turn.transform(final)
if _turn.folded_messages:
tokens_after = _turn.count_messages(final, tokens_after)
_after_counts_hook[0] = True
return (
result,
final,
@@ -10559,6 +10583,7 @@ class OpenAIHandlerMixin:
_tok = get_tokenizer(model_name)
raw_tokens_before = _tok.count_messages(messages)
final_tokens_after = _tok.count_messages(final)
_after_counts_hook[0] = True
except Exception as e:
# Fail-open, but LOUD: this fallback reverts to the
# pipeline's counts of the cache-swapped input, which
@@ -10609,7 +10634,19 @@ class OpenAIHandlerMixin:
ccr_hashes = _response_ccr_hashes(final_messages, result.markers_inserted)
tokens_saved = max(0, tokens_before - tokens_after)
# Tools and message text a turn hook added are sent too: they reduce
# the saving, and offset tools the hook removed (signed, clamped once).
_hook_tool_saved = 0
if _turn is not None:
tokens_after += max(0, int(getattr(_turn, "tool_growth_tokens", 0) or 0))
# Only a count of pre-hook messages lacks the text a hook added;
# a recount of the final messages (stateless fold, session) has it.
if not _after_counts_hook[0]:
tokens_after += max(0, int(getattr(_turn, "message_growth_tokens", 0) or 0))
_hook_tool_saved = int(getattr(_turn, "hook_tool_saved_tokens", 0) or 0)
tokens_saved = net_hook_tool_saving(
tokens_before - tokens_after, tags, _hook_tool_saved
)
latency_ms = (time.time() - start_time) * 1000
_transforms_applied = list(result.transforms_applied or ())
# The claimed turn adds its labels and response fields BEFORE the
@@ -183,6 +183,26 @@ def tool_schema_saved_from_tags(tags: object) -> int:
return total
def net_hook_tool_saving(raw_saved: int, tags: object, hook_tool_saved: int) -> int:
"""Net a request's message-side saving against the tool saving its turn
hooks booked on ``turn_hook_tools_saved_tokens``; return it clamped at 0.
The headline is ``tokens_saved`` (clamped) plus that tag. A hook that adds
message text while removing tools leaves ``raw_saved`` negative, and the
clamp alone would drop the growth while the tag kept the full removal. The
deficit is debited from the tag first (at most what the hooks booked on
this request), so headline == messages + tools, signed, clamped once.
"""
raw = int(raw_saved)
booked = max(0, int(hook_tool_saved or 0))
if raw < 0 and booked > 0 and isinstance(tags, dict):
offset = min(-raw, booked)
current = int(tags.get("turn_hook_tools_saved_tokens", 0) or 0)
tags["turn_hook_tools_saved_tokens"] = max(0, current - offset)
raw += offset
return max(0, raw)
def headline_tokens_saved(tokens_saved: object, tags: object) -> int:
"""Return the single "Tokens saved" figure for one request.
+42 -10
View File
@@ -207,6 +207,11 @@ def run_request_hooks(ctx: TurnContext, *, stream_safe_only: bool = False) -> No
# Deferral booked before or by a hook; reconciled once every hook has run,
# since a later hook may un-defer booked tools (tool search's hot tools).
booking = deferred_booking(ctx.tags, ctx.tools)
# Positive nets wait until every hook has run: a later hook that grows the
# request (adds a tool) offsets earlier savings, so the ledger never credits
# more than the whole sequence actually removed.
gains: list[tuple[str, int, dict[str, int]]] = []
sequence_net = 0
for hook in registered_turn_hooks():
if stream_safe_only and not getattr(hook, "stream_safe", False):
continue
@@ -221,24 +226,28 @@ def run_request_hooks(ctx: TurnContext, *, stream_safe_only: bool = False) -> No
if ctx.count_tools is not None:
before_tools = ctx.count_tools(without_deferral_flags(ctx.tools))
fn(ctx)
# Signed deltas, netted: a hook that folds messages but ADDS a tool
# (skill search's ``search_skills``) saved the difference, not the
# fold alone. Booked only when the net is a saving.
message_saved = (
max(0, before_messages - ctx.count_messages(ctx.messages))
before_messages - ctx.count_messages(ctx.messages)
if before_messages is not None and ctx.count_messages is not None
else 0
)
tool_saved = (
max(0, before_tools - ctx.count_tools(without_deferral_flags(ctx.tools)))
before_tools - ctx.count_tools(without_deferral_flags(ctx.tools))
if before_tools is not None and ctx.count_tools is not None
else 0
)
if message_saved or tool_saved:
ctx.record_savings(
getattr(hook, "savings_source", getattr(hook, "name", type(hook).__name__)),
tokens=message_saved + tool_saved,
details={
"message_tokens_saved": message_saved,
"tool_tokens_saved": tool_saved,
},
net_saved = message_saved + tool_saved
sequence_net += net_saved
if net_saved > 0:
gains.append(
(
getattr(hook, "savings_source", getattr(hook, "name", type(hook).__name__)),
net_saved,
{"message_tokens_saved": message_saved, "tool_tokens_saved": tool_saved},
)
)
except Exception: # a hook must never break the proxy
log.exception("turn hook %r on_request failed", getattr(hook, "name", hook))
@@ -246,6 +255,29 @@ def run_request_hooks(ctx: TurnContext, *, stream_safe_only: bool = False) -> No
# This hook booked (or re-booked) the deferral: it is the new baseline.
booking = deferred_booking(ctx.tags, ctx.tools)
reconcile_deferred_tokens(ctx.tags, booking, ctx.tools, ctx.count_tools)
# Book gains in run order up to the sequence's net saving; a loss that
# exceeds them all books nothing (never a negative entry).
budget = max(0, sequence_net)
for source, net_saved, details in gains:
tokens = min(net_saved, budget)
if tokens <= 0:
break
budget -= tokens
if tokens < net_saved:
# Later hooks' cost comes off this credit; share it across the
# message and tool parts so the breakdown still sums to ``tokens``.
# The uncapped deltas stay for diagnosis.
message_part = round(details["message_tokens_saved"] * tokens / net_saved)
details = {
"message_tokens_saved": message_part,
"tool_tokens_saved": tokens - message_part,
"uncapped_message_tokens_saved": details["message_tokens_saved"],
"uncapped_tool_tokens_saved": details["tool_tokens_saved"],
}
try:
ctx.record_savings(source, tokens=tokens, details=details)
except Exception:
log.exception("turn hook savings for %r not recorded", source)
async def run_response_hooks(
+124
View File
@@ -0,0 +1,124 @@
"""/v1/compress nets tool definitions a turn hook adds against its saving.
A hook that folds messages but adds a tool (skill search's search tool) sends
those tool tokens too; ``tokens_after`` must include them, as on the chat path.
"""
from __future__ import annotations
import json
from typing import Any
import pytest
from headroom.proxy.savings_attribution import from_tags
from headroom.proxy.turn_hooks import register_turn_hook
from tests.gateway.conftest import compress
_LONG = "line of tool output that a hook can fold away entirely " * 60
_ADDED_TOOL = {
"name": "search_skills",
"description": "Find a skill by what it does. " * 40,
"input_schema": {"type": "object", "properties": {"q": {"type": "string"}}},
}
class _FoldAndAddTool:
name = savings_source = "skill_search"
stream_safe = True
def __init__(self, add_tool: bool) -> None:
self.add_tool = add_tool
def on_request(self, ctx: Any) -> None:
ctx.messages = [{**m, "content": "folded"} for m in ctx.messages]
if self.add_tool:
ctx.tools = [*(ctx.tools or []), dict(_ADDED_TOOL)]
def _run(make_headroom_client: Any, *, add_tool: bool) -> dict[str, Any]:
register_turn_hook(_FoldAndAddTool(add_tool))
client = make_headroom_client(optimize=False)
body = {
"model": "claude-sonnet-4-5",
"messages": [{"role": "user", "content": _LONG}],
"tools": [{"name": "noop", "description": "x", "input_schema": {"type": "object"}}],
"gateway": {"can_redrive": False, "can_relay_response": False},
}
resp = compress(client, body)
assert resp.status_code == 200, resp.text
return resp.json()
def test_tools_a_hook_adds_count_as_sent(make_headroom_client) -> None:
from headroom.proxy.turn_hooks import clear_turn_hooks
plain = _run(make_headroom_client, add_tool=False)
clear_turn_hooks()
grown = _run(make_headroom_client, add_tool=True)
assert plain["tokens_saved"] > 0
added = grown["tokens_after"] - plain["tokens_after"]
assert added > 100 # the search tool's definition
assert grown["tokens_saved"] == plain["tokens_saved"] - added
assert grown["tokens_before"] == plain["tokens_before"]
class _AddTextRemoveTools:
name = savings_source = "rewriter"
stream_safe = True
def __init__(self, repeat: int) -> None:
self.repeat = repeat
def on_request(self, ctx: Any) -> None:
ctx.messages = [*ctx.messages, {"role": "user", "content": "steer. " * self.repeat}]
ctx.tools = []
@pytest.mark.parametrize("session", [False, True], ids=["stateless", "session"])
@pytest.mark.parametrize("repeat", [5, 2_000], ids=["net-positive", "net-negative"])
def test_message_growth_offsets_tools_a_hook_removed(
make_headroom_client, outcome_spy, repeat, session
) -> None:
"""The opposite signed case on /v1/compress: a hook appends message text
and drops the tools. The headline is removed tools minus added text,
clamped once, not the full tool removal."""
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
from headroom.tokenizers import get_tokenizer
register_turn_hook(_AddTextRemoveTools(repeat))
client = make_headroom_client(optimize=False)
outcomes = outcome_spy(client)
before = [{"role": "user", "content": "hi"}]
body = {
"model": "claude-sonnet-4-5",
"messages": before,
"tools": [dict(_ADDED_TOOL)],
"gateway": {"can_redrive": False, "can_relay_response": False},
}
if session:
# Session turns recount the final (post-hook) messages: the added
# text must not be counted a second time on top of that.
body["config"] = {"session_id": f"growth-{repeat}"}
resp = compress(client, body)
assert resp.status_code == 200, resp.text
tok = get_tokenizer("claude-sonnet-4-5")
growth = tok.count_messages(
[*before, {"role": "user", "content": "steer. " * repeat}]
) - tok.count_messages(before)
removed = tok.count_text(json.dumps([_ADDED_TOOL], default=str))
net = removed - growth
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
headline = headline_tokens_saved(outcome.tokens_saved, outcome.tags)
# The gateway compacts tool schemas before hooks run, so compare with the
# hook's own measured deltas (the runner's ledger), not the raw schema.
entries = from_tags(outcome.tags)
if net > 0:
(entry,) = entries
hook_net = entry["details"]["tool_tokens_saved"] + entry["details"]["message_tokens_saved"]
assert entry["details"]["message_tokens_saved"] < 0
assert headline == hook_net == entry["tokens"]
else:
assert entries == []
assert headline == 0
+388
View File
@@ -0,0 +1,388 @@
"""A turn hook that folds messages but ADDS a tool saved the difference.
headroom-skill-search rewrites the skill list in the messages and appends a
``search_skills`` tool. The runner used to clamp the message and tool halves
separately, so the added tool never reduced the booked saving, and the
handlers left it out of the forwarded count, so the headline overstated too.
"""
from __future__ import annotations
import json
from typing import Any
import httpx
import pytest
import respx
from headroom.proxy.savings_attribution import from_tags
from headroom.proxy.turn_hooks import (
TurnContext,
clear_turn_hooks,
register_turn_hook,
run_request_hooks,
)
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.loopback_guard import require_loopback # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
from headroom.tokenizers import get_tokenizer # noqa: E402
SEARCH_TOOL = {
"name": "search_skills",
"description": "Search the skill catalogue by keyword. " * 10,
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}}},
}
@pytest.fixture(autouse=True)
def _clean_registry():
clear_turn_hooks()
yield
clear_turn_hooks()
class _FoldAndAddTool:
"""Shaped like SkillSearchHook: shrink a message, append a search tool."""
name = savings_source = "skill_search"
stream_safe = True
def __init__(self, folded_text: str) -> None:
self.folded_text = folded_text
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [{"role": "user", "content": self.folded_text}]
ctx.tools = [*(ctx.tools or []), SEARCH_TOOL]
def _count(value: Any) -> int:
return len(json.dumps(value, default=str)) // 4 if value else 0
def _run(hook: Any, text: str) -> TurnContext:
register_turn_hook(hook)
ctx = TurnContext(
provider="anthropic",
model="claude-sonnet-4-6",
messages=[{"role": "user", "content": text}],
tools=[],
count_messages=_count,
count_tools=_count,
)
run_request_hooks(ctx)
return ctx
def test_ledger_entry_is_net_of_the_added_tool() -> None:
long_text = "skill line\n" * 400
ctx = _run(_FoldAndAddTool("short"), long_text)
(entry,) = from_tags(ctx.tags)
folded = _count([{"role": "user", "content": long_text}]) - _count(
[{"role": "user", "content": "short"}]
)
added = _count([SEARCH_TOOL])
assert entry["tokens"] == folded - added
assert entry["details"]["tool_tokens_saved"] == -added
def test_no_entry_when_the_added_tool_outweighs_the_fold() -> None:
ctx = _run(_FoldAndAddTool("short"), "a bit longer than short")
assert from_tags(ctx.tags) == []
@respx.mock
def test_anthropic_headline_nets_a_tool_the_hook_added(monkeypatch) -> None:
long_text = "skill line number one\n" * 300
register_turn_hook(_FoldAndAddTool("short"))
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw): # noqa: ANN001, ANN002, ANN003, ANN202
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-5",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 100, "output_tokens": 1},
},
)
)
with TestClient(app) as client:
result = client.post(
"/v1/messages",
json={
"model": "claude-sonnet-4-5",
"max_tokens": 16,
"messages": [{"role": "user", "content": long_text}],
},
headers={"x-api-key": "sk-ant-test", "anthropic-version": "2023-06-01"},
)
assert result.status_code == 200
tok = get_tokenizer("claude-sonnet-4-5")
folded = tok.count_messages([{"role": "user", "content": long_text}]) - tok.count_messages(
[{"role": "user", "content": "short"}]
)
added = tok.count_text(json.dumps([SEARCH_TOOL], default=str))
assert folded > added > 0
assert outcomes[-1].tokens_saved == folded - added
class _FoldAndAddFunction(_FoldAndAddTool):
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [{"role": "user", "content": self.folded_text}]
ctx.tools = [*(ctx.tools or []), {"type": "function", "function": SEARCH_TOOL}]
@respx.mock
def test_openai_chat_headline_nets_a_tool_the_hook_added(monkeypatch) -> None:
long_text = "skill line number one\n" * 300
register_turn_hook(_FoldAndAddFunction("short"))
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw): # noqa: ANN001, ANN002, ANN003, ANN202
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
respx.post("https://api.openai.com/v1/chat/completions").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"choices": [
{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 100, "completion_tokens": 1},
},
)
)
with TestClient(app) as client:
result = client.post(
"/v1/chat/completions",
json={"model": "gpt-4o", "messages": [{"role": "user", "content": long_text}]},
headers={"authorization": "Bearer sk-test"},
)
assert result.status_code == 200
tok = get_tokenizer("gpt-4o")
folded = tok.count_messages([{"role": "user", "content": long_text}]) - tok.count_messages(
[{"role": "user", "content": "short"}]
)
added = tok.count_text(json.dumps([{"type": "function", "function": SEARCH_TOOL}], default=str))
assert folded > added > 0
assert outcomes[-1].tokens_saved == folded - added
# ── a later hook's loss offsets an earlier hook's gain ───────────────────────
class _Fold:
name = savings_source = "fold"
stream_safe = True
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [{"role": "user", "content": "x"}]
class _AddBigTool:
name = savings_source = "adder"
stream_safe = True
def __init__(self, size: int) -> None:
self.size = size
def on_request(self, ctx: TurnContext) -> None:
ctx.tools = [*(ctx.tools or []), {"name": "t", "description": "d" * self.size}]
def _sequence(*hooks: object) -> tuple[list[dict], int]:
for hook in hooks:
register_turn_hook(hook)
count = lambda v: len(json.dumps(v, default=str)) // 4 if v else 0 # noqa: E731
messages = [{"role": "user", "content": "m" * 800}]
ctx = TurnContext(
provider="anthropic",
model="claude-sonnet-4-6",
messages=messages,
tools=[],
count_messages=count,
count_tools=count,
)
before = count(messages)
run_request_hooks(ctx)
net = before - count(ctx.messages) - count(ctx.tools)
return from_tags(ctx.tags), net
def test_a_later_hook_that_grows_more_than_was_saved_cancels_the_credit() -> None:
entries, net = _sequence(_Fold(), _AddBigTool(2_000))
assert net < 0
assert entries == []
def test_a_later_partial_loss_caps_the_earlier_credit_at_the_net() -> None:
entries, net = _sequence(_Fold(), _AddBigTool(200))
assert net > 0
assert [(e["source"], e["tokens"]) for e in entries] == [("fold", net)]
(entry,) = entries
details = entry["details"]
# The breakdown sums to the credited amount; the raw deltas are kept.
assert details["message_tokens_saved"] + details["tool_tokens_saved"] == net
assert details["uncapped_message_tokens_saved"] > net
assert details["uncapped_tool_tokens_saved"] == 0
# ── message growth offsets tool removal in the handler headline ──────────────
BIG_TOOL = {
"name": "big_tool",
"description": "Does many things in many ways. " * 40,
"input_schema": {"type": "object"},
}
class _AddTextRemoveTools:
"""The opposite signed case: append message text, drop every tool."""
name = savings_source = "rewriter"
stream_safe = True
def __init__(self, added_text: str) -> None:
self.added_text = added_text
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [*ctx.messages, {"role": "user", "content": self.added_text}]
ctx.tools = []
def _app_with_spy(monkeypatch) -> tuple[Any, list[Any]]:
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw): # noqa: ANN001, ANN002, ANN003, ANN202
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
return app, outcomes
def _expected(model: str, before: list, added_text: str, tools: list) -> int:
tok = get_tokenizer(model)
grown = tok.count_messages([*before, {"role": "user", "content": added_text}])
message_growth = grown - tok.count_messages(before)
removed = tok.count_text(json.dumps(tools, default=str))
return removed - message_growth
@pytest.mark.parametrize("repeat", [5, 400], ids=["net-positive", "net-negative"])
@respx.mock
def test_anthropic_headline_nets_added_text_against_removed_tools(monkeypatch, repeat) -> None:
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
added = "extra steering text. " * repeat
register_turn_hook(_AddTextRemoveTools(added))
app, outcomes = _app_with_spy(monkeypatch)
respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-5",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 100, "output_tokens": 1},
},
)
)
before = [{"role": "user", "content": "hi"}]
with TestClient(app) as client:
result = client.post(
"/v1/messages",
json={
"model": "claude-sonnet-4-5",
"max_tokens": 16,
"messages": before,
"tools": [BIG_TOOL],
},
headers={"x-api-key": "sk-ant-test", "anthropic-version": "2023-06-01"},
)
assert result.status_code == 200
net = _expected("claude-sonnet-4-5", before, added, [BIG_TOOL])
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
assert headline_tokens_saved(outcome.tokens_saved, outcome.tags) == max(0, net)
@pytest.mark.parametrize("repeat", [5, 400], ids=["net-positive", "net-negative"])
@respx.mock
def test_openai_chat_headline_nets_added_text_against_removed_tools(monkeypatch, repeat) -> None:
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
added = "extra steering text. " * repeat
register_turn_hook(_AddTextRemoveTools(added))
app, outcomes = _app_with_spy(monkeypatch)
respx.post("https://api.openai.com/v1/chat/completions").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"choices": [
{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 100, "completion_tokens": 1},
},
)
)
tools = [{"type": "function", "function": BIG_TOOL}]
before = [{"role": "user", "content": "hi"}]
with TestClient(app) as client:
result = client.post(
"/v1/chat/completions",
json={"model": "gpt-4o", "messages": before, "tools": tools},
headers={"authorization": "Bearer sk-test"},
)
assert result.status_code == 200
net = _expected("gpt-4o", before, added, tools)
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
assert headline_tokens_saved(outcome.tokens_saved, outcome.tags) == max(0, net)