Files
headroom/tests/test_hook_net_tool_additions.py
85520c2941 fix(proxy): net tools a turn hook adds against the saving it books (#4057)
## Description

A turn hook that folds messages but **adds** a tool definition saved the
difference, not the fold alone. headroom-skill-search does exactly this:
it rewrites the skill list in the messages and appends a `search_skills`
tool. Two places over-credited it:

- **Hook runner (`run_request_hooks`):**
  - It clamped the message and tool halves separately at zero.
- So the added tool never reduced the hook's ledger entry (`/stats`
`savings.by_source.skill_search`).
- **Anthropic and OpenAI chat handlers:**
- Tool shrinkage from a hook goes into `turn_hook_tools_saved_tokens`,
but tool growth went nowhere.
- So the headline `tokens_saved` ignored a tool that is sent on every
request.

Found while auditing the enterprise plugins for the provider-unit
savings work (#4052 / #4055).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `turn_hooks.run_request_hooks`: signed message and tool deltas,
netted. An entry is recorded only when the net is a saving.
`details.tool_tokens_saved` is negative when tools were added.
- Anthropic handler: tool growth from turn hooks (`_th_tool_growth`) is
added to `optimized_tokens` at the recount. This is the same place
compaction deltas are folded, so `tok_before - tok_after == tok_saved`
stays coherent.
- OpenAI chat handler: the same, in its post-hook recount.
- `turn_hook_tools_saved_tokens` keeps its non-negative meaning
(shrinkage only). Every surface that sums it is unchanged.

## Testing

- [x] Unit tests pass (`pytest`): related suites
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy`) on changed modules
- [x] New tests added for new functionality
- [x] Manual testing performed (plugin suites against this core)

### Test Output

```text
tests/test_hook_net_tool_additions.py: 6 passed
  - ledger entry is net of the added tool; no entry when the tool outweighs the fold
  - Anthropic /v1/messages and OpenAI /v1/chat/completions: tokens_saved == fold - added tool
  - a later hook that grows the request more than was saved cancels the earlier credit
  - a later partial loss caps the earlier credit at the sequence net, and the
    breakdown (message + tool) sums to the capped tokens (uncapped_* kept)
tests/gateway/test_hook_tool_growth.py: /v1/compress tokens_after includes the added tool
  -> each fails on the commit before its fix (e.g. 1499 == 1499 - 117, 0 > 100,
     ('fold', 200) == ('fold', 142), (200 + 0) == 142)
gateway + turn_hook / tool_search / tool_schema / savings_attribution / extension /
anthropic / openai / responses / compress suites: 2101 passed, 34 skipped
Plugin suites against this core (PYTHONPATH): skill-search 41, tool-search 144, lossless-guard 237 passed
ruff check / format: clean; mypy (changed modules): Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `LITELLM_LOCAL_MODEL_COST_MAP=True`;
the real proxy app (`create_app`) through FastAPI TestClient, upstream
mocked with respx.
- Exact command / steps: `pytest tests/test_hook_net_tool_additions.py
tests/gateway/test_hook_tool_growth.py` (a hook shaped like
SkillSearchHook folds messages and adds a search tool, on
`/v1/messages`, `/v1/chat/completions` and `/v1/compress`), then the
skill-search / tool-search / lossless-guard suites with `PYTHONPATH` set
to this branch.
- Observed result: on every path `tokens_saved` and the `skill_search`
ledger entry equal the message fold minus the added tool's tokens;
across several hooks the total credit never exceeds the sequence's net
saving and each entry's breakdown sums to its tokens.
- Live skill-search session (2026-10-15, branch head `7a81ec9f`):
- Setup: a real proxy run from this branch with headroom-skill-search
0.2.0 (`--proxy-extension skill_search`,
`HEADROOM_SKILL_SEARCH_HINT_MODE=first_sentence`,
`HEADROOM_BEACON=off`), real Anthropic upstream, `claude-haiku-4-5`.
Each request was a non-streaming curl to `/v1/messages` with 22 skills
listed the way Claude Code lists them.
  - Turn 1: HTTP 200, body 9116 B → 2744 B.
- The `skill_search` ledger booked `message_tokens_saved=1327` and
`tool_tokens_saved=-108`; the −108 is the added `search_skills` tool.
That gives `tokens=1219`, the signed net this PR books.
- Anthropic `count_tokens` on the original body: 2415. Input billed
through the proxy: 1153–1276.
  - Headline:
- On this branch alone it read `tok_before=424 tok_after=532
tok_saved=0`. The cause is a bug on main, not in this PR:
`snapshot_original_messages` aliases the live messages when only turn
hooks are registered, and skill search folds text in place, so the
recorded original was the folded one. The fix is #4241.
- With #4241 applied: `tok_before=1751 tok_after=532 tok_saved=1219` =
1327 − 108, the same as the ledger.
- Turn 2: the model called `search_skills` and the hook served
`image-resize`. After one reload round the final answer came back HTTP
200 `end_turn`. Headline `tok_saved=1219`; no new ledger entry was
booked.
  - `/stats` `savings.by_source.skill_search`: 1 event, 1219 tokens.
- Not tested: streaming (skill search acts only on buffered turns with
at least 18 skills), and the OpenAI path live.
- Merge order: the headline matches the ledger only once #4241 is on
main. Merge #4241 first.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, accounting only (requests are
byte-identical). Headline, `/v1/compress` `tokens_saved` and
`skill_search` attribution drop by the size of the tool the hook adds,
and a later hook's growth offsets earlier hook credits.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: one extra tool-array count per hooked OpenAI
chat request.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- **Docs:** none needed; this is an internal accounting fix.
- **Stacked on #4056** (base `fix/hook-tool-deferral-accounting`). Both
PRs change the same lines in `run_request_hooks`, so this branch carries
the resolution: the signed tool delta is measured with
`without_deferral_flags`, and the OpenAI tool-growth count strips the
flag too. Merge #4056 first. The combined branch passes 844 core tests
and the skill-search, tool-search and lossless-guard suites.

🤖 Generated with [Claude Code](https://claude.com/claude-code)



## Current maintainer verification (October 8)

Exact synchronized head: `de0208ae9dbabe7967d4b781e67364f6ee8b141d`,
containing main `01d9194854233ef6ed04005cf0a5b8c32bf65d6d`. The main
merge preserves the contributor's signed sequence accounting and
net-saving helper, along with main's deferral-filtered tool counters.
Parent #4056 has merged, so the PR now targets main directly. No
contributor logic was changed.

25 focused hook/gateway/deferral tests pass. Two additional isolated
reviewer probes force only the session's final message recount to fail;
positive and negative net-savings cases both pass, reaching the fallback
without breaking the hook counter. This is local TestClient/native-core
evidence, not a live skill-search session. Current-head hosted
qualification is pending.

The previously acknowledged live skill-search runtime proof remains
outstanding; the new tests close the accounting finding but do not close
that evidence requirement.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-16 14:30:00 -07:00

389 lines
13 KiB
Python

"""A turn hook that folds messages but ADDS a tool saved the difference.
headroom-skill-search rewrites the skill list in the messages and appends a
``search_skills`` tool. The runner used to clamp the message and tool halves
separately, so the added tool never reduced the booked saving, and the
handlers left it out of the forwarded count, so the headline overstated too.
"""
from __future__ import annotations
import json
from typing import Any
import httpx
import pytest
import respx
from headroom.proxy.savings_attribution import from_tags
from headroom.proxy.turn_hooks import (
TurnContext,
clear_turn_hooks,
register_turn_hook,
run_request_hooks,
)
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.loopback_guard import require_loopback # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
from headroom.tokenizers import get_tokenizer # noqa: E402
SEARCH_TOOL = {
"name": "search_skills",
"description": "Search the skill catalogue by keyword. " * 10,
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}}},
}
@pytest.fixture(autouse=True)
def _clean_registry():
clear_turn_hooks()
yield
clear_turn_hooks()
class _FoldAndAddTool:
"""Shaped like SkillSearchHook: shrink a message, append a search tool."""
name = savings_source = "skill_search"
stream_safe = True
def __init__(self, folded_text: str) -> None:
self.folded_text = folded_text
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [{"role": "user", "content": self.folded_text}]
ctx.tools = [*(ctx.tools or []), SEARCH_TOOL]
def _count(value: Any) -> int:
return len(json.dumps(value, default=str)) // 4 if value else 0
def _run(hook: Any, text: str) -> TurnContext:
register_turn_hook(hook)
ctx = TurnContext(
provider="anthropic",
model="claude-sonnet-4-6",
messages=[{"role": "user", "content": text}],
tools=[],
count_messages=_count,
count_tools=_count,
)
run_request_hooks(ctx)
return ctx
def test_ledger_entry_is_net_of_the_added_tool() -> None:
long_text = "skill line\n" * 400
ctx = _run(_FoldAndAddTool("short"), long_text)
(entry,) = from_tags(ctx.tags)
folded = _count([{"role": "user", "content": long_text}]) - _count(
[{"role": "user", "content": "short"}]
)
added = _count([SEARCH_TOOL])
assert entry["tokens"] == folded - added
assert entry["details"]["tool_tokens_saved"] == -added
def test_no_entry_when_the_added_tool_outweighs_the_fold() -> None:
ctx = _run(_FoldAndAddTool("short"), "a bit longer than short")
assert from_tags(ctx.tags) == []
@respx.mock
def test_anthropic_headline_nets_a_tool_the_hook_added(monkeypatch) -> None:
long_text = "skill line number one\n" * 300
register_turn_hook(_FoldAndAddTool("short"))
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw): # noqa: ANN001, ANN002, ANN003, ANN202
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-5",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 100, "output_tokens": 1},
},
)
)
with TestClient(app) as client:
result = client.post(
"/v1/messages",
json={
"model": "claude-sonnet-4-5",
"max_tokens": 16,
"messages": [{"role": "user", "content": long_text}],
},
headers={"x-api-key": "sk-ant-test", "anthropic-version": "2023-06-01"},
)
assert result.status_code == 200
tok = get_tokenizer("claude-sonnet-4-5")
folded = tok.count_messages([{"role": "user", "content": long_text}]) - tok.count_messages(
[{"role": "user", "content": "short"}]
)
added = tok.count_text(json.dumps([SEARCH_TOOL], default=str))
assert folded > added > 0
assert outcomes[-1].tokens_saved == folded - added
class _FoldAndAddFunction(_FoldAndAddTool):
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [{"role": "user", "content": self.folded_text}]
ctx.tools = [*(ctx.tools or []), {"type": "function", "function": SEARCH_TOOL}]
@respx.mock
def test_openai_chat_headline_nets_a_tool_the_hook_added(monkeypatch) -> None:
long_text = "skill line number one\n" * 300
register_turn_hook(_FoldAndAddFunction("short"))
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw): # noqa: ANN001, ANN002, ANN003, ANN202
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
respx.post("https://api.openai.com/v1/chat/completions").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"choices": [
{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 100, "completion_tokens": 1},
},
)
)
with TestClient(app) as client:
result = client.post(
"/v1/chat/completions",
json={"model": "gpt-4o", "messages": [{"role": "user", "content": long_text}]},
headers={"authorization": "Bearer sk-test"},
)
assert result.status_code == 200
tok = get_tokenizer("gpt-4o")
folded = tok.count_messages([{"role": "user", "content": long_text}]) - tok.count_messages(
[{"role": "user", "content": "short"}]
)
added = tok.count_text(json.dumps([{"type": "function", "function": SEARCH_TOOL}], default=str))
assert folded > added > 0
assert outcomes[-1].tokens_saved == folded - added
# ── a later hook's loss offsets an earlier hook's gain ───────────────────────
class _Fold:
name = savings_source = "fold"
stream_safe = True
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [{"role": "user", "content": "x"}]
class _AddBigTool:
name = savings_source = "adder"
stream_safe = True
def __init__(self, size: int) -> None:
self.size = size
def on_request(self, ctx: TurnContext) -> None:
ctx.tools = [*(ctx.tools or []), {"name": "t", "description": "d" * self.size}]
def _sequence(*hooks: object) -> tuple[list[dict], int]:
for hook in hooks:
register_turn_hook(hook)
count = lambda v: len(json.dumps(v, default=str)) // 4 if v else 0 # noqa: E731
messages = [{"role": "user", "content": "m" * 800}]
ctx = TurnContext(
provider="anthropic",
model="claude-sonnet-4-6",
messages=messages,
tools=[],
count_messages=count,
count_tools=count,
)
before = count(messages)
run_request_hooks(ctx)
net = before - count(ctx.messages) - count(ctx.tools)
return from_tags(ctx.tags), net
def test_a_later_hook_that_grows_more_than_was_saved_cancels_the_credit() -> None:
entries, net = _sequence(_Fold(), _AddBigTool(2_000))
assert net < 0
assert entries == []
def test_a_later_partial_loss_caps_the_earlier_credit_at_the_net() -> None:
entries, net = _sequence(_Fold(), _AddBigTool(200))
assert net > 0
assert [(e["source"], e["tokens"]) for e in entries] == [("fold", net)]
(entry,) = entries
details = entry["details"]
# The breakdown sums to the credited amount; the raw deltas are kept.
assert details["message_tokens_saved"] + details["tool_tokens_saved"] == net
assert details["uncapped_message_tokens_saved"] > net
assert details["uncapped_tool_tokens_saved"] == 0
# ── message growth offsets tool removal in the handler headline ──────────────
BIG_TOOL = {
"name": "big_tool",
"description": "Does many things in many ways. " * 40,
"input_schema": {"type": "object"},
}
class _AddTextRemoveTools:
"""The opposite signed case: append message text, drop every tool."""
name = savings_source = "rewriter"
stream_safe = True
def __init__(self, added_text: str) -> None:
self.added_text = added_text
def on_request(self, ctx: TurnContext) -> None:
ctx.messages = [*ctx.messages, {"role": "user", "content": self.added_text}]
ctx.tools = []
def _app_with_spy(monkeypatch) -> tuple[Any, list[Any]]:
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
outcomes: list[Any] = []
async def _spy(_self, outcome, *a, **kw): # noqa: ANN001, ANN002, ANN003, ANN202
outcomes.append(outcome)
monkeypatch.setattr(type(app.state.proxy), "_record_request_outcome", _spy, raising=True)
return app, outcomes
def _expected(model: str, before: list, added_text: str, tools: list) -> int:
tok = get_tokenizer(model)
grown = tok.count_messages([*before, {"role": "user", "content": added_text}])
message_growth = grown - tok.count_messages(before)
removed = tok.count_text(json.dumps(tools, default=str))
return removed - message_growth
@pytest.mark.parametrize("repeat", [5, 400], ids=["net-positive", "net-negative"])
@respx.mock
def test_anthropic_headline_nets_added_text_against_removed_tools(monkeypatch, repeat) -> None:
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
added = "extra steering text. " * repeat
register_turn_hook(_AddTextRemoveTools(added))
app, outcomes = _app_with_spy(monkeypatch)
respx.post("https://api.anthropic.com/v1/messages").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-5",
"content": [{"type": "text", "text": "ok"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 100, "output_tokens": 1},
},
)
)
before = [{"role": "user", "content": "hi"}]
with TestClient(app) as client:
result = client.post(
"/v1/messages",
json={
"model": "claude-sonnet-4-5",
"max_tokens": 16,
"messages": before,
"tools": [BIG_TOOL],
},
headers={"x-api-key": "sk-ant-test", "anthropic-version": "2023-06-01"},
)
assert result.status_code == 200
net = _expected("claude-sonnet-4-5", before, added, [BIG_TOOL])
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
assert headline_tokens_saved(outcome.tokens_saved, outcome.tags) == max(0, net)
@pytest.mark.parametrize("repeat", [5, 400], ids=["net-positive", "net-negative"])
@respx.mock
def test_openai_chat_headline_nets_added_text_against_removed_tools(monkeypatch, repeat) -> None:
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
added = "extra steering text. " * repeat
register_turn_hook(_AddTextRemoveTools(added))
app, outcomes = _app_with_spy(monkeypatch)
respx.post("https://api.openai.com/v1/chat/completions").mock(
return_value=httpx.Response(
200,
json={
"id": "a",
"choices": [
{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}
],
"usage": {"prompt_tokens": 100, "completion_tokens": 1},
},
)
)
tools = [{"type": "function", "function": BIG_TOOL}]
before = [{"role": "user", "content": "hi"}]
with TestClient(app) as client:
result = client.post(
"/v1/chat/completions",
json={"model": "gpt-4o", "messages": before, "tools": tools},
headers={"authorization": "Bearer sk-test"},
)
assert result.status_code == 200
net = _expected("gpt-4o", before, added, tools)
assert (net > 0) == (repeat == 5)
outcome = outcomes[-1]
assert headline_tokens_saved(outcome.tokens_saved, outcome.tags) == max(0, net)