mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description Gateway-mode `/v1/compress` now reports the same savings as the proxy path. The why is in #4466. Closes #4466 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made Two commits, one per gap: 1. **`fix(proxy): count tool-schema compaction ...`** - The gateway turn counts the tool definitions before and after compaction. - The handler adds both counts to `tokens_before` / `tokens_after`, with the proxy path's guard (`0 < after < before`). - Tools the client already marked `defer_loading` are skipped. 2. **`feat(proxy): report tool_tokens_saved and tool_tokens_deferred ...`** - Gateway-mode answers add `tool_tokens_saved` (from the tags `/stats` sums) and `tool_tokens_deferred` (the estimated share). - Docs: `proxy.mdx`, `wiki/proxy.md`. Non-gateway and fail-open answers do not change. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`, on the changed files) - [x] Type checking passes (`mypy`, on the changed files) - [x] New tests added for new functionality ```text $ pytest -q tests/gateway/ tests/test_gateway_turn_unit.py tests/test_proxy_compress_endpoint.py 380 passed, 5 skipped ``` The tests also pass on the first commit alone (`tests/gateway/`: 280 passed). The new test for client-deferred tools fails without its fix. ## Real Behavior Proof - Environment: macOS, Python 3.12. Before: image `headroom:0.41.0-nonroot`. After: this branch, `HEADROOM_OFFLINE=1 headroom proxy`. - Exact command / steps: start each build, then send it the same gateway-mode `POST /v1/compress` — one message plus 5 tools whose schemas carry `title` / `examples` / `default` (the annotation keys schema compaction strips), with `"gateway": {"can_redrive": false, "can_relay_response": false}` so the request is a single call. Read `tokens_before`, `tokens_after`, `tokens_saved` and the `tool_*` fields out of the JSON answer. No provider is contacted. - Observed result: same request, same tools. On 0.41.0 the tool definitions are invisible to the counts, so `tokens_saved` is 0 and neither `tool_*` field is present. On this branch the compaction is counted, so `tokens_before` / `tokens_after` carry the tool bytes and `tokens_saved` is the 240 tokens compaction stripped (`tool_tokens_saved` / `tool_tokens_deferred` stay 0 here: no tool-search extension ran, and the extension is what books deferral). ```text 0.41.0: tokens_before 13, tokens_after 13, tokens_saved 0 (no tool_* fields) branch: tokens_before 830, tokens_after 590, tokens_saved 240, tool_tokens_saved 0, tool_tokens_deferred 0 ``` - Not tested: a live run with the headroom-tool-search extension (deferral is covered by unit tests), and Responses-API bodies. ## Runtime Rollout Safety - Rollout-managed feature(s): None gate this change — the gateway `/v1/compress` accounting is not behind a registry feature. One interaction: `proxy_output_shaper` (registry feature, available `stable`, deliberately not default-enabled) consumes the augmented count. `tokens_before` is folded at `headroom/proxy/handlers/openai.py:11064` and then handed to the turn at `:11083`, which passes it as `input_tokens` to `shape_gateway_body` (`headroom/proxy/gateway_extension.py:213`) → `stratum_key` → `input_bucket`. A turn whose tool definitions push it across one of the 2k / 8k / 32k / 128k boundaries therefore lands in a bigger stratum, which picks a different output-savings baseline. The chat path already has this coupling. - Minimum rollout channel: n/a — no channel gate, so the change takes effect wherever gateway mode is served, on every channel. - Stable/default behavior changed: Yes, gateway-mode answers only. `tokens_before` / `tokens_after` now include the tool definitions before and after schema compaction, so `tokens_saved` covers the compaction saving the same way the chat handlers already do, and two fields are added. Non-gateway and fail-open answers are byte-identical. - Kill switch / disable path: None. Without compaction there is nothing to fold, and the two fields are additive on answers; a gateway that does not want them ignores them. - Unsafe override required: No. Nothing crosses a channel boundary. - Qualification impact: None. No registry entry is added, so `registry_digest` and `snapshot_digest` are unchanged. The gateway `tokens_saved` figure itself changes, which a benchmark comparing arms across this commit will see. - Rollback path: Revert the two commits. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I have added tests that my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - The fields are emitted for any claimed compress turn. Only the gateway extension claims turns today. - If a response-half hook rewrote the tool tags during a re-drive, `/stats` would no longer match the answer. Core has no such hook. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
134 lines
5.1 KiB
Python
134 lines
5.1 KiB
Python
"""/v1/compress nets tool definitions a turn hook adds against its saving.
|
|
|
|
A hook that folds messages but adds a tool (skill search's search tool) sends
|
|
those tool tokens too; ``tokens_after`` must include them, as on the chat path.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.savings_attribution import from_tags
|
|
from headroom.proxy.turn_hooks import register_turn_hook
|
|
from tests.gateway.conftest import compress
|
|
|
|
_LONG = "line of tool output that a hook can fold away entirely " * 60
|
|
_ADDED_TOOL = {
|
|
"name": "search_skills",
|
|
"description": "Find a skill by what it does. " * 40,
|
|
"input_schema": {"type": "object", "properties": {"q": {"type": "string"}}},
|
|
}
|
|
|
|
|
|
class _FoldAndAddTool:
|
|
name = savings_source = "skill_search"
|
|
stream_safe = True
|
|
|
|
def __init__(self, add_tool: bool) -> None:
|
|
self.add_tool = add_tool
|
|
|
|
def on_request(self, ctx: Any) -> None:
|
|
ctx.messages = [{**m, "content": "folded"} for m in ctx.messages]
|
|
if self.add_tool:
|
|
ctx.tools = [*(ctx.tools or []), dict(_ADDED_TOOL)]
|
|
|
|
|
|
def _run(make_headroom_client: Any, *, add_tool: bool) -> dict[str, Any]:
|
|
register_turn_hook(_FoldAndAddTool(add_tool))
|
|
client = make_headroom_client(optimize=False)
|
|
body = {
|
|
"model": "claude-sonnet-4-5",
|
|
"messages": [{"role": "user", "content": _LONG}],
|
|
"tools": [{"name": "noop", "description": "x", "input_schema": {"type": "object"}}],
|
|
"gateway": {"can_redrive": False, "can_relay_response": False},
|
|
}
|
|
resp = compress(client, body)
|
|
assert resp.status_code == 200, resp.text
|
|
return resp.json()
|
|
|
|
|
|
def test_tools_a_hook_adds_count_as_sent(make_headroom_client) -> None:
|
|
from headroom.proxy.turn_hooks import clear_turn_hooks
|
|
|
|
plain = _run(make_headroom_client, add_tool=False)
|
|
clear_turn_hooks()
|
|
grown = _run(make_headroom_client, add_tool=True)
|
|
assert plain["tokens_saved"] > 0
|
|
added = grown["tokens_after"] - plain["tokens_after"]
|
|
assert added > 100 # the search tool's definition
|
|
assert grown["tokens_saved"] == plain["tokens_saved"] - added
|
|
assert grown["tokens_before"] == plain["tokens_before"]
|
|
|
|
|
|
class _AddTextRemoveTools:
|
|
name = savings_source = "rewriter"
|
|
stream_safe = True
|
|
|
|
def __init__(self, repeat: int) -> None:
|
|
self.repeat = repeat
|
|
|
|
def on_request(self, ctx: Any) -> None:
|
|
ctx.messages = [*ctx.messages, {"role": "user", "content": "steer. " * self.repeat}]
|
|
ctx.tools = []
|
|
|
|
|
|
@pytest.mark.parametrize("session", [False, True], ids=["stateless", "session"])
|
|
@pytest.mark.parametrize("repeat", [5, 2_000], ids=["net-positive", "net-negative"])
|
|
def test_message_growth_offsets_tools_a_hook_removed(
|
|
make_headroom_client, outcome_spy, repeat, session
|
|
) -> None:
|
|
"""The opposite signed case on /v1/compress: a hook appends message text
|
|
and drops the tools. The headline is removed tools minus added text,
|
|
clamped once, not the full tool removal."""
|
|
from headroom.proxy.tool_schema_savings_policy import headline_tokens_saved
|
|
from headroom.tokenizers import get_tokenizer
|
|
|
|
register_turn_hook(_AddTextRemoveTools(repeat))
|
|
client = make_headroom_client(optimize=False)
|
|
outcomes = outcome_spy(client)
|
|
before = [{"role": "user", "content": "hi"}]
|
|
body = {
|
|
"model": "claude-sonnet-4-5",
|
|
"messages": before,
|
|
"tools": [dict(_ADDED_TOOL)],
|
|
"gateway": {"can_redrive": False, "can_relay_response": False},
|
|
}
|
|
if session:
|
|
# Session turns recount the final (post-hook) messages: the added
|
|
# text must not be counted a second time on top of that.
|
|
body["config"] = {"session_id": f"growth-{repeat}"}
|
|
resp = compress(client, body)
|
|
assert resp.status_code == 200, resp.text
|
|
tok = get_tokenizer("claude-sonnet-4-5")
|
|
growth = tok.count_messages(
|
|
[*before, {"role": "user", "content": "steer. " * repeat}]
|
|
) - tok.count_messages(before)
|
|
removed = tok.count_text(json.dumps([_ADDED_TOOL], default=str))
|
|
net = removed - growth
|
|
assert (net > 0) == (repeat == 5)
|
|
outcome = outcomes[-1]
|
|
headline = headline_tokens_saved(outcome.tokens_saved, outcome.tags)
|
|
# The answer reports the same headline, none of it estimated.
|
|
data = resp.json()
|
|
assert data["tokens_saved"] + data["tool_tokens_saved"] == headline
|
|
assert data["tool_tokens_deferred"] == 0
|
|
# The gateway compacts tool schemas before hooks run: the hook's own ledger
|
|
# covers the compacted schema, and the compaction delta is inside
|
|
# tokens_saved, so the headline is the two together.
|
|
from headroom.proxy.gateway_turn import compact_gateway_tools
|
|
|
|
compacted, _ = compact_gateway_tools([dict(_ADDED_TOOL)])
|
|
compaction = removed - tok.count_text(json.dumps(compacted, default=str))
|
|
entries = from_tags(outcome.tags)
|
|
if net > 0:
|
|
(entry,) = entries
|
|
hook_net = entry["details"]["tool_tokens_saved"] + entry["details"]["message_tokens_saved"]
|
|
assert entry["details"]["message_tokens_saved"] < 0
|
|
assert headline == hook_net + compaction == entry["tokens"] + compaction
|
|
else:
|
|
assert entries == []
|
|
assert headline == 0
|