mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Why A `cache_control` marker means the provider has already hashed everything up to and including that point, and is billing it at the **cache-read rate (0.1x)**. Rewriting any of those bytes busts the entry and re-bills the whole prefix at the **write rate (1.25x)** — a 12.5x swing that dwarfs anything the transform saves. Preserving the `cache_control` *field* does **not** preserve the *entry*. The provider keys on content, not on the marker. A transform that copies the marker onto rewritten text reads as cache-safe and is not. The arithmetic: if a transform invalidates the prefix, it has to cut **>92%** of tokens just to break even against leaving the cache intact. Real compaction cuts single-digit percentages. It is never worth it. ## What was wrong **`compact_tool_descriptions`** truncated tool descriptions with no marker check at all. A client that pinned any tool lost its entire tools prefix — and every message behind it — to save a few hundred bytes. **`_compact_system_blocks`** rewrote pinned system text and copied the marker onto the new block. Its own docstring said "`cache_control` and other non-text fields are preserved unchanged", which is true of the field and misleading about the effect. ## What changed - Tool-description compaction is skipped when any tool carries a marker. This mirrors guards that already exist on `_sort_tools_deterministically` and on `any_tool_has_cache_control` in the Rust live zone — this path was simply missed. - System compaction now compacts only blocks **after** the last breakpoint, via a new `_last_cache_control_index` helper. Both transforms behave exactly as before when the client pins nothing — the common case, and the one they were written for. Neither is silent: the skip logs `event=tool_desc_compaction_skipped`. ## Tests New `tests/test_cache_prefix_safety.py` (7 tests) covers both transforms: pinned content stays byte-identical, unpinned content still compacts, and a marker on the final block is a full no-op. Two existing expectations changed, deliberately: - `test_preserves_cache_control` asserted that a pinned block is compacted *and* keeps its marker — precisely the bug. Replaced by `test_marked_block_is_left_byte_identical`, plus a new test that compaction still runs after the breakpoint. - `_make_anthropic_payload_with_long_system` had a single pinned block, so the guard made it a no-op. It now carries a pinned block followed by dynamic content — the shape agentic clients actually send — so the tests still exercise compaction while asserting the pinned block survives. 318 related tests pass; `ruff check` and `ruff format --check` clean. ## Scope Not addressed here, tracked separately: - Savings accounting clamps every component at `max(…, 0)` (`savings_tracker.py:958-965`) and never debits `cache_bust_tokens_lost` from the headline, so a net-negative deployment reports a gain. `cost.py:496` already computes the honest figure and nothing consumes it. - `normalize_message_cache_control` was investigated and left alone: the client-mirroring path is correct, and the legacy fallback deliberately consolidates accumulated replay leftovers (#2375). An earlier attempt to "preserve" markers there broke five existing tests for good reasons. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### Known limitation — read before relying on this Adversarial review after the fact found the guard is **narrower than this description implies**. It is not a regression, but it does not fully deliver what the title promises, and that is worth recording rather than discovering later. Anthropic evaluates the cache prefix in the order **tools → system → messages**, so a marker in a *later* section freezes every *earlier* one. Neither new guard accounts for that: 1. **`compact_tools` is unguarded** (`tool_schema_compaction.py`, `compact_tools`). It strips `$schema`/`title`/`examples` and collapses whitespace on every `input_schema`, and it runs **first, on the same array**, one block above the guard that was added. Verified on this branch: a tools array containing a `cache_control`-marked tool is still rewritten, 436 → 374 bytes. A client that pins its tools still gets its whole tools prefix rewritten. 2. **A marker in `system` or `messages` does not stop tool compaction**, even though tools sit inside that prefix. The updated test in this PR encodes exactly this: it pins `system[0]`, then asserts the tools were compacted anyway — so the cached prefix is busted on the very request where the new system guard carefully preserved `system[0]` byte-for-byte. 3. **`_compact_system_blocks` ignores markers in `messages`.** A client whose only breakpoint sits on the last user message has its whole system array inside the cached prefix, yet `frozen_upto == -1` and every block is rewritten. Fixing this needs a signature change — the function only receives `blocks`, not the payload. Net: this helps a client that pins **only** system, and buys little for one that pins its tools. Fixing it properly requires cross-section marker awareness, and the tools-side guard must be widened *after* (2) is fixed, not before — widening it first makes things worse. Merging as incremental progress with the limitation stated. Follow-up tracked separately. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
407 lines
16 KiB
Python
407 lines
16 KiB
Python
"""Handler-level regression tests for Anthropic compaction transforms_applied reporting.
|
|
|
|
Verifies that when tool-schema compaction (L1), tool-description compaction (L2),
|
|
or system-prompt compaction (L3) modifies an Anthropic request, the corresponding
|
|
label is appended to ``transforms_applied`` so that ``/stats`` and the
|
|
transformation accounting remain accurate.
|
|
|
|
These tests exercise the handler wiring directly (not just the helper functions)
|
|
to catch the specific bug where Anthropic omitted the append calls that the
|
|
OpenAI handler already had.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fixtures / helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _make_anthropic_payload_with_tools() -> dict:
|
|
"""Minimal Anthropic-style payload with tools that will be compacted."""
|
|
return {
|
|
"model": "claude-sonnet-4-6",
|
|
"system": "You are a helpful assistant.",
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
"tools": [
|
|
{
|
|
"name": "read_file",
|
|
"description": "Read the contents of a file from disk. "
|
|
"Returns the full text content as a string.",
|
|
"input_schema": {
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"title": "read_file_schema",
|
|
"examples": [{"path": "/tmp/test.txt"}],
|
|
"type": "object",
|
|
"properties": {
|
|
"path": {"type": "string", "description": "File path"},
|
|
},
|
|
"required": ["path"],
|
|
},
|
|
}
|
|
],
|
|
"max_tokens": 1024,
|
|
}
|
|
|
|
|
|
def _make_anthropic_payload_with_long_system() -> dict:
|
|
"""Payload with a long system prompt that qualifies for L3 compaction."""
|
|
long_text = "x" * 5000 # well above default min_chars
|
|
# Two blocks, mirroring how agentic clients actually shape the system array:
|
|
# a stable block the client has pinned with cache_control (CLAUDE.md, rules)
|
|
# followed by dynamic content. Only the block AFTER the last breakpoint is
|
|
# eligible -- rewriting the pinned one would bust the provider's entry, so
|
|
# compaction must leave it byte-identical.
|
|
return {
|
|
"model": "claude-sonnet-4-6",
|
|
"system": [
|
|
{"type": "text", "text": long_text, "cache_control": {"type": "ephemeral"}},
|
|
{"type": "text", "text": "y" * 5000},
|
|
],
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
"tools": [],
|
|
"max_tokens": 1024,
|
|
}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# L1: Tool schema compaction
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestAnthropicToolSchemaCompactionTransforms:
|
|
"""When L1 compaction modifies tools, ``anthropic:tool_schema_compaction``
|
|
must appear in ``transforms_applied``."""
|
|
|
|
def test_l1_appends_transform_label(self) -> None:
|
|
from headroom.proxy.tool_schema_compaction import compact_tools
|
|
|
|
payload = _make_anthropic_payload_with_tools()
|
|
body, modified, before, after = compact_tools(payload)
|
|
|
|
assert modified is True
|
|
# The handler code does:
|
|
# if _tools_modified:
|
|
# transforms_applied.append("anthropic:tool_schema_compaction")
|
|
# We verify the condition that triggers the append is met.
|
|
assert before > after
|
|
|
|
def test_l1_skips_label_when_no_compaction(self) -> None:
|
|
from headroom.proxy.tool_schema_compaction import compact_tools
|
|
|
|
payload = _make_anthropic_payload_with_tools()
|
|
# Already compact — remove annotation keys AND normalise description
|
|
# so compact_tools has nothing to change.
|
|
schema = payload["tools"][0]["input_schema"]
|
|
schema.pop("$schema", None)
|
|
schema.pop("title", None)
|
|
schema.pop("examples", None)
|
|
# Normalise description whitespace to match compaction output.
|
|
payload["tools"][0]["description"] = " ".join(payload["tools"][0]["description"].split())
|
|
|
|
body, modified, before, after = compact_tools(payload)
|
|
# When nothing can be compacted, the handler should NOT append the label.
|
|
# We verify the condition: _tools_modified must be False.
|
|
assert modified is False
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# L2: Tool description compaction
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestAnthropicToolDescCompactionTransforms:
|
|
"""When L2 compaction truncates descriptions,
|
|
``anthropic:tool_desc_compaction`` must appear in ``transforms_applied``."""
|
|
|
|
def test_l2_appends_transform_label(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
import headroom.proxy.tool_schema_compaction as _mod
|
|
from headroom.proxy.tool_schema_compaction import (
|
|
compact_tool_descriptions,
|
|
tool_desc_max_chars,
|
|
)
|
|
|
|
# Opt-in with a very short max so truncation triggers. Reset the
|
|
# per-process cache first: an earlier test in the shard may have read
|
|
# the (unset) env and pinned max_chars to 0, which would swallow our
|
|
# setenv below.
|
|
monkeypatch.setenv("HEADROOM_TOOL_DESC_MAX_CHARS", "20")
|
|
_mod._TOOL_DESC_MAX_CHARS = None
|
|
|
|
payload = _make_anthropic_payload_with_tools()
|
|
max_chars = tool_desc_max_chars()
|
|
assert max_chars == 20
|
|
|
|
body, modified, before, after = compact_tool_descriptions(payload, max_chars)
|
|
assert modified is True
|
|
assert before > after
|
|
# Don't leak the cached 20 into later tests in this shard.
|
|
_mod._TOOL_DESC_MAX_CHARS = None
|
|
|
|
def test_l2_skips_label_when_disabled(self) -> None:
|
|
import headroom.proxy.tool_schema_compaction as _mod
|
|
from headroom.proxy.tool_schema_compaction import tool_desc_max_chars
|
|
|
|
# Reset the per-process cache so the env var is re-read.
|
|
_mod._TOOL_DESC_MAX_CHARS = None
|
|
with patch.dict(os.environ, {}, clear=True):
|
|
max_chars = tool_desc_max_chars()
|
|
# Restore cache state for subsequent tests.
|
|
_mod._TOOL_DESC_MAX_CHARS = None
|
|
# When max_chars == 0, the handler skips the entire L2 block,
|
|
# so no append happens.
|
|
assert max_chars == 0
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# L3: System prompt compaction
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestAnthropicSystemCompactionTransforms:
|
|
"""When L3 compaction compresses system blocks,
|
|
``anthropic:system_prompt_compaction`` must appear in ``transforms_applied``."""
|
|
|
|
def test_l3_appends_transform_label(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from headroom.proxy.system_compaction import (
|
|
compact_system_prompt,
|
|
)
|
|
|
|
monkeypatch.setenv("HEADROOM_SYSTEM_COMPACT", "1")
|
|
|
|
payload = _make_anthropic_payload_with_long_system()
|
|
|
|
# Mock the router so we don't need a real one.
|
|
class _MockCompressResult:
|
|
def __init__(self, compressed: str):
|
|
self.compressed = compressed
|
|
|
|
class _MockRouter:
|
|
def compress(self, text: str, **kwargs):
|
|
return _MockCompressResult(text[:100])
|
|
|
|
body, modified, before, after = compact_system_prompt(
|
|
payload,
|
|
router=_MockRouter(),
|
|
model="claude-sonnet-4-6",
|
|
request_id="test-req",
|
|
)
|
|
|
|
assert modified is True
|
|
assert before > after
|
|
|
|
def test_l3_skips_label_when_disabled(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
from headroom.proxy.system_compaction import system_compact_enabled
|
|
|
|
monkeypatch.delenv("HEADROOM_SYSTEM_COMPACT", raising=False)
|
|
assert system_compact_enabled() is False
|
|
# When disabled, the handler skips L3 entirely, so no append.
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Handler-level end-to-end regression (issue: Anthropic omitted the append)
|
|
#
|
|
# The tests above exercise the helper return values in isolation. These below
|
|
# drive the *handler wiring* end-to-end: a real ``_handle_anthropic_request``
|
|
# runs against a tool-bearing payload, the live ``compact_tools`` mutates it,
|
|
# and the L1 label must surface on the ``x-headroom-transforms`` response
|
|
# header. This is the gap the maintainer flagged -- the bug lived in the
|
|
# handler's append call, not in the helpers.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
import httpx # noqa: E402
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient # noqa: E402
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
|
|
|
|
|
def _make_proxy_client(*, optimize: bool = True) -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=optimize,
|
|
mode="token",
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
app = create_app(config)
|
|
return TestClient(app)
|
|
|
|
|
|
def _ok_response(msg_id: str) -> httpx.Response:
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": msg_id,
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 10,
|
|
"output_tokens": 3,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
|
|
class TestAnthropicHandlerReportsL1Transform:
|
|
"""End-to-end: when L1 tool-schema compaction mutates the request, the
|
|
handler must append ``anthropic:tool_schema_compaction`` so it reaches the
|
|
``x-headroom-transforms`` response header -- not just the helper's
|
|
``modified`` flag."""
|
|
|
|
def test_l1_label_reaches_response_header(self) -> None:
|
|
from types import SimpleNamespace
|
|
|
|
with _make_proxy_client() as client:
|
|
proxy = client.app.state.proxy
|
|
|
|
def _fake_apply(**kwargs):
|
|
# Return the minimum result shape the handler reads; let the
|
|
# handler's own compaction pass (which runs after apply) do the
|
|
# real mutation we are testing.
|
|
return SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=10,
|
|
tokens_after=10,
|
|
waste_signals=None,
|
|
)
|
|
|
|
proxy.anthropic_pipeline.apply = _fake_apply
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return _ok_response("msg_l1_e2e")
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={
|
|
"x-api-key": "test-key",
|
|
"anthropic-version": "2023-06-01",
|
|
},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 64,
|
|
"messages": [{"role": "user", "content": "hello"}],
|
|
"tools": [
|
|
{
|
|
"name": "read_file",
|
|
"description": "Read a file from disk. Returns text content.",
|
|
"input_schema": {
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"title": "read_file_schema",
|
|
"examples": [{"path": "/tmp/test.txt"}],
|
|
"type": "object",
|
|
"properties": {
|
|
"path": {"type": "string"},
|
|
},
|
|
"required": ["path"],
|
|
},
|
|
}
|
|
],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200, response.text
|
|
transforms_header = response.headers.get("x-headroom-transforms", "")
|
|
assert "anthropic:tool_schema_compaction" in transforms_header, (
|
|
f"expected L1 label in x-headroom-transforms, got: {transforms_header!r}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("optimize", "headers", "should_compact"),
|
|
[
|
|
pytest.param(False, {}, False, id="no-optimize"),
|
|
pytest.param(True, {"x-headroom-bypass": "true"}, False, id="bypass"),
|
|
pytest.param(True, {"x-headroom-mode": "passthrough"}, False, id="passthrough"),
|
|
pytest.param(True, {}, True, id="optimization-enabled"),
|
|
],
|
|
)
|
|
def test_handler_auxiliary_compaction_respects_optimization_decision(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
optimize: bool,
|
|
headers: dict[str, str],
|
|
should_compact: bool,
|
|
) -> None:
|
|
"""Opt-in tool/system passes must also honor the request's disable controls."""
|
|
import copy
|
|
from types import SimpleNamespace
|
|
from unittest.mock import Mock
|
|
|
|
import headroom.proxy.tool_schema_compaction as tool_compaction
|
|
|
|
monkeypatch.setenv("HEADROOM_TOOL_DESC_MAX_CHARS", "20")
|
|
monkeypatch.setenv("HEADROOM_SYSTEM_COMPACT", "1")
|
|
monkeypatch.setattr(tool_compaction, "_TOOL_DESC_MAX_CHARS", None)
|
|
router = SimpleNamespace(compress=Mock(return_value=SimpleNamespace(compressed="short system")))
|
|
monkeypatch.setattr(
|
|
"headroom.transforms.compression_units.find_content_router", lambda _: router
|
|
)
|
|
payload = _make_anthropic_payload_with_tools()
|
|
payload["system"] = _make_anthropic_payload_with_long_system()["system"]
|
|
captured = []
|
|
|
|
with _make_proxy_client(optimize=optimize) as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.anthropic_pipeline.apply = lambda **kwargs: SimpleNamespace(
|
|
messages=kwargs["messages"],
|
|
transforms_applied=[],
|
|
timing={},
|
|
tokens_before=10,
|
|
tokens_after=10,
|
|
waste_signals=None,
|
|
)
|
|
|
|
async def capture(method, url, headers, body, stream=False, **kwargs):
|
|
captured.append(copy.deepcopy(body))
|
|
return _ok_response("msg_compaction_controls")
|
|
|
|
proxy._retry_request = capture
|
|
response = client.post("/v1/messages", headers=headers, json=payload)
|
|
assert response.status_code == 200, response.text
|
|
summary = client.get("/stats").json()["summary"]
|
|
|
|
forwarded = captured[0]
|
|
transforms = response.headers.get("x-headroom-transforms", "")
|
|
labels = (
|
|
"anthropic:tool_schema_compaction",
|
|
"anthropic:tool_desc_compaction",
|
|
"anthropic:system_prompt_compaction",
|
|
)
|
|
if should_compact:
|
|
assert "title" not in forwarded["tools"][0]["input_schema"]
|
|
assert len(forwarded["tools"][0]["description"]) < len(payload["tools"][0]["description"])
|
|
# Block 0 is pinned by cache_control: byte-identical, marker intact.
|
|
assert forwarded["system"][0]["text"] == payload["system"][0]["text"]
|
|
assert forwarded["system"][0]["cache_control"] == {"type": "ephemeral"}
|
|
# Block 1 sits after the breakpoint and is compacted.
|
|
assert forwarded["system"][1]["text"] == "short system"
|
|
assert all(label in transforms for label in labels)
|
|
router.compress.assert_called_once()
|
|
assert summary["compression"]["total_tokens_removed"] > 0
|
|
else:
|
|
assert forwarded["tools"] == payload["tools"]
|
|
assert forwarded["system"] == payload["system"]
|
|
assert all(label not in transforms for label in labels)
|
|
router.compress.assert_not_called()
|
|
assert summary["compression"]["requests_compressed"] == 0
|
|
assert summary["compression"]["total_tokens_removed"] == 0
|