Files
headroom/tests/test_anthropic_compaction_transforms.py
Tejas ChopraandClaude Opus 5 a2bf5ed1ba fix(cache): stop prefix transforms from busting the provider cache (#3735)
## Why

A `cache_control` marker means the provider has already hashed
everything up to and including that point, and is billing it at the
**cache-read rate (0.1x)**. Rewriting any of those bytes busts the entry
and re-bills the whole prefix at the **write rate (1.25x)** — a 12.5x
swing that dwarfs anything the transform saves.

Preserving the `cache_control` *field* does **not** preserve the
*entry*. The provider keys on content, not on the marker. A transform
that copies the marker onto rewritten text reads as cache-safe and is
not.

The arithmetic: if a transform invalidates the prefix, it has to cut
**>92%** of tokens just to break even against leaving the cache intact.
Real compaction cuts single-digit percentages. It is never worth it.

## What was wrong

**`compact_tool_descriptions`** truncated tool descriptions with no
marker check at all. A client that pinned any tool lost its entire tools
prefix — and every message behind it — to save a few hundred bytes.

**`_compact_system_blocks`** rewrote pinned system text and copied the
marker onto the new block. Its own docstring said "`cache_control` and
other non-text fields are preserved unchanged", which is true of the
field and misleading about the effect.

## What changed

- Tool-description compaction is skipped when any tool carries a marker.
This mirrors guards that already exist on
`_sort_tools_deterministically` and on `any_tool_has_cache_control` in
the Rust live zone — this path was simply missed.
- System compaction now compacts only blocks **after** the last
breakpoint, via a new `_last_cache_control_index` helper.

Both transforms behave exactly as before when the client pins nothing —
the common case, and the one they were written for. Neither is silent:
the skip logs `event=tool_desc_compaction_skipped`.

## Tests

New `tests/test_cache_prefix_safety.py` (7 tests) covers both
transforms: pinned content stays byte-identical, unpinned content still
compacts, and a marker on the final block is a full no-op.

Two existing expectations changed, deliberately:

- `test_preserves_cache_control` asserted that a pinned block is
compacted *and* keeps its marker — precisely the bug. Replaced by
`test_marked_block_is_left_byte_identical`, plus a new test that
compaction still runs after the breakpoint.
- `_make_anthropic_payload_with_long_system` had a single pinned block,
so the guard made it a no-op. It now carries a pinned block followed by
dynamic content — the shape agentic clients actually send — so the tests
still exercise compaction while asserting the pinned block survives.

318 related tests pass; `ruff check` and `ruff format --check` clean.

## Scope

Not addressed here, tracked separately:

- Savings accounting clamps every component at `max(…, 0)`
(`savings_tracker.py:958-965`) and never debits `cache_bust_tokens_lost`
from the headline, so a net-negative deployment reports a gain.
`cost.py:496` already computes the honest figure and nothing consumes
it.
- `normalize_message_cache_control` was investigated and left alone: the
client-mirroring path is correct, and the legacy fallback deliberately
consolidates accumulated replay leftovers (#2375). An earlier attempt to
"preserve" markers there broke five existing tests for good reasons.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

### Known limitation — read before relying on this

Adversarial review after the fact found the guard is **narrower than
this description implies**. It is not a regression, but it does not
fully deliver what the title promises, and that is worth recording
rather than discovering later.

Anthropic evaluates the cache prefix in the order **tools → system →
messages**, so a marker in a *later* section freezes every *earlier*
one. Neither new guard accounts for that:

1. **`compact_tools` is unguarded** (`tool_schema_compaction.py`,
`compact_tools`). It strips `$schema`/`title`/`examples` and collapses
whitespace on every `input_schema`, and it runs **first, on the same
array**, one block above the guard that was added. Verified on this
branch: a tools array containing a `cache_control`-marked tool is still
rewritten, 436 → 374 bytes. A client that pins its tools still gets its
whole tools prefix rewritten.
2. **A marker in `system` or `messages` does not stop tool compaction**,
even though tools sit inside that prefix. The updated test in this PR
encodes exactly this: it pins `system[0]`, then asserts the tools were
compacted anyway — so the cached prefix is busted on the very request
where the new system guard carefully preserved `system[0]`
byte-for-byte.
3. **`_compact_system_blocks` ignores markers in `messages`.** A client
whose only breakpoint sits on the last user message has its whole system
array inside the cached prefix, yet `frozen_upto == -1` and every block
is rewritten. Fixing this needs a signature change — the function only
receives `blocks`, not the payload.

Net: this helps a client that pins **only** system, and buys little for
one that pins its tools. Fixing it properly requires cross-section
marker awareness, and the tools-side guard must be widened *after* (2)
is fixed, not before — widening it first makes things worse.

Merging as incremental progress with the limitation stated. Follow-up
tracked separately.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 22:39:33 +02:00

407 lines
16 KiB
Python

"""Handler-level regression tests for Anthropic compaction transforms_applied reporting.
Verifies that when tool-schema compaction (L1), tool-description compaction (L2),
or system-prompt compaction (L3) modifies an Anthropic request, the corresponding
label is appended to ``transforms_applied`` so that ``/stats`` and the
transformation accounting remain accurate.
These tests exercise the handler wiring directly (not just the helper functions)
to catch the specific bug where Anthropic omitted the append calls that the
OpenAI handler already had.
"""
from __future__ import annotations
import os
from unittest.mock import patch
import pytest
# ---------------------------------------------------------------------------
# Fixtures / helpers
# ---------------------------------------------------------------------------
def _make_anthropic_payload_with_tools() -> dict:
"""Minimal Anthropic-style payload with tools that will be compacted."""
return {
"model": "claude-sonnet-4-6",
"system": "You are a helpful assistant.",
"messages": [{"role": "user", "content": "hello"}],
"tools": [
{
"name": "read_file",
"description": "Read the contents of a file from disk. "
"Returns the full text content as a string.",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "read_file_schema",
"examples": [{"path": "/tmp/test.txt"}],
"type": "object",
"properties": {
"path": {"type": "string", "description": "File path"},
},
"required": ["path"],
},
}
],
"max_tokens": 1024,
}
def _make_anthropic_payload_with_long_system() -> dict:
"""Payload with a long system prompt that qualifies for L3 compaction."""
long_text = "x" * 5000 # well above default min_chars
# Two blocks, mirroring how agentic clients actually shape the system array:
# a stable block the client has pinned with cache_control (CLAUDE.md, rules)
# followed by dynamic content. Only the block AFTER the last breakpoint is
# eligible -- rewriting the pinned one would bust the provider's entry, so
# compaction must leave it byte-identical.
return {
"model": "claude-sonnet-4-6",
"system": [
{"type": "text", "text": long_text, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "y" * 5000},
],
"messages": [{"role": "user", "content": "hello"}],
"tools": [],
"max_tokens": 1024,
}
# ---------------------------------------------------------------------------
# L1: Tool schema compaction
# ---------------------------------------------------------------------------
class TestAnthropicToolSchemaCompactionTransforms:
"""When L1 compaction modifies tools, ``anthropic:tool_schema_compaction``
must appear in ``transforms_applied``."""
def test_l1_appends_transform_label(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tools
payload = _make_anthropic_payload_with_tools()
body, modified, before, after = compact_tools(payload)
assert modified is True
# The handler code does:
# if _tools_modified:
# transforms_applied.append("anthropic:tool_schema_compaction")
# We verify the condition that triggers the append is met.
assert before > after
def test_l1_skips_label_when_no_compaction(self) -> None:
from headroom.proxy.tool_schema_compaction import compact_tools
payload = _make_anthropic_payload_with_tools()
# Already compact — remove annotation keys AND normalise description
# so compact_tools has nothing to change.
schema = payload["tools"][0]["input_schema"]
schema.pop("$schema", None)
schema.pop("title", None)
schema.pop("examples", None)
# Normalise description whitespace to match compaction output.
payload["tools"][0]["description"] = " ".join(payload["tools"][0]["description"].split())
body, modified, before, after = compact_tools(payload)
# When nothing can be compacted, the handler should NOT append the label.
# We verify the condition: _tools_modified must be False.
assert modified is False
# ---------------------------------------------------------------------------
# L2: Tool description compaction
# ---------------------------------------------------------------------------
class TestAnthropicToolDescCompactionTransforms:
"""When L2 compaction truncates descriptions,
``anthropic:tool_desc_compaction`` must appear in ``transforms_applied``."""
def test_l2_appends_transform_label(self, monkeypatch: pytest.MonkeyPatch) -> None:
import headroom.proxy.tool_schema_compaction as _mod
from headroom.proxy.tool_schema_compaction import (
compact_tool_descriptions,
tool_desc_max_chars,
)
# Opt-in with a very short max so truncation triggers. Reset the
# per-process cache first: an earlier test in the shard may have read
# the (unset) env and pinned max_chars to 0, which would swallow our
# setenv below.
monkeypatch.setenv("HEADROOM_TOOL_DESC_MAX_CHARS", "20")
_mod._TOOL_DESC_MAX_CHARS = None
payload = _make_anthropic_payload_with_tools()
max_chars = tool_desc_max_chars()
assert max_chars == 20
body, modified, before, after = compact_tool_descriptions(payload, max_chars)
assert modified is True
assert before > after
# Don't leak the cached 20 into later tests in this shard.
_mod._TOOL_DESC_MAX_CHARS = None
def test_l2_skips_label_when_disabled(self) -> None:
import headroom.proxy.tool_schema_compaction as _mod
from headroom.proxy.tool_schema_compaction import tool_desc_max_chars
# Reset the per-process cache so the env var is re-read.
_mod._TOOL_DESC_MAX_CHARS = None
with patch.dict(os.environ, {}, clear=True):
max_chars = tool_desc_max_chars()
# Restore cache state for subsequent tests.
_mod._TOOL_DESC_MAX_CHARS = None
# When max_chars == 0, the handler skips the entire L2 block,
# so no append happens.
assert max_chars == 0
# ---------------------------------------------------------------------------
# L3: System prompt compaction
# ---------------------------------------------------------------------------
class TestAnthropicSystemCompactionTransforms:
"""When L3 compaction compresses system blocks,
``anthropic:system_prompt_compaction`` must appear in ``transforms_applied``."""
def test_l3_appends_transform_label(self, monkeypatch: pytest.MonkeyPatch) -> None:
from headroom.proxy.system_compaction import (
compact_system_prompt,
)
monkeypatch.setenv("HEADROOM_SYSTEM_COMPACT", "1")
payload = _make_anthropic_payload_with_long_system()
# Mock the router so we don't need a real one.
class _MockCompressResult:
def __init__(self, compressed: str):
self.compressed = compressed
class _MockRouter:
def compress(self, text: str, **kwargs):
return _MockCompressResult(text[:100])
body, modified, before, after = compact_system_prompt(
payload,
router=_MockRouter(),
model="claude-sonnet-4-6",
request_id="test-req",
)
assert modified is True
assert before > after
def test_l3_skips_label_when_disabled(self, monkeypatch: pytest.MonkeyPatch) -> None:
from headroom.proxy.system_compaction import system_compact_enabled
monkeypatch.delenv("HEADROOM_SYSTEM_COMPACT", raising=False)
assert system_compact_enabled() is False
# When disabled, the handler skips L3 entirely, so no append.
# ---------------------------------------------------------------------------
# Handler-level end-to-end regression (issue: Anthropic omitted the append)
#
# The tests above exercise the helper return values in isolation. These below
# drive the *handler wiring* end-to-end: a real ``_handle_anthropic_request``
# runs against a tool-bearing payload, the live ``compact_tools`` mutates it,
# and the L1 label must surface on the ``x-headroom-transforms`` response
# header. This is the gap the maintainer flagged -- the bug lived in the
# handler's append call, not in the helpers.
# ---------------------------------------------------------------------------
import httpx # noqa: E402
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
def _make_proxy_client(*, optimize: bool = True) -> TestClient:
config = ProxyConfig(
optimize=optimize,
mode="token",
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
app = create_app(config)
return TestClient(app)
def _ok_response(msg_id: str) -> httpx.Response:
return httpx.Response(
200,
json={
"id": msg_id,
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 10,
"output_tokens": 3,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
class TestAnthropicHandlerReportsL1Transform:
"""End-to-end: when L1 tool-schema compaction mutates the request, the
handler must append ``anthropic:tool_schema_compaction`` so it reaches the
``x-headroom-transforms`` response header -- not just the helper's
``modified`` flag."""
def test_l1_label_reaches_response_header(self) -> None:
from types import SimpleNamespace
with _make_proxy_client() as client:
proxy = client.app.state.proxy
def _fake_apply(**kwargs):
# Return the minimum result shape the handler reads; let the
# handler's own compaction pass (which runs after apply) do the
# real mutation we are testing.
return SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=10,
tokens_after=10,
waste_signals=None,
)
proxy.anthropic_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return _ok_response("msg_l1_e2e")
proxy._retry_request = _fake_retry
response = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 64,
"messages": [{"role": "user", "content": "hello"}],
"tools": [
{
"name": "read_file",
"description": "Read a file from disk. Returns text content.",
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "read_file_schema",
"examples": [{"path": "/tmp/test.txt"}],
"type": "object",
"properties": {
"path": {"type": "string"},
},
"required": ["path"],
},
}
],
},
)
assert response.status_code == 200, response.text
transforms_header = response.headers.get("x-headroom-transforms", "")
assert "anthropic:tool_schema_compaction" in transforms_header, (
f"expected L1 label in x-headroom-transforms, got: {transforms_header!r}"
)
@pytest.mark.parametrize(
("optimize", "headers", "should_compact"),
[
pytest.param(False, {}, False, id="no-optimize"),
pytest.param(True, {"x-headroom-bypass": "true"}, False, id="bypass"),
pytest.param(True, {"x-headroom-mode": "passthrough"}, False, id="passthrough"),
pytest.param(True, {}, True, id="optimization-enabled"),
],
)
def test_handler_auxiliary_compaction_respects_optimization_decision(
monkeypatch: pytest.MonkeyPatch,
optimize: bool,
headers: dict[str, str],
should_compact: bool,
) -> None:
"""Opt-in tool/system passes must also honor the request's disable controls."""
import copy
from types import SimpleNamespace
from unittest.mock import Mock
import headroom.proxy.tool_schema_compaction as tool_compaction
monkeypatch.setenv("HEADROOM_TOOL_DESC_MAX_CHARS", "20")
monkeypatch.setenv("HEADROOM_SYSTEM_COMPACT", "1")
monkeypatch.setattr(tool_compaction, "_TOOL_DESC_MAX_CHARS", None)
router = SimpleNamespace(compress=Mock(return_value=SimpleNamespace(compressed="short system")))
monkeypatch.setattr(
"headroom.transforms.compression_units.find_content_router", lambda _: router
)
payload = _make_anthropic_payload_with_tools()
payload["system"] = _make_anthropic_payload_with_long_system()["system"]
captured = []
with _make_proxy_client(optimize=optimize) as client:
proxy = client.app.state.proxy
proxy.anthropic_pipeline.apply = lambda **kwargs: SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
timing={},
tokens_before=10,
tokens_after=10,
waste_signals=None,
)
async def capture(method, url, headers, body, stream=False, **kwargs):
captured.append(copy.deepcopy(body))
return _ok_response("msg_compaction_controls")
proxy._retry_request = capture
response = client.post("/v1/messages", headers=headers, json=payload)
assert response.status_code == 200, response.text
summary = client.get("/stats").json()["summary"]
forwarded = captured[0]
transforms = response.headers.get("x-headroom-transforms", "")
labels = (
"anthropic:tool_schema_compaction",
"anthropic:tool_desc_compaction",
"anthropic:system_prompt_compaction",
)
if should_compact:
assert "title" not in forwarded["tools"][0]["input_schema"]
assert len(forwarded["tools"][0]["description"]) < len(payload["tools"][0]["description"])
# Block 0 is pinned by cache_control: byte-identical, marker intact.
assert forwarded["system"][0]["text"] == payload["system"][0]["text"]
assert forwarded["system"][0]["cache_control"] == {"type": "ephemeral"}
# Block 1 sits after the breakpoint and is compacted.
assert forwarded["system"][1]["text"] == "short system"
assert all(label in transforms for label in labels)
router.compress.assert_called_once()
assert summary["compression"]["total_tokens_removed"] > 0
else:
assert forwarded["tools"] == payload["tools"]
assert forwarded["system"] == payload["system"]
assert all(label not in transforms for label in labels)
router.compress.assert_not_called()
assert summary["compression"]["requests_compressed"] == 0
assert summary["compression"]["total_tokens_removed"] == 0