fix(cache): stop prefix transforms from busting the provider cache (#3735)

## Why

A `cache_control` marker means the provider has already hashed
everything up to and including that point, and is billing it at the
**cache-read rate (0.1x)**. Rewriting any of those bytes busts the entry
and re-bills the whole prefix at the **write rate (1.25x)** — a 12.5x
swing that dwarfs anything the transform saves.

Preserving the `cache_control` *field* does **not** preserve the
*entry*. The provider keys on content, not on the marker. A transform
that copies the marker onto rewritten text reads as cache-safe and is
not.

The arithmetic: if a transform invalidates the prefix, it has to cut
**>92%** of tokens just to break even against leaving the cache intact.
Real compaction cuts single-digit percentages. It is never worth it.

## What was wrong

**`compact_tool_descriptions`** truncated tool descriptions with no
marker check at all. A client that pinned any tool lost its entire tools
prefix — and every message behind it — to save a few hundred bytes.

**`_compact_system_blocks`** rewrote pinned system text and copied the
marker onto the new block. Its own docstring said "`cache_control` and
other non-text fields are preserved unchanged", which is true of the
field and misleading about the effect.

## What changed

- Tool-description compaction is skipped when any tool carries a marker.
This mirrors guards that already exist on
`_sort_tools_deterministically` and on `any_tool_has_cache_control` in
the Rust live zone — this path was simply missed.
- System compaction now compacts only blocks **after** the last
breakpoint, via a new `_last_cache_control_index` helper.

Both transforms behave exactly as before when the client pins nothing —
the common case, and the one they were written for. Neither is silent:
the skip logs `event=tool_desc_compaction_skipped`.

## Tests

New `tests/test_cache_prefix_safety.py` (7 tests) covers both
transforms: pinned content stays byte-identical, unpinned content still
compacts, and a marker on the final block is a full no-op.

Two existing expectations changed, deliberately:

- `test_preserves_cache_control` asserted that a pinned block is
compacted *and* keeps its marker — precisely the bug. Replaced by
`test_marked_block_is_left_byte_identical`, plus a new test that
compaction still runs after the breakpoint.
- `_make_anthropic_payload_with_long_system` had a single pinned block,
so the guard made it a no-op. It now carries a pinned block followed by
dynamic content — the shape agentic clients actually send — so the tests
still exercise compaction while asserting the pinned block survives.

318 related tests pass; `ruff check` and `ruff format --check` clean.

## Scope

Not addressed here, tracked separately:

- Savings accounting clamps every component at `max(…, 0)`
(`savings_tracker.py:958-965`) and never debits `cache_bust_tokens_lost`
from the headline, so a net-negative deployment reports a gain.
`cost.py:496` already computes the honest figure and nothing consumes
it.
- `normalize_message_cache_control` was investigated and left alone: the
client-mirroring path is correct, and the legacy fallback deliberately
consolidates accumulated replay leftovers (#2375). An earlier attempt to
"preserve" markers there broke five existing tests for good reasons.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

### Known limitation — read before relying on this

Adversarial review after the fact found the guard is **narrower than
this description implies**. It is not a regression, but it does not
fully deliver what the title promises, and that is worth recording
rather than discovering later.

Anthropic evaluates the cache prefix in the order **tools → system →
messages**, so a marker in a *later* section freezes every *earlier*
one. Neither new guard accounts for that:

1. **`compact_tools` is unguarded** (`tool_schema_compaction.py`,
`compact_tools`). It strips `$schema`/`title`/`examples` and collapses
whitespace on every `input_schema`, and it runs **first, on the same
array**, one block above the guard that was added. Verified on this
branch: a tools array containing a `cache_control`-marked tool is still
rewritten, 436 → 374 bytes. A client that pins its tools still gets its
whole tools prefix rewritten.
2. **A marker in `system` or `messages` does not stop tool compaction**,
even though tools sit inside that prefix. The updated test in this PR
encodes exactly this: it pins `system[0]`, then asserts the tools were
compacted anyway — so the cached prefix is busted on the very request
where the new system guard carefully preserved `system[0]`
byte-for-byte.
3. **`_compact_system_blocks` ignores markers in `messages`.** A client
whose only breakpoint sits on the last user message has its whole system
array inside the cached prefix, yet `frozen_upto == -1` and every block
is rewritten. Fixing this needs a signature change — the function only
receives `blocks`, not the payload.

Net: this helps a client that pins **only** system, and buys little for
one that pins its tools. Fixing it properly requires cross-section
marker awareness, and the tools-side guard must be widened *after* (2)
is fixed, not before — widening it first makes things worse.

Merging as incremental progress with the limitation stated. Follow-up
tracked separately.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-09-23 22:39:33 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent 1455f00222
commit a2bf5ed1ba
5 changed files with 215 additions and 5 deletions
+28 -1
View File
@@ -41,6 +41,21 @@ def _json_byte_len(value: Any) -> int:
return len(json.dumps(value, ensure_ascii=False, default=str, separators=(",", ":")))
def _last_cache_control_index(blocks: list[Any]) -> int:
"""Index of the last block carrying ``cache_control``, or ``-1`` if none.
Everything at or before this index is inside a prefix the provider has
already hashed and is billing at the cache-read rate. Mutating those bytes
busts the entry and re-bills the whole prefix at write rate (1.25x list on
Anthropic), which costs far more than the compaction saves.
"""
last = -1
for i, block in enumerate(blocks):
if isinstance(block, dict) and block.get("cache_control"):
last = i
return last
def _compact_system_blocks(
blocks: list[dict[str, Any]],
router: Any,
@@ -57,11 +72,23 @@ def _compact_system_blocks(
modified = False
updated: list[dict[str, Any]] = []
for block in blocks:
# A cache_control marker anywhere in the system array freezes every block at
# or before it: the provider hashed those exact bytes, so rewriting any of
# them re-bills the whole prefix as cache creation. Preserving the marker
# field (as this function does) does NOT preserve the cache entry -- the key
# is the content, not the marker. Compact only what sits after the last
# breakpoint. See ``_last_cache_control_index``.
frozen_upto = _last_cache_control_index(blocks)
for idx, block in enumerate(blocks):
if not isinstance(block, dict) or block.get("type") != "text":
updated.append(block)
continue
if idx <= frozen_upto:
updated.append(block)
continue
text = block.get("text", "")
if not isinstance(text, str) or len(text) < min_chars:
updated.append(block)
+19
View File
@@ -34,11 +34,14 @@ from __future__ import annotations
import copy
import hashlib
import json
import logging
import os
import re
import threading
from typing import Any
logger = logging.getLogger(__name__)
# Keys that are JSON Schema annotations, not constraints.
# Removing them does not change the set of valid inputs.
TOOL_SCHEMA_DROP_KEYS: frozenset[str] = frozenset(
@@ -392,6 +395,22 @@ def compact_tool_descriptions(
if not isinstance(tools, list) or not tools:
return payload, False, 0, 0
# A cache_control marker on any tool means "cache everything up to and
# including this one". Truncating a description inside that span changes the
# bytes the provider hashed, so the whole tools prefix -- and every message
# after it -- re-bills as cache creation. The saving is a few hundred bytes;
# the bust is the entire conversation. Mirrors the same guard on
# ``_sort_tools_deterministically`` (handlers/anthropic.py) and
# ``any_tool_has_cache_control`` in the Rust live zone.
marked = sum(1 for t in tools if isinstance(t, dict) and t.get("cache_control"))
if marked:
logger.info(
"event=tool_desc_compaction_skipped reason=marker_present tool_count=%d marked=%d",
len(tools),
marked,
)
return payload, False, 0, 0
strip_sem = strip_semantic_params()
key = _cache_key(tools, "L2", max_chars, strip_sem)
cached = _cache_get(key)
+11 -1
View File
@@ -52,10 +52,16 @@ def _make_anthropic_payload_with_tools() -> dict:
def _make_anthropic_payload_with_long_system() -> dict:
"""Payload with a long system prompt that qualifies for L3 compaction."""
long_text = "x" * 5000 # well above default min_chars
# Two blocks, mirroring how agentic clients actually shape the system array:
# a stable block the client has pinned with cache_control (CLAUDE.md, rules)
# followed by dynamic content. Only the block AFTER the last breakpoint is
# eligible -- rewriting the pinned one would bust the provider's entry, so
# compaction must leave it byte-identical.
return {
"model": "claude-sonnet-4-6",
"system": [
{"type": "text", "text": long_text, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "y" * 5000},
],
"messages": [{"role": "user", "content": "hello"}],
"tools": [],
@@ -383,7 +389,11 @@ def test_handler_auxiliary_compaction_respects_optimization_decision(
if should_compact:
assert "title" not in forwarded["tools"][0]["input_schema"]
assert len(forwarded["tools"][0]["description"]) < len(payload["tools"][0]["description"])
assert forwarded["system"][0]["text"] == "short system"
# Block 0 is pinned by cache_control: byte-identical, marker intact.
assert forwarded["system"][0]["text"] == payload["system"][0]["text"]
assert forwarded["system"][0]["cache_control"] == {"type": "ephemeral"}
# Block 1 sits after the breakpoint and is compacted.
assert forwarded["system"][1]["text"] == "short system"
assert all(label in transforms for label in labels)
router.compress.assert_called_once()
assert summary["compression"]["total_tokens_removed"] > 0
+125
View File
@@ -0,0 +1,125 @@
"""Transforms must never rewrite bytes the provider has already hashed.
A ``cache_control`` marker means the provider is billing everything up to and
including that point at the cache-read rate (0.1x on Anthropic). Rewriting any
of those bytes busts the entry and re-bills the whole prefix at the write rate
(1.25x) -- a 12.5x swing that dwarfs anything the transform saves.
Preserving the ``cache_control`` *field* does not preserve the *entry*: the
provider's key is the content, not the marker.
"""
from __future__ import annotations
from typing import Any
from headroom.proxy.system_compaction import _compact_system_blocks
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
def _text(chars: int, seed: str = "x") -> str:
return (seed * 10 + " context line. ") * (chars // 25 + 1)
class _StubRouter:
"""Router whose compression is obvious and always shrinks."""
def compress(self, text: str, context: str = "") -> Any: # noqa: ARG002
class _R:
compressed = "SHRUNK"
return _R()
# --------------------------------------------------------------------------
# tool description compaction
# --------------------------------------------------------------------------
def test_tool_desc_compaction_skips_when_a_tool_is_marked() -> None:
"""A marker on any tool freezes the whole tools prefix."""
payload = {
"tools": [
{"name": "a", "description": _text(4000), "input_schema": {}},
{
"name": "b",
"description": _text(4000, "y"),
"input_schema": {},
"cache_control": {"type": "ephemeral"},
},
]
}
before = [dict(t) for t in payload["tools"]]
out, modified, _, _ = compact_tool_descriptions(payload, max_chars=50)
assert modified is False
assert out is payload
assert out["tools"] == before
def test_tool_desc_compaction_still_runs_when_nothing_is_marked() -> None:
"""The common case -- no markers -- must keep working."""
payload = {
"tools": [
{"name": "a", "description": _text(4000), "input_schema": {}},
{"name": "b", "description": _text(4000, "y"), "input_schema": {}},
]
}
out, modified, before_bytes, after_bytes = compact_tool_descriptions(payload, max_chars=50)
assert modified is True
assert after_bytes < before_bytes
assert all(len(t["description"]) <= 80 for t in out["tools"])
# --------------------------------------------------------------------------
# system prompt compaction
# --------------------------------------------------------------------------
def test_system_compaction_leaves_blocks_at_or_before_the_marker() -> None:
"""Blocks inside the cached span keep their exact bytes."""
blocks = [
{"type": "text", "text": _text(2000, "a")},
{"type": "text", "text": _text(2000, "b"), "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": _text(2000, "c")},
]
original = [dict(b) for b in blocks]
out, modified = _compact_system_blocks(blocks, _StubRouter(), "m", "rid", 500)
# frozen: identical bytes, marker intact
assert out[0] == original[0]
assert out[1] == original[1]
assert out[1]["cache_control"] == {"type": "ephemeral"}
# after the breakpoint: free to compact
assert out[2]["text"] == "SHRUNK"
assert modified is True
def test_system_compaction_unrestricted_without_markers() -> None:
blocks = [
{"type": "text", "text": _text(2000, "a")},
{"type": "text", "text": _text(2000, "b")},
]
out, modified = _compact_system_blocks(blocks, _StubRouter(), "m", "rid", 500)
assert modified is True
assert [b["text"] for b in out] == ["SHRUNK", "SHRUNK"]
def test_system_compaction_noop_when_marker_is_last() -> None:
"""Marker on the final block freezes everything."""
blocks = [
{"type": "text", "text": _text(2000, "a")},
{"type": "text", "text": _text(2000, "b"), "cache_control": {"type": "ephemeral"}},
]
original = [dict(b) for b in blocks]
out, modified = _compact_system_blocks(blocks, _StubRouter(), "m", "rid", 500)
assert modified is False
assert out == original
+32 -3
View File
@@ -86,8 +86,13 @@ class TestCompactSystemPromptContentBlocks:
assert modified is False
assert result["system"][0]["text"] == "Short instruction."
def test_preserves_cache_control(self) -> None:
"""cache_control must survive compaction."""
def test_marked_block_is_left_byte_identical(self) -> None:
"""A cached block must not be rewritten, marker or not.
Preserving the ``cache_control`` field does not preserve the cache
entry: the provider keys on the content. Rewriting a block it has
already hashed re-bills the whole prefix at the write rate.
"""
payload = {
"system": [
{
@@ -103,10 +108,34 @@ class TestCompactSystemPromptContentBlocks:
model="m",
request_id="test3",
)
assert modified is True
assert modified is False
block = result["system"][0]
assert block["text"] == "A" * 1000
assert block["cache_control"] == {"type": "ephemeral"}
def test_compacts_after_the_breakpoint_and_keeps_the_marker(self) -> None:
"""Blocks past the last marker are still fair game."""
payload = {
"system": [
{
"type": "text",
"text": "A" * 1000,
"cache_control": {"type": "ephemeral"},
},
{"type": "text", "text": "B" * 1000},
],
}
result, modified, _, _ = compact_system_prompt(
payload,
router=_MockRouter(),
model="m",
request_id="test3b",
)
assert modified is True
assert result["system"][0]["text"] == "A" * 1000
assert result["system"][0]["cache_control"] == {"type": "ephemeral"}
assert result["system"][1]["text"] != "B" * 1000
def test_preserves_non_text_blocks(self) -> None:
payload = {
"system": [