fix(cache): stop skipping tool compaction on a pinned tools array (#3750)

Follow-up to #3735, which I reviewed against the real API before
extending. The extension turned out to be the wrong move — the guard
itself is.

## The hole in the original reasoning

#3735 skipped tool-description compaction whenever any tool carried a
`cache_control` marker, on the grounds that *"a marker means the
provider hashed these bytes, so rewriting them re-bills the prefix."*

**The provider never saw the client's bytes.** It only ever sees ours.
Compaction is a pure deterministic transform over a fixed key set, so
turn 2 sends exactly what turn 1 cached — and hits.

## Measured

30 pinned tools, 6 turns, each run salted so they cannot share cache
(haiku-4-5):

| | turn 0 | turns 1–5 | billed prefix |
|---|---|---|---|
| always raw (the guard) | write 12,536 | read 12,536 ×5 | **21,938** |
| always compacted | write 8,726 | read 8,726 ×5 | **15,270** |

The guard made the cached prefix **30% larger for the life of every
session that pins its tools**, and every single read paid it — to avoid
a bust that never occurred.

## The hazard that *is* real, and why it doesn't justify the guard

Inconsistency — compacting on some turns and not others, which
`_decision.should_compress` can produce via a per-request
`x-headroom-bypass` header or a license gate flipping mid-session.

That's bounded. Anthropic keeps **both** prefixes alive, so alternating
turns each hit their own entry:

```
TOGGLING   turn 0  write 12,506
           turn 1  write  8,666
           turn 2  read  12,506
           turn 3  read   8,666      <- alternating hits, not busts
```

One extra write, once — not a bust per flip. Not worth a permanent 30%
tax.

## What this changes

* Removes the marker guard from `compact_tool_descriptions`.
* Leaves `compact_tools` (layer 1) unguarded **deliberately**. Extending
the guard there was the obvious symmetry and is what I started with; it
would have made things worse, because `normalize_message_cache_control`
runs *before* compaction, so the guard would have seen the breakpoint
**Headroom itself placed** and disabled tool compaction on essentially
every request.
* **Keeps the system-prompt guard from #3735 untouched.** The
determinism argument is specific to a pure byte transform; system
compaction goes through a pluggable compressor I have not established to
be deterministic.

## Testing

Byte-stability is now **asserted, not assumed** —
`test_compaction_is_byte_stable_across_calls` fails if compaction ever
stops being deterministic, which is the condition that would require the
guard back. 1125 tests pass across the compaction, cache-prefix,
transform and byte-faithful-forwarding suites; ruff clean.

Reproduction script used for the numbers is in the session scratchpad
(`compaction_cache.py`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-09-24 00:24:26 +02:00
committed by GitHub
co-authored by Claude Opus 5
parent 665b73df1b
commit a6a9cef945
2 changed files with 91 additions and 23 deletions
+22 -15
View File
@@ -395,21 +395,28 @@ def compact_tool_descriptions(
if not isinstance(tools, list) or not tools:
return payload, False, 0, 0
# A cache_control marker on any tool means "cache everything up to and
# including this one". Truncating a description inside that span changes the
# bytes the provider hashed, so the whole tools prefix -- and every message
# after it -- re-bills as cache creation. The saving is a few hundred bytes;
# the bust is the entire conversation. Mirrors the same guard on
# ``_sort_tools_deterministically`` (handlers/anthropic.py) and
# ``any_tool_has_cache_control`` in the Rust live zone.
marked = sum(1 for t in tools if isinstance(t, dict) and t.get("cache_control"))
if marked:
logger.info(
"event=tool_desc_compaction_skipped reason=marker_present tool_count=%d marked=%d",
len(tools),
marked,
)
return payload, False, 0, 0
# NO cache_control guard here, deliberately. The obvious reasoning -- "a
# marker means the provider hashed these bytes, so rewriting them busts the
# prefix" -- does not survive contact with the API, because the provider
# never saw the client's bytes. It only ever sees ours. Compaction is a pure
# deterministic transform, so turn 2 sends exactly what turn 1 cached and
# hits. Measured against the real API, 30 pinned tools over 6 turns:
#
# always raw write 12,536 then read 12,536 x5 billed 21,938
# always compacted write 8,726 then read 8,726 x5 billed 15,270 (-30%)
#
# Skipping made the cached prefix 30% larger for the life of every session
# that pins its tools, to avoid a bust that never happened.
#
# The one real hazard is INCONSISTENCY -- compacting on some turns and not
# others, which ``_decision.should_compress`` can produce via a per-request
# bypass header or a license gate flipping. That is bounded too: Anthropic
# keeps both prefixes alive, so alternating turns each hit their own entry
# and the cost is one extra write, once, not a bust per flip.
#
# This reasoning is specific to a pure byte transform over a fixed key set.
# It does NOT extend to system-prompt compaction, whose compressor is
# pluggable and not established to be deterministic -- that guard stays.
strip_sem = strip_semantic_params()
key = _cache_key(tools, "L2", max_chars, strip_sem)
+69 -8
View File
@@ -11,10 +11,11 @@ provider's key is the content, not the marker.
from __future__ import annotations
import json
from typing import Any
from headroom.proxy.system_compaction import _compact_system_blocks
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions
from headroom.proxy.tool_schema_compaction import compact_tool_descriptions, compact_tools
def _text(chars: int, seed: str = "x") -> str:
@@ -36,8 +37,16 @@ class _StubRouter:
# --------------------------------------------------------------------------
def test_tool_desc_compaction_skips_when_a_tool_is_marked() -> None:
"""A marker on any tool freezes the whole tools prefix."""
def test_tool_desc_compaction_runs_even_when_a_tool_is_marked() -> None:
"""A marker does NOT freeze the tools, and skipping was the expensive choice.
The provider never saw the client's bytes -- it only ever sees ours -- so
there is nothing to preserve. Compaction is deterministic, so every turn
sends the same compacted bytes and the cache hits. Measured against the real
API with 30 pinned tools over 6 turns, always-compacted billed 15,270 in
prefix tokens against 21,938 for always-raw: skipping made the cached prefix
30% larger for the whole session to avoid a bust that never happened.
"""
payload = {
"tools": [
{"name": "a", "description": _text(4000), "input_schema": {}},
@@ -49,13 +58,65 @@ def test_tool_desc_compaction_skips_when_a_tool_is_marked() -> None:
},
]
}
before = [dict(t) for t in payload["tools"]]
out, modified, _, _ = compact_tool_descriptions(payload, max_chars=50)
out, modified, before_bytes, after_bytes = compact_tool_descriptions(payload, max_chars=50)
assert modified is False
assert out is payload
assert out["tools"] == before
assert modified is True
assert after_bytes < before_bytes
# The marker itself is carried through untouched -- we shrink the schema,
# we do not take over the client's cache management.
assert out["tools"][1]["cache_control"] == {"type": "ephemeral"}
def test_compaction_is_byte_stable_across_calls() -> None:
"""The property the whole argument rests on.
Cache-safety here is not "we avoided the bytes", it is "we produce the same
bytes every time". If this ever stops holding, the removed guard has to come
back -- so it is asserted rather than assumed.
"""
def _payload() -> dict[str, Any]:
return {
"tools": [
{
"name": f"t{i}",
"description": " Lots of whitespace. " * 20,
"input_schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Args",
"examples": [{"a": 1}],
"type": "object",
"properties": {"a": {"type": "string", "description": " x y "}},
},
}
for i in range(5)
]
}
first, _, _, _ = compact_tools(_payload())
second, _, _, _ = compact_tools(_payload())
assert json.dumps(first["tools"], sort_keys=True) == json.dumps(second["tools"], sort_keys=True)
def test_a_marked_tools_array_is_compacted_by_layer_one_too() -> None:
"""Layer 1 never had a guard, and now it deliberately never will."""
payload = {
"tools": [
{
"name": "a",
"description": "d",
"input_schema": {"type": "object", "title": "T", "properties": {}},
"cache_control": {"type": "ephemeral"},
}
]
}
out, modified, _, _ = compact_tools(payload)
assert modified is True
assert "title" not in out["tools"][0]["input_schema"]
def test_tool_desc_compaction_still_runs_when_nothing_is_marked() -> None: