fix(cache): count a tool_result with no text as replayable in the frozen prefix (#4523)

## Description

In Claude Code sessions that load deferred tools, the Anthropic
token-mode pipeline freezes only the messages before the first
ToolSearch result, however far the provider's cache has moved. The log
shows `Pipeline: freezing first 11/166 messages`, and the count stays
there for the rest of the session. Every turn then re-runs compression
over the whole history after that point, only for `finalize_turn` to
overwrite the result with its replay of the provider-confirmed prefix.
Read maturation also trusts that freeze, so it matures Reads the
provider has already cached and the cache re-writes from the Read onward
(#4522).

Root cause: `CompressionCache.compute_frozen_count`
(`headroom/cache/compression_cache.py:293`) stops at a tool_result whose
extracted text is None (`compression_cache.py:323`, "treat as
unstable"). `_extract_text_from_blocks` returns None for a block list
without text (`compression_cache.py:52`): a ToolSearch `tool_reference`
list, or an image. Nothing ever stores such a result or marks it stable,
because `update_from_result` and `mark_stable_from_messages` skip None
too. So it stays a miss for the whole session, and `prepare_turn`'s
`min(tracker_frozen, cache_count)`
(`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to
it.

With this change, a tool_result message with no text in any of its
tool_results counts as stable: the cache has nothing to replay and the
text pipeline has nothing to compress. A message that mixes a textless
first result with a text one still stops the count. The cache keys a
message by its first tool_result, so that text was never stored and
cannot be byte-replayed; fixing that means keying every tool_result,
which belongs in its own change.

What it changes: in such sessions the pipeline freeze follows the
provider-confirmed prefix again, so the pipeline compresses only the new
turn instead of re-running the whole history each turn. Forwarded bytes
inside the confirmed prefix are unchanged, because `finalize_turn`
already replays them. Compression of new content is unchanged.

Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token
mode):
- 299 of the 484 Anthropic requests in its current log froze more than
12 messages below the request size.
- In a sample of that proxy's feed, every long session stopped at a
ToolSearch result. Those holding only `tool_reference` blocks (11/139,
11/166, 11/173) are fixed here. One whose ToolSearch result shares a
message with a text result (13/566 to 13/573) is the mixed case this PR
leaves alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a
tool_result message with no text as stable, through a new
`_carries_tool_result_text` helper (both formats). It still stops at
text it cannot replay, including text beside a textless first result.
- `tests/test_compression_cache.py`: `tool_reference`, image and empty
results no longer stop the count, and a text result beside a textless
one still does.
- `tests/test_session_engine.py`: `prepare_turn` follows the tracker
past a ToolSearch result. The new tests fail on main.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q
90 passed in 4.28s
# without the change, the 4 new tests fail

$ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main>
16 failed, 6 passed
# the same 16 fail on main (f10725e6) on this machine: tests/test_proxy_uds.py (11),
# tests/test_cli/test_wrap_persistent.py (4), test_image_compression.py::TestOnnxRouter::test_full_classify_with_image
# The full suite was not run locally on this branch; CI runs it.

$ ruff check . && ruff format --check .
All checks passed!
1834 files already formatted

$ mypy headroom --ignore-missing-imports
Found 2 errors in 2 files (checked 578 source files)
# the same 2 as main, in files this PR does not touch:
# integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389
```

## Real Behavior Proof

- Environment: macOS 27 on an M2; this branch and upstream main
(f10725e6), each from its own worktree; `headroom proxy` started with
Extra Headroom's flags and env (token mode, read maturation on via the
beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated
HOME, telemetry off), in front of a local mock Anthropic upstream that
reports cache_read for the longest message prefix shared with an earlier
request and cache_write for the rest.
- Exact command / steps: one scripted Claude Code session per build: a
ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB
file, then 8 Bash turns, each a 300-entry JSON listing that the router
compresses. The pipeline freeze per request comes from the `Pipeline:
freezing first N/M` log lines. Each forwarded request is diffed against
the previous one with `cache_control` stripped.
- Observed result: the freeze follows the confirmed prefix instead of
staying at the ToolSearch result, and on the last turn the router
processes 1 content-block message instead of 17.

| Build | Pipeline freeze per request | Router content-block msgs, last
turn | Cached prefix changed | cache_read/forwarded | Cache writes |
tok_saved/tok_before |
  |---|---|---|---|---|---|---|
| main | 2, 2, 2, 2, 2, 2, 2, 2, 2 | 17 | request 8, message 4 (a Read
matured) | 67.0% | 39,360 | 43.2% |
| this PR | 2, 4, 4, 8, 10, 12, 14, 16, 18 | 1 | never | 79.5% | 25,994
| 42.2% |
- Not tested: Linux and Windows; real Anthropic traffic on this branch;
the OpenAI token path and `/v1/compress`, which use the replayable
policy, where a larger count only freezes more and cannot bust (their
suites pass); a session where nothing compresses, where the freeze moves
on only as far as the first uncompressed text result (2 to 4 in the same
session with short Bash output), which is why #4522 is needed as well.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; in token mode the Anthropic
pipeline now freezes past a tool_result with no text, up to the
provider-confirmed prefix, instead of stopping there for the session.
- Kill switch / disable path: none; revert the commit
- Unsafe override required: no
- Qualification impact: none expected; forwarded bytes inside the
confirmed prefix were already replayed by finalize_turn
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Related: #4522 is the fix for the maturation rewrite and works on its
own; this PR removes one of the two reasons the freeze sticks. The other
is a text tool_result that never went through a pipeline run that
changed something: `update_from_result` only marks unchanged results
stable on such a run (`headroom/proxy/handlers/anthropic.py`, `if
result.messages != working_messages`). That is a broader change, left
for its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4523"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-28 23:45:04 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 55602fc1e2
commit fba09f674c
3 changed files with 105 additions and 5 deletions
+28 -5
View File
@@ -76,6 +76,22 @@ def _extract_tool_result_content(msg: dict) -> str | None:
return None
def _carries_tool_result_text(msg: dict) -> bool:
"""Whether any tool result in ``msg`` holds text (both formats)."""
if msg.get("role") == "tool":
results = [msg.get("content")]
else:
results = [
b.get("content")
for b in msg.get("content") or []
if isinstance(b, dict) and b.get("type") == "tool_result"
]
return any(
isinstance(c, str) or (isinstance(c, list) and _extract_text_from_blocks(c) is not None)
for c in results
)
def _swap_tool_result_content(msg: dict, new_content: str) -> dict:
"""Deep copy msg and replace tool result content with new_content."""
new_msg = copy.deepcopy(msg)
@@ -294,9 +310,10 @@ class CompressionCache:
"""Count consecutive stable messages from the start.
A message is stable if it is a plain user/assistant/system message,
an assistant message with tool_use blocks, or a tool_result whose
content hash is already in the cache. The first unstable tool_result
(cache miss) stops the count.
an assistant message with tool_use blocks, a tool_result whose
content hash is already in the cache, or a tool_result with no text
(nothing to replay). The first unstable tool_result (cache miss)
stops the count.
The trailing message is *always* excluded from the frozen prefix
(cap of ``len(messages) - 1``). The trailing message represents
@@ -319,9 +336,15 @@ class CompressionCache:
h = self.content_hash(content)
if h not in self._cache and h not in self._stable_hashes:
break
else:
# tool_result with non-string content; treat as unstable
elif _carries_tool_result_text(msg):
# Text after a textless first tool_result: the cache
# keys a message by its first one, so this text was
# never stored and cannot be byte-replayed.
break
# A tool_result with no text at all (a ToolSearch
# tool_reference list, an image) gives the cache nothing
# to replay, so it is stable. As a miss it stopped the
# count for the rest of the session.
# Regular user/assistant/system messages and assistant+tool_use
# are always stable — fall through.
count += 1
+54
View File
@@ -310,6 +310,60 @@ class TestCompressionCacheFrozenCount:
# count = 3, then capped at len-1 = 2 (live-zone reservation).
assert cache.compute_frozen_count(messages) == 2
@pytest.mark.parametrize(
"blocks",
[
[{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
[
{
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": "iVBO"},
}
],
[],
],
ids=["tool-reference", "image", "empty"],
)
def test_tool_result_without_text_does_not_stop_frozen(
self, cache: CompressionCache, blocks: list
) -> None:
"""Claude Code's ToolSearch answers with tool_reference blocks only. The
cache has nothing to replay for such a result; as a miss it stopped the
count at that message for the rest of the session."""
messages = [
{"role": "user", "content": "hello"},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": blocks}],
},
{"role": "assistant", "content": "found it"},
{"role": "user", "content": "follow up"},
]
assert cache.compute_frozen_count(messages) == 3
def test_text_beside_a_textless_tool_result_still_stops_frozen(
self, cache: CompressionCache
) -> None:
"""Parallel calls put several tool_results in one message, and the cache
keys it by the first. Text after a textless first result was never
stored, so it cannot be replayed."""
messages = [
{"role": "user", "content": "hello"},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "t1",
"content": [{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
},
{"type": "tool_result", "tool_use_id": "t2", "content": "uncached output"},
],
},
{"role": "user", "content": "follow up"},
]
assert cache.compute_frozen_count(messages) == 1
def test_update_from_result_identical_content_marks_stable(
self, cache: CompressionCache
) -> None:
+23
View File
@@ -89,6 +89,29 @@ def test_proxy_policy_clamps_by_cache_count() -> None:
assert prep.frozen_message_count == 3
def test_proxy_policy_follows_the_tracker_past_a_tool_search_result() -> None:
"""A ToolSearch result holds only tool_reference blocks. It used to stop
the cache count, so the freeze sat there for the whole session and the
pipeline re-ran everything after it on every turn."""
cache = CompressionCache()
search = {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "ts",
"content": [{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
}
],
}
messages = [{"role": "user", "content": "find the notes tool"}, search]
messages += [{"role": "assistant", "content": f"step {i}"} for i in range(6)]
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=8)
assert prep.frozen_message_count == 8
def test_proxy_policy_clamps_by_tracker() -> None:
"""Local state could replay 3, but the provider only confirmed 1: content
past the confirmed prefix stays compressible (the #327 posture)."""