mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): count a tool_result with no text as replayable in the frozen prefix (#4523)
## Description In Claude Code sessions that load deferred tools, the Anthropic token-mode pipeline freezes only the messages before the first ToolSearch result, however far the provider's cache has moved. The log shows `Pipeline: freezing first 11/166 messages`, and the count stays there for the rest of the session. Every turn then re-runs compression over the whole history after that point, only for `finalize_turn` to overwrite the result with its replay of the provider-confirmed prefix. Read maturation also trusts that freeze, so it matures Reads the provider has already cached and the cache re-writes from the Read onward (#4522). Root cause: `CompressionCache.compute_frozen_count` (`headroom/cache/compression_cache.py:293`) stops at a tool_result whose extracted text is None (`compression_cache.py:323`, "treat as unstable"). `_extract_text_from_blocks` returns None for a block list without text (`compression_cache.py:52`): a ToolSearch `tool_reference` list, or an image. Nothing ever stores such a result or marks it stable, because `update_from_result` and `mark_stable_from_messages` skip None too. So it stays a miss for the whole session, and `prepare_turn`'s `min(tracker_frozen, cache_count)` (`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to it. With this change, a tool_result message with no text in any of its tool_results counts as stable: the cache has nothing to replay and the text pipeline has nothing to compress. A message that mixes a textless first result with a text one still stops the count. The cache keys a message by its first tool_result, so that text was never stored and cannot be byte-replayed; fixing that means keying every tool_result, which belongs in its own change. What it changes: in such sessions the pipeline freeze follows the provider-confirmed prefix again, so the pipeline compresses only the new turn instead of re-running the whole history each turn. Forwarded bytes inside the confirmed prefix are unchanged, because `finalize_turn` already replays them. Compression of new content is unchanged. Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token mode): - 299 of the 484 Anthropic requests in its current log froze more than 12 messages below the request size. - In a sample of that proxy's feed, every long session stopped at a ToolSearch result. Those holding only `tool_reference` blocks (11/139, 11/166, 11/173) are fixed here. One whose ToolSearch result shares a message with a text result (13/566 to 13/573) is the mixed case this PR leaves alone. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a tool_result message with no text as stable, through a new `_carries_tool_result_text` helper (both formats). It still stops at text it cannot replay, including text beside a textless first result. - `tests/test_compression_cache.py`: `tool_reference`, image and empty results no longer stop the count, and a text result beside a textless one still does. - `tests/test_session_engine.py`: `prepare_turn` follows the tracker past a ToolSearch result. The new tests fail on main. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q 90 passed in 4.28s # without the change, the 4 new tests fail $ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main> 16 failed, 6 passed # the same 16 fail on main (f10725e6) on this machine: tests/test_proxy_uds.py (11), # tests/test_cli/test_wrap_persistent.py (4), test_image_compression.py::TestOnnxRouter::test_full_classify_with_image # The full suite was not run locally on this branch; CI runs it. $ ruff check . && ruff format --check . All checks passed! 1834 files already formatted $ mypy headroom --ignore-missing-imports Found 2 errors in 2 files (checked 578 source files) # the same 2 as main, in files this PR does not touch: # integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389 ``` ## Real Behavior Proof - Environment: macOS 27 on an M2; this branch and upstream main (f10725e6), each from its own worktree; `headroom proxy` started with Extra Headroom's flags and env (token mode, read maturation on via the beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated HOME, telemetry off), in front of a local mock Anthropic upstream that reports cache_read for the longest message prefix shared with an earlier request and cache_write for the rest. - Exact command / steps: one scripted Claude Code session per build: a ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB file, then 8 Bash turns, each a 300-entry JSON listing that the router compresses. The pipeline freeze per request comes from the `Pipeline: freezing first N/M` log lines. Each forwarded request is diffed against the previous one with `cache_control` stripped. - Observed result: the freeze follows the confirmed prefix instead of staying at the ToolSearch result, and on the last turn the router processes 1 content-block message instead of 17. | Build | Pipeline freeze per request | Router content-block msgs, last turn | Cached prefix changed | cache_read/forwarded | Cache writes | tok_saved/tok_before | |---|---|---|---|---|---|---| | main | 2, 2, 2, 2, 2, 2, 2, 2, 2 | 17 | request 8, message 4 (a Read matured) | 67.0% | 39,360 | 43.2% | | this PR | 2, 4, 4, 8, 10, 12, 14, 16, 18 | 1 | never | 79.5% | 25,994 | 42.2% | - Not tested: Linux and Windows; real Anthropic traffic on this branch; the OpenAI token path and `/v1/compress`, which use the replayable policy, where a larger count only freezes more and cannot bust (their suites pass); a session where nothing compresses, where the freeze moves on only as far as the first uncompressed text result (2 to 4 in the same session with short Bash output), which is why #4522 is needed as well. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, not rollout-managed - Stable/default behavior changed: yes; in token mode the Anthropic pipeline now freezes past a tool_result with no text, up to the provider-confirmed prefix, instead of stopping there for the session. - Kill switch / disable path: none; revert the commit - Unsafe override required: no - Qualification impact: none expected; forwarded bytes inside the confirmed prefix were already replayed by finalize_turn - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Related: #4522 is the fix for the maturation rewrite and works on its own; this PR removes one of the two reasons the freeze sticks. The other is a text tool_result that never went through a pipeline run that changed something: `update_from_result` only marks unchanged results stable on such a run (`headroom/proxy/handlers/anthropic.py`, `if result.messages != working_messages`). That is a broader change, left for its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4523" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
55602fc1e2
commit
fba09f674c
Vendored
+28
-5
@@ -76,6 +76,22 @@ def _extract_tool_result_content(msg: dict) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def _carries_tool_result_text(msg: dict) -> bool:
|
||||
"""Whether any tool result in ``msg`` holds text (both formats)."""
|
||||
if msg.get("role") == "tool":
|
||||
results = [msg.get("content")]
|
||||
else:
|
||||
results = [
|
||||
b.get("content")
|
||||
for b in msg.get("content") or []
|
||||
if isinstance(b, dict) and b.get("type") == "tool_result"
|
||||
]
|
||||
return any(
|
||||
isinstance(c, str) or (isinstance(c, list) and _extract_text_from_blocks(c) is not None)
|
||||
for c in results
|
||||
)
|
||||
|
||||
|
||||
def _swap_tool_result_content(msg: dict, new_content: str) -> dict:
|
||||
"""Deep copy msg and replace tool result content with new_content."""
|
||||
new_msg = copy.deepcopy(msg)
|
||||
@@ -294,9 +310,10 @@ class CompressionCache:
|
||||
"""Count consecutive stable messages from the start.
|
||||
|
||||
A message is stable if it is a plain user/assistant/system message,
|
||||
an assistant message with tool_use blocks, or a tool_result whose
|
||||
content hash is already in the cache. The first unstable tool_result
|
||||
(cache miss) stops the count.
|
||||
an assistant message with tool_use blocks, a tool_result whose
|
||||
content hash is already in the cache, or a tool_result with no text
|
||||
(nothing to replay). The first unstable tool_result (cache miss)
|
||||
stops the count.
|
||||
|
||||
The trailing message is *always* excluded from the frozen prefix
|
||||
(cap of ``len(messages) - 1``). The trailing message represents
|
||||
@@ -319,9 +336,15 @@ class CompressionCache:
|
||||
h = self.content_hash(content)
|
||||
if h not in self._cache and h not in self._stable_hashes:
|
||||
break
|
||||
else:
|
||||
# tool_result with non-string content; treat as unstable
|
||||
elif _carries_tool_result_text(msg):
|
||||
# Text after a textless first tool_result: the cache
|
||||
# keys a message by its first one, so this text was
|
||||
# never stored and cannot be byte-replayed.
|
||||
break
|
||||
# A tool_result with no text at all (a ToolSearch
|
||||
# tool_reference list, an image) gives the cache nothing
|
||||
# to replay, so it is stable. As a miss it stopped the
|
||||
# count for the rest of the session.
|
||||
# Regular user/assistant/system messages and assistant+tool_use
|
||||
# are always stable — fall through.
|
||||
count += 1
|
||||
|
||||
@@ -310,6 +310,60 @@ class TestCompressionCacheFrozenCount:
|
||||
# count = 3, then capped at len-1 = 2 (live-zone reservation).
|
||||
assert cache.compute_frozen_count(messages) == 2
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"blocks",
|
||||
[
|
||||
[{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
|
||||
[
|
||||
{
|
||||
"type": "image",
|
||||
"source": {"type": "base64", "media_type": "image/png", "data": "iVBO"},
|
||||
}
|
||||
],
|
||||
[],
|
||||
],
|
||||
ids=["tool-reference", "image", "empty"],
|
||||
)
|
||||
def test_tool_result_without_text_does_not_stop_frozen(
|
||||
self, cache: CompressionCache, blocks: list
|
||||
) -> None:
|
||||
"""Claude Code's ToolSearch answers with tool_reference blocks only. The
|
||||
cache has nothing to replay for such a result; as a miss it stopped the
|
||||
count at that message for the rest of the session."""
|
||||
messages = [
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": blocks}],
|
||||
},
|
||||
{"role": "assistant", "content": "found it"},
|
||||
{"role": "user", "content": "follow up"},
|
||||
]
|
||||
assert cache.compute_frozen_count(messages) == 3
|
||||
|
||||
def test_text_beside_a_textless_tool_result_still_stops_frozen(
|
||||
self, cache: CompressionCache
|
||||
) -> None:
|
||||
"""Parallel calls put several tool_results in one message, and the cache
|
||||
keys it by the first. Text after a textless first result was never
|
||||
stored, so it cannot be replayed."""
|
||||
messages = [
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "t1",
|
||||
"content": [{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
|
||||
},
|
||||
{"type": "tool_result", "tool_use_id": "t2", "content": "uncached output"},
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "follow up"},
|
||||
]
|
||||
assert cache.compute_frozen_count(messages) == 1
|
||||
|
||||
def test_update_from_result_identical_content_marks_stable(
|
||||
self, cache: CompressionCache
|
||||
) -> None:
|
||||
|
||||
@@ -89,6 +89,29 @@ def test_proxy_policy_clamps_by_cache_count() -> None:
|
||||
assert prep.frozen_message_count == 3
|
||||
|
||||
|
||||
def test_proxy_policy_follows_the_tracker_past_a_tool_search_result() -> None:
|
||||
"""A ToolSearch result holds only tool_reference blocks. It used to stop
|
||||
the cache count, so the freeze sat there for the whole session and the
|
||||
pipeline re-ran everything after it on every turn."""
|
||||
cache = CompressionCache()
|
||||
search = {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "ts",
|
||||
"content": [{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
|
||||
}
|
||||
],
|
||||
}
|
||||
messages = [{"role": "user", "content": "find the notes tool"}, search]
|
||||
messages += [{"role": "assistant", "content": f"step {i}"} for i in range(6)]
|
||||
messages.append({"role": "user", "content": "next"})
|
||||
|
||||
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=8)
|
||||
assert prep.frozen_message_count == 8
|
||||
|
||||
|
||||
def test_proxy_policy_clamps_by_tracker() -> None:
|
||||
"""Local state could replay 3, but the provider only confirmed 1: content
|
||||
past the confirmed prefix stays compressible (the #327 posture)."""
|
||||
|
||||
Reference in New Issue
Block a user