Files
headroom/tests/test_session_engine.py
fba09f674c fix(cache): count a tool_result with no text as replayable in the frozen prefix (#4523)
## Description

In Claude Code sessions that load deferred tools, the Anthropic
token-mode pipeline freezes only the messages before the first
ToolSearch result, however far the provider's cache has moved. The log
shows `Pipeline: freezing first 11/166 messages`, and the count stays
there for the rest of the session. Every turn then re-runs compression
over the whole history after that point, only for `finalize_turn` to
overwrite the result with its replay of the provider-confirmed prefix.
Read maturation also trusts that freeze, so it matures Reads the
provider has already cached and the cache re-writes from the Read onward
(#4522).

Root cause: `CompressionCache.compute_frozen_count`
(`headroom/cache/compression_cache.py:293`) stops at a tool_result whose
extracted text is None (`compression_cache.py:323`, "treat as
unstable"). `_extract_text_from_blocks` returns None for a block list
without text (`compression_cache.py:52`): a ToolSearch `tool_reference`
list, or an image. Nothing ever stores such a result or marks it stable,
because `update_from_result` and `mark_stable_from_messages` skip None
too. So it stays a miss for the whole session, and `prepare_turn`'s
`min(tracker_frozen, cache_count)`
(`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to
it.

With this change, a tool_result message with no text in any of its
tool_results counts as stable: the cache has nothing to replay and the
text pipeline has nothing to compress. A message that mixes a textless
first result with a text one still stops the count. The cache keys a
message by its first tool_result, so that text was never stored and
cannot be byte-replayed; fixing that means keying every tool_result,
which belongs in its own change.

What it changes: in such sessions the pipeline freeze follows the
provider-confirmed prefix again, so the pipeline compresses only the new
turn instead of re-running the whole history each turn. Forwarded bytes
inside the confirmed prefix are unchanged, because `finalize_turn`
already replays them. Compression of new content is unchanged.

Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token
mode):
- 299 of the 484 Anthropic requests in its current log froze more than
12 messages below the request size.
- In a sample of that proxy's feed, every long session stopped at a
ToolSearch result. Those holding only `tool_reference` blocks (11/139,
11/166, 11/173) are fixed here. One whose ToolSearch result shares a
message with a text result (13/566 to 13/573) is the mixed case this PR
leaves alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a
tool_result message with no text as stable, through a new
`_carries_tool_result_text` helper (both formats). It still stops at
text it cannot replay, including text beside a textless first result.
- `tests/test_compression_cache.py`: `tool_reference`, image and empty
results no longer stop the count, and a text result beside a textless
one still does.
- `tests/test_session_engine.py`: `prepare_turn` follows the tracker
past a ToolSearch result. The new tests fail on main.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q
90 passed in 4.28s
# without the change, the 4 new tests fail

$ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main>
16 failed, 6 passed
# the same 16 fail on main (f10725e6) on this machine: tests/test_proxy_uds.py (11),
# tests/test_cli/test_wrap_persistent.py (4), test_image_compression.py::TestOnnxRouter::test_full_classify_with_image
# The full suite was not run locally on this branch; CI runs it.

$ ruff check . && ruff format --check .
All checks passed!
1834 files already formatted

$ mypy headroom --ignore-missing-imports
Found 2 errors in 2 files (checked 578 source files)
# the same 2 as main, in files this PR does not touch:
# integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389
```

## Real Behavior Proof

- Environment: macOS 27 on an M2; this branch and upstream main
(f10725e6), each from its own worktree; `headroom proxy` started with
Extra Headroom's flags and env (token mode, read maturation on via the
beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated
HOME, telemetry off), in front of a local mock Anthropic upstream that
reports cache_read for the longest message prefix shared with an earlier
request and cache_write for the rest.
- Exact command / steps: one scripted Claude Code session per build: a
ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB
file, then 8 Bash turns, each a 300-entry JSON listing that the router
compresses. The pipeline freeze per request comes from the `Pipeline:
freezing first N/M` log lines. Each forwarded request is diffed against
the previous one with `cache_control` stripped.
- Observed result: the freeze follows the confirmed prefix instead of
staying at the ToolSearch result, and on the last turn the router
processes 1 content-block message instead of 17.

| Build | Pipeline freeze per request | Router content-block msgs, last
turn | Cached prefix changed | cache_read/forwarded | Cache writes |
tok_saved/tok_before |
  |---|---|---|---|---|---|---|
| main | 2, 2, 2, 2, 2, 2, 2, 2, 2 | 17 | request 8, message 4 (a Read
matured) | 67.0% | 39,360 | 43.2% |
| this PR | 2, 4, 4, 8, 10, 12, 14, 16, 18 | 1 | never | 79.5% | 25,994
| 42.2% |
- Not tested: Linux and Windows; real Anthropic traffic on this branch;
the OpenAI token path and `/v1/compress`, which use the replayable
policy, where a larger count only freezes more and cannot bust (their
suites pass); a session where nothing compresses, where the freeze moves
on only as far as the first uncompressed text result (2 to 4 in the same
session with short Bash output), which is why #4522 is needed as well.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; in token mode the Anthropic
pipeline now freezes past a tool_result with no text, up to the
provider-confirmed prefix, instead of stopping there for the session.
- Kill switch / disable path: none; revert the commit
- Unsafe override required: no
- Qualification impact: none expected; forwarded bytes inside the
confirmed prefix were already replayed by finalize_turn
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Related: #4522 is the fix for the maturation rewrite and works on its
own; this PR removes one of the two reasons the freeze sticks. The other
is a text tool_result that never went through a pipeline run that
changed something: `update_from_result` only marks unchanged results
stable on such a run (`headroom/proxy/handlers/anthropic.py`, `if
result.messages != working_messages`). That is a broader change, left
for its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4523"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-28 23:45:04 -05:00

279 lines
11 KiB
Python

"""Unit tests for the shared session-turn engine (headroom/proxy/session_engine).
The engine is the single cache-management brain for the proxy request paths
and the sidecar /v1/compress path; these tests pin its two freeze policies
and the overlay finalization directly, without an HTTP harness.
"""
from __future__ import annotations
import json
import pytest
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import (
FREEZE_POLICY_CONFIRMED_CLAMP,
FREEZE_POLICY_REPLAYABLE,
finalize_turn,
prepare_turn,
)
def _tool_msg(content: str, call_id: str = "c1") -> dict:
return {"role": "tool", "tool_call_id": call_id, "content": content}
def _history_with_cached_tool(
cache: CompressionCache, original: str, compressed: str
) -> list[dict]:
"""A 3-message history whose tool result has a cached compressed form."""
cache.store_compressed(cache.content_hash(original), compressed, tokens_saved=10)
return [
{"role": "user", "content": "get items"},
{"role": "assistant", "content": "calling"},
_tool_msg(original),
]
# --------------------------------------------------------------------------- #
# prepare_turn: freeze policies #
# --------------------------------------------------------------------------- #
def test_sidecar_policy_freezes_full_replayable_prefix() -> None:
cache = CompressionCache()
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
# user, assistant, cached tool are all stable; the trailing message is
# always excluded by compute_frozen_count.
assert prep.frozen_message_count == 3
# The swap replaced the tool result with its cached compressed form.
assert prep.pipeline_input[2]["content"] == "[compressed]"
# The caller's list is never mutated.
assert messages[2]["content"].startswith("ORIGINAL")
def test_sidecar_policy_explicit_pin_wins_when_larger() -> None:
cache = CompressionCache()
messages = [
{"role": "user", "content": "a"},
_tool_msg("never seen before " * 50), # not in cache -> derived stops here
{"role": "user", "content": "next"},
]
derived = cache.compute_frozen_count(messages)
assert derived == 1 # only the leading plain message
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE, explicit_frozen=2)
assert prep.frozen_message_count == 2
def test_sidecar_policy_derived_wins_when_explicit_smaller() -> None:
cache = CompressionCache()
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE, explicit_frozen=1)
assert prep.frozen_message_count == 3
def test_proxy_policy_clamps_by_cache_count() -> None:
"""Provider says 5 messages are cached, but local state can only replay 3:
freezing past the replayable bound would forward raw bytes."""
cache = CompressionCache()
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
messages.append(_tool_msg("uncached " * 50, "c2"))
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=5)
assert prep.frozen_message_count == 3
def test_proxy_policy_follows_the_tracker_past_a_tool_search_result() -> None:
"""A ToolSearch result holds only tool_reference blocks. It used to stop
the cache count, so the freeze sat there for the whole session and the
pipeline re-ran everything after it on every turn."""
cache = CompressionCache()
search = {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "ts",
"content": [{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
}
],
}
messages = [{"role": "user", "content": "find the notes tool"}, search]
messages += [{"role": "assistant", "content": f"step {i}"} for i in range(6)]
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=8)
assert prep.frozen_message_count == 8
def test_proxy_policy_clamps_by_tracker() -> None:
"""Local state could replay 3, but the provider only confirmed 1: content
past the confirmed prefix stays compressible (the #327 posture)."""
cache = CompressionCache()
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=1)
assert prep.frozen_message_count == 1
def test_proxy_policy_none_tracker_freezes_nothing() -> None:
cache = CompressionCache()
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=None)
assert prep.frozen_message_count == 0
def test_unknown_policy_rejected() -> None:
cache = CompressionCache()
with pytest.raises(ValueError):
prepare_turn(cache, [], policy="wat")
def test_prepare_marks_frozen_tool_results_stable() -> None:
cache = CompressionCache()
original = "ORIGINAL " * 100
messages = _history_with_cached_tool(cache, original, "[compressed]")
messages.append({"role": "user", "content": "next"})
prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
assert cache.content_hash(original) in cache._stable_hashes
# --------------------------------------------------------------------------- #
# finalize_turn: overlay + recount hook #
# --------------------------------------------------------------------------- #
def _prev_pair() -> tuple[list[dict], list[dict]]:
prev_original = [
{"role": "user", "content": "ORIGINAL " * 100},
{"role": "assistant", "content": "ok"},
]
prev_returned = [
{"role": "user", "content": "[returned-form]"},
{"role": "assistant", "content": "ok"},
]
return prev_original, prev_returned
def test_finalize_replays_previous_returned_prefix() -> None:
prev_original, prev_returned = _prev_pair()
current = prev_original + [{"role": "user", "content": "next"}]
# The pipeline "drifted": it emitted the raw original for message 0.
drifted = [dict(m) for m in current]
counted: list[int] = []
def _count(msgs: list[dict]) -> int:
counted.append(len(json.dumps(msgs)))
return 42
turn = finalize_turn(drifted, current, prev_original, prev_returned, count_tokens=_count)
assert turn.replayed
assert turn.messages[0]["content"] == "[returned-form]"
assert turn.messages[-1]["content"] == "next"
assert turn.tokens == 42
assert len(counted) == 1
def test_finalize_noop_without_prev_snapshots() -> None:
current = [{"role": "user", "content": "hi"}]
calls: list[int] = []
turn = finalize_turn(current, current, [], [], count_tokens=lambda m: calls.append(1) or 1)
assert not turn.replayed
assert turn.messages == current
assert turn.tokens is None
assert not calls # count_tokens only runs when the overlay fired
def test_finalize_count_hook_failure_falls_back() -> None:
prev_original, prev_returned = _prev_pair()
current = prev_original + [{"role": "user", "content": "next"}]
def _boom(_msgs: list[dict]) -> int:
raise RuntimeError("tokenizer down")
turn = finalize_turn(
[dict(m) for m in current], current, prev_original, prev_returned, count_tokens=_boom
)
assert turn.replayed
assert turn.tokens is None
def test_finalize_declines_inflating_replay_by_default() -> None:
prev_original, prev_returned = _prev_pair()
current = prev_original + [{"role": "user", "content": "next"}]
# The pipeline recompressed message 0 SMALLER than the returned form.
recompressed = [
{"role": "user", "content": "[t]"},
{"role": "assistant", "content": "ok"},
{"role": "user", "content": "next"},
]
turn = finalize_turn(recompressed, current, prev_original, prev_returned)
assert not turn.replayed
assert turn.messages == recompressed
def test_finalize_confirmed_floor_replays_confirmed_bytes() -> None:
prev_original, prev_returned = _prev_pair()
current = prev_original + [{"role": "user", "content": "next"}]
recompressed = [
{"role": "user", "content": "[t]"},
{"role": "assistant", "content": "ok"},
{"role": "user", "content": "next"},
]
turn = finalize_turn(
recompressed, current, prev_original, prev_returned, confirmed_frozen_count=2
)
assert turn.replayed
assert turn.messages[0]["content"] == "[returned-form]"
assert turn.messages[-1]["content"] == "next"
# --------------------------------------------------------------------------- #
# OpenAI proxy token-path migration: formula identity + marking benefit. #
# --------------------------------------------------------------------------- #
def test_replayable_without_pin_equals_bare_cache_count() -> None:
"""The OpenAI proxy token path historically froze on compute_frozen_count
alone; REPLAYABLE with no explicit pin must be formula-identical, so its
migration onto the engine is a pure extraction."""
cache = CompressionCache()
messages = _history_with_cached_tool(cache, "AAAA " * 50, "[c1]")
messages.append(_tool_msg("uncached content", call_id="c2"))
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
assert prep.frozen_message_count == cache.compute_frozen_count(messages)
# And that count stops at the uncached tool_result (index 3).
assert prep.frozen_message_count == 3
def test_marking_preserves_freeze_across_entry_eviction() -> None:
"""The one real benefit mark_stable_from_messages adds on the migrated
path: an in-prefix tool_result stays stable via `_stable_hashes` even
after its compressed ENTRY is evicted by the per-cache LRU, so the frozen
count does not collapse at that position on the next turn."""
cache = CompressionCache(max_entries=100)
original = "BBBB " * 50
messages = _history_with_cached_tool(cache, original, "[c1]")
messages.append({"role": "user", "content": "next"})
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
assert prep.frozen_message_count == 3 # tool in prefix, marked stable
# Simulate entry LRU turnover: the compressed entry disappears.
h = cache.content_hash(original)
with cache._lock:
cache._cache.pop(h, None)
# Without marking, the frozen count would collapse to 2 here; the
# stable-hash record keeps the position frozen.
assert cache.compute_frozen_count(messages) == 3