Files
headroom/tests/test_cache_replay_shapes.py
Yuchuan HeandJerrettDavis 973c12d2af fix(cache): replay every parallel tool result (#4818)
## Description

An Anthropic user message can hold several parallel `tool_result`
blocks. The session compression cache recorded and replayed only the
first one, but a hit for that result could freeze the whole message.
Later results then reached the provider in their raw form instead of the
previously compressed form.

Visit every result when recording, replaying and marking stable content.
Freeze a message only if every result is replayable or textless. Reuse
the existing single-result shape guard, so images, tool references,
multiblock boundaries and text-block metadata retain their established
protection. Cache seeding checks the paired tool-call IDs before
associating original and compressed text.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Cache and replay each parallel result independently without mutating
caller input.
- Stop the frozen prefix on a later miss or unsupported text layout.
- Cover a textless first result, later text-block metadata, changed IDs
and partial replay. Update the existing later-result limitation test to
assert actual replay.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
New regressions against original production code: 5 failed, 1 passed
pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_cache_replay_shapes.py tests/test_token_headroom_mode.py tests/test_cache_prefix_safety.py tests/test_proxy_anthropic_cache_stability.py tests/test_prefix_tracker_restart.py:
  189 passed
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs. The restart tests also report the
environment's existing Starlette/httpx deprecation warning.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`a09ad47dc8ceb04b42858a67eecb83a7ee246485`. Explicit checkout on
PYTHONPATH; no provider credentials. Headroom runs offline; the parallel
proof uses a real SDK to send only to its bound loopback receiver.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.

```python
"""Production cache/session APIs followed by real Anthropic SDK loopback HTTP."""
import copy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import threading
from anthropic import Anthropic
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import FREEZE_POLICY_REPLAYABLE, prepare_turn

captured = []
class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        captured.append(json.loads(self.rfile.read(int(self.headers['Content-Length']))))
        data = json.dumps({'id':'local','type':'message','role':'assistant','model':'local-fixture','content':[{'type':'text','text':'received'}],'stop_reason':'end_turn','stop_sequence':None,'usage':{'input_tokens':1,'output_tokens':1}}).encode()
        self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
    def log_message(self,*args): pass

def message(a,b):
    return {'role':'user','content':[{'type':'tool_result','tool_use_id':'t1','content':a},{'type':'tool_result','tool_use_id':'t2','content':b}]}

server = ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread = threading.Thread(target=server.serve_forever,daemon=True);thread.start()
raw = message('first original '*20,[{'type':'text','text':'second original '*20,'cache_control':{'type':'ephemeral'}}])
short = message('first compressed',[{'type':'text','text':'second compressed','cache_control':{'type':'ephemeral'}}])
messages=[{'role':'assistant','content':[{'type':'tool_use','id':'t1','name':'inspect','input':{}},{'type':'tool_use','id':'t2','name':'inspect','input':{}}]},raw,{'role':'user','content':'continue'}]
before = copy.deepcopy(messages)
cache = CompressionCache();cache.update_from_result([raw],[short])
prep = prepare_turn(cache,messages,policy=FREEZE_POLICY_REPLAYABLE)
try:
    with Anthropic(api_key='local-fixture',base_url=f'http://127.0.0.1:{server.server_port}',max_retries=0) as client:
        client.messages.create(model='local-fixture',max_tokens=16,messages=prep.pipeline_input)
    wire = captured[-1]['messages'][1]
    print(json.dumps({'all_results_replayed':wire==short,'second_result':wire['content'][1]['content'],'frozen_count':prep.frozen_message_count,'caller_unchanged':messages==before,'requests':len(captured)},indent=2))
finally:
    server.shutdown();server.server_close();thread.join(5)
```

- Observed result: With the same original/compressed message pair, clean
main freezes the two-message prefix but the real Anthropic SDK sends the
second result as its original text. With the patch, the HTTP receiver
gets both compressed results, including the second text block's
cache_control; frozen_count remains 2 and caller_unchanged remains true.
- Not tested: Live Anthropic service acceptance, model inference,
billing, task accuracy, or the complete Python suite. The receiver
records actual SDK JSON and returns a synthetic response.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Cached replay covers every tool
result in a message; a later cache miss or unsupported shape keeps that
message mutable.
- Kill switch / disable path: Revert this commit to restore the prior
replay behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Developed and validated with Codex assistance. I reviewed the changes
and runtime evidence.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4818"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:00:39 -06:00

168 lines
7.0 KiB
Python

"""Freeze eligibility must match the bytes the cache can actually replay."""
from __future__ import annotations
import copy
import unittest
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import (
FREEZE_POLICY_CONFIRMED_CLAMP,
FREEZE_POLICY_REPLAYABLE,
prepare_turn,
)
class TestCacheReplayShapes(unittest.TestCase):
def setUp(self) -> None:
self.cache = CompressionCache()
self.cache.store_compressed(self.cache.content_hash("A\nB"), "short", 1)
self.blocks = [{"type": "text", "text": "A"}, {"type": "text", "text": "B"}]
self.messages = [
{"role": "tool", "tool_call_id": "c1", "content": self.blocks},
{"role": "user", "content": "next"},
]
def test_multiblock_cache_hit_remains_mutable(self) -> None:
before = copy.deepcopy(self.messages)
prep = prepare_turn(self.cache, self.messages, policy=FREEZE_POLICY_REPLAYABLE)
self.assertEqual(prep.frozen_message_count, 0)
self.assertEqual(prep.pipeline_input, before)
self.assertEqual(self.messages, before)
self.assertEqual(self.cache.get_stats()["hits"], 0)
self.assertNotIn(self.cache.content_hash("A\nB"), self.cache._stable_hashes)
def test_stable_joined_hash_does_not_prove_shape_replay(self) -> None:
self.cache.mark_stable(self.cache.content_hash("A\nB"))
self.assertEqual(self.cache.compute_frozen_count(self.messages), 0)
def test_prefix_stops_before_unsupported_shape(self) -> None:
messages = [{"role": "user", "content": "earlier"}, *self.messages]
self.assertEqual(self.cache.compute_frozen_count(messages), 1)
def test_confirmed_policy_is_still_clamped_by_replayability(self) -> None:
prep = prepare_turn(
self.cache,
self.messages,
policy=FREEZE_POLICY_CONFIRMED_CLAMP,
tracker_frozen=1,
)
self.assertEqual(prep.frozen_message_count, 0)
def test_explicit_caller_pin_still_wins(self) -> None:
prep = prepare_turn(
self.cache, self.messages, policy=FREEZE_POLICY_REPLAYABLE, explicit_frozen=1
)
self.assertEqual(prep.frozen_message_count, 1)
self.assertEqual(prep.pipeline_input, self.messages)
def test_supported_string_and_single_block_replay(self) -> None:
for content in ["A\nB", [{"type": "text", "text": "A\nB", "custom": [1]}]]:
with self.subTest(content=content):
messages = [{"role": "tool", "content": content}, self.messages[-1]]
prep = prepare_turn(self.cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
self.assertEqual(prep.frozen_message_count, 1)
expected = (
"short"
if isinstance(content, str)
else [{"type": "text", "text": "short", "custom": [1]}]
)
self.assertEqual(prep.pipeline_input[0]["content"], expected)
def test_anthropic_replay_contract_is_unchanged(self) -> None:
content = [{"type": "tool_result", "tool_use_id": "c1", "content": "A\nB"}]
prep = prepare_turn(
self.cache,
[{"role": "user", "content": content}, self.messages[-1]],
policy=FREEZE_POLICY_REPLAYABLE,
)
self.assertEqual(prep.frozen_message_count, 1)
self.assertEqual(prep.pipeline_input[0]["content"][0]["content"], "short")
def test_malformed_and_media_shapes_remain_mutable(self) -> None:
for content in [
None,
{},
[],
[{"type": "text", "text": 7}],
[
{"type": "image_url", "image_url": {"url": "synthetic"}},
{"type": "text", "text": "A\nB"},
],
]:
with self.subTest(content=content):
messages = [{"role": "tool", "content": content}, self.messages[-1]]
self.assertEqual(self.cache.compute_frozen_count(messages), 0)
def test_textless_anthropic_prefix_stops_at_unsupported_openai_shape(self) -> None:
textless = {
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t0", "content": []}],
}
for blocks in [
self.blocks,
[
{"type": "image_url", "image_url": {"url": "synthetic"}},
{"type": "text", "text": "A\nB"},
],
]:
with self.subTest(blocks=blocks):
messages = [textless, {"role": "tool", "content": blocks}, self.messages[-1]]
before = copy.deepcopy(messages)
stats_before = self.cache.get_stats()
prep = prepare_turn(self.cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
self.assertEqual(prep.frozen_message_count, 1)
self.assertEqual(prep.pipeline_input, before)
self.assertEqual(messages, before)
self.assertEqual(self.cache.get_stats()["hits"], 0)
self.assertEqual(
self.cache.get_stats()["tokens_saved"], stats_before["tokens_saved"]
)
def test_textless_anthropic_prefix_still_obeys_confirmed_clamp(self) -> None:
textless = {
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "t0", "content": []}],
}
messages = [textless, *self.messages]
for confirmed in [0, 1, 2]:
with self.subTest(confirmed=confirmed):
prep = prepare_turn(
self.cache,
messages,
policy=FREEZE_POLICY_CONFIRMED_CLAMP,
tracker_frozen=confirmed,
)
self.assertEqual(prep.frozen_message_count, min(confirmed, 1))
def test_cached_text_after_textless_anthropic_result_is_replayed(self) -> None:
messages = [
{
"role": "user",
"content": [
{"type": "tool_result", "tool_use_id": "t0", "content": []},
{"type": "tool_result", "tool_use_id": "t1", "content": "A\nB"},
],
},
self.messages[-1],
]
before = copy.deepcopy(messages)
prep = prepare_turn(self.cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
self.assertEqual(prep.frozen_message_count, 1)
expected = copy.deepcopy(before)
expected[0]["content"][1]["content"] = "short"
self.assertEqual(prep.pipeline_input, expected)
self.assertEqual(messages, before)
self.assertEqual(self.cache.get_stats()["hits"], 1)
def test_supported_stable_hash_and_trailing_live_zone_are_unchanged(self) -> None:
cache = CompressionCache()
cache.mark_stable(cache.content_hash("stable"))
messages = [{"role": "tool", "content": "stable"}, self.messages[-1]]
self.assertEqual(cache.compute_frozen_count(messages), 1)
self.assertEqual(cache.compute_frozen_count(messages[:1]), 0)
if __name__ == "__main__":
unittest.main()