mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
### Maintainer qualification refresh (2026-10-09) Current head: `a91a1ec24ab20b6dcea0f6e7f8d41feb7d0beea0`. The newly supplied live Claude Code/Anthropic before-after evidence resolves the previous runtime-proof hold. Clean synchronization with current main `01d9194` preserves contributor logic. Independently reran all 88 fork-lineage/prefix tests with the existing native extension; current-head hosted checks are green. Approved for human merge review. The live capture below remains correctly attributed to the preceding implementation head, not represented as a new capture of this synchronization commit. No merge performed. ## Description Claude Code sends side requests alongside an agent's turns. The subagent progress summary and the parent session's side query both re-send the history plus one prompt, with the breakpoint one message earlier, so they read the prompt cache without writing it. resolve_tracker matched such a fork as a continuation and replaced the lineage snapshot with it. The next real turn extends the history before the fork, not the fork, so it matched no lineage. It started a cold tracker (frozen=0), and the token-mode pipeline rewrote history the provider had already cached. Production, 2026-10-08: a review subagent busted Anthropic's cache on 35 of 81 turns. cache_read_input_tokens fell to the 6,065-token system prompt while 100-150k tokens were re-written. Its 600k-token parent did the same after its own side queries. In the proxy log, every bust is the first main request after a fork and shows no frozen prefix. Fork-free sessions on the same proxy were 96/96 and 130/130 full hits. A lineage now keeps its last three request snapshots. A request that extends an earlier one resumes that turn instead of starting cold. Each response is also recorded against the turn that sent it, through a request-scoped ContextVar (the same pattern as the tenant key and CCR store). A fork's response can no longer replace the main line's frozen prefix or overlay source, whether it lands before or after the in-flight main turn's response. Templated fan-out siblings that extend a shared earlier snapshot now stay on one tracker and each resumes its own turn's state. Splitting them would mint a new lineage per fork and exhaust the per-session cap. The fan-out test asserts the per-line state instead of object identity. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: each lineage keeps a `deque(maxlen=_LINEAGE_HISTORY)` (3) of request snapshots instead of one. `resolve_tracker` resumes the turn a request extends, and the turn it resolved is held in a request-scoped `ContextVar` (`_resolved_turn`) so the response is recorded against that turn. - `tests/test_anthropic_fork_lineage_cache.py` (new): handler-level regression. A main turn after a fork (progress summary or side query) must replay its cached prefix byte for byte, and a fork must not replace the lineage it branched from, whichever response lands first. - `tests/test_cache/test_prefix_tracker.py`: the fan-out test asserts per-line state instead of tracker object identity. ## Testing - [x] Unit tests pass (`pytest tests/test_anthropic_fork_lineage_cache.py tests/test_cache/ tests/test_proxy_anthropic_cache_stability.py`: 272 passed, with `test_cache/test_dynamic_detector.py` run on its own: 57 passed, 2 skipped) - [x] Linting passes (`ruff check` on the changed files) - [x] Type checking passes for the changed module (`mypy headroom/cache/prefix_tracker.py`: no issues) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # new test file, without the fix (parent commit) FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[summary] FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[side-query] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[fork-first] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[main-first] AssertionError: main turn 2 re-sent turn 1's history with different bytes ============================== 4 failed ============================== # with the fix ============================== 4 passed ============================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.16, real Claude Code 2.1.295 sessions against the Anthropic API through `headroom proxy --mode token` (details below). - Exact command / steps: the scripted Claude Code session described below, run once on `17abee27` (before) and once on `9b27664a` (after). - Observed result: main turns right after a fork arrived with a cold tracker 5/5 before, 0/5 after; cached tokens lost 3,363 before, 0 after. - Not tested: the full collapse to the system prompt seen in production fork-heavy sessions (did not reproduce in this short session); a deployed proxy running this fix. Live before/after capture on 2026-10-09. Real Claude Code 2.1.295 sessions against the Anthropic API (subscription login, `claude-sonnet-5-5`), through `headroom proxy --mode token` on Windows 11 / Python 3.13.16, with the native extension built in each tree. - **Before:** `17abee27` (merge base with `main`) - **After:** `9b27664a` (this PR) Both ran the same scripted session: a stream-json SDK client with `agentProgressSummaries` and `promptSuggestions` on, one general-purpose subagent that `cat`s ten ~40 KB JSON files one tool call at a time, then three parent follow-up turns. Each request's kind comes from Claude Code's own debug log (`[API REQUEST] source=…`). `sdk` and `agent:*` are main turns; `agent_summary` and `prompt_suggestion` are forks. Token counts are the upstream `usage` values from the proxy's PERF lines. | Main turns right after a fork | Before | After | |---|---|---| | Count | 5 | 5 | | Arrived with a cold tracker (no `Pipeline: freezing` line) | **5/5** | **0/5** | | Read the full prior prefix | 4/5 | **5/5** | | Cached tokens lost | 3,363 | 0 | - **Before**, subagent turn #15 (after its 23-message `agent_summary` fork): cold, `cache_read` 22,137 against the 25,500 cached by the previous turn, `cache_creation` 4,902. - **After**, the matching subagent turn #17 (after a 21-message `agent_summary` fork): frozen 20/23, `cache_read` 25,311 = 24,192 + 1,119, exactly the previous turn's read + write. This short session did not reproduce the full collapse to the system prompt (~6k read) seen in production fork-heavy sessions. In most cold turns the pipeline happened to regenerate identical bytes. The capture does show the mechanism deterministically: before, every fork leaves the next main turn of that conversation starting cold, and one of those cold turns rewrote 3.4k cached tokens. After, none start cold and every post-fork turn reads its whole prior prefix. <details><summary>Per-request tables (before / after)</summary> **Before (`17abee27`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 0 | 15,072 | | 2 | main (subagent) | 2 | - | 5,406 | 7,859 | | 3 | main (subagent) | 5 | 3 | 13,265 | 660 | | 4 | main (subagent) | 7 | 3 | 13,925 | 1,176 | | 5 | main (subagent) | 9 | 3 | 15,101 | 1,319 | | 6 | main (subagent) | 11 | 3 | 16,420 | 1,138 | | 7 | fork (agent_summary) | 11 | 3 | 16,476 | 0 | | 8 | main (subagent) <- after fork | 13 | - | 17,558 | 1,127 | | 9 | main (subagent) | 15 | 3 | 18,685 | 1,134 | | 10 | main (subagent) | 17 | 3 | 19,819 | 1,126 | | 11 | main (subagent) | 19 | 3 | 20,945 | 1,136 | | 12 | main (subagent) | 21 | 3 | 22,081 | 1,133 | | 13 | main (subagent) | 23 | 3 | 23,214 | 2,286 | | 14 | fork (agent_summary) | 23 | 22 | 23,321 | 0 | | 15 | main (subagent) <- after fork | 25 | - | 22,137 | 4,902 | | 16 | main (subagent) | 27 | 24 | 27,039 | 1,260 | | 17 | main (subagent) | 29 | 24 | 28,299 | 7,523 | | 18 | main (parent) | 4 | 3 | 15,072 | 2,204 | | 19 | fork (prompt_suggestion) | 6 | 5 | 17,276 | 335 | | 20 | main (parent) <- after fork | 6 | - | 17,611 | 31 | | 21 | fork (prompt_suggestion) | 8 | 7 | 17,642 | 362 | | 22 | main (parent) <- after fork | 8 | - | 18,004 | 29 | | 23 | main (parent) | 10 | 9 | 18,033 | 242 | | 24 | fork (prompt_suggestion) | 12 | 9 | 18,275 | 128 | | 25 | main (parent) <- after fork | 12 | - | 18,403 | 18 | | 26 | fork (prompt_suggestion) | 14 | 9 | 18,421 | 142 | **After (`9b27664a`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 5,406 | 9,666 | | 2 | main (subagent) | 2 | - | 10,523 | 2,742 | | 3 | main (parent) | 4 | 3 | 15,072 | 692 | | 4 | main (subagent) | 5 | 3 | 13,265 | 1,681 | | 5 | fork (prompt_suggestion) | 6 | 3 | 15,764 | 25 | | 6 | main (subagent) | 7 | 3 | 14,946 | 1,246 | | 7 | main (subagent) | 9 | 3 | 16,192 | 1,138 | | 8 | main (subagent) | 11 | 3 | 17,330 | 2,303 | | 9 | main (subagent) | 13 | 3 | 19,633 | 1,167 | | 10 | main (subagent) | 15 | 3 | 20,800 | 1,137 | | 11 | main (subagent) | 17 | 3 | 21,937 | 1,129 | | 12 | main (subagent) | 19 | 3 | 23,066 | 1,126 | | 13 | main (subagent) | 21 | 20 | 24,192 | 1,119 | | 14 | main (parent) <- after fork | 6 | 3 | 15,789 | 31 | | 15 | fork (prompt_suggestion) | 8 | 3 | 15,820 | 108 | | 16 | fork (agent_summary) | 21 | 20 | 24,248 | 0 | | 17 | main (subagent) <- after fork | 23 | 20 | 25,311 | 710 | | 18 | main (parent) <- after fork | 8 | 3 | 15,928 | 851 | | 19 | fork (prompt_suggestion) | 10 | 3 | 16,779 | 688 | | 20 | main (parent) <- after fork | 10 | 3 | 17,467 | 29 | | 21 | main (parent) | 12 | 3 | 17,496 | 130 | | 22 | main (parent) | 14 | 3 | 17,626 | 249 | | 23 | fork (prompt_suggestion) | 16 | 3 | 17,875 | 131 | | 24 | main (parent) <- after fork | 16 | 3 | 18,006 | 18 | | 25 | fork (prompt_suggestion) | 18 | 3 | 18,024 | 136 | </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, prefix-tracker lineage matching for requests that extend an earlier snapshot of the same lineage - Kill switch / disable path: none - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Not covered here: a few small partial misses in production that forks don't explain, and a synthetic-only risk that an older tool result gets recompressed once it falls past a lagging frozen boundary. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4043" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
1395 lines
58 KiB
Python
1395 lines
58 KiB
Python
"""Tests for PrefixCacheTracker — cache-aware compression."""
|
|
|
|
import contextvars
|
|
import time
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.prefix_tracker import (
|
|
MISS_COLD_START,
|
|
MISS_PREFIX_CHANGE,
|
|
MISS_TTL_EXPIRY,
|
|
MISS_UNKNOWN,
|
|
FreezeStats,
|
|
PrefixCacheTracker,
|
|
PrefixFreezeConfig,
|
|
SessionTrackerStore,
|
|
extract_cache_stable_delta,
|
|
overlay_cached_prefix,
|
|
)
|
|
|
|
|
|
class TestPrefixCacheTracker:
|
|
"""Test PrefixCacheTracker core functionality."""
|
|
|
|
@pytest.fixture
|
|
def tracker(self):
|
|
return PrefixCacheTracker("anthropic")
|
|
|
|
@pytest.fixture
|
|
def openai_tracker(self):
|
|
return PrefixCacheTracker("openai")
|
|
|
|
def test_turn_0_no_freeze(self, tracker):
|
|
"""First turn should never freeze — no cache state yet."""
|
|
assert tracker.get_frozen_message_count() == 0
|
|
|
|
def test_turn_1_with_cache_hit_freezes(self, tracker):
|
|
"""After turn 1 with cache hits, turn 2 should freeze."""
|
|
messages = [
|
|
{"role": "system", "content": "You are a helpful assistant." * 100},
|
|
{"role": "user", "content": "Hello"},
|
|
{"role": "assistant", "content": "Hi there!"},
|
|
]
|
|
# Simulate: provider cached 2000 tokens (system + user)
|
|
token_counts = [1500, 50, 500]
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=2050,
|
|
messages=messages,
|
|
message_token_counts=token_counts,
|
|
)
|
|
|
|
# On turn 2, the first 2 messages (1500 + 50 = 1550 <= 2050) are frozen
|
|
assert tracker.get_frozen_message_count() == 3 # All 3 fit within 2050
|
|
|
|
def test_partial_freeze(self, tracker):
|
|
"""Only messages that fit within cached tokens are frozen."""
|
|
messages = [
|
|
{"role": "system", "content": "System prompt" * 50},
|
|
{"role": "user", "content": "First question" * 50},
|
|
{"role": "assistant", "content": "First answer" * 50},
|
|
{"role": "user", "content": "Second question"},
|
|
]
|
|
token_counts = [2000, 500, 500, 50]
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=2500,
|
|
cache_write_tokens=0,
|
|
messages=messages,
|
|
message_token_counts=token_counts,
|
|
)
|
|
|
|
# 2000 + 500 = 2500 <= 2500, but 2000 + 500 + 500 = 3000 > 2500
|
|
assert tracker.get_frozen_message_count() == 2
|
|
|
|
def test_cold_start_no_freeze(self, tracker):
|
|
"""If cache_read=0 and cache_write=0, don't freeze."""
|
|
messages = [{"role": "user", "content": "Hello"}]
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=0,
|
|
messages=messages,
|
|
)
|
|
|
|
assert tracker.get_frozen_message_count() == 0
|
|
|
|
def test_cache_write_freezes_next_turn(self, tracker):
|
|
"""Cache writes (new cache entries) should be frozen on the next turn."""
|
|
messages = [
|
|
{"role": "system", "content": "System" * 200},
|
|
{"role": "user", "content": "Hello"},
|
|
]
|
|
token_counts = [1500, 50]
|
|
|
|
# Turn 1: provider writes to cache (above min threshold)
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=1550,
|
|
messages=messages,
|
|
message_token_counts=token_counts,
|
|
)
|
|
|
|
# Turn 2: should freeze what was written
|
|
assert tracker.get_frozen_message_count() == 2
|
|
|
|
def test_min_cached_tokens_threshold(self):
|
|
"""Below min_cached_tokens, no freeze."""
|
|
config = PrefixFreezeConfig(min_cached_tokens=2000)
|
|
tracker = PrefixCacheTracker("anthropic", config)
|
|
|
|
messages = [{"role": "user", "content": "Hello"}]
|
|
|
|
# Turn 1: only 500 tokens cached — below threshold
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=500,
|
|
messages=messages,
|
|
message_token_counts=[500],
|
|
)
|
|
|
|
assert tracker.get_frozen_message_count() == 0
|
|
|
|
def test_disabled_config(self):
|
|
"""Disabled config always returns 0."""
|
|
config = PrefixFreezeConfig(enabled=False)
|
|
tracker = PrefixCacheTracker("anthropic", config)
|
|
|
|
messages = [{"role": "system", "content": "System" * 500}]
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=5000,
|
|
cache_write_tokens=0,
|
|
messages=messages,
|
|
message_token_counts=[5000],
|
|
)
|
|
|
|
assert tracker.get_frozen_message_count() == 0
|
|
|
|
def test_turn_number_increments(self, tracker):
|
|
"""Turn number should increment on each update."""
|
|
messages = [{"role": "user", "content": "Hello"}]
|
|
|
|
assert tracker._turn_number == 0
|
|
|
|
tracker.update_from_response(0, 0, messages)
|
|
assert tracker._turn_number == 1
|
|
|
|
tracker.update_from_response(0, 0, messages)
|
|
assert tracker._turn_number == 2
|
|
|
|
def test_stats_tracking(self, tracker):
|
|
"""Stats should reflect tracker state."""
|
|
stats = tracker.stats
|
|
assert isinstance(stats, FreezeStats)
|
|
assert stats.busts_avoided == 0
|
|
assert stats.tokens_preserved == 0
|
|
assert stats.turn_number == 0
|
|
|
|
def test_record_bust_avoided(self, tracker):
|
|
"""Recording bust avoided should update stats."""
|
|
tracker.record_bust_avoided(tokens_preserved=5000, compression_foregone=500)
|
|
tracker.record_bust_avoided(tokens_preserved=3000, compression_foregone=200)
|
|
|
|
stats = tracker.stats
|
|
assert stats.busts_avoided == 2
|
|
assert stats.tokens_preserved == 8000
|
|
assert stats.compression_foregone_tokens == 700
|
|
assert stats.net_benefit_tokens == 7300
|
|
|
|
def test_should_force_compress_outside_frozen(self, tracker):
|
|
"""Messages outside frozen prefix should always be compressed."""
|
|
tracker._cached_message_count = 3
|
|
assert tracker.should_force_compress(5, 1000, 200) is True
|
|
|
|
def test_should_force_compress_when_savings_exceed_discount(self, tracker):
|
|
"""For Anthropic (90% discount), compression must save >90% to be worth it."""
|
|
tracker._cached_message_count = 5
|
|
|
|
# 95% savings > 90% discount — should force compress
|
|
assert tracker.should_force_compress(2, 1000, 50) is True
|
|
|
|
# 50% savings < 90% discount — should NOT force compress
|
|
assert tracker.should_force_compress(2, 1000, 500) is False
|
|
|
|
def test_should_force_compress_openai(self, openai_tracker):
|
|
"""For OpenAI (50% discount), compression must save >50% to be worth it."""
|
|
openai_tracker._cached_message_count = 5
|
|
|
|
# 60% savings > 50% discount — should force compress
|
|
assert openai_tracker.should_force_compress(2, 1000, 400) is True
|
|
|
|
# 40% savings < 50% discount — should NOT force compress
|
|
assert openai_tracker.should_force_compress(2, 1000, 600) is False
|
|
|
|
def test_estimate_message_tokens(self):
|
|
"""Token estimation should roughly match character / 3.5."""
|
|
messages = [
|
|
{"role": "system", "content": "A" * 350}, # ~100 tokens
|
|
{"role": "user", "content": "B" * 70}, # ~20 tokens
|
|
]
|
|
counts = PrefixCacheTracker._estimate_message_tokens(messages)
|
|
assert len(counts) == 2
|
|
assert counts[0] > counts[1] # System should have more tokens
|
|
|
|
def test_estimate_content_blocks(self):
|
|
"""Token estimation should handle Anthropic content blocks."""
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "A" * 350},
|
|
{"type": "text", "text": "B" * 350},
|
|
],
|
|
},
|
|
]
|
|
counts = PrefixCacheTracker._estimate_message_tokens(messages)
|
|
assert len(counts) == 1
|
|
assert counts[0] > 100
|
|
|
|
def test_estimate_tool_result_content(self):
|
|
"""Token estimation should count tool_result content field."""
|
|
tool_content = "x" * 3500 # ~1000 tokens
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "t1",
|
|
"content": tool_content,
|
|
}
|
|
],
|
|
},
|
|
]
|
|
counts = PrefixCacheTracker._estimate_message_tokens(messages)
|
|
assert len(counts) == 1
|
|
# Should be ~1000 tokens, definitely > 100
|
|
assert counts[0] > 100
|
|
|
|
def test_estimate_tool_use_input(self):
|
|
"""Token estimation should count tool_use input field."""
|
|
messages = [
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "t1",
|
|
"name": "Read",
|
|
"input": {"file_path": "/very/long/path/" + "x" * 700},
|
|
}
|
|
],
|
|
},
|
|
]
|
|
counts = PrefixCacheTracker._estimate_message_tokens(messages)
|
|
assert len(counts) == 1
|
|
# Should count the serialized input dict
|
|
assert counts[0] > 50
|
|
|
|
def test_estimate_tool_result_nested_blocks(self):
|
|
"""Token estimation should handle nested content blocks in tool_result."""
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "t1",
|
|
"content": [
|
|
{"type": "text", "text": "A" * 3500},
|
|
],
|
|
}
|
|
],
|
|
},
|
|
]
|
|
counts = PrefixCacheTracker._estimate_message_tokens(messages)
|
|
assert len(counts) == 1
|
|
assert counts[0] > 100
|
|
|
|
def test_session_ttl_expiry(self):
|
|
"""Tracker should report as expired after TTL."""
|
|
config = PrefixFreezeConfig(session_ttl_seconds=1)
|
|
tracker = PrefixCacheTracker("anthropic", config)
|
|
|
|
assert tracker.is_expired is False
|
|
|
|
# Simulate time passing
|
|
tracker._last_activity = time.time() - 2
|
|
assert tracker.is_expired is True
|
|
|
|
|
|
class TestSessionTrackerStore:
|
|
"""Test SessionTrackerStore management."""
|
|
|
|
@pytest.fixture
|
|
def store(self):
|
|
return SessionTrackerStore()
|
|
|
|
def test_get_or_create_new(self, store):
|
|
"""Should create a new tracker for unknown session."""
|
|
tracker = store.get_or_create("session-1", "anthropic")
|
|
assert isinstance(tracker, PrefixCacheTracker)
|
|
assert tracker.provider == "anthropic"
|
|
|
|
def test_get_or_create_existing(self, store):
|
|
"""Should return the same tracker for the same session."""
|
|
tracker1 = store.get_or_create("session-1", "anthropic")
|
|
tracker2 = store.get_or_create("session-1", "anthropic")
|
|
assert tracker1 is tracker2
|
|
|
|
def test_different_sessions(self, store):
|
|
"""Different sessions should get different trackers."""
|
|
tracker1 = store.get_or_create("session-1", "anthropic")
|
|
tracker2 = store.get_or_create("session-2", "openai")
|
|
assert tracker1 is not tracker2
|
|
assert tracker1.provider == "anthropic"
|
|
assert tracker2.provider == "openai"
|
|
|
|
def test_active_sessions_count(self, store):
|
|
"""Should track the number of active sessions."""
|
|
assert store.active_sessions == 0
|
|
|
|
store.get_or_create("s1", "anthropic")
|
|
assert store.active_sessions == 1
|
|
|
|
store.get_or_create("s2", "openai")
|
|
assert store.active_sessions == 2
|
|
|
|
def test_cleanup_expired(self, store):
|
|
"""Should remove expired sessions on cleanup."""
|
|
config = PrefixFreezeConfig(session_ttl_seconds=1)
|
|
store = SessionTrackerStore(default_config=config)
|
|
|
|
tracker = store.get_or_create("expired-session", "anthropic")
|
|
tracker._last_activity = time.time() - 2
|
|
|
|
# Force cleanup
|
|
store._last_cleanup = 0
|
|
store._maybe_cleanup()
|
|
|
|
assert store.active_sessions == 0
|
|
|
|
def test_request_cache_ttl_preserves_frozen_lineage_through_cleanup(self, monkeypatch):
|
|
"""A warm one-hour cache lineage survives cleanup and replays its prefix."""
|
|
now = [1_000.0]
|
|
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
|
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1))
|
|
original = [{"role": "user", "content": "original prompt " + "x" * 2000}]
|
|
forwarded = [{"role": "user", "content": "compressed cached prompt"}]
|
|
tracker = store.resolve_tracker(
|
|
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
|
|
)
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=1500,
|
|
messages=forwarded,
|
|
message_token_counts=[1500],
|
|
original_messages=original,
|
|
)
|
|
assert tracker.get_frozen_message_count() == 1
|
|
|
|
now[0] += 660
|
|
store._last_cleanup = now[0] - store._cleanup_interval
|
|
resumed = store.resolve_tracker(
|
|
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
|
|
)
|
|
assert resumed is tracker
|
|
assert resumed.get_frozen_message_count() == 1
|
|
assert (
|
|
overlay_cached_prefix(
|
|
original,
|
|
original,
|
|
resumed.get_last_original_messages(),
|
|
resumed.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=resumed.get_frozen_message_count(),
|
|
)
|
|
== forwarded
|
|
)
|
|
|
|
now[0] += 3901
|
|
store._last_cleanup = now[0] - store._cleanup_interval
|
|
expired = store.resolve_tracker(
|
|
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
|
|
)
|
|
assert expired is not tracker
|
|
expired.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=1500,
|
|
messages=forwarded,
|
|
message_token_counts=[1500],
|
|
original_messages=original,
|
|
)
|
|
unrelated_messages = [{"role": "user", "content": "unrelated conversation"}]
|
|
unrelated = store.resolve_tracker(
|
|
"one-hour", "anthropic", messages=unrelated_messages, cache_ttl_seconds=3600
|
|
)
|
|
|
|
assert unrelated is not expired
|
|
assert (
|
|
overlay_cached_prefix(
|
|
unrelated_messages,
|
|
unrelated_messages,
|
|
unrelated.get_last_original_messages(),
|
|
unrelated.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=unrelated.get_frozen_message_count(),
|
|
)
|
|
== unrelated_messages
|
|
)
|
|
|
|
def test_configured_cache_ttl_preserves_and_expires_tracker_without_request_ttl(
|
|
self, monkeypatch
|
|
):
|
|
"""Configured provider lifetime retains cached bytes without a per-request TTL."""
|
|
now = [1_000.0]
|
|
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
|
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, cache_ttl_seconds=3600))
|
|
original = [{"role": "user", "content": "original prompt " + "x" * 2000}]
|
|
forwarded = [{"role": "user", "content": "configured cache payload"}]
|
|
tracker = store.resolve_tracker("configured-hour", "anthropic", messages=original)
|
|
now[0] += 660
|
|
store._last_cleanup = now[0]
|
|
assert store.peek("configured-hour") is tracker
|
|
assert store.resolve_tracker("configured-hour", "anthropic", messages=original) is tracker
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=1500,
|
|
messages=forwarded,
|
|
message_token_counts=[1500],
|
|
original_messages=original,
|
|
)
|
|
assert tracker.get_frozen_message_count() == 1
|
|
|
|
now[0] += 660
|
|
store._last_cleanup = now[0]
|
|
resumed = store.resolve_tracker("configured-hour", "anthropic", messages=original)
|
|
assert resumed is tracker
|
|
assert (
|
|
overlay_cached_prefix(
|
|
original,
|
|
original,
|
|
resumed.get_last_original_messages(),
|
|
resumed.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=resumed.get_frozen_message_count(),
|
|
)
|
|
== forwarded
|
|
)
|
|
|
|
now[0] += 3_661
|
|
expired = store.resolve_tracker("configured-hour", "anthropic", messages=original)
|
|
assert expired is not tracker
|
|
assert expired.get_frozen_message_count() == 0
|
|
|
|
def test_request_ttl_does_not_resurrect_expired_sibling_affinity(self, monkeypatch):
|
|
"""A new long request TTL cannot revive an expired short-cache prefix."""
|
|
now = [1_000.0]
|
|
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
|
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1))
|
|
short_history = [{"role": "user", "content": "short profile"}]
|
|
long_history = [{"role": "user", "content": "long profile"}]
|
|
short = store.resolve_tracker(
|
|
"mixed", "anthropic", short_history, cache_affinity="short", cache_ttl_seconds=300
|
|
)
|
|
short.update_from_response(
|
|
0,
|
|
1000,
|
|
[{"role": "user", "content": "old short-cache bytes"}],
|
|
message_token_counts=[1000],
|
|
original_messages=short_history,
|
|
)
|
|
long = store.resolve_tracker(
|
|
"mixed", "anthropic", long_history, cache_affinity="long", cache_ttl_seconds=3600
|
|
)
|
|
|
|
now[0] += 661
|
|
store._last_cleanup = now[0]
|
|
continued = store.resolve_tracker(
|
|
"mixed",
|
|
"anthropic",
|
|
long_history + [{"role": "assistant", "content": "next"}],
|
|
cache_affinity="long",
|
|
cache_ttl_seconds=3600,
|
|
)
|
|
assert continued is long
|
|
restarted = store.resolve_tracker(
|
|
"mixed", "anthropic", short_history, cache_affinity="short", cache_ttl_seconds=3600
|
|
)
|
|
assert restarted.get_frozen_message_count() == 0
|
|
current = [{"role": "user", "content": "current short-profile bytes"}]
|
|
assert (
|
|
overlay_cached_prefix(
|
|
current,
|
|
short_history,
|
|
restarted.get_last_original_messages(),
|
|
restarted.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=restarted.get_frozen_message_count(),
|
|
)
|
|
== current
|
|
)
|
|
|
|
def test_request_cache_ttl_is_isolated_per_session(self, monkeypatch):
|
|
"""A one-hour cache in one session must not extend another session."""
|
|
now = [1_000.0]
|
|
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
|
store = SessionTrackerStore()
|
|
messages = [{"role": "user", "content": "prompt " + "x" * 1000}]
|
|
long_lived = store.resolve_tracker(
|
|
"one-hour", "anthropic", messages=messages, cache_ttl_seconds=3600
|
|
)
|
|
short_lived = store.resolve_tracker(
|
|
"five-minute", "anthropic", messages=messages, cache_ttl_seconds=300
|
|
)
|
|
|
|
now[0] += 661
|
|
store._last_cleanup = now[0] - store._cleanup_interval
|
|
assert (
|
|
store.resolve_tracker(
|
|
"one-hour", "anthropic", messages=messages, cache_ttl_seconds=3600
|
|
)
|
|
is long_lived
|
|
)
|
|
replacement = store.resolve_tracker(
|
|
"five-minute", "anthropic", messages=messages, cache_ttl_seconds=300
|
|
)
|
|
assert replacement is not short_lived
|
|
|
|
def test_compute_session_id_from_header(self, store):
|
|
"""Should use x-headroom-session-id header if present."""
|
|
|
|
class MockRequest:
|
|
headers = {"x-headroom-session-id": "explicit-id-123"}
|
|
|
|
session_id = store.compute_session_id(
|
|
MockRequest(), "claude-3", [{"role": "user", "content": "Hi"}]
|
|
)
|
|
assert session_id == "explicit-id-123"
|
|
|
|
def test_compute_session_id_from_hash(self, store):
|
|
"""Should hash model + system prompt as fallback."""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are helpful."},
|
|
{"role": "user", "content": "Hi"},
|
|
]
|
|
|
|
id1 = store.compute_session_id(MockRequest(), "claude-3", messages)
|
|
id2 = store.compute_session_id(MockRequest(), "claude-3", messages)
|
|
assert id1 == id2 # Stable hash
|
|
assert len(id1) == 16
|
|
|
|
# Different model = different session
|
|
id3 = store.compute_session_id(MockRequest(), "gpt-4", messages)
|
|
assert id3 != id1
|
|
|
|
def test_compute_session_id_distinguishes_leading_system_run(self, store):
|
|
"""Different dynamic LEADING system messages should not collide."""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
static_prompt = "framework prompt " * 80
|
|
conv_a = [
|
|
{"role": "system", "content": [{"type": "text", "text": static_prompt}]},
|
|
{"role": "system", "content": [{"type": "text", "text": "context: session A"}]},
|
|
{"role": "user", "content": "hello"},
|
|
]
|
|
conv_b = [
|
|
{"role": "system", "content": [{"type": "text", "text": static_prompt}]},
|
|
{"role": "system", "content": [{"type": "text", "text": "context: session B"}]},
|
|
{"role": "user", "content": "hello"},
|
|
]
|
|
|
|
id_a = store.compute_session_id(MockRequest(), "claude-3", conv_a)
|
|
id_b = store.compute_session_id(MockRequest(), "claude-3", conv_b)
|
|
|
|
assert id_a != id_b
|
|
|
|
def test_compute_session_id_distinguishes_top_level_system(self, store):
|
|
"""Anthropic carries the system prompt as a top-level field (not a
|
|
role:'system' message). The handler folds it in as a synthetic system
|
|
message so two conversations with the same model and turns but different
|
|
system prompts get distinct ids — otherwise they share one tracker and
|
|
their sticky state cross-contaminates. This exercises that mechanism."""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
turns = [{"role": "user", "content": "hello"}]
|
|
|
|
def with_system(system):
|
|
# Mirror what handlers/anthropic.py does for the top-level system.
|
|
return [{"role": "system", "content": system}, *turns]
|
|
|
|
id_a = store.compute_session_id(
|
|
MockRequest(), "claude-3", with_system("You are a Python expert.")
|
|
)
|
|
id_b = store.compute_session_id(
|
|
MockRequest(), "claude-3", with_system("You are a Rust expert.")
|
|
)
|
|
assert id_a != id_b
|
|
|
|
# A list-of-text-blocks system folds the same text as the string form.
|
|
id_a_list = store.compute_session_id(
|
|
MockRequest(),
|
|
"claude-3",
|
|
with_system([{"type": "text", "text": "You are a Python expert."}]),
|
|
)
|
|
assert id_a_list == id_a
|
|
|
|
def test_compute_session_id_is_stable_when_only_non_system_turns_change(self, store):
|
|
"""Appending non-system turns should keep the same fallback session id."""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
base_messages = [
|
|
{"role": "system", "content": [{"type": "text", "text": "framework prompt"}]},
|
|
{"role": "system", "content": [{"type": "text", "text": "context: session A"}]},
|
|
{"role": "user", "content": "hello"},
|
|
]
|
|
extended_messages = base_messages + [{"role": "assistant", "content": "hi there"}]
|
|
|
|
id1 = store.compute_session_id(MockRequest(), "claude-3", base_messages)
|
|
id2 = store.compute_session_id(MockRequest(), "claude-3", extended_messages)
|
|
|
|
assert id1 == id2
|
|
|
|
def test_compute_session_id_no_system(self, store):
|
|
"""Should work without system messages."""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
messages = [{"role": "user", "content": "Hi"}]
|
|
session_id = store.compute_session_id(MockRequest(), "claude-3", messages)
|
|
assert isinstance(session_id, str)
|
|
assert len(session_id) == 16
|
|
|
|
def test_mid_conversation_system_turns_do_not_rotate_session_id(self, store):
|
|
"""Claude Code sends <system-reminder> turns as role:"system" MESSAGES
|
|
interleaved into the history (hook outputs, skills lists, truncation
|
|
notices). Hashing those into the fallback id rotates the session id
|
|
mid-conversation — orphaning the prefix tracker and every other
|
|
session-sticky subsystem (beta headers, CCR/memory registries, the
|
|
compression cache) each time a reminder lands. Only the LEADING run of
|
|
system messages is session identity; later system turns are content.
|
|
"""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
leading = {"role": "system", "content": "You are an agent. " * 40}
|
|
turn1 = [leading, {"role": "user", "content": "read file A"}]
|
|
turn2 = turn1 + [
|
|
{"role": "assistant", "content": "read it"},
|
|
{"role": "user", "content": "tool result ..."},
|
|
{
|
|
"role": "system",
|
|
"content": "<system-reminder>Truncated: PARTIAL view</system-reminder>",
|
|
},
|
|
{"role": "user", "content": "continue"},
|
|
]
|
|
|
|
id1 = store.compute_session_id(MockRequest(), "claude-sonnet-5", turn1)
|
|
id2 = store.compute_session_id(MockRequest(), "claude-sonnet-5", turn2)
|
|
assert id1 == id2
|
|
|
|
|
|
class TestConversationLineageResolution:
|
|
"""resolve_tracker: one PrefixCacheTracker per conversation lineage (#2085).
|
|
|
|
Concurrent conversations that share a fallback session id (same model +
|
|
same system prompt — e.g. a Claude Code session and its parallel subagents)
|
|
must not thrash one tracker's frozen-prefix state: interleaved turns would
|
|
each see the *other* conversation's prefix, freeze never stabilizes, and
|
|
the provider prompt cache is re-written on every call.
|
|
|
|
resolve_tracker keys trackers by message lineage instead: an incoming
|
|
history that extends a known lineage reuses its tracker; a diverging or
|
|
rewritten history gets a fresh one. The session id itself is never changed,
|
|
so session-sticky state keyed on it elsewhere (beta headers, CCR/memory
|
|
registries, the compression cache) is unaffected.
|
|
"""
|
|
|
|
@pytest.fixture
|
|
def store(self):
|
|
return SessionTrackerStore()
|
|
|
|
@staticmethod
|
|
def _history(name: str, turn: int) -> list[dict]:
|
|
"""Client-shaped request messages for `turn` (1-based): u0,a0,...,u_{turn-1}."""
|
|
messages: list[dict] = []
|
|
for t in range(turn):
|
|
messages.append({"role": "user", "content": f"[{name}] user {t} " + "x" * 200})
|
|
if t < turn - 1:
|
|
messages.append(
|
|
{"role": "assistant", "content": f"[{name}] assistant {t} " + "y" * 200}
|
|
)
|
|
return messages
|
|
|
|
@staticmethod
|
|
def _block_history(turn: int, cc_on: int | None) -> list[dict]:
|
|
"""Block-content history; cache_control breakpoint on message `cc_on` (or none)."""
|
|
messages: list[dict] = []
|
|
for t in range(turn):
|
|
messages.append(
|
|
{"role": "user", "content": [{"type": "text", "text": f"user {t} " + "x" * 200}]}
|
|
)
|
|
if t < turn - 1:
|
|
messages.append(
|
|
{
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": f"assistant {t} " + "y" * 200}],
|
|
}
|
|
)
|
|
if cc_on is not None:
|
|
msg = messages[cc_on]
|
|
blocks = [dict(b) for b in msg["content"]]
|
|
blocks[-1] = {**blocks[-1], "cache_control": {"type": "ephemeral"}}
|
|
messages[cc_on] = {**msg, "content": blocks}
|
|
return messages
|
|
|
|
def test_interleaved_conversations_resolve_to_independent_trackers(self, store):
|
|
"""The #2085 production shape: two conversations, one session id,
|
|
alternating requests. Each must keep its own tracker and per-turn state
|
|
(on a shared tracker, _turn_number would count both conversations)."""
|
|
sid = "shared-fallback-id"
|
|
trackers: dict[str, PrefixCacheTracker] = {}
|
|
for turn in range(1, 5):
|
|
for name in ("A", "B"):
|
|
history = self._history(name, turn)
|
|
tracker = store.resolve_tracker(sid, "anthropic", messages=history)
|
|
trackers.setdefault(name, tracker)
|
|
assert tracker is trackers[name], f"[{name}] turn {turn} switched trackers"
|
|
tracker.update_from_response(
|
|
cache_read_tokens=1000 * turn,
|
|
cache_write_tokens=500,
|
|
messages=history,
|
|
)
|
|
assert trackers["A"] is not trackers["B"]
|
|
assert trackers["A"]._turn_number == 4
|
|
assert trackers["B"]._turn_number == 4
|
|
|
|
def test_identical_first_turns_keep_per_line_state_after_divergence(self, store):
|
|
"""Templated fan-outs send byte-identical first turns, then diverge.
|
|
Each line extends its own earlier snapshot and must resume that turn's
|
|
frozen prefix and replay source, never its sibling's. (A sibling that
|
|
extends an earlier snapshot is indistinguishable from a main turn after
|
|
a Claude Code fork, so they share the lineage rather than split.)
|
|
|
|
Each request runs in its own context, as each proxied request runs in
|
|
its own asyncio task."""
|
|
sid = "shared"
|
|
first = [{"role": "user", "content": "verify the fix " + "p" * 300}]
|
|
t_a1 = store.resolve_tracker(sid, "anthropic", messages=first)
|
|
t_b1 = store.resolve_tracker(sid, "anthropic", messages=first)
|
|
assert t_b1 is t_a1
|
|
|
|
def turn(history: list[dict], frozen: int) -> PrefixCacheTracker:
|
|
def send() -> PrefixCacheTracker:
|
|
tracker = store.resolve_tracker(sid, "anthropic", messages=history)
|
|
tracker.update_from_response(
|
|
cache_read_tokens=2000 * frozen,
|
|
cache_write_tokens=0,
|
|
messages=history,
|
|
message_token_counts=[2000] * len(history),
|
|
)
|
|
return tracker
|
|
|
|
return contextvars.Context().run(send)
|
|
|
|
def reply(history: list[dict], text: str) -> list[dict]:
|
|
return history + [
|
|
{"role": "assistant", "content": f"answer {text}"},
|
|
{"role": "user", "content": f"next {text}"},
|
|
]
|
|
|
|
a2, b2 = reply(first, "A"), reply(first, "B")
|
|
turn(a2, frozen=3)
|
|
turn(b2, frozen=2)
|
|
|
|
a3 = store.resolve_tracker(sid, "anthropic", messages=reply(a2, "A2"))
|
|
assert a3.get_last_original_messages() == a2
|
|
assert a3.get_frozen_message_count() == 3
|
|
b3 = store.resolve_tracker(sid, "anthropic", messages=reply(b2, "B2"))
|
|
assert b3.get_last_original_messages() == b2
|
|
assert b3.get_frozen_message_count() == 2
|
|
|
|
@pytest.mark.parametrize(
|
|
"cc_turn2",
|
|
[0, -1, None],
|
|
ids=["breakpoint-stays", "breakpoint-moved-to-last", "breakpoint-removed"],
|
|
)
|
|
def test_cache_control_movement_does_not_split_lineage(self, store, cc_turn2):
|
|
"""Clients move the cache_control breakpoint every turn; that must not
|
|
read as a rewritten history."""
|
|
sid = "shared"
|
|
t1 = store.resolve_tracker(sid, "anthropic", messages=self._block_history(1, cc_on=0))
|
|
t2 = store.resolve_tracker(
|
|
sid, "anthropic", messages=self._block_history(2, cc_on=cc_turn2)
|
|
)
|
|
assert t2 is t1
|
|
|
|
@pytest.mark.parametrize(
|
|
"requote",
|
|
[
|
|
lambda m: {**m, "content": [{"type": "text", "text": m["content"]}]},
|
|
lambda m: {
|
|
**m,
|
|
"content": [{"type": "text", "text": m["content"], "index": 0}],
|
|
},
|
|
lambda m: {
|
|
**m,
|
|
"content": [
|
|
{"type": "text", "text": m["content"]},
|
|
{"cachePoint": {"type": "default"}},
|
|
],
|
|
},
|
|
],
|
|
ids=["string-to-block-sugar", "streaming-index-annotation", "bedrock-cachepoint-block"],
|
|
)
|
|
def test_representation_churn_does_not_split_lineage(self, store, requote):
|
|
"""Clients re-encode history turn-to-turn without changing content
|
|
(litellm flips string<->block sugar, streaming assembly adds `index`,
|
|
Bedrock moves its cachePoint block). Lineage matching must use the
|
|
same canonical equivalence as the cache-stable delta path."""
|
|
sid = "shared"
|
|
first = {"role": "user", "content": "hello " + "x" * 200}
|
|
t1 = store.resolve_tracker(sid, "anthropic", messages=[first])
|
|
grown = [
|
|
requote(first),
|
|
{"role": "assistant", "content": "hi"},
|
|
{"role": "user", "content": "next"},
|
|
]
|
|
assert store.resolve_tracker(sid, "anthropic", messages=grown) is t1
|
|
|
|
@pytest.mark.parametrize(
|
|
"rewrite",
|
|
[
|
|
lambda h: [{"role": "user", "content": "[summary of the conversation so far]"}],
|
|
lambda h: [h[0], {"role": "assistant", "content": "EDITED"}, *h[2:]],
|
|
lambda h: h[:-2],
|
|
],
|
|
ids=["compacted", "middle-edited", "truncated"],
|
|
)
|
|
def test_rewritten_history_gets_fresh_tracker(self, store, rewrite):
|
|
"""A rewritten history (client-side /compact, edits, truncation) means
|
|
the provider cache line is gone anyway: start a fresh lineage, keep the
|
|
old tracker until TTL, and never touch the session id."""
|
|
sid = "shared"
|
|
history = self._history("A", 3)
|
|
tracker = store.resolve_tracker(sid, "anthropic", messages=history)
|
|
fresh = store.resolve_tracker(sid, "anthropic", messages=rewrite(history))
|
|
assert fresh is not tracker
|
|
assert store.active_sessions == 2
|
|
|
|
@pytest.mark.parametrize("messages", [None, []], ids=["none", "empty"])
|
|
def test_resolve_without_messages_matches_legacy_get_or_create(self, store, messages):
|
|
tracker = store.resolve_tracker("sid", "anthropic", messages=messages)
|
|
assert tracker is store.get_or_create("sid", "anthropic")
|
|
|
|
def test_resolve_with_freeze_disabled_matches_legacy_get_or_create(self):
|
|
"""With prefix freeze off there is no frozen state to protect — skip
|
|
lineage bookkeeping entirely."""
|
|
store = SessionTrackerStore(PrefixFreezeConfig(enabled=False))
|
|
t_a = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 1))
|
|
t_b = store.resolve_tracker("sid", "anthropic", messages=self._history("B", 1))
|
|
assert t_a is t_b
|
|
assert t_a is store.get_or_create("sid", "anthropic")
|
|
|
|
def test_over_cap_conversations_share_one_overflow_tracker(self):
|
|
"""A fan-out storm on one session id must not grow trackers unbounded:
|
|
past the cap, new conversations share one overflow tracker instead of
|
|
evicting an established lineage."""
|
|
store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=4))
|
|
sid = "storm"
|
|
overflow = set()
|
|
for i in range(9):
|
|
tracker = store.resolve_tracker(
|
|
sid, "anthropic", messages=[{"role": "user", "content": f"task {i} " + "z" * 100}]
|
|
)
|
|
if i >= 4:
|
|
overflow.add(id(tracker))
|
|
assert store.active_sessions == 5 # 4 lineages + 1 shared overflow
|
|
assert len(overflow) == 1
|
|
|
|
def test_established_lineages_survive_cap_overflow(self):
|
|
"""Filling the family must never evict an established conversation —
|
|
under round-robin any eviction victim is the next requester, which
|
|
would degrade EVERY conversation to a cold tracker per turn."""
|
|
store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=2))
|
|
sid = "shared"
|
|
parent_history = self._history("parent", 4)
|
|
parent = store.resolve_tracker(sid, "anthropic", messages=parent_history)
|
|
shorty = store.resolve_tracker(sid, "anthropic", messages=self._history("shorty", 1))
|
|
# Third divergent conversation lands on the overflow tracker; both
|
|
# established lineages keep their trackers.
|
|
newcomer = store.resolve_tracker(sid, "anthropic", messages=self._history("new", 1))
|
|
assert newcomer is not parent
|
|
assert newcomer is not shorty
|
|
grown = parent_history + [
|
|
{"role": "assistant", "content": "[parent] assistant 3 " + "y" * 200},
|
|
{"role": "user", "content": "[parent] user 4 " + "x" * 200},
|
|
]
|
|
assert store.resolve_tracker(sid, "anthropic", messages=grown) is parent
|
|
assert (
|
|
store.resolve_tracker(sid, "anthropic", messages=self._history("shorty", 2)) is shorty
|
|
)
|
|
|
|
def test_no_cliff_at_cap_plus_one_round_robin(self):
|
|
"""cap+1 conversations round-robining turns: the in-cap conversations
|
|
keep their trackers on every round (no eviction churn); only the
|
|
over-cap tail shares the overflow tracker."""
|
|
store = SessionTrackerStore(PrefixFreezeConfig(max_lineages_per_session=3))
|
|
sid = "shared"
|
|
trackers: dict[str, PrefixCacheTracker] = {}
|
|
for turn in range(1, 4):
|
|
for name in ("A", "B", "C", "D"):
|
|
tracker = store.resolve_tracker(
|
|
sid, "anthropic", messages=self._history(name, turn)
|
|
)
|
|
if turn == 1:
|
|
trackers[name] = tracker
|
|
elif name != "D":
|
|
assert tracker is trackers[name], f"[{name}] turn {turn} lost its tracker"
|
|
else:
|
|
assert tracker is trackers["D"] # stable overflow tracker
|
|
|
|
@pytest.mark.parametrize("new_affinity", ["profile-a", "profile-b"])
|
|
def test_expired_unswept_tracker_cannot_replay_prefix_across_affinities(
|
|
self, monkeypatch, new_affinity
|
|
):
|
|
"""An expired bare lineage is cold before lookup, without waiting for sweep."""
|
|
now = [1_000.0]
|
|
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
|
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, session_ttl_seconds=1))
|
|
history = self._history("shared", 1)
|
|
stale_forwarded = [{"role": "user", "content": "profile A's frozen forwarded bytes"}]
|
|
stale = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-a")
|
|
stale.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=1_000,
|
|
messages=stale_forwarded,
|
|
message_token_counts=[1_000],
|
|
original_messages=history,
|
|
)
|
|
assert stale.get_frozen_message_count() == 1
|
|
|
|
now[0] += 2
|
|
# Keep the periodic sweep not due: lookup/pruning itself must enforce expiry.
|
|
store._last_cleanup = now[0]
|
|
fresh = store.resolve_tracker("shared", "anthropic", history, cache_affinity=new_affinity)
|
|
assert fresh is not stale
|
|
assert fresh.get_frozen_message_count() == 0
|
|
processed = [{"role": "user", "content": "current profile bytes"}]
|
|
replayed = overlay_cached_prefix(
|
|
processed,
|
|
history,
|
|
fresh.get_last_original_messages(),
|
|
fresh.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=fresh.get_frozen_message_count(),
|
|
)
|
|
assert replayed == processed
|
|
assert replayed != stale_forwarded
|
|
|
|
def test_expired_legacy_tracker_rejoins_index_without_cross_affinity_replay(self, monkeypatch):
|
|
"""Evicting a legacy bare tracker must not lose the new lineage index."""
|
|
now = [1_000.0]
|
|
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
|
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, session_ttl_seconds=1))
|
|
store.get_or_create("shared", "anthropic")
|
|
now[0] += 2
|
|
store._last_cleanup = now[0]
|
|
history = self._history("shared", 1)
|
|
first = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-a")
|
|
first.update_from_response(
|
|
0,
|
|
1000,
|
|
[{"role": "user", "content": "profile A frozen bytes"}],
|
|
message_token_counts=[1000],
|
|
original_messages=history,
|
|
)
|
|
assert first.get_frozen_message_count() == 1
|
|
|
|
second = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-b")
|
|
current = [{"role": "user", "content": "profile B bytes"}]
|
|
assert (
|
|
overlay_cached_prefix(
|
|
current,
|
|
history,
|
|
second.get_last_original_messages(),
|
|
second.get_last_forwarded_messages(),
|
|
confirmed_frozen_count=second.get_frozen_message_count(),
|
|
)
|
|
== current
|
|
)
|
|
|
|
def test_empty_canonical_history_falls_back_to_legacy(self):
|
|
"""A history whose every message projects away (pure directive
|
|
content) carries no lineage signal — behave like get_or_create."""
|
|
store = SessionTrackerStore()
|
|
tracker = store.resolve_tracker("sid", "anthropic", messages=[{}])
|
|
assert tracker is store.get_or_create("sid", "anthropic")
|
|
|
|
def test_nan_in_tool_payload_does_not_split_lineage(self):
|
|
"""json.loads accepts bare NaN, and NaN != NaN — a resent history
|
|
containing one must still read as the same conversation."""
|
|
store = SessionTrackerStore()
|
|
sid = "shared"
|
|
turn1 = [
|
|
{"role": "user", "content": "run the tool " + "x" * 200},
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{"type": "tool_use", "id": "t1", "name": "score", "input": {"v": float("nan")}}
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": "ok"}],
|
|
},
|
|
]
|
|
t1 = store.resolve_tracker(sid, "anthropic", messages=turn1)
|
|
turn2 = turn1 + [
|
|
{"role": "assistant", "content": "done"},
|
|
{"role": "user", "content": "next"},
|
|
]
|
|
assert store.resolve_tracker(sid, "anthropic", messages=turn2) is t1
|
|
|
|
def test_expired_lineages_are_cleaned_up(self):
|
|
config = PrefixFreezeConfig(session_ttl_seconds=1)
|
|
store = SessionTrackerStore(default_config=config)
|
|
for name in ("A", "B"):
|
|
tracker = store.resolve_tracker("sid", "anthropic", messages=self._history(name, 1))
|
|
tracker._last_activity = time.time() - 2
|
|
|
|
store._last_cleanup = 0
|
|
store._maybe_cleanup()
|
|
assert store.active_sessions == 0
|
|
|
|
# The lineage index must not resurrect evicted trackers: extending an
|
|
# evicted conversation starts cold.
|
|
fresh = store.resolve_tracker("sid", "anthropic", messages=self._history("A", 2))
|
|
assert fresh._turn_number == 0
|
|
|
|
@staticmethod
|
|
def _reminder_turns() -> tuple[list[dict], list[dict]]:
|
|
"""Claude Code shape: a trailing system reminder replaced by the next turn."""
|
|
history = [
|
|
{"role": "user", "content": "Inspect example.py."},
|
|
{"role": "assistant", "content": "Reading the file."},
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "read-1",
|
|
"content": "original source line\n" * 400,
|
|
}
|
|
],
|
|
},
|
|
]
|
|
reminder = {"role": "system", "content": "Batch independent tools this turn."}
|
|
previous = [*history, reminder]
|
|
current = [
|
|
*history,
|
|
{"role": "assistant", "content": "Now run the checks."},
|
|
{"role": "user", "content": "The checks passed."},
|
|
dict(reminder),
|
|
]
|
|
return previous, current
|
|
|
|
def test_replaced_system_tail_keeps_confirmed_tracker(self, store):
|
|
"""The previous request ended in a system reminder the client swapped
|
|
for new turns. Its unchanged history must keep the tracker and its
|
|
confirmed frozen prefix instead of starting a cold lineage."""
|
|
previous, current = self._reminder_turns()
|
|
forwarded = [dict(m) for m in previous]
|
|
forwarded[2] = {**previous[2], "content": "previously cached compressed result"}
|
|
tracker = store.resolve_tracker("sid", "anthropic", previous, cache_affinity="a")
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=1_000_000,
|
|
messages=forwarded,
|
|
original_messages=previous,
|
|
message_token_counts=[100] * len(previous),
|
|
)
|
|
frozen = tracker.get_frozen_message_count()
|
|
assert frozen > 0
|
|
|
|
assert store.resolve_tracker("sid", "anthropic", current, cache_affinity="a") is tracker
|
|
assert tracker.get_frozen_message_count() == frozen
|
|
# The existing replay restores the unchanged prefix and stops at the
|
|
# replaced reminder; this turn's tail is forwarded as processed.
|
|
processed = [dict(m) for m in current]
|
|
processed[2] = {**current[2], "content": "a different compressed result"}
|
|
replayed = overlay_cached_prefix(
|
|
processed, current, previous, forwarded, confirmed_frozen_count=frozen
|
|
)
|
|
assert replayed[:3] == forwarded[:3]
|
|
assert replayed[3:] == processed[3:]
|
|
|
|
def test_moving_system_tail_records_full_snapshots(self, store):
|
|
"""The reminder moves and changes every turn; each turn stays on one
|
|
tracker and the store records the full current history."""
|
|
previous, current = self._reminder_turns()
|
|
tracker = store.resolve_tracker("sid", "anthropic", previous)
|
|
for i in range(4):
|
|
current[-1] = {"role": "system", "content": f"reminder {i}"}
|
|
assert store.resolve_tracker("sid", "anthropic", current) is tracker
|
|
assert store._lineages["sid"]["sid"][-1]["content"][0]["text"] == f"reminder {i}"
|
|
assert len(store._lineages["sid"]["sid"]) == len(current)
|
|
current = [
|
|
*current[:-1],
|
|
{"role": "assistant", "content": f"reply {i}"},
|
|
{"role": "user", "content": f"next {i}"},
|
|
current[-1],
|
|
]
|
|
|
|
def test_system_tail_fallback_respects_affinity_and_provider(self):
|
|
previous, current = self._reminder_turns()
|
|
for provider, affinity in (("anthropic", "other-tools"), ("openai", "a")):
|
|
store = SessionTrackerStore()
|
|
tracker = store.resolve_tracker("sid", provider, previous, cache_affinity="a")
|
|
resolved = store.resolve_tracker("sid", provider, current, cache_affinity=affinity)
|
|
assert resolved is not tracker
|
|
|
|
@pytest.mark.parametrize(
|
|
("index", "role"),
|
|
[(0, "user"), (0, "system"), (2, "system")],
|
|
ids=["changed-history", "changed-leading-system", "changed-historical-system"],
|
|
)
|
|
def test_system_tail_fallback_requires_unchanged_history(self, store, index, role):
|
|
"""Only the trailing reminder is set aside: any other change, including
|
|
a changed leading or historical system instruction, still diverges."""
|
|
previous, current = self._reminder_turns()
|
|
previous.insert(index, {"role": role, "content": "original"})
|
|
current.insert(index, {"role": role, "content": "changed"})
|
|
tracker = store.resolve_tracker("sid", "anthropic", previous)
|
|
assert store.resolve_tracker("sid", "anthropic", current) is not tracker
|
|
|
|
def test_swapped_final_instruction_is_not_a_continuation(self, store):
|
|
"""Same-length siblings that differ only in their final system message
|
|
stay separate: the new history must extend past the old reminder."""
|
|
first = [{"role": "user", "content": "task"}, {"role": "system", "content": "A"}]
|
|
second = [{"role": "user", "content": "task"}, {"role": "system", "content": "B"}]
|
|
tracker = store.resolve_tracker("sid", "anthropic", first)
|
|
assert store.resolve_tracker("sid", "anthropic", second) is not tracker
|
|
assert store.resolve_tracker("sid", "anthropic", first) is tracker
|
|
|
|
def test_ambiguous_system_tail_siblings_are_not_merged(self, store, monkeypatch):
|
|
"""Two lineages that differ only in their reminder (recorded here with
|
|
the fallback disabled) are an ambiguous match: start a fresh lineage
|
|
instead of guessing, and an exact match still wins."""
|
|
previous, current = self._reminder_turns()
|
|
sibling = [*previous[:-1], {"role": "system", "content": "other reminder"}]
|
|
monkeypatch.setenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE", "0")
|
|
tracker = store.resolve_tracker("sid", "anthropic", previous)
|
|
other = store.resolve_tracker("sid", "anthropic", sibling)
|
|
assert other is not tracker
|
|
monkeypatch.delenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE")
|
|
assert store.resolve_tracker("sid", "anthropic", current) not in (tracker, other)
|
|
assert store.resolve_tracker("sid", "anthropic", sibling) is other
|
|
|
|
def test_cache_delta_replays_prefix_past_replaced_system_tail(self, monkeypatch):
|
|
"""Cache mode's delta split leaves the replaced reminder out of the
|
|
replayed prefix, with or without the model reply the handlers record
|
|
after the request, and compresses everything after it as new."""
|
|
previous, current = self._reminder_turns()
|
|
reply = current[3]
|
|
forwarded = [*previous[:2], {**previous[2], "content": "compressed"}, previous[3], reply]
|
|
for recorded in ([*previous, reply], previous):
|
|
replay = forwarded[: len(recorded)]
|
|
prefix, delta = extract_cache_stable_delta(current, recorded, replay)
|
|
assert prefix == replay[:3] + replay[4:]
|
|
assert delta == current[len(prefix) :]
|
|
|
|
# A system message followed by a client turn is history, not a reminder.
|
|
changed = [*previous, {"role": "user", "content": "Continue."}]
|
|
dropped = [*previous[:3], changed[4], *current[3:]]
|
|
assert extract_cache_stable_delta(dropped, changed, changed) is None
|
|
# The client's copy of the reply must match the recorded one.
|
|
other = [*previous, {"role": "assistant", "content": "Something else."}]
|
|
assert extract_cache_stable_delta(current, other, other) is None
|
|
|
|
monkeypatch.setenv("HEADROOM_TRANSIENT_SYSTEM_LINEAGE", "0")
|
|
assert extract_cache_stable_delta(current, [*previous, reply], forwarded) is None
|
|
|
|
def test_shared_session_id_is_not_rotated(self, store):
|
|
"""Composition guard: lineage resolution must not leak into session-id
|
|
derivation — beta stickiness and the compression cache key on it."""
|
|
|
|
class MockRequest:
|
|
headers = {}
|
|
|
|
msgs_a = [{"role": "system", "content": "S"}, *self._history("A", 1)]
|
|
msgs_b = [{"role": "system", "content": "S"}, *self._history("B", 1)]
|
|
id_a = store.compute_session_id(MockRequest(), "claude-3", msgs_a)
|
|
id_b = store.compute_session_id(MockRequest(), "claude-3", msgs_b)
|
|
assert id_a == id_b
|
|
|
|
|
|
class TestMultiTurnScenario:
|
|
"""Integration-style tests simulating multi-turn conversations."""
|
|
|
|
def test_five_turn_conversation(self):
|
|
"""Simulate a 5-turn conversation with growing prefix."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
|
|
# Turn 1: System + User (cold start, no cache)
|
|
messages_t1 = [
|
|
{"role": "system", "content": "System prompt" * 200},
|
|
{"role": "user", "content": "Question 1"},
|
|
]
|
|
token_counts_t1 = [2000, 50]
|
|
|
|
assert tracker.get_frozen_message_count() == 0 # No freeze on turn 1
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=2050,
|
|
messages=messages_t1,
|
|
message_token_counts=token_counts_t1,
|
|
)
|
|
|
|
# Turn 2: Previous messages cached, new user message added
|
|
messages_t2 = messages_t1 + [
|
|
{"role": "assistant", "content": "Answer 1"},
|
|
{"role": "user", "content": "Question 2"},
|
|
]
|
|
token_counts_t2 = [2000, 50, 200, 50]
|
|
|
|
frozen = tracker.get_frozen_message_count()
|
|
assert frozen == 2 # System + User1 frozen
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=2050,
|
|
cache_write_tokens=250,
|
|
messages=messages_t2,
|
|
message_token_counts=token_counts_t2,
|
|
)
|
|
|
|
# Turn 3: Even more cached
|
|
messages_t3 = messages_t2 + [
|
|
{"role": "assistant", "content": "Answer 2"},
|
|
{"role": "user", "content": "Question 3"},
|
|
]
|
|
token_counts_t3 = [2000, 50, 200, 50, 200, 50]
|
|
|
|
frozen = tracker.get_frozen_message_count()
|
|
assert frozen == 4 # System + User1 + Asst1 + User2 frozen
|
|
|
|
tracker.update_from_response(
|
|
cache_read_tokens=2300,
|
|
cache_write_tokens=250,
|
|
messages=messages_t3,
|
|
message_token_counts=token_counts_t3,
|
|
)
|
|
|
|
# Verify turn count
|
|
assert tracker._turn_number == 3
|
|
|
|
def test_cache_bust_resets_freeze(self):
|
|
"""If cache is busted (0 read, 0 write), freeze should reset."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
|
|
messages = [
|
|
{"role": "system", "content": "System" * 200},
|
|
{"role": "user", "content": "Hello"},
|
|
]
|
|
|
|
# Turn 1: Cache established
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=2000,
|
|
messages=messages,
|
|
message_token_counts=[1500, 500],
|
|
)
|
|
assert tracker.get_frozen_message_count() == 2 # Both fit within 2000
|
|
|
|
# Turn 2: Cache bust (0 reads, system prompt changed)
|
|
tracker.update_from_response(
|
|
cache_read_tokens=0,
|
|
cache_write_tokens=0,
|
|
messages=messages,
|
|
message_token_counts=[1500, 500],
|
|
)
|
|
|
|
# After a bust with 0 total, freeze should reset
|
|
assert tracker.get_frozen_message_count() == 0
|
|
|
|
|
|
class TestClassifyCacheMiss:
|
|
"""Cache-miss attribution (#1313): TTL lapse vs prefix change vs unknown."""
|
|
|
|
BASE = [
|
|
{"role": "system", "content": "x" * 4000},
|
|
{"role": "user", "content": "hello"},
|
|
]
|
|
CHANGED = [
|
|
{"role": "system", "content": "DIFFERENT" * 400},
|
|
{"role": "user", "content": "hello"},
|
|
]
|
|
|
|
def _warm(self, tracker, messages, read=500, write=500):
|
|
"""Simulate a turn that left `messages` cached."""
|
|
tracker.update_from_response(
|
|
cache_read_tokens=read, cache_write_tokens=write, messages=messages
|
|
)
|
|
|
|
def test_cold_start_is_not_a_miss(self):
|
|
"""No prior cached prefix → cold start, is_miss False."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
result = tracker.classify_cache_miss(0, self.BASE)
|
|
assert result.is_miss is False
|
|
assert result.reason == MISS_COLD_START
|
|
|
|
def test_cache_read_is_a_hit(self):
|
|
"""A non-zero read on an expected-cached prefix is a hit, not a miss."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
self._warm(tracker, self.BASE)
|
|
result = tracker.classify_cache_miss(800, self.BASE)
|
|
assert result.is_miss is False
|
|
assert result.reason == "hit"
|
|
|
|
def test_ttl_expiry_when_idle_exceeds_ttl(self):
|
|
"""Idle longer than the cache TTL → ttl_expiry."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
self._warm(tracker, self.BASE)
|
|
result = tracker.classify_cache_miss(0, self.BASE, idle_seconds=400)
|
|
assert result.is_miss is True
|
|
assert result.reason == MISS_TTL_EXPIRY
|
|
assert result.ttl_exceeded is True
|
|
assert result.cache_ttl_seconds == 300
|
|
|
|
def test_ttl_wins_tie_when_prefix_also_changed(self):
|
|
"""When idle past TTL AND prefix changed, TTL expiry wins (docstring)."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
self._warm(tracker, self.BASE)
|
|
result = tracker.classify_cache_miss(0, self.CHANGED, idle_seconds=400)
|
|
assert result.reason == MISS_TTL_EXPIRY
|
|
assert result.ttl_exceeded is True
|
|
assert result.prefix_changed is True
|
|
|
|
def test_prefix_change_within_ttl(self):
|
|
"""Within TTL but the forwarded prefix differs → prefix_change."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
self._warm(tracker, self.BASE)
|
|
result = tracker.classify_cache_miss(0, self.CHANGED, idle_seconds=10)
|
|
assert result.is_miss is True
|
|
assert result.reason == MISS_PREFIX_CHANGE
|
|
assert result.prefix_changed is True
|
|
assert result.ttl_exceeded is False
|
|
|
|
def test_unknown_when_stable_prefix_within_ttl(self):
|
|
"""Within TTL, prefix unchanged, but still no read → unknown."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
self._warm(tracker, self.BASE)
|
|
result = tracker.classify_cache_miss(0, self.BASE, idle_seconds=10)
|
|
assert result.is_miss is True
|
|
assert result.reason == MISS_UNKNOWN
|
|
|
|
def test_growing_prefix_is_stable(self):
|
|
"""A turn that appends to last turn's forwarded prefix is not a change."""
|
|
tracker = PrefixCacheTracker("anthropic")
|
|
self._warm(tracker, self.BASE)
|
|
grown = self.BASE + [{"role": "assistant", "content": "hi back"}]
|
|
result = tracker.classify_cache_miss(0, grown, idle_seconds=10)
|
|
# Prefix preserved (only appended) → not a prefix_change.
|
|
assert result.prefix_changed is False
|
|
assert result.reason == MISS_UNKNOWN
|
|
|
|
def test_one_hour_ttl_override(self):
|
|
"""cache_ttl_seconds override widens the TTL window (1h breakpoint)."""
|
|
tracker = PrefixCacheTracker("anthropic", PrefixFreezeConfig(cache_ttl_seconds=3600))
|
|
self._warm(tracker, self.BASE)
|
|
# 400s idle is past the 300s default but within 3600s → not TTL expiry.
|
|
result = tracker.classify_cache_miss(0, self.BASE, idle_seconds=400)
|
|
assert result.cache_ttl_seconds == 3600
|
|
assert result.ttl_exceeded is False
|
|
assert result.reason == MISS_UNKNOWN
|
|
|
|
def test_resolved_ttl_falls_back_to_provider_default(self):
|
|
assert PrefixCacheTracker("anthropic").resolved_cache_ttl_seconds() == 300
|
|
assert (
|
|
PrefixCacheTracker(
|
|
"anthropic", PrefixFreezeConfig(cache_ttl_seconds=3600)
|
|
).resolved_cache_ttl_seconds()
|
|
== 3600
|
|
)
|