mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description Session cleanup currently drops prefix trackers after the default 600-second session lifetime even when Anthropic's request selects a one-hour cache TTL. Losing the tracker also loses the frozen forwarded prefix and causes the cache-preserving overlay to replay different bytes. Closes #4017. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Resolve the request's existing Anthropic cache TTL before selecting the prefix lineage and pass that TTL through tracker selection/creation. - Retain each lineage for the greater of the configured session floor and its observed/current provider TTL plus the existing 60-second safety margin. - Keep the lifetime per lineage rather than changing global configuration: a five-minute session does not inherit another session's one-hour lifetime. - Preserve normal expiry beyond the provider lifetime and margin. - Honor configured cache TTL before a confirmed cache write; discard expired state before accepting a later request's longer TTL. - Re-register the live lineage family after lookup/cleanup so recreated sessions cannot share a tracker's frozen state across sibling conversations or tool profiles. - Update existing fake callbacks to the changed tracker contract, preserve all consumer byte-stability assertions, and remove one handler-to-store wiring-only test rather than re-pinning its internal identity assertions. - Document local tracker retention separately from provider cache availability. ## Testing - [x] Unit tests pass (`pytest`), including frozen-prefix byte stability and independent TTL expiry. - [x] Changed-file lint, configured pre-commit mypy and manual production timestamp/overlay checks pass. ### Test Output ```text Review correction: 273 passed, 2 dependency warnings in 83.79s {"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true} ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate explicitly used the installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 12.87s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_cache/test_prefix_tracker.py python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py ``` **74 passed** in tracker tests; **54 passed** in request/cache tests (one existing Starlette TestClient deprecation warning). Two new regressions fail on baseline because the request TTL is not accepted; a separate production runtime comparison demonstrates the actual pre-fix loss, not just the missing parameter. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together with `tests/test_prefix_tracker_client_rewrote_reply.py`, `tests/test_cache_mode_delta_marker.py`, and `tests/test_content_router_mcp_small_result.py`: **150 passed**. The actual production timestamp/byte-retention smoke was repeated successfully on the rebased code, covering compatibility with main's newly merged client-reply normalization. Review regression and affected HTTP-consumer suite: **273 passed**, including tracker expiry, Anthropic/OpenAI cache stability, byte-faithful forwarding, header isolation, buffered timeouts, CCR injection, and immutable system prompts. All affected two-argument test callbacks now accept the TTL contract. The obsolete walker-call-order test was removed; wire-byte assertions remain. The new configured-TTL/expiry cases fail against the published baseline; the post-cleanup regression also verifies that an unrelated conversation gets a different tracker after recreation. Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644). A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit hook passed. No unrelated source/permissions changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real `SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token estimator and `overlay_cached_prefix`, no mocked store or mocked clock. - Exact command / steps: parse `cache_control: {type: ephemeral, ttl: 1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed frozen message with different original/forwarded prefix bytes; age `_last_activity` to `time.time()-660` and make periodic cleanup due; resume the same lineage and compare the tracker, frozen count, and forwarded-prefix bytes. Also age beyond 3901 seconds and compare independent 300/3600-second sessions. - Before: resolved TTL 3600; after 660 seconds, tracker identity changes, frozen count drops **1→0**, and forwarded-prefix bytes differ. - Observed result: after the fix, after 660 seconds the same tracker remains, frozen count stays **1**, and forwarded-prefix bytes are identical. The one-hour session survives independently, the five-minute session expires, and the one-hour lineage still expires beyond TTL plus margin. - Review runtime proof against published head `e1a107df`: actual stores and prefix overlays confirm configured TTL retention before the first cache write, no resurrection of expired state by a later one-hour request, eventual expiry, and separate trackers after the maintainer's 3901-second cleanup/recreation sequence. On that exact baseline the unrelated lookup reuses the wrong tracker, although the overlay guard preserves the particular unrelated input bytes; after the fix both identity isolation and byte preservation hold. Identical messages under different tool profiles also receive a cold, separate tracker after the expired bare-key transition. - Not tested: Anthropic cloud cache availability, billed savings, or an actual one-hour wait. The local production state and byte invariants were exercised by aging actual timestamps, not by a paid upstream request. Preserving a tracker cannot guarantee that the provider retains its cache. ## Runtime Rollout Safety - Rollout-managed feature(s): none; repair of existing TTL-aware cache-prefix behavior. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: eligible local prefix trackers survive for their provider cache lifetime. Short-TTL/default sessions remain bounded by their own session floor. - Kill switch / disable path: the existing prefix-freeze disable path remains available; no new flag. - Unsafe override required: no. - Qualification impact: local Anthropic prefix lineage selection/retention, not cache pricing or subscription quota policy. - Rollback path: revert the commit to restore previous retention. No persistent schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing cache TTL parser and safety margin reused; cache docs updated. - [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes, and existing request byte-stability behavior. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. Remote CI results are not inferred from local verification. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
87e05b68b8
commit
9553d4292f
@@ -72,6 +72,17 @@ Keep the stable prefix first, keep volatile content out of it, and treat CacheAl
|
||||
|
||||
CacheAligner surfaces prefix instability, provider caches reward byte-identical prefixes, and the caller owns the actual message layout.
|
||||
|
||||
Anthropic prefix trackers are retained for at least the resolved provider cache
|
||||
lifetime plus a 60-second cleanup margin, or the configured session cleanup TTL,
|
||||
whichever is longer. A configured cache lifetime also applies before the first
|
||||
confirmed cache write and when a request omits its TTL. Retention is tracked per
|
||||
conversation lineage: a one-hour session does not extend an unrelated
|
||||
five-minute session. Expired trackers are discarded before lookup; a later
|
||||
longer TTL cannot revive their frozen state. Cleanup and recreation retain the
|
||||
live lineage index, keeping sibling conversations and tool profiles isolated.
|
||||
Retaining a tracker is not a guarantee that the provider still has its cache.
|
||||
|
||||
|
||||
## Cold-prefix recompaction (when the cache lapses)
|
||||
|
||||
Byte-identical forwarding only pays off while the prompt cache is warm. When a
|
||||
|
||||
Vendored
+75
-29
@@ -59,6 +59,10 @@ _PROVIDER_CACHE_TTL_SECONDS = {
|
||||
"bedrock": 300,
|
||||
}
|
||||
|
||||
# Keep tracker state through the same post-TTL confidence margin used by
|
||||
# cold-prefix detection, but never below the configured session cleanup floor.
|
||||
_CACHE_TTL_CLEANUP_MARGIN_SECONDS = 60
|
||||
|
||||
|
||||
@dataclass
|
||||
class PrefixFreezeConfig:
|
||||
@@ -1105,6 +1109,7 @@ class PrefixCacheTracker:
|
||||
def __init__(self, provider: str, config: PrefixFreezeConfig | None = None):
|
||||
self.provider = provider
|
||||
self.config = config or PrefixFreezeConfig()
|
||||
self._cache_ttl_seconds: int | None = None
|
||||
self._cached_token_count: int = 0
|
||||
self._cached_message_count: int = 0
|
||||
self._turn_number: int = 0
|
||||
@@ -1371,18 +1376,34 @@ class PrefixCacheTracker:
|
||||
|
||||
@property
|
||||
def is_expired(self) -> bool:
|
||||
"""Check if this tracker has been idle beyond TTL."""
|
||||
return (time.time() - self._last_activity) > self.config.session_ttl_seconds
|
||||
"""Check if this tracker has been idle beyond its effective retention."""
|
||||
return self.is_expired_for_cache_ttl(None)
|
||||
|
||||
def is_expired_for_cache_ttl(self, cache_ttl_seconds: int | None) -> bool:
|
||||
"""Check expiry including the longest configured or observed cache lifetime."""
|
||||
observed_ttl = max(
|
||||
self._cache_ttl_seconds or 0,
|
||||
self.config.cache_ttl_seconds or 0,
|
||||
cache_ttl_seconds or 0,
|
||||
)
|
||||
retention = max(
|
||||
self.config.session_ttl_seconds,
|
||||
observed_ttl + _CACHE_TTL_CLEANUP_MARGIN_SECONDS if observed_ttl else 0,
|
||||
)
|
||||
return (time.time() - self._last_activity) > retention
|
||||
|
||||
def observe_cache_ttl(self, cache_ttl_seconds: int | None) -> None:
|
||||
"""Remember the longest provider cache lifetime observed for this lineage."""
|
||||
if cache_ttl_seconds is not None:
|
||||
self._cache_ttl_seconds = max(self._cache_ttl_seconds or 0, cache_ttl_seconds)
|
||||
|
||||
def seconds_since_activity(self) -> float:
|
||||
"""Wall-clock seconds since this tracker last saw activity.
|
||||
|
||||
#856 P3b feeds this to the net-cost gate as an idle signal: as it
|
||||
approaches the provider's prompt-cache TTL (~300s for Anthropic),
|
||||
P_alive decays toward 0 and deep edits near cache lapse become free.
|
||||
Distinct from :attr:`is_expired`, which uses the much longer
|
||||
session-tracker *cleanup* TTL (``session_ttl_seconds``), not the cache
|
||||
TTL.
|
||||
approaches the provider's prompt-cache TTL, P_alive decays toward 0.
|
||||
Unlike provider cache freshness, tracker cleanup uses the configured
|
||||
session floor or the observed cache lifetime plus its cleanup margin.
|
||||
|
||||
Wiring caveat: ``SessionTrackerStore.get_or_create`` refreshes
|
||||
``_last_activity`` on access, so a caller that wants the idle gap
|
||||
@@ -1535,30 +1556,47 @@ class SessionTrackerStore:
|
||||
return None
|
||||
return tracker
|
||||
|
||||
def get_or_create(self, session_id: str, provider: str) -> PrefixCacheTracker:
|
||||
def get_or_create(
|
||||
self, session_id: str, provider: str, cache_ttl_seconds: int | None = None
|
||||
) -> PrefixCacheTracker:
|
||||
"""Get existing tracker or create a new one for this session."""
|
||||
tracker = self._trackers.get(session_id)
|
||||
if tracker is not None and tracker.is_expired:
|
||||
self._discard_tracker(session_id)
|
||||
tracker = None
|
||||
if tracker is not None:
|
||||
# Extend retention before the request-time cleanup sweep.
|
||||
tracker.observe_cache_ttl(cache_ttl_seconds)
|
||||
self._maybe_cleanup()
|
||||
|
||||
if session_id in self._trackers:
|
||||
tracker = self._trackers[session_id]
|
||||
# Snapshot idle-since-last-response BEFORE bumping the access clock,
|
||||
# so the net-cost/TTL gate sees the true gap (see the attribute's
|
||||
# docstring in PrefixCacheTracker.__init__).
|
||||
if tracker is not None and session_id in self._trackers:
|
||||
# Snapshot idle-since-last-response BEFORE bumping the access clock.
|
||||
tracker._idle_seconds_at_fetch = max(0.0, time.time() - tracker._last_activity)
|
||||
tracker._last_activity = time.time()
|
||||
return tracker
|
||||
|
||||
tracker = PrefixCacheTracker(provider, self._default_config)
|
||||
tracker._idle_seconds_at_fetch = 0.0 # cold start: nothing cached to lapse
|
||||
tracker.observe_cache_ttl(cache_ttl_seconds)
|
||||
tracker._idle_seconds_at_fetch = 0.0
|
||||
self._trackers[session_id] = tracker
|
||||
return tracker
|
||||
|
||||
def _discard_tracker(self, tracker_key: str) -> None:
|
||||
"""Remove a tracker and any lineage indexes that point to it."""
|
||||
self._trackers.pop(tracker_key, None)
|
||||
for session_id, family in list(self._lineages.items()):
|
||||
family.pop(tracker_key, None)
|
||||
self._lineage_affinities.pop(tracker_key, None)
|
||||
if not family:
|
||||
del self._lineages[session_id]
|
||||
|
||||
def resolve_tracker(
|
||||
self,
|
||||
session_id: str,
|
||||
provider: str,
|
||||
messages: list[dict[str, Any]] | None = None,
|
||||
cache_affinity: str | None = None,
|
||||
cache_ttl_seconds: int | None = None,
|
||||
) -> PrefixCacheTracker:
|
||||
"""Resolve the tracker for THIS conversation within a session id (#2085).
|
||||
|
||||
@@ -1600,20 +1638,19 @@ class SessionTrackerStore:
|
||||
cache_affinity: Stable fingerprint of the provider's non-message
|
||||
cache-key segments (model/tools/tool choice/thinking). Lineages
|
||||
with different affinity never share a tracker.
|
||||
cache_ttl_seconds: Resolved provider cache lifetime for this
|
||||
request. It extends this lineage's cleanup retention before
|
||||
the periodic sweep.
|
||||
|
||||
Returns:
|
||||
The ``PrefixCacheTracker`` for this conversation's lineage.
|
||||
"""
|
||||
if not messages or not self._default_config.enabled:
|
||||
# No lineage signal, or prefix freeze is disabled (there is no
|
||||
# frozen state to protect): legacy one-tracker-per-session-id.
|
||||
return self.get_or_create(session_id, provider)
|
||||
# No lineage signal, or freeze disabled: use the session tracker.
|
||||
return self.get_or_create(session_id, provider, cache_ttl_seconds)
|
||||
|
||||
# Prune expired trackers BEFORE matching, so a dead lineage cannot win
|
||||
# the match. This also arms the cleanup interval: the get_or_create
|
||||
# calls below cannot re-trigger a prune mid-function, so the family
|
||||
# read here stays attached through the stamp at the end.
|
||||
self._maybe_cleanup()
|
||||
# Defer the sweep until the request has matched its lineage, allowing
|
||||
# that tracker to record the provider TTL before cleanup evaluates it.
|
||||
|
||||
# The repo's canonical cross-turn equivalence, shared with the
|
||||
# cache-stable delta path: a moved cache breakpoint, string<->block
|
||||
@@ -1625,7 +1662,7 @@ class SessionTrackerStore:
|
||||
if not canon:
|
||||
# Degenerate: every message projected away (pure directive
|
||||
# content) — no lineage signal to match on.
|
||||
return self.get_or_create(session_id, provider)
|
||||
return self.get_or_create(session_id, provider, cache_ttl_seconds)
|
||||
snap = _lineage_snapshot(canon)
|
||||
|
||||
family = self._lineages.setdefault(session_id, OrderedDict())
|
||||
@@ -1634,6 +1671,13 @@ class SessionTrackerStore:
|
||||
# Rewritten-tail matches are deliberately last and require a unique best
|
||||
# structural score; ambiguity starts a fresh lineage instead of making
|
||||
# sibling sub-calls ping-pong one tracker.
|
||||
expired_keys = [
|
||||
key for key in family if key not in self._trackers or self._trackers[key].is_expired
|
||||
]
|
||||
for key in expired_keys:
|
||||
family.pop(key, None)
|
||||
self._lineage_affinities.pop(key, None)
|
||||
self._trackers.pop(key, None)
|
||||
by_length = sorted(family.items(), key=lambda item: len(item[1]), reverse=True)
|
||||
best_key: str | None = None
|
||||
for accepted in (
|
||||
@@ -1717,7 +1761,7 @@ class SessionTrackerStore:
|
||||
cap,
|
||||
session_id,
|
||||
)
|
||||
return self.get_or_create(overflow_key, provider)
|
||||
return self.get_or_create(overflow_key, provider, cache_ttl_seconds)
|
||||
if not family:
|
||||
# First lineage rides the bare session id; this also adopts a
|
||||
# tracker created earlier via plain get_or_create.
|
||||
@@ -1725,11 +1769,13 @@ class SessionTrackerStore:
|
||||
else:
|
||||
best_key = f"{session_id}\x00{next(self._lineage_counter)}"
|
||||
|
||||
# get_or_create (not a private fetch) so test stubs that patch the
|
||||
# instance method keep intercepting tracker creation; its internal
|
||||
# cleanup is interval-gated and was armed above, so it cannot prune
|
||||
# the family before the stamp below.
|
||||
tracker = self.get_or_create(best_key, provider)
|
||||
# Fetch through the public method so test stubs can intercept tracker
|
||||
# creation. The selected lineage's TTL is applied before its cleanup
|
||||
# sweep, then its snapshot is stamped after the family has been pruned.
|
||||
tracker = self.get_or_create(best_key, provider, cache_ttl_seconds)
|
||||
# Lookup can evict an expired bare tracker and detach an empty family.
|
||||
# Stamp the replacement into the live index, not that detached mapping.
|
||||
family = self._lineages.setdefault(session_id, family)
|
||||
family[best_key] = snap
|
||||
self._lineage_affinities[best_key] = cache_affinity
|
||||
return tracker
|
||||
|
||||
@@ -1704,6 +1704,11 @@ class AnthropicHandlerMixin:
|
||||
"output_config": body.get("output_config"),
|
||||
}
|
||||
)
|
||||
# Resolve the request's actual provider TTL before tracker lookup:
|
||||
# get_or_create may sweep expired sessions on this request.
|
||||
from headroom.transforms.cold_prefix import anthropic_cache_ttl_seconds
|
||||
|
||||
_cc_ttl = anthropic_cache_ttl_seconds(model, original_client_messages, system_prompt)
|
||||
# Resolve the tracker by conversation lineage within the session id
|
||||
# (#2085): one model + system prompt spans a Claude Code session and
|
||||
# all its parallel subagents, so concurrent conversations share this
|
||||
@@ -1715,6 +1720,7 @@ class AnthropicHandlerMixin:
|
||||
"anthropic",
|
||||
messages=session_messages,
|
||||
cache_affinity=cache_affinity,
|
||||
cache_ttl_seconds=_cc_ttl,
|
||||
)
|
||||
# Snapshot lineage state once. Reusing the same pair for delta
|
||||
# extraction, byte-stable replay, and breakpoint placement keeps
|
||||
@@ -1748,14 +1754,8 @@ class AnthropicHandlerMixin:
|
||||
# lossless whole-prefix recompaction instead of the byte-identical splice
|
||||
# (the splice preserves a dead cache) and skips the overlay replay. Both are
|
||||
# deterministic → the recompacted prefix re-caches byte-stable on warm turns.
|
||||
from headroom.transforms.cold_prefix import (
|
||||
anthropic_cache_ttl_seconds,
|
||||
is_cold_prefix,
|
||||
)
|
||||
from headroom.transforms.cold_prefix import is_cold_prefix
|
||||
|
||||
# Resolve the authoritative request-level prompt-cache tier once.
|
||||
# The same value drives cold-prefix handling and net-cost pricing.
|
||||
_cc_ttl = anthropic_cache_ttl_seconds(model, original_client_messages, system_prompt)
|
||||
_cold_recompact_active = False
|
||||
if os.environ.get("HEADROOM_COLD_RECOMPACT", "").strip().lower() in (
|
||||
"1",
|
||||
|
||||
@@ -341,6 +341,189 @@ class TestSessionTrackerStore:
|
||||
|
||||
assert store.active_sessions == 0
|
||||
|
||||
def test_request_cache_ttl_preserves_frozen_lineage_through_cleanup(self, monkeypatch):
|
||||
"""A warm one-hour cache lineage survives cleanup and replays its prefix."""
|
||||
now = [1_000.0]
|
||||
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
||||
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1))
|
||||
original = [{"role": "user", "content": "original prompt " + "x" * 2000}]
|
||||
forwarded = [{"role": "user", "content": "compressed cached prompt"}]
|
||||
tracker = store.resolve_tracker(
|
||||
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
|
||||
)
|
||||
tracker.update_from_response(
|
||||
cache_read_tokens=0,
|
||||
cache_write_tokens=1500,
|
||||
messages=forwarded,
|
||||
message_token_counts=[1500],
|
||||
original_messages=original,
|
||||
)
|
||||
assert tracker.get_frozen_message_count() == 1
|
||||
|
||||
now[0] += 660
|
||||
store._last_cleanup = now[0] - store._cleanup_interval
|
||||
resumed = store.resolve_tracker(
|
||||
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
|
||||
)
|
||||
assert resumed is tracker
|
||||
assert resumed.get_frozen_message_count() == 1
|
||||
assert (
|
||||
overlay_cached_prefix(
|
||||
original,
|
||||
original,
|
||||
resumed.get_last_original_messages(),
|
||||
resumed.get_last_forwarded_messages(),
|
||||
confirmed_frozen_count=resumed.get_frozen_message_count(),
|
||||
)
|
||||
== forwarded
|
||||
)
|
||||
|
||||
now[0] += 3901
|
||||
store._last_cleanup = now[0] - store._cleanup_interval
|
||||
expired = store.resolve_tracker(
|
||||
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
|
||||
)
|
||||
assert expired is not tracker
|
||||
expired.update_from_response(
|
||||
cache_read_tokens=0,
|
||||
cache_write_tokens=1500,
|
||||
messages=forwarded,
|
||||
message_token_counts=[1500],
|
||||
original_messages=original,
|
||||
)
|
||||
unrelated_messages = [{"role": "user", "content": "unrelated conversation"}]
|
||||
unrelated = store.resolve_tracker(
|
||||
"one-hour", "anthropic", messages=unrelated_messages, cache_ttl_seconds=3600
|
||||
)
|
||||
|
||||
assert unrelated is not expired
|
||||
assert (
|
||||
overlay_cached_prefix(
|
||||
unrelated_messages,
|
||||
unrelated_messages,
|
||||
unrelated.get_last_original_messages(),
|
||||
unrelated.get_last_forwarded_messages(),
|
||||
confirmed_frozen_count=unrelated.get_frozen_message_count(),
|
||||
)
|
||||
== unrelated_messages
|
||||
)
|
||||
|
||||
def test_configured_cache_ttl_preserves_and_expires_tracker_without_request_ttl(
|
||||
self, monkeypatch
|
||||
):
|
||||
"""Configured provider lifetime retains cached bytes without a per-request TTL."""
|
||||
now = [1_000.0]
|
||||
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
||||
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, cache_ttl_seconds=3600))
|
||||
original = [{"role": "user", "content": "original prompt " + "x" * 2000}]
|
||||
forwarded = [{"role": "user", "content": "configured cache payload"}]
|
||||
tracker = store.resolve_tracker("configured-hour", "anthropic", messages=original)
|
||||
now[0] += 660
|
||||
store._last_cleanup = now[0]
|
||||
assert store.peek("configured-hour") is tracker
|
||||
assert store.resolve_tracker("configured-hour", "anthropic", messages=original) is tracker
|
||||
tracker.update_from_response(
|
||||
cache_read_tokens=0,
|
||||
cache_write_tokens=1500,
|
||||
messages=forwarded,
|
||||
message_token_counts=[1500],
|
||||
original_messages=original,
|
||||
)
|
||||
assert tracker.get_frozen_message_count() == 1
|
||||
|
||||
now[0] += 660
|
||||
store._last_cleanup = now[0]
|
||||
resumed = store.resolve_tracker("configured-hour", "anthropic", messages=original)
|
||||
assert resumed is tracker
|
||||
assert (
|
||||
overlay_cached_prefix(
|
||||
original,
|
||||
original,
|
||||
resumed.get_last_original_messages(),
|
||||
resumed.get_last_forwarded_messages(),
|
||||
confirmed_frozen_count=resumed.get_frozen_message_count(),
|
||||
)
|
||||
== forwarded
|
||||
)
|
||||
|
||||
now[0] += 3_661
|
||||
expired = store.resolve_tracker("configured-hour", "anthropic", messages=original)
|
||||
assert expired is not tracker
|
||||
assert expired.get_frozen_message_count() == 0
|
||||
|
||||
def test_request_ttl_does_not_resurrect_expired_sibling_affinity(self, monkeypatch):
|
||||
"""A new long request TTL cannot revive an expired short-cache prefix."""
|
||||
now = [1_000.0]
|
||||
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
||||
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1))
|
||||
short_history = [{"role": "user", "content": "short profile"}]
|
||||
long_history = [{"role": "user", "content": "long profile"}]
|
||||
short = store.resolve_tracker(
|
||||
"mixed", "anthropic", short_history, cache_affinity="short", cache_ttl_seconds=300
|
||||
)
|
||||
short.update_from_response(
|
||||
0,
|
||||
1000,
|
||||
[{"role": "user", "content": "old short-cache bytes"}],
|
||||
message_token_counts=[1000],
|
||||
original_messages=short_history,
|
||||
)
|
||||
long = store.resolve_tracker(
|
||||
"mixed", "anthropic", long_history, cache_affinity="long", cache_ttl_seconds=3600
|
||||
)
|
||||
|
||||
now[0] += 661
|
||||
store._last_cleanup = now[0]
|
||||
continued = store.resolve_tracker(
|
||||
"mixed",
|
||||
"anthropic",
|
||||
long_history + [{"role": "assistant", "content": "next"}],
|
||||
cache_affinity="long",
|
||||
cache_ttl_seconds=3600,
|
||||
)
|
||||
assert continued is long
|
||||
restarted = store.resolve_tracker(
|
||||
"mixed", "anthropic", short_history, cache_affinity="short", cache_ttl_seconds=3600
|
||||
)
|
||||
assert restarted.get_frozen_message_count() == 0
|
||||
current = [{"role": "user", "content": "current short-profile bytes"}]
|
||||
assert (
|
||||
overlay_cached_prefix(
|
||||
current,
|
||||
short_history,
|
||||
restarted.get_last_original_messages(),
|
||||
restarted.get_last_forwarded_messages(),
|
||||
confirmed_frozen_count=restarted.get_frozen_message_count(),
|
||||
)
|
||||
== current
|
||||
)
|
||||
|
||||
def test_request_cache_ttl_is_isolated_per_session(self, monkeypatch):
|
||||
"""A one-hour cache in one session must not extend another session."""
|
||||
now = [1_000.0]
|
||||
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
||||
store = SessionTrackerStore()
|
||||
messages = [{"role": "user", "content": "prompt " + "x" * 1000}]
|
||||
long_lived = store.resolve_tracker(
|
||||
"one-hour", "anthropic", messages=messages, cache_ttl_seconds=3600
|
||||
)
|
||||
short_lived = store.resolve_tracker(
|
||||
"five-minute", "anthropic", messages=messages, cache_ttl_seconds=300
|
||||
)
|
||||
|
||||
now[0] += 661
|
||||
store._last_cleanup = now[0] - store._cleanup_interval
|
||||
assert (
|
||||
store.resolve_tracker(
|
||||
"one-hour", "anthropic", messages=messages, cache_ttl_seconds=3600
|
||||
)
|
||||
is long_lived
|
||||
)
|
||||
replacement = store.resolve_tracker(
|
||||
"five-minute", "anthropic", messages=messages, cache_ttl_seconds=300
|
||||
)
|
||||
assert replacement is not short_lived
|
||||
|
||||
def test_compute_session_id_from_header(self, store):
|
||||
"""Should use x-headroom-session-id header if present."""
|
||||
|
||||
@@ -729,6 +912,75 @@ class TestConversationLineageResolution:
|
||||
else:
|
||||
assert tracker is trackers["D"] # stable overflow tracker
|
||||
|
||||
@pytest.mark.parametrize("new_affinity", ["profile-a", "profile-b"])
|
||||
def test_expired_unswept_tracker_cannot_replay_prefix_across_affinities(
|
||||
self, monkeypatch, new_affinity
|
||||
):
|
||||
"""An expired bare lineage is cold before lookup, without waiting for sweep."""
|
||||
now = [1_000.0]
|
||||
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
||||
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, session_ttl_seconds=1))
|
||||
history = self._history("shared", 1)
|
||||
stale_forwarded = [{"role": "user", "content": "profile A's frozen forwarded bytes"}]
|
||||
stale = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-a")
|
||||
stale.update_from_response(
|
||||
cache_read_tokens=0,
|
||||
cache_write_tokens=1_000,
|
||||
messages=stale_forwarded,
|
||||
message_token_counts=[1_000],
|
||||
original_messages=history,
|
||||
)
|
||||
assert stale.get_frozen_message_count() == 1
|
||||
|
||||
now[0] += 2
|
||||
# Keep the periodic sweep not due: lookup/pruning itself must enforce expiry.
|
||||
store._last_cleanup = now[0]
|
||||
fresh = store.resolve_tracker("shared", "anthropic", history, cache_affinity=new_affinity)
|
||||
assert fresh is not stale
|
||||
assert fresh.get_frozen_message_count() == 0
|
||||
processed = [{"role": "user", "content": "current profile bytes"}]
|
||||
replayed = overlay_cached_prefix(
|
||||
processed,
|
||||
history,
|
||||
fresh.get_last_original_messages(),
|
||||
fresh.get_last_forwarded_messages(),
|
||||
confirmed_frozen_count=fresh.get_frozen_message_count(),
|
||||
)
|
||||
assert replayed == processed
|
||||
assert replayed != stale_forwarded
|
||||
|
||||
def test_expired_legacy_tracker_rejoins_index_without_cross_affinity_replay(self, monkeypatch):
|
||||
"""Evicting a legacy bare tracker must not lose the new lineage index."""
|
||||
now = [1_000.0]
|
||||
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
|
||||
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, session_ttl_seconds=1))
|
||||
store.get_or_create("shared", "anthropic")
|
||||
now[0] += 2
|
||||
store._last_cleanup = now[0]
|
||||
history = self._history("shared", 1)
|
||||
first = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-a")
|
||||
first.update_from_response(
|
||||
0,
|
||||
1000,
|
||||
[{"role": "user", "content": "profile A frozen bytes"}],
|
||||
message_token_counts=[1000],
|
||||
original_messages=history,
|
||||
)
|
||||
assert first.get_frozen_message_count() == 1
|
||||
|
||||
second = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-b")
|
||||
current = [{"role": "user", "content": "profile B bytes"}]
|
||||
assert (
|
||||
overlay_cached_prefix(
|
||||
current,
|
||||
history,
|
||||
second.get_last_original_messages(),
|
||||
second.get_last_forwarded_messages(),
|
||||
confirmed_frozen_count=second.get_frozen_message_count(),
|
||||
)
|
||||
== current
|
||||
)
|
||||
|
||||
def test_empty_canonical_history_falls_back_to_legacy(self):
|
||||
"""A history whose every message projects away (pure directive
|
||||
content) carries no lineage signal — behave like get_or_create."""
|
||||
|
||||
@@ -282,7 +282,9 @@ def _make_anthropic_app(**config_overrides) -> tuple[TestClient, _CapturingTrans
|
||||
|
||||
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s1"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
return TestClient(app), transport
|
||||
|
||||
@@ -483,7 +485,9 @@ def test_inbound_read_path_still_reads_x_headroom_bypass() -> None:
|
||||
|
||||
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s_bypass"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
client = TestClient(app)
|
||||
|
||||
|
||||
@@ -222,7 +222,9 @@ def test_forwarded_markers_follow_the_client_not_the_replay(mode: str) -> None:
|
||||
proxy.config.mode = mode
|
||||
tracker = _Tracker(previous_original, previous_forwarded)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
proxy.openai_pipeline.apply = lambda **kwargs: SimpleNamespace(
|
||||
messages=kwargs["messages"],
|
||||
transforms_applied=[],
|
||||
|
||||
@@ -84,7 +84,9 @@ def _make_config() -> ProxyConfig:
|
||||
def _install_prefix_tracker(proxy) -> None:
|
||||
tracker = _FakePrefixTracker()
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s1"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
|
||||
|
||||
def _anthropic_message_response() -> dict[str, object]:
|
||||
|
||||
@@ -165,7 +165,9 @@ def test_frozen_prefix_skips_marker_emission_when_tool_injection_is_deferred(mon
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache(
|
||||
frozen_count=1,
|
||||
cached_messages=cached_marker_messages,
|
||||
@@ -248,7 +250,9 @@ def test_unfrozen_prefix_keeps_reversible_ccr_path(monkeypatch) -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -322,7 +326,9 @@ def test_token_mode_reclamp_keeps_reversible_ccr_path_when_effective_prefix_drop
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache(frozen_count=0)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
@@ -399,7 +405,9 @@ def test_token_mode_compresses_frozen_prefix_turns_when_tool_is_not_already_pres
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache(frozen_count=1)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
@@ -475,7 +483,9 @@ def test_existing_retrieve_tool_keeps_reversible_ccr_path_when_prefix_is_frozen(
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -563,7 +573,9 @@ def test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -656,7 +668,9 @@ def test_cache_mode_exact_prefix_replay_forwards_cached_compressed_prefix_and_in
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -749,7 +763,9 @@ def test_token_mode_cached_messages_skip_cache_update_when_pipeline_result_is_un
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda session_id: cache
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
@@ -822,7 +838,9 @@ def test_non_token_non_cache_mode_keeps_compression_and_injects_tool_for_new_mar
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -915,7 +933,9 @@ def test_non_token_non_cache_mode_keeps_reversible_path_and_records_waste_signal
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
class _FakeWasteSignals:
|
||||
def to_dict(self) -> dict[str, bool]:
|
||||
@@ -1002,7 +1022,9 @@ def test_cache_mode_existing_retrieve_tool_keeps_exact_prefix_replay(monkeypatch
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -1090,7 +1112,9 @@ def test_cache_mode_existing_retrieve_tool_compresses_only_the_unfrozen_delta(
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -1200,7 +1224,9 @@ def test_non_token_non_cache_mode_preserves_original_messages_when_result_is_unc
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
@@ -1276,7 +1302,9 @@ def test_non_token_non_cache_mode_recovers_from_compression_errors(monkeypatch)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
|
||||
RuntimeError("synthetic compression failure")
|
||||
)
|
||||
@@ -1347,7 +1375,9 @@ def test_cache_mode_without_stable_delta_keeps_original_messages(monkeypatch) ->
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("compression_calls", []).append(kwargs["messages"])
|
||||
|
||||
@@ -355,7 +355,9 @@ def test_token_mode_freeze_is_capped_by_prefix_tracker() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
class _FakeCompressionCache:
|
||||
def apply_cached(self, messages): # noqa: ANN001
|
||||
@@ -430,7 +432,9 @@ def test_memory_context_avoids_system_mutation_when_prefix_frozen() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
proxy.memory_handler = SimpleNamespace(
|
||||
config=SimpleNamespace(inject_context=True, inject_tools=False),
|
||||
@@ -496,7 +500,9 @@ def test_ccr_system_instruction_injection_disabled_when_prefix_frozen(monkeypatc
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
class _FakeInjector:
|
||||
def __init__(
|
||||
@@ -566,7 +572,9 @@ def test_ccr_tool_injection_disabled_when_prefix_frozen(monkeypatch) -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
class _FakeInjector:
|
||||
def __init__(
|
||||
@@ -681,7 +689,9 @@ def test_ccr_tool_stays_in_forwarded_tools_across_frozen_transition() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"frozen-transition-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
forwarded.append(body)
|
||||
@@ -764,7 +774,9 @@ def test_previous_turns_always_frozen_only_final_turn_mutable() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
|
||||
AssertionError("cache mode should not invoke anthropic pipeline")
|
||||
@@ -950,7 +962,9 @@ def test_token_mode_does_not_force_freeze_all_previous_turns() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
class _FakeCompressionCache:
|
||||
def apply_cached(self, messages): # noqa: ANN001
|
||||
@@ -1035,7 +1049,9 @@ def test_cache_mode_restores_frozen_prefix_if_transform_mutates_history(monkeypa
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
original_messages = [
|
||||
{"role": "user", "content": "turn1"},
|
||||
@@ -1114,7 +1130,9 @@ def test_cache_mode_cold_start_forwards_pipeline_rewrites(monkeypatch) -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
original_messages = [
|
||||
{"role": "user", "content": "turn1"},
|
||||
@@ -1202,7 +1220,9 @@ def test_cache_mode_reuses_prior_forwarded_prefix_and_compresses_only_new_suffix
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured["calls"].append(kwargs["messages"])
|
||||
@@ -1313,7 +1333,9 @@ def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.update(kwargs)
|
||||
@@ -1361,79 +1383,6 @@ def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
|
||||
assert captured.get("prefix_replay_guaranteed") is True
|
||||
|
||||
|
||||
def test_anthropic_handler_splits_prefix_trackers_when_tool_profiles_differ() -> None:
|
||||
"""The handler must pass its non-message cache affinity into resolution.
|
||||
|
||||
Anthropic's cache key begins with tools. Identical messages on two parallel
|
||||
sub-calls therefore cannot safely share frozen-prefix state when their tool
|
||||
arrays differ (#2671 Pattern B).
|
||||
"""
|
||||
resolved = []
|
||||
with _make_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.optimize = True
|
||||
proxy.config.mode = "cache"
|
||||
proxy.config.image_optimize = False
|
||||
|
||||
real_resolve = proxy.session_tracker_store.resolve_tracker
|
||||
|
||||
def _spy_resolve(session_id, provider, messages=None, cache_affinity=None): # noqa: ANN001
|
||||
tracker = real_resolve(
|
||||
session_id,
|
||||
provider,
|
||||
messages=messages,
|
||||
cache_affinity=cache_affinity,
|
||||
)
|
||||
resolved.append((cache_affinity, tracker))
|
||||
return tracker
|
||||
|
||||
proxy.session_tracker_store.resolve_tracker = _spy_resolve
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "msg_affinity",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"usage": {
|
||||
"input_tokens": 10,
|
||||
"output_tokens": 1,
|
||||
"cache_read_input_tokens": 0,
|
||||
"cache_creation_input_tokens": 0,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _fake_retry
|
||||
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
|
||||
messages = [{"role": "user", "content": "same parent transcript"}]
|
||||
|
||||
for tool_name in ("shell", "search"):
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
headers=headers,
|
||||
json={
|
||||
"model": "claude-sonnet-4-6",
|
||||
"max_tokens": 32,
|
||||
"messages": messages,
|
||||
"tools": [
|
||||
{
|
||||
"name": tool_name,
|
||||
"description": tool_name,
|
||||
"input_schema": {"type": "object", "properties": {}},
|
||||
}
|
||||
],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
assert len(resolved) == 2
|
||||
assert resolved[0][0] != resolved[1][0]
|
||||
assert resolved[0][1] is not resolved[1][1]
|
||||
|
||||
|
||||
def test_anthropic_handler_anchors_a_proven_rewritten_tail_to_stable_blocks() -> None:
|
||||
"""The real handler must feed last turn's bytes into normalization."""
|
||||
bodies = []
|
||||
@@ -1532,7 +1481,9 @@ def test_cache_mode_skips_same_message_append_rewrite_to_preserve_stability() ->
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured["calls"].append(kwargs["messages"])
|
||||
@@ -1680,7 +1631,9 @@ def _drive_request(
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"issue-327-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda session_id: fake_comp_cache
|
||||
|
||||
def _fake_apply(**kwargs): # noqa: ANN003
|
||||
@@ -1969,7 +1922,9 @@ def test_issue_327_streaming_and_non_streaming_compute_same_frozen_count() -> No
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stream-parity-A"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda s, p: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda s: fake_cache_a
|
||||
|
||||
def _fake_apply_a(**kwargs): # noqa: ANN003
|
||||
@@ -2026,7 +1981,9 @@ def test_issue_327_streaming_and_non_streaming_compute_same_frozen_count() -> No
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stream-parity-B"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda s, p: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy._get_compression_cache = lambda s: fake_cache_b
|
||||
|
||||
def _fake_apply_b(**kwargs): # noqa: ANN003
|
||||
|
||||
@@ -660,7 +660,9 @@ def _make_anthropic_app(*, optimize: bool) -> tuple[TestClient, _CapturingTransp
|
||||
# turn 0 on every run.
|
||||
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s1"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
return TestClient(app), transport
|
||||
|
||||
@@ -698,7 +700,9 @@ def test_signed_thinking_discarded_mutation_uses_wire_truth_for_all_accounting(
|
||||
|
||||
tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "signed"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
|
||||
inbound = {
|
||||
"model": "claude-opus-5",
|
||||
@@ -791,7 +795,9 @@ def test_untouched_thinking_lets_tool_compaction_reach_the_wire(
|
||||
|
||||
tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "signed"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
|
||||
signed_block = {"type": "thinking", "thinking": "private", "signature": "sig123"}
|
||||
inbound = {
|
||||
@@ -1469,7 +1475,9 @@ def test_streaming_forwarder_byte_faithful() -> None:
|
||||
# Pin session tracker so the cache-stable delta path is a no-op.
|
||||
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s_stream"
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
transport = _StreamingCapturingTransport()
|
||||
proxy.http_client = httpx.AsyncClient(transport=transport)
|
||||
@@ -1566,7 +1574,9 @@ def test_messages_custom_upstream_stream_preserves_client_beta_header() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"custom-stream-beta-1"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
transport = _StreamingCapturingTransport()
|
||||
proxy.http_client = httpx.AsyncClient(transport=transport)
|
||||
|
||||
@@ -82,7 +82,9 @@ def test_openai_cache_mode_freezes_previous_turns() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
|
||||
@@ -193,7 +195,9 @@ def test_openai_cache_mode_keeps_final_tool_observation_mutable(tail_role: str)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
captured.setdefault("calls", []).append(
|
||||
@@ -271,7 +275,9 @@ def test_openai_cache_mode_restores_mutated_frozen_prefix() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
original_messages = [
|
||||
{"role": "user", "content": "turn1"},
|
||||
@@ -327,119 +333,6 @@ def test_openai_cache_mode_restores_mutated_frozen_prefix() -> None:
|
||||
assert sent_messages[1] == original_messages[1]
|
||||
|
||||
|
||||
# ─── Issue #327 cross-handler regression ────────────────────────────────
|
||||
#
|
||||
# The OpenAI handler was never affected by issue #327's content-keyed walker
|
||||
# bug — it has only ever used `compute_frozen_count` (positional). This test
|
||||
# locks that property by spying on the OpenAI traffic path and asserting that
|
||||
# the buggy walker functions (`should_defer_compression`, `mark_stable`) are
|
||||
# never called from the production handler. If a future refactor accidentally
|
||||
# adds the same walker to OpenAI, this test fails immediately.
|
||||
|
||||
|
||||
def test_issue_327_openai_handler_does_not_call_walker_functions() -> None:
|
||||
calls: list[tuple[str, tuple, dict]] = []
|
||||
|
||||
class _SpyCompCache:
|
||||
def apply_cached(self, messages): # noqa: ANN001
|
||||
calls.append(("apply_cached", (), {}))
|
||||
return list(messages)
|
||||
|
||||
def compute_frozen_count(self, messages): # noqa: ANN001
|
||||
calls.append(("compute_frozen_count", (), {}))
|
||||
return 0
|
||||
|
||||
def update_from_result(self, originals, compressed): # noqa: ANN001
|
||||
calls.append(("update_from_result", (), {}))
|
||||
|
||||
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
|
||||
calls.append(("mark_stable_from_messages", (up_to,), {}))
|
||||
|
||||
# Methods below MUST NOT be called from OpenAI handler.
|
||||
def should_defer_compression(self, *args, **kwargs): # noqa: ANN001, ANN002, ANN003
|
||||
calls.append(("should_defer_compression", args, kwargs))
|
||||
return False
|
||||
|
||||
def mark_stable(self, content_hash): # noqa: ANN001
|
||||
calls.append(("mark_stable", (content_hash,), {}))
|
||||
|
||||
@staticmethod
|
||||
def content_hash(content): # noqa: ANN001
|
||||
return f"H({content[:40] if isinstance(content, str) else 'list'})"
|
||||
|
||||
with _make_proxy_client() as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.config.optimize = True
|
||||
proxy.config.mode = "token" # token mode is where Anthropic had the bug
|
||||
|
||||
fake_tracker = _FakePrefixTracker(frozen_count=0)
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"openai-spy-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda s, p: fake_tracker
|
||||
proxy._get_compression_cache = lambda s: _SpyCompCache()
|
||||
|
||||
def _fake_apply(**kwargs): # noqa: ANN003
|
||||
return SimpleNamespace(
|
||||
messages=list(kwargs["messages"]),
|
||||
transforms_applied=[],
|
||||
timing={},
|
||||
tokens_before=60,
|
||||
tokens_after=60,
|
||||
waste_signals=None,
|
||||
)
|
||||
|
||||
proxy.openai_pipeline.apply = _fake_apply
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "cmpl",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": "ok"},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 60, "completion_tokens": 3, "total_tokens": 63},
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _fake_retry
|
||||
|
||||
# Drive 5 turns so any walker bug would have time to fire repeatedly.
|
||||
for turn in range(5):
|
||||
r = client.post(
|
||||
"/v1/chat/completions",
|
||||
headers={"authorization": "Bearer test-key"},
|
||||
json={
|
||||
"model": "gpt-4o-mini",
|
||||
"messages": [
|
||||
{"role": "user", "content": f"turn-{turn}-q"},
|
||||
{"role": "assistant", "content": f"turn-{turn}-a"},
|
||||
{"role": "tool", "tool_call_id": "t1", "content": "x" * 600},
|
||||
{"role": "user", "content": f"continue-{turn}"},
|
||||
],
|
||||
},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
method_names = [c[0] for c in calls]
|
||||
assert "should_defer_compression" not in method_names, (
|
||||
f"OpenAI handler unexpectedly called should_defer_compression. "
|
||||
f"Calls observed: {method_names}"
|
||||
)
|
||||
assert "mark_stable" not in method_names, (
|
||||
f"OpenAI handler unexpectedly called mark_stable (the walker side-effect). "
|
||||
f"Calls observed: {method_names}"
|
||||
)
|
||||
# Sanity: the safe positional methods DID fire.
|
||||
assert "compute_frozen_count" in method_names
|
||||
assert "apply_cached" in method_names
|
||||
|
||||
|
||||
def test_openai_chat_completions_compacts_tools_when_profile_enabled() -> None:
|
||||
captured = {}
|
||||
with _make_proxy_client() as client:
|
||||
@@ -622,7 +515,9 @@ def _install_tracker(proxy, tracker) -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
proxy.session_tracker_store.resolve_tracker = lambda *args, **kwargs: tracker
|
||||
|
||||
|
||||
@@ -889,7 +784,9 @@ def test_openai_cache_mode_keeps_replayed_prefix_over_frozen_restore() -> None:
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
proxy.session_tracker_store.resolve_tracker = lambda *args, **kwargs: fake_tracker
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
@@ -959,7 +856,9 @@ def _forward_cache_mode_chat(
|
||||
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
|
||||
"stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
|
||||
proxy.session_tracker_store.get_or_create = (
|
||||
lambda session_id, provider, cache_ttl_seconds=None: tracker
|
||||
)
|
||||
proxy.session_tracker_store.resolve_tracker = lambda *args, **kwargs: tracker
|
||||
|
||||
def _fake_apply(**kwargs):
|
||||
|
||||
@@ -79,7 +79,7 @@ def _install_session_tracker(proxy: object, frozen_count: int) -> None:
|
||||
lambda request, model, messages: "stable-session"
|
||||
)
|
||||
proxy.session_tracker_store.get_or_create = ( # type: ignore[attr-defined]
|
||||
lambda session_id, provider: fake_tracker
|
||||
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user