fix(cache): retain prefix trackers for the provider cache lifetime (#4023)

## Description

Session cleanup currently drops prefix trackers after the default
600-second session lifetime even when Anthropic's request selects a
one-hour cache TTL. Losing the tracker also loses the frozen forwarded
prefix and causes the cache-preserving overlay to replay different
bytes.

Closes #4017.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Resolve the request's existing Anthropic cache TTL before selecting
the prefix lineage and pass that TTL through tracker selection/creation.
- Retain each lineage for the greater of the configured session floor
and its observed/current provider TTL plus the existing 60-second safety
margin.
- Keep the lifetime per lineage rather than changing global
configuration: a five-minute session does not inherit another session's
one-hour lifetime.
- Preserve normal expiry beyond the provider lifetime and margin.
- Honor configured cache TTL before a confirmed cache write; discard
expired state before accepting a later request's longer TTL.
- Re-register the live lineage family after lookup/cleanup so recreated
sessions cannot share a tracker's frozen state across sibling
conversations or tool profiles.
- Update existing fake callbacks to the changed tracker contract,
preserve all consumer byte-stability assertions, and remove one
handler-to-store wiring-only test rather than re-pinning its internal
identity assertions.
- Document local tracker retention separately from provider cache
availability.

## Testing

- [x] Unit tests pass (`pytest`), including frozen-prefix byte stability
and independent TTL expiry.
- [x] Changed-file lint, configured pre-commit mypy and manual
production timestamp/overlay checks pass.

### Test Output

```text
Review correction: 273 passed, 2 dependency warnings in 83.79s
{"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true}
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 12.87s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_cache/test_prefix_tracker.py
python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py
```

**74 passed** in tracker tests; **54 passed** in request/cache tests
(one existing Starlette TestClient deprecation warning). Two new
regressions fail on baseline because the request TTL is not accepted; a
separate production runtime comparison demonstrates the actual pre-fix
loss, not just the missing parameter.

After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together
with `tests/test_prefix_tracker_client_rewrote_reply.py`,
`tests/test_cache_mode_delta_marker.py`, and
`tests/test_content_router_mcp_small_result.py`: **150 passed**. The
actual production timestamp/byte-retention smoke was repeated
successfully on the rebased code, covering compatibility with main's
newly merged client-reply normalization.

Review regression and affected HTTP-consumer suite: **273 passed**,
including tracker expiry, Anthropic/OpenAI cache stability,
byte-faithful forwarding, header isolation, buffered timeouts, CCR
injection, and immutable system prompts. All affected two-argument test
callbacks now accept the TTL contract. The obsolete walker-call-order
test was removed; wire-byte assertions remain. The new
configured-TTL/expiry cases fail against the published baseline; the
post-cleanup regression also verifies that an unrelated conversation
gets a different tracker after recreation.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff is locally blocked by executable
filesystem permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. No unrelated source/permissions changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real
`SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token
estimator and `overlay_cached_prefix`, no mocked store or mocked clock.
- Exact command / steps: parse `cache_control: {type: ephemeral, ttl:
1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed
frozen message with different original/forwarded prefix bytes; age
`_last_activity` to `time.time()-660` and make periodic cleanup due;
resume the same lineage and compare the tracker, frozen count, and
forwarded-prefix bytes. Also age beyond 3901 seconds and compare
independent 300/3600-second sessions.
- Before: resolved TTL 3600; after 660 seconds, tracker identity
changes, frozen count drops **1→0**, and forwarded-prefix bytes differ.
- Observed result: after the fix, after 660 seconds the same tracker
remains, frozen count stays **1**, and forwarded-prefix bytes are
identical. The one-hour session survives independently, the five-minute
session expires, and the one-hour lineage still expires beyond TTL plus
margin.
- Review runtime proof against published head `e1a107df`: actual stores
and prefix overlays confirm configured TTL retention before the first
cache write, no resurrection of expired state by a later one-hour
request, eventual expiry, and separate trackers after the maintainer's
3901-second cleanup/recreation sequence. On that exact baseline the
unrelated lookup reuses the wrong tracker, although the overlay guard
preserves the particular unrelated input bytes; after the fix both
identity isolation and byte preservation hold. Identical messages under
different tool profiles also receive a cold, separate tracker after the
expired bare-key transition.
- Not tested: Anthropic cloud cache availability, billed savings, or an
actual one-hour wait. The local production state and byte invariants
were exercised by aging actual timestamps, not by a paid upstream
request. Preserving a tracker cannot guarantee that the provider retains
its cache.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; repair of existing TTL-aware
cache-prefix behavior.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: eligible local prefix trackers
survive for their provider cache lifetime. Short-TTL/default sessions
remain bounded by their own session floor.
- Kill switch / disable path: the existing prefix-freeze disable path
remains available; no new flag.
- Unsafe override required: no.
- Qualification impact: local Anthropic prefix lineage
selection/retention, not cache pricing or subscription quota policy.
- Rollback path: revert the commit to restore previous retention. No
persistent schema change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing cache TTL parser and safety margin reused; cache docs
updated.
- [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes,
and existing request byte-stability behavior.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are not inferred from local verification.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Mansurshakh Japarov
2026-10-07 17:46:21 -05:00
committed by GitHub
co-authored by JD Davis
parent 87e05b68b8
commit 9553d4292f
12 changed files with 481 additions and 268 deletions
+11
View File
@@ -72,6 +72,17 @@ Keep the stable prefix first, keep volatile content out of it, and treat CacheAl
CacheAligner surfaces prefix instability, provider caches reward byte-identical prefixes, and the caller owns the actual message layout.
Anthropic prefix trackers are retained for at least the resolved provider cache
lifetime plus a 60-second cleanup margin, or the configured session cleanup TTL,
whichever is longer. A configured cache lifetime also applies before the first
confirmed cache write and when a request omits its TTL. Retention is tracked per
conversation lineage: a one-hour session does not extend an unrelated
five-minute session. Expired trackers are discarded before lookup; a later
longer TTL cannot revive their frozen state. Cleanup and recreation retain the
live lineage index, keeping sibling conversations and tool profiles isolated.
Retaining a tracker is not a guarantee that the provider still has its cache.
## Cold-prefix recompaction (when the cache lapses)
Byte-identical forwarding only pays off while the prompt cache is warm. When a
+75 -29
View File
@@ -59,6 +59,10 @@ _PROVIDER_CACHE_TTL_SECONDS = {
"bedrock": 300,
}
# Keep tracker state through the same post-TTL confidence margin used by
# cold-prefix detection, but never below the configured session cleanup floor.
_CACHE_TTL_CLEANUP_MARGIN_SECONDS = 60
@dataclass
class PrefixFreezeConfig:
@@ -1105,6 +1109,7 @@ class PrefixCacheTracker:
def __init__(self, provider: str, config: PrefixFreezeConfig | None = None):
self.provider = provider
self.config = config or PrefixFreezeConfig()
self._cache_ttl_seconds: int | None = None
self._cached_token_count: int = 0
self._cached_message_count: int = 0
self._turn_number: int = 0
@@ -1371,18 +1376,34 @@ class PrefixCacheTracker:
@property
def is_expired(self) -> bool:
"""Check if this tracker has been idle beyond TTL."""
return (time.time() - self._last_activity) > self.config.session_ttl_seconds
"""Check if this tracker has been idle beyond its effective retention."""
return self.is_expired_for_cache_ttl(None)
def is_expired_for_cache_ttl(self, cache_ttl_seconds: int | None) -> bool:
"""Check expiry including the longest configured or observed cache lifetime."""
observed_ttl = max(
self._cache_ttl_seconds or 0,
self.config.cache_ttl_seconds or 0,
cache_ttl_seconds or 0,
)
retention = max(
self.config.session_ttl_seconds,
observed_ttl + _CACHE_TTL_CLEANUP_MARGIN_SECONDS if observed_ttl else 0,
)
return (time.time() - self._last_activity) > retention
def observe_cache_ttl(self, cache_ttl_seconds: int | None) -> None:
"""Remember the longest provider cache lifetime observed for this lineage."""
if cache_ttl_seconds is not None:
self._cache_ttl_seconds = max(self._cache_ttl_seconds or 0, cache_ttl_seconds)
def seconds_since_activity(self) -> float:
"""Wall-clock seconds since this tracker last saw activity.
#856 P3b feeds this to the net-cost gate as an idle signal: as it
approaches the provider's prompt-cache TTL (~300s for Anthropic),
P_alive decays toward 0 and deep edits near cache lapse become free.
Distinct from :attr:`is_expired`, which uses the much longer
session-tracker *cleanup* TTL (``session_ttl_seconds``), not the cache
TTL.
approaches the provider's prompt-cache TTL, P_alive decays toward 0.
Unlike provider cache freshness, tracker cleanup uses the configured
session floor or the observed cache lifetime plus its cleanup margin.
Wiring caveat: ``SessionTrackerStore.get_or_create`` refreshes
``_last_activity`` on access, so a caller that wants the idle gap
@@ -1535,30 +1556,47 @@ class SessionTrackerStore:
return None
return tracker
def get_or_create(self, session_id: str, provider: str) -> PrefixCacheTracker:
def get_or_create(
self, session_id: str, provider: str, cache_ttl_seconds: int | None = None
) -> PrefixCacheTracker:
"""Get existing tracker or create a new one for this session."""
tracker = self._trackers.get(session_id)
if tracker is not None and tracker.is_expired:
self._discard_tracker(session_id)
tracker = None
if tracker is not None:
# Extend retention before the request-time cleanup sweep.
tracker.observe_cache_ttl(cache_ttl_seconds)
self._maybe_cleanup()
if session_id in self._trackers:
tracker = self._trackers[session_id]
# Snapshot idle-since-last-response BEFORE bumping the access clock,
# so the net-cost/TTL gate sees the true gap (see the attribute's
# docstring in PrefixCacheTracker.__init__).
if tracker is not None and session_id in self._trackers:
# Snapshot idle-since-last-response BEFORE bumping the access clock.
tracker._idle_seconds_at_fetch = max(0.0, time.time() - tracker._last_activity)
tracker._last_activity = time.time()
return tracker
tracker = PrefixCacheTracker(provider, self._default_config)
tracker._idle_seconds_at_fetch = 0.0 # cold start: nothing cached to lapse
tracker.observe_cache_ttl(cache_ttl_seconds)
tracker._idle_seconds_at_fetch = 0.0
self._trackers[session_id] = tracker
return tracker
def _discard_tracker(self, tracker_key: str) -> None:
"""Remove a tracker and any lineage indexes that point to it."""
self._trackers.pop(tracker_key, None)
for session_id, family in list(self._lineages.items()):
family.pop(tracker_key, None)
self._lineage_affinities.pop(tracker_key, None)
if not family:
del self._lineages[session_id]
def resolve_tracker(
self,
session_id: str,
provider: str,
messages: list[dict[str, Any]] | None = None,
cache_affinity: str | None = None,
cache_ttl_seconds: int | None = None,
) -> PrefixCacheTracker:
"""Resolve the tracker for THIS conversation within a session id (#2085).
@@ -1600,20 +1638,19 @@ class SessionTrackerStore:
cache_affinity: Stable fingerprint of the provider's non-message
cache-key segments (model/tools/tool choice/thinking). Lineages
with different affinity never share a tracker.
cache_ttl_seconds: Resolved provider cache lifetime for this
request. It extends this lineage's cleanup retention before
the periodic sweep.
Returns:
The ``PrefixCacheTracker`` for this conversation's lineage.
"""
if not messages or not self._default_config.enabled:
# No lineage signal, or prefix freeze is disabled (there is no
# frozen state to protect): legacy one-tracker-per-session-id.
return self.get_or_create(session_id, provider)
# No lineage signal, or freeze disabled: use the session tracker.
return self.get_or_create(session_id, provider, cache_ttl_seconds)
# Prune expired trackers BEFORE matching, so a dead lineage cannot win
# the match. This also arms the cleanup interval: the get_or_create
# calls below cannot re-trigger a prune mid-function, so the family
# read here stays attached through the stamp at the end.
self._maybe_cleanup()
# Defer the sweep until the request has matched its lineage, allowing
# that tracker to record the provider TTL before cleanup evaluates it.
# The repo's canonical cross-turn equivalence, shared with the
# cache-stable delta path: a moved cache breakpoint, string<->block
@@ -1625,7 +1662,7 @@ class SessionTrackerStore:
if not canon:
# Degenerate: every message projected away (pure directive
# content) — no lineage signal to match on.
return self.get_or_create(session_id, provider)
return self.get_or_create(session_id, provider, cache_ttl_seconds)
snap = _lineage_snapshot(canon)
family = self._lineages.setdefault(session_id, OrderedDict())
@@ -1634,6 +1671,13 @@ class SessionTrackerStore:
# Rewritten-tail matches are deliberately last and require a unique best
# structural score; ambiguity starts a fresh lineage instead of making
# sibling sub-calls ping-pong one tracker.
expired_keys = [
key for key in family if key not in self._trackers or self._trackers[key].is_expired
]
for key in expired_keys:
family.pop(key, None)
self._lineage_affinities.pop(key, None)
self._trackers.pop(key, None)
by_length = sorted(family.items(), key=lambda item: len(item[1]), reverse=True)
best_key: str | None = None
for accepted in (
@@ -1717,7 +1761,7 @@ class SessionTrackerStore:
cap,
session_id,
)
return self.get_or_create(overflow_key, provider)
return self.get_or_create(overflow_key, provider, cache_ttl_seconds)
if not family:
# First lineage rides the bare session id; this also adopts a
# tracker created earlier via plain get_or_create.
@@ -1725,11 +1769,13 @@ class SessionTrackerStore:
else:
best_key = f"{session_id}\x00{next(self._lineage_counter)}"
# get_or_create (not a private fetch) so test stubs that patch the
# instance method keep intercepting tracker creation; its internal
# cleanup is interval-gated and was armed above, so it cannot prune
# the family before the stamp below.
tracker = self.get_or_create(best_key, provider)
# Fetch through the public method so test stubs can intercept tracker
# creation. The selected lineage's TTL is applied before its cleanup
# sweep, then its snapshot is stamped after the family has been pruned.
tracker = self.get_or_create(best_key, provider, cache_ttl_seconds)
# Lookup can evict an expired bare tracker and detach an empty family.
# Stamp the replacement into the live index, not that detached mapping.
family = self._lineages.setdefault(session_id, family)
family[best_key] = snap
self._lineage_affinities[best_key] = cache_affinity
return tracker
+7 -7
View File
@@ -1704,6 +1704,11 @@ class AnthropicHandlerMixin:
"output_config": body.get("output_config"),
}
)
# Resolve the request's actual provider TTL before tracker lookup:
# get_or_create may sweep expired sessions on this request.
from headroom.transforms.cold_prefix import anthropic_cache_ttl_seconds
_cc_ttl = anthropic_cache_ttl_seconds(model, original_client_messages, system_prompt)
# Resolve the tracker by conversation lineage within the session id
# (#2085): one model + system prompt spans a Claude Code session and
# all its parallel subagents, so concurrent conversations share this
@@ -1715,6 +1720,7 @@ class AnthropicHandlerMixin:
"anthropic",
messages=session_messages,
cache_affinity=cache_affinity,
cache_ttl_seconds=_cc_ttl,
)
# Snapshot lineage state once. Reusing the same pair for delta
# extraction, byte-stable replay, and breakpoint placement keeps
@@ -1748,14 +1754,8 @@ class AnthropicHandlerMixin:
# lossless whole-prefix recompaction instead of the byte-identical splice
# (the splice preserves a dead cache) and skips the overlay replay. Both are
# deterministic → the recompacted prefix re-caches byte-stable on warm turns.
from headroom.transforms.cold_prefix import (
anthropic_cache_ttl_seconds,
is_cold_prefix,
)
from headroom.transforms.cold_prefix import is_cold_prefix
# Resolve the authoritative request-level prompt-cache tier once.
# The same value drives cold-prefix handling and net-cost pricing.
_cc_ttl = anthropic_cache_ttl_seconds(model, original_client_messages, system_prompt)
_cold_recompact_active = False
if os.environ.get("HEADROOM_COLD_RECOMPACT", "").strip().lower() in (
"1",
+252
View File
@@ -341,6 +341,189 @@ class TestSessionTrackerStore:
assert store.active_sessions == 0
def test_request_cache_ttl_preserves_frozen_lineage_through_cleanup(self, monkeypatch):
"""A warm one-hour cache lineage survives cleanup and replays its prefix."""
now = [1_000.0]
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1))
original = [{"role": "user", "content": "original prompt " + "x" * 2000}]
forwarded = [{"role": "user", "content": "compressed cached prompt"}]
tracker = store.resolve_tracker(
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
)
tracker.update_from_response(
cache_read_tokens=0,
cache_write_tokens=1500,
messages=forwarded,
message_token_counts=[1500],
original_messages=original,
)
assert tracker.get_frozen_message_count() == 1
now[0] += 660
store._last_cleanup = now[0] - store._cleanup_interval
resumed = store.resolve_tracker(
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
)
assert resumed is tracker
assert resumed.get_frozen_message_count() == 1
assert (
overlay_cached_prefix(
original,
original,
resumed.get_last_original_messages(),
resumed.get_last_forwarded_messages(),
confirmed_frozen_count=resumed.get_frozen_message_count(),
)
== forwarded
)
now[0] += 3901
store._last_cleanup = now[0] - store._cleanup_interval
expired = store.resolve_tracker(
"one-hour", "anthropic", messages=original, cache_ttl_seconds=3600
)
assert expired is not tracker
expired.update_from_response(
cache_read_tokens=0,
cache_write_tokens=1500,
messages=forwarded,
message_token_counts=[1500],
original_messages=original,
)
unrelated_messages = [{"role": "user", "content": "unrelated conversation"}]
unrelated = store.resolve_tracker(
"one-hour", "anthropic", messages=unrelated_messages, cache_ttl_seconds=3600
)
assert unrelated is not expired
assert (
overlay_cached_prefix(
unrelated_messages,
unrelated_messages,
unrelated.get_last_original_messages(),
unrelated.get_last_forwarded_messages(),
confirmed_frozen_count=unrelated.get_frozen_message_count(),
)
== unrelated_messages
)
def test_configured_cache_ttl_preserves_and_expires_tracker_without_request_ttl(
self, monkeypatch
):
"""Configured provider lifetime retains cached bytes without a per-request TTL."""
now = [1_000.0]
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, cache_ttl_seconds=3600))
original = [{"role": "user", "content": "original prompt " + "x" * 2000}]
forwarded = [{"role": "user", "content": "configured cache payload"}]
tracker = store.resolve_tracker("configured-hour", "anthropic", messages=original)
now[0] += 660
store._last_cleanup = now[0]
assert store.peek("configured-hour") is tracker
assert store.resolve_tracker("configured-hour", "anthropic", messages=original) is tracker
tracker.update_from_response(
cache_read_tokens=0,
cache_write_tokens=1500,
messages=forwarded,
message_token_counts=[1500],
original_messages=original,
)
assert tracker.get_frozen_message_count() == 1
now[0] += 660
store._last_cleanup = now[0]
resumed = store.resolve_tracker("configured-hour", "anthropic", messages=original)
assert resumed is tracker
assert (
overlay_cached_prefix(
original,
original,
resumed.get_last_original_messages(),
resumed.get_last_forwarded_messages(),
confirmed_frozen_count=resumed.get_frozen_message_count(),
)
== forwarded
)
now[0] += 3_661
expired = store.resolve_tracker("configured-hour", "anthropic", messages=original)
assert expired is not tracker
assert expired.get_frozen_message_count() == 0
def test_request_ttl_does_not_resurrect_expired_sibling_affinity(self, monkeypatch):
"""A new long request TTL cannot revive an expired short-cache prefix."""
now = [1_000.0]
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1))
short_history = [{"role": "user", "content": "short profile"}]
long_history = [{"role": "user", "content": "long profile"}]
short = store.resolve_tracker(
"mixed", "anthropic", short_history, cache_affinity="short", cache_ttl_seconds=300
)
short.update_from_response(
0,
1000,
[{"role": "user", "content": "old short-cache bytes"}],
message_token_counts=[1000],
original_messages=short_history,
)
long = store.resolve_tracker(
"mixed", "anthropic", long_history, cache_affinity="long", cache_ttl_seconds=3600
)
now[0] += 661
store._last_cleanup = now[0]
continued = store.resolve_tracker(
"mixed",
"anthropic",
long_history + [{"role": "assistant", "content": "next"}],
cache_affinity="long",
cache_ttl_seconds=3600,
)
assert continued is long
restarted = store.resolve_tracker(
"mixed", "anthropic", short_history, cache_affinity="short", cache_ttl_seconds=3600
)
assert restarted.get_frozen_message_count() == 0
current = [{"role": "user", "content": "current short-profile bytes"}]
assert (
overlay_cached_prefix(
current,
short_history,
restarted.get_last_original_messages(),
restarted.get_last_forwarded_messages(),
confirmed_frozen_count=restarted.get_frozen_message_count(),
)
== current
)
def test_request_cache_ttl_is_isolated_per_session(self, monkeypatch):
"""A one-hour cache in one session must not extend another session."""
now = [1_000.0]
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
store = SessionTrackerStore()
messages = [{"role": "user", "content": "prompt " + "x" * 1000}]
long_lived = store.resolve_tracker(
"one-hour", "anthropic", messages=messages, cache_ttl_seconds=3600
)
short_lived = store.resolve_tracker(
"five-minute", "anthropic", messages=messages, cache_ttl_seconds=300
)
now[0] += 661
store._last_cleanup = now[0] - store._cleanup_interval
assert (
store.resolve_tracker(
"one-hour", "anthropic", messages=messages, cache_ttl_seconds=3600
)
is long_lived
)
replacement = store.resolve_tracker(
"five-minute", "anthropic", messages=messages, cache_ttl_seconds=300
)
assert replacement is not short_lived
def test_compute_session_id_from_header(self, store):
"""Should use x-headroom-session-id header if present."""
@@ -729,6 +912,75 @@ class TestConversationLineageResolution:
else:
assert tracker is trackers["D"] # stable overflow tracker
@pytest.mark.parametrize("new_affinity", ["profile-a", "profile-b"])
def test_expired_unswept_tracker_cannot_replay_prefix_across_affinities(
self, monkeypatch, new_affinity
):
"""An expired bare lineage is cold before lookup, without waiting for sweep."""
now = [1_000.0]
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, session_ttl_seconds=1))
history = self._history("shared", 1)
stale_forwarded = [{"role": "user", "content": "profile A's frozen forwarded bytes"}]
stale = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-a")
stale.update_from_response(
cache_read_tokens=0,
cache_write_tokens=1_000,
messages=stale_forwarded,
message_token_counts=[1_000],
original_messages=history,
)
assert stale.get_frozen_message_count() == 1
now[0] += 2
# Keep the periodic sweep not due: lookup/pruning itself must enforce expiry.
store._last_cleanup = now[0]
fresh = store.resolve_tracker("shared", "anthropic", history, cache_affinity=new_affinity)
assert fresh is not stale
assert fresh.get_frozen_message_count() == 0
processed = [{"role": "user", "content": "current profile bytes"}]
replayed = overlay_cached_prefix(
processed,
history,
fresh.get_last_original_messages(),
fresh.get_last_forwarded_messages(),
confirmed_frozen_count=fresh.get_frozen_message_count(),
)
assert replayed == processed
assert replayed != stale_forwarded
def test_expired_legacy_tracker_rejoins_index_without_cross_affinity_replay(self, monkeypatch):
"""Evicting a legacy bare tracker must not lose the new lineage index."""
now = [1_000.0]
monkeypatch.setattr("headroom.cache.prefix_tracker.time.time", lambda: now[0])
store = SessionTrackerStore(PrefixFreezeConfig(min_cached_tokens=1, session_ttl_seconds=1))
store.get_or_create("shared", "anthropic")
now[0] += 2
store._last_cleanup = now[0]
history = self._history("shared", 1)
first = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-a")
first.update_from_response(
0,
1000,
[{"role": "user", "content": "profile A frozen bytes"}],
message_token_counts=[1000],
original_messages=history,
)
assert first.get_frozen_message_count() == 1
second = store.resolve_tracker("shared", "anthropic", history, cache_affinity="profile-b")
current = [{"role": "user", "content": "profile B bytes"}]
assert (
overlay_cached_prefix(
current,
history,
second.get_last_original_messages(),
second.get_last_forwarded_messages(),
confirmed_frozen_count=second.get_frozen_message_count(),
)
== current
)
def test_empty_canonical_history_falls_back_to_legacy(self):
"""A history whose every message projects away (pure directive
content) carries no lineage signal — behave like get_or_create."""
+6 -2
View File
@@ -282,7 +282,9 @@ def _make_anthropic_app(**config_overrides) -> tuple[TestClient, _CapturingTrans
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s1"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
return TestClient(app), transport
@@ -483,7 +485,9 @@ def test_inbound_read_path_still_reads_x_headroom_bypass() -> None:
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s_bypass"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
client = TestClient(app)
@@ -222,7 +222,9 @@ def test_forwarded_markers_follow_the_client_not_the_replay(mode: str) -> None:
proxy.config.mode = mode
tracker = _Tracker(previous_original, previous_forwarded)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
proxy.openai_pipeline.apply = lambda **kwargs: SimpleNamespace(
messages=kwargs["messages"],
transforms_applied=[],
@@ -84,7 +84,9 @@ def _make_config() -> ProxyConfig:
def _install_prefix_tracker(proxy) -> None:
tracker = _FakePrefixTracker()
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s1"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _anthropic_message_response() -> dict[str, object]:
@@ -165,7 +165,9 @@ def test_frozen_prefix_skips_marker_emission_when_tool_injection_is_deferred(mon
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache(
frozen_count=1,
cached_messages=cached_marker_messages,
@@ -248,7 +250,9 @@ def test_unfrozen_prefix_keeps_reversible_ccr_path(monkeypatch) -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -322,7 +326,9 @@ def test_token_mode_reclamp_keeps_reversible_ccr_path_when_effective_prefix_drop
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache(frozen_count=0)
def _fake_apply(**kwargs):
@@ -399,7 +405,9 @@ def test_token_mode_compresses_frozen_prefix_turns_when_tool_is_not_already_pres
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda session_id: _FakeCompressionCache(frozen_count=1)
def _fake_apply(**kwargs):
@@ -475,7 +483,9 @@ def test_existing_retrieve_tool_keeps_reversible_ccr_path_when_prefix_is_frozen(
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -563,7 +573,9 @@ def test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -656,7 +668,9 @@ def test_cache_mode_exact_prefix_replay_forwards_cached_compressed_prefix_and_in
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -749,7 +763,9 @@ def test_token_mode_cached_messages_skip_cache_update_when_pipeline_result_is_un
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda session_id: cache
def _fake_apply(**kwargs):
@@ -822,7 +838,9 @@ def test_non_token_non_cache_mode_keeps_compression_and_injects_tool_for_new_mar
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -915,7 +933,9 @@ def test_non_token_non_cache_mode_keeps_reversible_path_and_records_waste_signal
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeWasteSignals:
def to_dict(self) -> dict[str, bool]:
@@ -1002,7 +1022,9 @@ def test_cache_mode_existing_retrieve_tool_keeps_exact_prefix_replay(monkeypatch
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -1090,7 +1112,9 @@ def test_cache_mode_existing_retrieve_tool_compresses_only_the_unfrozen_delta(
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -1200,7 +1224,9 @@ def test_non_token_non_cache_mode_preserves_original_messages_when_result_is_unc
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
@@ -1276,7 +1302,9 @@ def test_non_token_non_cache_mode_recovers_from_compression_errors(monkeypatch)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
RuntimeError("synthetic compression failure")
)
@@ -1347,7 +1375,9 @@ def test_cache_mode_without_stable_delta_keeps_original_messages(monkeypatch) ->
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("compression_calls", []).append(kwargs["messages"])
+45 -88
View File
@@ -355,7 +355,9 @@ def test_token_mode_freeze_is_capped_by_prefix_tracker() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeCompressionCache:
def apply_cached(self, messages): # noqa: ANN001
@@ -430,7 +432,9 @@ def test_memory_context_avoids_system_mutation_when_prefix_frozen() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy.memory_handler = SimpleNamespace(
config=SimpleNamespace(inject_context=True, inject_tools=False),
@@ -496,7 +500,9 @@ def test_ccr_system_instruction_injection_disabled_when_prefix_frozen(monkeypatc
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeInjector:
def __init__(
@@ -566,7 +572,9 @@ def test_ccr_tool_injection_disabled_when_prefix_frozen(monkeypatch) -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeInjector:
def __init__(
@@ -681,7 +689,9 @@ def test_ccr_tool_stays_in_forwarded_tools_across_frozen_transition() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"frozen-transition-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
forwarded.append(body)
@@ -764,7 +774,9 @@ def test_previous_turns_always_frozen_only_final_turn_mutable() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy.anthropic_pipeline.apply = lambda **kwargs: (_ for _ in ()).throw(
AssertionError("cache mode should not invoke anthropic pipeline")
@@ -950,7 +962,9 @@ def test_token_mode_does_not_force_freeze_all_previous_turns() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
class _FakeCompressionCache:
def apply_cached(self, messages): # noqa: ANN001
@@ -1035,7 +1049,9 @@ def test_cache_mode_restores_frozen_prefix_if_transform_mutates_history(monkeypa
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
original_messages = [
{"role": "user", "content": "turn1"},
@@ -1114,7 +1130,9 @@ def test_cache_mode_cold_start_forwards_pipeline_rewrites(monkeypatch) -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
original_messages = [
{"role": "user", "content": "turn1"},
@@ -1202,7 +1220,9 @@ def test_cache_mode_reuses_prior_forwarded_prefix_and_compresses_only_new_suffix
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _fake_apply(**kwargs):
captured["calls"].append(kwargs["messages"])
@@ -1313,7 +1333,9 @@ def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _fake_apply(**kwargs):
captured.update(kwargs)
@@ -1361,79 +1383,6 @@ def test_cache_mode_delta_keeps_newest_user_prompt_verbatim() -> None:
assert captured.get("prefix_replay_guaranteed") is True
def test_anthropic_handler_splits_prefix_trackers_when_tool_profiles_differ() -> None:
"""The handler must pass its non-message cache affinity into resolution.
Anthropic's cache key begins with tools. Identical messages on two parallel
sub-calls therefore cannot safely share frozen-prefix state when their tool
arrays differ (#2671 Pattern B).
"""
resolved = []
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "cache"
proxy.config.image_optimize = False
real_resolve = proxy.session_tracker_store.resolve_tracker
def _spy_resolve(session_id, provider, messages=None, cache_affinity=None): # noqa: ANN001
tracker = real_resolve(
session_id,
provider,
messages=messages,
cache_affinity=cache_affinity,
)
resolved.append((cache_affinity, tracker))
return tracker
proxy.session_tracker_store.resolve_tracker = _spy_resolve
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "msg_affinity",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "ok"}],
"usage": {
"input_tokens": 10,
"output_tokens": 1,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0,
},
},
)
proxy._retry_request = _fake_retry
headers = {"x-api-key": "test-key", "anthropic-version": "2023-06-01"}
messages = [{"role": "user", "content": "same parent transcript"}]
for tool_name in ("shell", "search"):
response = client.post(
"/v1/messages",
headers=headers,
json={
"model": "claude-sonnet-4-6",
"max_tokens": 32,
"messages": messages,
"tools": [
{
"name": tool_name,
"description": tool_name,
"input_schema": {"type": "object", "properties": {}},
}
],
},
)
assert response.status_code == 200
assert len(resolved) == 2
assert resolved[0][0] != resolved[1][0]
assert resolved[0][1] is not resolved[1][1]
def test_anthropic_handler_anchors_a_proven_rewritten_tail_to_stable_blocks() -> None:
"""The real handler must feed last turn's bytes into normalization."""
bodies = []
@@ -1532,7 +1481,9 @@ def test_cache_mode_skips_same_message_append_rewrite_to_preserve_stability() ->
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
def _fake_apply(**kwargs):
captured["calls"].append(kwargs["messages"])
@@ -1680,7 +1631,9 @@ def _drive_request(
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"issue-327-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda session_id: fake_comp_cache
def _fake_apply(**kwargs): # noqa: ANN003
@@ -1969,7 +1922,9 @@ def test_issue_327_streaming_and_non_streaming_compute_same_frozen_count() -> No
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stream-parity-A"
)
proxy.session_tracker_store.get_or_create = lambda s, p: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda s: fake_cache_a
def _fake_apply_a(**kwargs): # noqa: ANN003
@@ -2026,7 +1981,9 @@ def test_issue_327_streaming_and_non_streaming_compute_same_frozen_count() -> No
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stream-parity-B"
)
proxy.session_tracker_store.get_or_create = lambda s, p: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy._get_compression_cache = lambda s: fake_cache_b
def _fake_apply_b(**kwargs): # noqa: ANN003
+15 -5
View File
@@ -660,7 +660,9 @@ def _make_anthropic_app(*, optimize: bool) -> tuple[TestClient, _CapturingTransp
# turn 0 on every run.
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s1"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
return TestClient(app), transport
@@ -698,7 +700,9 @@ def test_signed_thinking_discarded_mutation_uses_wire_truth_for_all_accounting(
tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "signed"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
inbound = {
"model": "claude-opus-5",
@@ -791,7 +795,9 @@ def test_untouched_thinking_lets_tool_compaction_reach_the_wire(
tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "signed"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
signed_block = {"type": "thinking", "thinking": "private", "signature": "sig123"}
inbound = {
@@ -1469,7 +1475,9 @@ def test_streaming_forwarder_byte_faithful() -> None:
# Pin session tracker so the cache-stable delta path is a no-op.
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: "s_stream"
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
transport = _StreamingCapturingTransport()
proxy.http_client = httpx.AsyncClient(transport=transport)
@@ -1566,7 +1574,9 @@ def test_messages_custom_upstream_stream_preserves_client_beta_header() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"custom-stream-beta-1"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
transport = _StreamingCapturingTransport()
proxy.http_client = httpx.AsyncClient(transport=transport)
+18 -119
View File
@@ -82,7 +82,9 @@ def test_openai_cache_mode_freezes_previous_turns() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured["frozen_message_count"] = kwargs.get("frozen_message_count")
@@ -193,7 +195,9 @@ def test_openai_cache_mode_keeps_final_tool_observation_mutable(tail_role: str)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
def _fake_apply(**kwargs):
captured.setdefault("calls", []).append(
@@ -271,7 +275,9 @@ def test_openai_cache_mode_restores_mutated_frozen_prefix() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
original_messages = [
{"role": "user", "content": "turn1"},
@@ -327,119 +333,6 @@ def test_openai_cache_mode_restores_mutated_frozen_prefix() -> None:
assert sent_messages[1] == original_messages[1]
# ─── Issue #327 cross-handler regression ────────────────────────────────
#
# The OpenAI handler was never affected by issue #327's content-keyed walker
# bug — it has only ever used `compute_frozen_count` (positional). This test
# locks that property by spying on the OpenAI traffic path and asserting that
# the buggy walker functions (`should_defer_compression`, `mark_stable`) are
# never called from the production handler. If a future refactor accidentally
# adds the same walker to OpenAI, this test fails immediately.
def test_issue_327_openai_handler_does_not_call_walker_functions() -> None:
calls: list[tuple[str, tuple, dict]] = []
class _SpyCompCache:
def apply_cached(self, messages): # noqa: ANN001
calls.append(("apply_cached", (), {}))
return list(messages)
def compute_frozen_count(self, messages): # noqa: ANN001
calls.append(("compute_frozen_count", (), {}))
return 0
def update_from_result(self, originals, compressed): # noqa: ANN001
calls.append(("update_from_result", (), {}))
def mark_stable_from_messages(self, messages, up_to): # noqa: ANN001
calls.append(("mark_stable_from_messages", (up_to,), {}))
# Methods below MUST NOT be called from OpenAI handler.
def should_defer_compression(self, *args, **kwargs): # noqa: ANN001, ANN002, ANN003
calls.append(("should_defer_compression", args, kwargs))
return False
def mark_stable(self, content_hash): # noqa: ANN001
calls.append(("mark_stable", (content_hash,), {}))
@staticmethod
def content_hash(content): # noqa: ANN001
return f"H({content[:40] if isinstance(content, str) else 'list'})"
with _make_proxy_client() as client:
proxy = client.app.state.proxy
proxy.config.optimize = True
proxy.config.mode = "token" # token mode is where Anthropic had the bug
fake_tracker = _FakePrefixTracker(frozen_count=0)
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"openai-spy-session"
)
proxy.session_tracker_store.get_or_create = lambda s, p: fake_tracker
proxy._get_compression_cache = lambda s: _SpyCompCache()
def _fake_apply(**kwargs): # noqa: ANN003
return SimpleNamespace(
messages=list(kwargs["messages"]),
transforms_applied=[],
timing={},
tokens_before=60,
tokens_after=60,
waste_signals=None,
)
proxy.openai_pipeline.apply = _fake_apply
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
return httpx.Response(
200,
json={
"id": "cmpl",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 60, "completion_tokens": 3, "total_tokens": 63},
},
)
proxy._retry_request = _fake_retry
# Drive 5 turns so any walker bug would have time to fire repeatedly.
for turn in range(5):
r = client.post(
"/v1/chat/completions",
headers={"authorization": "Bearer test-key"},
json={
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": f"turn-{turn}-q"},
{"role": "assistant", "content": f"turn-{turn}-a"},
{"role": "tool", "tool_call_id": "t1", "content": "x" * 600},
{"role": "user", "content": f"continue-{turn}"},
],
},
)
assert r.status_code == 200
method_names = [c[0] for c in calls]
assert "should_defer_compression" not in method_names, (
f"OpenAI handler unexpectedly called should_defer_compression. "
f"Calls observed: {method_names}"
)
assert "mark_stable" not in method_names, (
f"OpenAI handler unexpectedly called mark_stable (the walker side-effect). "
f"Calls observed: {method_names}"
)
# Sanity: the safe positional methods DID fire.
assert "compute_frozen_count" in method_names
assert "apply_cached" in method_names
def test_openai_chat_completions_compacts_tools_when_profile_enabled() -> None:
captured = {}
with _make_proxy_client() as client:
@@ -622,7 +515,9 @@ def _install_tracker(proxy, tracker) -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
proxy.session_tracker_store.resolve_tracker = lambda *args, **kwargs: tracker
@@ -889,7 +784,9 @@ def test_openai_cache_mode_keeps_replayed_prefix_over_frozen_restore() -> None:
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: fake_tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)
proxy.session_tracker_store.resolve_tracker = lambda *args, **kwargs: fake_tracker
def _fake_apply(**kwargs):
@@ -959,7 +856,9 @@ def _forward_cache_mode_chat(
proxy.session_tracker_store.compute_session_id = lambda request, model, messages: (
"stable-session"
)
proxy.session_tracker_store.get_or_create = lambda session_id, provider: tracker
proxy.session_tracker_store.get_or_create = (
lambda session_id, provider, cache_ttl_seconds=None: tracker
)
proxy.session_tracker_store.resolve_tracker = lambda *args, **kwargs: tracker
def _fake_apply(**kwargs):
+1 -1
View File
@@ -79,7 +79,7 @@ def _install_session_tracker(proxy: object, frozen_count: int) -> None:
lambda request, model, messages: "stable-session"
)
proxy.session_tracker_store.get_or_create = ( # type: ignore[attr-defined]
lambda session_id, provider: fake_tracker
lambda session_id, provider, cache_ttl_seconds=None: fake_tracker
)