fix(cache): keep prefix-tracker state across a graceful proxy restart (#4299)

## Description

Every proxy restart rewrites the prompt cache of every warm
conversation. After a restart the proxy has no `PrefixCacheTracker` for
the conversation, so it compresses the whole history afresh, the early
messages come out as different bytes than the prefix it forwarded before
the restart, and the provider re-writes the cache from the first changed
message.

Measured on 2026-10-19 on one Mac running Claude Code through the proxy:
after each graceful restart (app relaunches for updates: proxy SIGTERM'd
and respawned), the next request of every active session read only ~25k
cached tokens (tools + system) and re-wrote the rest, up to 247k, 310k,
361k and 432k tokens at a time. The restart-aligned rewrites came to
about 4.6M tokens of 1h cache writes that day (2x the input price),
about a quarter of all cache writes. A client-side integrity probe
reported `kind=tracker_reset, stable_messages=861,
first_changed_message=3`.

This adds an opt-in `HEADROOM_PREFIX_STATE_FILE=<path>`. When set, the
proxy saves the `SessionTrackerStore` on graceful shutdown and restores
it at startup, before serving, so the first request after a restart
replays byte-identical frozen-prefix bytes. Unset, nothing changes. The
file holds conversation content, so it is never a default.

Related but different: #4156 (away recap replaces the lineage), #4287
(CCR proactive expansion rewrites), #4204 (deferred tool loads in the
lineage key), #4284 (mid-history divergence starts a cold lineage).

No existing issue; found while auditing one machine's token usage.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `SessionTrackerStore.save_state` / `load_state`
(`headroom/cache/prefix_tracker.py`). Per tracker: turn number, cached
token and message counts, last original and last forwarded messages, the
per-turn states that lineage resolution resumes, provider, last activity
and observed cache TTL; per lineage: session id, cache affinity and the
recent snapshot history. So `resolve_tracker` matches the next request
to the restored tracker, forks included. Each distinct message is stored
once per tracker (successive turns, snapshots and the original/forwarded
pairs mostly repeat), and new lineage keys and turn ids continue past
the restored ones.
- CCR entries ride along (review: retrieval integrity). Per tracker,
`save_state` collects every CCR reference in its saved messages and
read-maturation state (`hash=<hex>` markers and SmartCrusher's
`<<ccr:<hex>`) and writes the live `CompressionStore` entries behind
them (new `CompressionStore.export_entries`, which reads without
counting a retrieval). A tracker and its entries count against the size
cap together, so a tracker is never saved without them. `load_state`
re-stores them (`import_entries`) with their original expiry, before any
tracker is installed, so a replayed marker resolves from the first
request even with `HEADROOM_CCR_BACKEND=memory`. An entry a persistent
backend still holds is left as it is; a malformed one rejects the file.
- Restore keeps only trackers still inside the store's own retention
(last activity plus the observed or configured cache TTL and its cleanup
margin, at least the session TTL), the same rule the live store uses.
NaN or future timestamps are dropped.
- Versioned schema (`version: 1`). Any read, parse or validation error
logs one warning and leaves the store empty; the whole file is validated
before anything is installed. The file is written atomically
(`headroom.fsutil.write_text`: temp file, fsync, rename) at mode 0600:
an existing file is tightened to 0600 in place, never unlinked first, so
a failed write (ENOSPC) keeps the previous snapshot. Deleted after a
successful load. Stateless mode (`persistence_allowed`) neither writes
nor reads it (a stateless start used to load, and so delete, a
snapshot). Ignored with multiple workers (one file cannot hold N
per-process stores).
- Size cap: the 256 most recently active trackers and at most 64 MB,
newest first; whatever does not fit starts cold, as everything did
before.
- `HeadroomProxy.startup()` loads, `HeadroomProxy.shutdown()` saves as
its first step, before any await, so the shutdown step timeout cannot
cut it short. No periodic writes.
- Token-mode `CompressionCache` seed: a restored tracker has no
`CompressionCache` behind it in the new process, so the freeze computed
from that cache froze nothing: the pipeline recompressed messages the
provider cached, read maturation rewrote Reads inside the cached prefix,
and beyond the provider-confirmed floor the overlay declined a larger
previous form. One shared `seed_restored_compression_cache` seeds the
session's cache from the restored original/forwarded pair
(`update_from_result`) on the first request, on all three token-mode
paths: Anthropic `/v1/messages`, OpenAI `/v1/chat/completions`, and the
session-aware `/v1/compress` (sidecar). It gives the same freeze count
as before the restart.
- `ReadMaturationManager.export_state` / `validate_state` /
`restore_state`: matured markers, booked ids and replay token deltas
ride along with the tracker, so a matured Read's removal is not booked
as savings again on every request after a restart. The saved state is
validated at load with the rest of the file, so a bad entry rejects the
file instead of failing a request.
- `tests/test_anthropic_fork_lineage_cache.py`: `_drive` accepts a
shared provider-cache model and config overrides, so a second call is a
proxy restart in front of a warm provider cache.
- Documented the variable in `docs/content/docs/configuration.mdx`.
- `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp`
constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and
GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock
--upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and
yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves
3.14.3, so the same commit is on #4299, #4286, #4285 and #4269.

Remaining gaps (unchanged by this PR, none of them changes forwarded
message bytes):
- Session-sticky `anthropic-beta` union (`HEADROOM_BETA_HEADER_STICKY`):
in memory, so after a restart a session's sticky set starts from the
client's own header again. Headers only.
- Output shaper: stateless per request (byte-stable steering text,
effort from the current request's structure), so nothing to persist.
- Gemini and Bedrock paths resume the restored tracker and its overlay
replay; they keep no token-mode `CompressionCache`, so there is nothing
to seed. A crash or SIGKILL still loses everything, by design (no
periodic writes).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New regression tests for the review findings, run against the c141e2ec source
# (headroom/ stashed, new tests kept): all six fail, the 15 earlier tests pass.
$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q
FAILED tests/test_prefix_tracker_restart.py::test_restored_retrieval_references_resolve_to_their_originals
FAILED tests/test_prefix_tracker_restart.py::test_a_stateless_proxy_leaves_a_saved_snapshot_alone
FAILED tests/test_prefix_tracker_restart.py::test_a_failed_save_keeps_the_previous_snapshot
FAILED tests/test_prefix_tracker_restart.py::test_saved_ccr_entries_resolve_after_a_restart
FAILED tests/test_prefix_tracker_restart.py::test_openai_token_mode_replays_its_prefix_after_a_restart
FAILED tests/test_prefix_tracker_restart.py::test_compress_sidecar_session_replays_its_prefix_after_a_restart
6 failed, 15 passed in 8.10s
# Why, in order: hash=cc146b21626392f0dee9890c no longer resolves; the stateless start
# deleted the snapshot; ENOSPC left no snapshot; retrieve() returns None; the OpenAI chat
# and /v1/compress paths re-compressed message 2 under the new query.

# Head c379245e:
$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q
21 passed in 9.50s

$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py tests/test_anthropic_fork_lineage_cache.py \
    tests/test_cache/test_prefix_tracker.py tests/test_compression_store.py tests/test_ccr_sqlite_backend.py \
    tests/test_issue_2671_block_growth_cache.py tests/test_cache_mode_cold_start.py tests/test_cache_prefix_overlay.py \
    tests/test_cross_turn_cache_safety.py tests/test_session_engine.py tests/test_read_maturation.py \
    tests/test_read_maturation_handler_nobust.py tests/test_graceful_shutdown.py tests/test_proxy_handlers_batch.py \
    tests/test_cache_control_move_bust.py tests/test_startup_log_noise.py tests/test_proxy_openai_cache_stability.py \
    tests/test_compress_session_mode.py tests/test_proxy_anthropic_cache_stability.py tests/test_ccr_row_drop_store_bridge.py \
    tests/test_bedrock_prefix_tracker_wiring.py -q
486 passed, 1 skipped in 46.92s

$ uv run --frozen --extra dev pytest tests/test_anthropic_*.py tests/test_proxy_anthropic_*.py tests/test_proxy_openai*.py \
    tests/test_openai_chat_*.py tests/test_cache_aligner_prefix_stability.py tests/test_cache_prefix_safety.py tests/test_cold_prefix.py \
    tests/test_gemini_prefix_freeze.py tests/test_prefix_tracker_client_rewrote_reply.py tests/test_tool_injection*.py tests/test_ccr_*.py -q
737 passed, 16 skipped in 51.45s

$ uvx ruff@0.17.0 check headroom tests
All checks passed!
$ uvx ruff@0.17.0 format --check headroom tests
1606 files already formatted
$ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/cache/compression_store.py \
    headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/server.py
Success: no issues found in 5 source files
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), this branch at c379245e, the real proxy
process (`python -m headroom.cli proxy --port 6811 --no-http2
--no-rate-limit --mode token --read-maturation
--read-maturation-quiesce-turns 1 --read-maturation-min-size-bytes 200
--disable-kompress --no-subscription-tracking`,
`HEADROOM_ROLLOUT_CHANNEL=beta`, clean `HOME` and workspace,
`HEADROOM_PREFIX_STATE_FILE` set, `HEADROOM_CCR_BACKEND=memory`,
`HEADROOM_BACKGROUND_COMPRESSION=1`) in front of the live Anthropic API,
driven by Claude Code 2.1.303 (`claude -p --model haiku`,
`ANTHROPIC_BASE_URL=http://127.0.0.1:6811`). Each restart is a SIGTERM
(graceful shutdown) and a fresh process.
- Exact command / steps: one Claude Code session through the proxy,
three graceful restarts, the last one without the state file as the
control:
1. Turn 1: `claude -p` reads three files (4.8, 17 and 8.9 KB) one per
turn, 5 requests through the proxy. SIGTERM: `SessionTrackerStore: saved
2 of 2 prefix trackers (181266 bytes)`; file mode `-rw-------`; it holds
2 CCR entries, for the two Reads that read maturation had replaced with
`Retrieve original: hash=` markers.
2. Restart: `restored 2 of 2 prefix trackers`, file deleted. `GET
/v1/retrieve/3a739d6ec5febcd49d0cfc0e` and
`/v1/retrieve/2ee4d4d435432b4cca9b72af` on the new process: HTTP 200,
`original_content` equal to the pre-restart entries (5,129 and 18,525
chars, the Read outputs of the two files). With the in-process CCR
store, c141e2ec loses such entries at a restart (the reviewer's
reproduction;
`test_restored_retrieval_references_resolve_to_their_originals` fails on
it).
3. `claude -p --resume`, then SIGTERM + restart with the state file
again, then `claude -p --resume` again (one request each, no tools).
4. Control: SIGTERM, snapshot moved aside, restart without
`HEADROOM_PREFIX_STATE_FILE`, `claude -p --resume`.
- Observed result: after a restart with the state file the first request
read 99% of its prefix from the provider cache and both restored
retrieval hashes resolved; the control re-wrote everything after the
system prompt (proxy `PERF` lines, which match Claude Code's reported
usage):

  | First request after a restart | cache_read | cache_write | hit |
  |---|---|---|---|
  | With the state file, resume after a resume | 40,406 | 514 | 99% |
| Without the state file (control), same history | 26,053 | 11,250 | 70%
|

With the state file the proxy forwarded messages 0-10 byte-identical to
what it forwarded before the restart (compared from the saved snapshots,
`cache_control` positions excluded since Claude Code moves its
breakpoint every turn), and froze 14 of 15 messages. In the control, the
forwarded bytes changed from message 1, and the third Read, verbatim
inside the cached prefix, matured into a new marker (10,205 -> 368
chars), so everything after the system prompt was re-written.
- Not tested: Kompress itself (`--disable-kompress`; the unit suite uses
a deterministic stand-in). Windows file modes. The OpenAI chat and
`/v1/compress` seeds against a live OpenAI endpoint (covered by the
handler-level restart tests above with a query-dependent compressor).
The very first `--resume` of a `claude -p` session (step 3's first
request) read only 26,053 tokens and wrote 14,845, although the client's
messages and the proxy's forwarded messages 0-10 were byte-identical to
the pre-restart request; the read stopped at the system-prompt
breakpoint, so the change is most likely in the system/tools part of the
request, which I did not capture. The resume after it, across another
restart, read 99%.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; opt-in environment variable
`HEADROOM_PREFIX_STATE_FILE`, unset by default.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No; with the variable unset nothing
is read or written and no code path changes (the new handler branches
only fire on a tracker restored from the file, and the CCR export/import
only runs inside save/load).
- Kill switch / disable path: Unset `HEADROOM_PREFIX_STATE_FILE`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Unset the variable or revert this PR; a leftover state
file is inert without the variable. The file format is still `version:
1` (now with a `ccr` list); a file written by c141e2ec has no `ccr` list
and is rejected at load, which starts cold.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Review follow-up (c379245e): CCR entries are saved with their trackers
(P1), the seed covers the OpenAI chat and `/v1/compress` token paths, a
failed save keeps the previous snapshot, and stateless mode no longer
loads (and deletes) a snapshot. Each has a regression test that fails on
c141e2ec.
- Caller isolation: the restored store has exactly the keys, lineages
and retention the in-memory store had at shutdown, and replay still
requires the caller to send the same original messages; see the reply on
that thread for the full reasoning.
- The full `pytest tests` run was not repeated for this head (shared
machine); the focused runs above cover every module the change touches.
- Size cap choice: 256 trackers covers many concurrent Claude Code
sessions plus their subagents; 64 MB bounds shutdown time (encoding is
synchronous) and disk use.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4299"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-10-20 19:44:00 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 08b78f45f1
commit 62b3bedaba
11 changed files with 951 additions and 7 deletions
+1
View File
@@ -401,6 +401,7 @@ headroom proxy --learn --min-evidence 3
| `HEADROOM_THINKING_COMPACT` | Compact plain-text reasoning that models re-send every turn (Kimi/GLM/DeepSeek `reasoning_content` / inline `<think>`): Kompress it on warm turns, drop it on cold turns. No-op for Claude/Codex/OpenAI (encrypted reasoning). See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
| `HEADROOM_THINKING_COMPACT_KEEP_LAST` | Most-recent assistant turns whose reasoning is left intact (the active reasoning the model still uses). Only applies with `HEADROOM_THINKING_COMPACT`. | `1` |
| `HEADROOM_COLD_RECOMPACT` | On a confirmed-cold turn (idle past the real cache TTL), recompact the whole prefix — cross-turn dedupe + superseded-read drop + lossless folds — instead of forwarding a byte-identical prefix to a dead cache. Warm turns unchanged. Pair with `HEADROOM_DEDUPE`. See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
| `HEADROOM_PREFIX_STATE_FILE` | Path where the proxy saves its prefix-cache trackers on a graceful shutdown and restores them on the next start (then deletes the file), so warm conversations replay the exact prefix the provider cached instead of re-writing it. The file holds conversation content: it is written `0600`, capped at the 256 newest conversations and 64 MB, and only trackers still inside their cache lifetime are restored, with the retrieval (CCR) entries their markers point at, so those markers resolve even with `HEADROOM_CCR_BACKEND=memory`. Not supported with multiple workers; ignored in stateless mode (neither read nor written). | -- |
| `HEADROOM_DEDUPE` | Whole-conversation verbatim cross-turn dedup in the router (cache-safe, information-preserving via retrieval markers). Superseded-read drop + lossless folds run without it; this adds verbatim dedup. | `off` |
| `HEADROOM_CACHE_TTL_LEARN` | Append per-turn cache-outcome observations (provider, model, idle, hit/miss) to `cache_ttl_observations.jsonl` for the offline `headroom-cache-ttl` learner. Observation-only (no request-behavior change); respects `HEADROOM_STATELESS`; the log is size-bounded. | `off` |
| `HEADROOM_KOMPRESS_ENDPOINT` / `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | Offload ML compression (Kompress) to a remote endpoint instead of the local ONNX model — used by reasoning compaction and the router when set. | -- |
+43 -1
View File
@@ -40,7 +40,7 @@ import threading
import time
from collections import deque
from contextvars import ContextVar
from dataclasses import dataclass, field, replace
from dataclasses import asdict, dataclass, field, replace
from typing import TYPE_CHECKING, Any
from ..log_safety import describe_exception, safe_id
@@ -737,6 +737,48 @@ class CompressionStore:
return list(reversed(events_copy[-limit:]))
def export_entries(self, hashes: list[str]) -> list[dict[str, Any]]:
"""The live entries for ``hashes`` as JSON-safe dicts, without counting
a retrieval. ``SessionTrackerStore.save_state`` carries them across a
restart so the markers it replays still resolve (:meth:`import_entries`)."""
with self._lock:
found = [self._backend.get(hash_key) for hash_key in hashes]
return [asdict(e) for e in found if e is not None and not e.is_expired()]
def import_entries(self, entries: list[CompressionEntry]) -> int:
"""Re-store :meth:`export_entries` output, each with its original expiry.
An entry the backend still holds (a persistent backend kept it) is left
as it is. Raises on a malformed entry. Returns the entries stored.
"""
now = time.time()
stored = 0
for e in entries:
if not all(
isinstance(v, str) for v in (e.hash, e.original_content, e.compressed_content)
):
raise ValueError(f"malformed CCR entry {e.hash!r}")
ttl = int(e.created_at + e.ttl - now)
if ttl <= 0 or self.exists(e.hash):
continue
self.store(
e.original_content,
e.compressed_content,
original_tokens=e.original_tokens,
compressed_tokens=e.compressed_tokens,
original_item_count=e.original_item_count,
compressed_item_count=e.compressed_item_count,
tool_name=e.tool_name,
tool_call_id=e.tool_call_id,
query_context=e.query_context,
tool_signature_hash=e.tool_signature_hash,
compression_strategy=e.compression_strategy,
ttl=ttl,
explicit_hash=e.hash,
)
stored += 1
return stored
def clear(self) -> None:
"""Clear all entries. Mainly for testing."""
with self._lock:
+277
View File
@@ -16,16 +16,19 @@ pipeline skips them entirely.
from __future__ import annotations
import contextlib
import copy
import hashlib
import itertools
import json
import logging
import os
import re
import time
from collections import OrderedDict, deque
from contextvars import ContextVar
from dataclasses import dataclass
from pathlib import Path
from typing import Any
logger = logging.getLogger(__name__)
@@ -71,6 +74,23 @@ _CACHE_TTL_CLEANUP_MARGIN_SECONDS = 60
# of forks off one turn leaves that turn resumable.
_LINEAGE_HISTORY = 3
# Opt-in file for SessionTrackerStore state across a graceful restart. It holds
# conversation content, so nothing is written unless an operator sets it.
PREFIX_STATE_FILE_ENV = "HEADROOM_PREFIX_STATE_FILE"
_PREFIX_STATE_VERSION = 1
# Newest trackers first; whatever does not fit starts cold, as all did before.
_PREFIX_STATE_MAX_TRACKERS = 256
_PREFIX_STATE_MAX_BYTES = 64 * 1024 * 1024
# Any CCR reference in saved messages: ``hash=<hex>`` markers (Retrieve
# more/original, compressed-item counts) and SmartCrusher's ``<<ccr:<hex>``.
_CCR_HASH_RE = re.compile(r"(?:hash=|<<ccr:)([0-9a-f]{12,24})")
def prefix_state_file() -> str | None:
"""The configured tracker state file, or None when persistence is off."""
return os.environ.get(PREFIX_STATE_FILE_ENV, "").strip() or None
# (tracker, turn, resumed state) the current request resolved to. Each proxied
# request runs in its own asyncio task, so it reads the state it resumed and
# ``update_from_response`` records its response against its own turn, even
@@ -1180,6 +1200,12 @@ class PrefixCacheTracker:
# lazily by the handler when read maturation is enabled. Rides
# here so it shares the session's affinity and TTL cleanup.
self.read_maturation_manager: Any = None
# Set by SessionTrackerStore.load_state: this process has no
# CompressionCache entries behind the restored prefix yet, and the
# handler adopts the saved maturation state (validated at load) when it
# builds the manager.
self.restored_from_disk = False
self.restored_maturation_state: dict[str, Any] | None = None
# Stats
self._busts_avoided: int = 0
@@ -1609,6 +1635,42 @@ def _lineage_snapshot(obj: Any) -> Any:
return obj
def _deref(table: list[Any], refs: list[int]) -> list[Any]:
"""Rebuild a saved message list. Restored lists share message objects,
which is safe: tracker and lineage lists are never mutated in place."""
if not all(type(i) is int and 0 <= i < len(table) for i in refs):
raise ValueError("message reference out of range")
return [table[i] for i in refs]
def _decode_turn_state(table: list[Any], state: list[Any]) -> tuple[Any, ...]:
turn, cached_tokens, cached_messages, original, forwarded = state
return (
int(turn),
int(cached_tokens),
int(cached_messages),
_deref(table, original),
_deref(table, forwarded),
)
def seed_restored_compression_cache(tracker: Any, comp_cache: Any) -> None:
"""Seed a token-mode CompressionCache on a tracker's first request after
:meth:`SessionTrackerStore.load_state`.
The new process has no cache entries behind the restored prefix, so the
freeze computed from that cache freezes nothing: the pipeline recompresses
(and read maturation rewrites) messages the provider cached, and beyond the
provider-confirmed floor the overlay declines a larger replay. Seeding with
what was forwarded gives the freeze the pre-restart count.
"""
if getattr(tracker, "restored_from_disk", False):
tracker.restored_from_disk = False
comp_cache.update_from_result(
tracker.get_last_original_messages(), tracker.get_last_forwarded_messages()
)
class SessionTrackerStore:
"""Manages PrefixCacheTracker instances across sessions.
@@ -2007,3 +2069,218 @@ class SessionTrackerStore:
def active_sessions(self) -> int:
"""Number of active session trackers (one per conversation lineage)."""
return len(self._trackers)
def save_state(self, path: str) -> int:
"""Write the live trackers to ``path`` for the next process to resume.
Called on graceful shutdown when ``HEADROOM_PREFIX_STATE_FILE`` is set.
Without it, a restart leaves every warm conversation with no tracker:
its next request recompresses the whole history into bytes that differ
from the prefix the provider cached, and the cache is re-written from
the first changed message. Newest trackers first, capped at
``_PREFIX_STATE_MAX_TRACKERS`` and ``_PREFIX_STATE_MAX_BYTES``; a
tracker whose messages are not JSON is skipped. Each tracker carries
the CCR entries its messages reference, so its replayed markers still
resolve when the CCR backend does not survive the restart (memory).
Never raises; returns the number of trackers written.
"""
from headroom.cache.compression_store import get_compression_store
from headroom.fsutil import write_text
from headroom.paths import persistence_allowed
if not persistence_allowed("prefix-cache tracker state"):
return 0
sessions = {key: sid for sid, family in self._lineages.items() for key in family}
live = sorted(
(item for item in self._trackers.items() if not item[1].is_expired),
key=lambda item: item[1]._last_activity,
reverse=True,
)
entries: list[str] = []
ccr: dict[str, str] = {}
size = 0
for key, tracker in live[:_PREFIX_STATE_MAX_TRACKERS]:
try:
entry = self._encode_tracker(key, tracker, sessions.get(key))
refs = [h for h in dict.fromkeys(_CCR_HASH_RE.findall(entry)) if h not in ccr]
records = get_compression_store().export_entries(refs) if refs else []
payloads = {r["hash"]: json.dumps(r) for r in records}
except Exception: # an unsaveable tracker starts cold, as all did before
continue
grow = len(entry) + sum(len(p) + 1 for p in payloads.values())
if size + grow > _PREFIX_STATE_MAX_BYTES:
continue
size += grow + 1
entries.append(entry)
ccr.update(payloads)
try:
target = Path(path)
target.parent.mkdir(parents=True, exist_ok=True)
# write_text keeps the target's mode, so tighten it in place rather
# than unlinking: a failed write must leave the last snapshot.
with contextlib.suppress(FileNotFoundError):
target.chmod(0o600)
write_text(
target,
f'{{"version":{_PREFIX_STATE_VERSION},"trackers":[{",".join(entries)}],'
f'"ccr":[{",".join(ccr.values())}]}}',
)
except OSError as exc:
logger.warning("SessionTrackerStore: prefix state not saved to %s: %s", path, exc)
return 0
logger.info(
"SessionTrackerStore: saved %d of %d prefix trackers (%d bytes) to %s",
len(entries),
len(self._trackers),
size,
path,
)
return len(entries)
def _encode_tracker(self, key: str, tracker: PrefixCacheTracker, session_id: str | None) -> str:
"""One tracker as JSON, each distinct message stored once.
Successive turns, the lineage snapshots and the original/forwarded
pairs mostly repeat the same messages, so this keeps the file near one
copy of each conversation instead of up to eleven.
"""
index: dict[str, int] = {}
memo: dict[int, list[int]] = {}
def ref(messages: list[Any]) -> list[int]:
if id(messages) not in memo:
refs = []
for message in messages:
blob = json.dumps(message)
refs.append(index.setdefault(blob, len(index)))
memo[id(messages)] = refs
return memo[id(messages)]
def state(s: tuple[Any, ...]) -> list[Any]:
return [s[0], s[1], s[2], ref(s[3]), ref(s[4])]
maturation = tracker.read_maturation_manager
meta = {
"key": key,
"session": session_id,
"affinity": self._lineage_affinities.get(key),
"provider": tracker.provider,
"last_activity": tracker._last_activity,
"cache_ttl": tracker._cache_ttl_seconds,
"state": state(tracker._turn_state()),
"turns": [[t, state(s)] for t, s in tracker._turn_states.items() if s is not None],
"history": [[ref(snap), t] for snap, t in self._lineage_history.get(key, ())],
"maturation": None if maturation is None else maturation.export_state(),
}
return f'{{"meta":{json.dumps(meta)},"messages":[{",".join(index)}]}}'
def load_state(self, path: str) -> int:
"""Resume the trackers :meth:`save_state` wrote, before serving.
Trackers past their retention are dropped. The file is removed once
loaded, so a later crash cannot bring back state older than what it
lost. A missing file is normal; an unreadable or invalid one is logged
and leaves the store empty. Never raises; returns trackers restored.
"""
from headroom.paths import persistence_allowed
# Loading deletes the file, so stateless mode leaves it for a later run.
if not persistence_allowed("prefix-cache tracker state"):
return 0
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
restored = self._decode_state(data)
except FileNotFoundError:
return 0
except Exception as exc: # any bad file must leave boot unaffected
logger.warning(
"SessionTrackerStore: ignoring prefix state %s (%s: %s)",
path,
type(exc).__name__,
exc,
)
return 0
try:
os.remove(path)
except OSError as exc:
logger.warning("SessionTrackerStore: could not remove %s: %s", path, exc)
logger.info(
"SessionTrackerStore: restored %d of %d prefix trackers from %s",
restored,
len(data["trackers"]),
path,
)
return restored
def _decode_state(self, data: Any) -> int:
"""Validate the whole file first, then install it; a bad file installs nothing."""
from headroom.cache.compression_store import CompressionEntry, get_compression_store
if data.get("version") != _PREFIX_STATE_VERSION:
raise ValueError(f"unsupported version {data.get('version')!r}")
ccr = [CompressionEntry(**record) for record in data["ccr"]]
trackers: dict[str, PrefixCacheTracker] = {}
families: dict[str, OrderedDict[str, list[Any]]] = {}
affinities: dict[str, str | None] = {}
histories: dict[str, deque[tuple[list[Any], int]]] = {}
now = time.time()
# Oldest first, so each family keeps its lineages in creation order.
for entry in reversed(data["trackers"]):
meta, table = entry["meta"], entry["messages"]
tracker = PrefixCacheTracker(str(meta["provider"]), self._default_config)
tracker._last_activity = float(meta["last_activity"])
tracker.observe_cache_ttl(meta["cache_ttl"])
# NaN or a future stamp would never expire.
if tracker.is_expired or not tracker._last_activity <= now:
continue
(
tracker._turn_number,
tracker._cached_token_count,
tracker._cached_message_count,
tracker._last_original_messages,
tracker._last_forwarded_messages,
) = _decode_turn_state(table, meta["state"])
tracker._turn_states = {int(t): _decode_turn_state(table, s) for t, s in meta["turns"]}
tracker.restored_from_disk = True
if meta["maturation"] is not None:
from headroom.transforms.read_maturation import ReadMaturationManager
tracker.restored_maturation_state = ReadMaturationManager.validate_state(
meta["maturation"]
)
key = str(meta["key"])
trackers[key] = tracker
history = deque(
((_deref(table, snap), int(t)) for snap, t in meta["history"]),
maxlen=_LINEAGE_HISTORY,
)
if not history:
continue
histories[key] = history
if meta["session"] is not None:
families.setdefault(str(meta["session"]), OrderedDict())[key] = history[0][0]
affinity = meta["affinity"]
affinities[key] = None if affinity is None else str(affinity)
if ccr and trackers:
# First, so a marker a restored tracker replays resolves from its
# first request; a failure here still installs no tracker.
get_compression_store().import_entries(ccr)
# New turns and lineages must not reuse a restored id.
turns = [t for h in histories.values() for _, t in h]
turns += [t for tracker in trackers.values() for t in tracker._turn_states]
lineage_ns = [
int(n)
for _, sep, n in (key.rpartition("\x00") for key in trackers)
if sep and n.isdecimal()
]
self._turn_counter = itertools.count(max([next(self._turn_counter), *turns]) + 1)
self._lineage_counter = itertools.count(max([next(self._lineage_counter), *lineage_ns]) + 1)
self._trackers.update(trackers)
self._lineage_history.update(histories)
self._lineage_affinities.update(affinities)
for session_id, family in families.items():
self._lineages.setdefault(session_id, OrderedDict()).update(family)
return len(trackers)
+13
View File
@@ -2029,6 +2029,15 @@ class AnthropicHandlerMixin:
compression_policy = resolve_policy(getattr(request.state, "auth_mode", None))
if is_token_mode(self.config.mode):
comp_cache = self._get_compression_cache(session_id)
from headroom.cache.prefix_tracker import (
seed_restored_compression_cache,
)
# Seeds the empty cache after a restart
# (HEADROOM_PREFIX_STATE_FILE); without it the clamp below
# freezes nothing and read maturation rewrites Reads
# inside the cached prefix.
seed_restored_compression_cache(prefix_tracker, comp_cache)
# Freeze + stable marking + Zone-1 swap now live in the
# shared session engine (PROXY policy: clamp by BOTH the
@@ -2609,6 +2618,10 @@ class AnthropicHandlerMixin:
compression_store=get_compression_store(),
)
prefix_tracker.read_maturation_manager = maturation_mgr
state = getattr(prefix_tracker, "restored_maturation_state", None)
if state is not None:
prefix_tracker.restored_maturation_state = None
maturation_mgr.restore_state(state)
_maturation_mgr = maturation_mgr
maturation = maturation_mgr.apply(
optimized_messages,
+9
View File
@@ -3821,6 +3821,12 @@ class OpenAIHandlerMixin:
if is_token_mode(self.config.mode):
comp_cache = self._get_compression_cache(openai_session_id)
from headroom.cache.prefix_tracker import seed_restored_compression_cache
# Seeds the empty cache after a restart
# (HEADROOM_PREFIX_STATE_FILE); without it the freeze below
# freezes nothing.
seed_restored_compression_cache(openai_prefix_tracker, comp_cache)
if not is_cache_mode(self.config.mode):
# Token mode: shared engine, REPLAYABLE policy — its
@@ -10695,6 +10701,9 @@ class OpenAIHandlerMixin:
session_tracker = self.session_tracker_store.get_or_create(
_session_key, _tracker_provider
)
from headroom.cache.prefix_tracker import seed_restored_compression_cache
seed_restored_compression_cache(session_tracker, comp_cache)
# A claimed turn prepares its executor-side work here (tokenizer,
# hooks, scope binding) — CPU-bound parts run inside the closures
+16
View File
@@ -1203,6 +1203,8 @@ class HeadroomProxy(
session_ttl_seconds=config.prefix_freeze_session_ttl,
)
)
# Set by startup() when HEADROOM_PREFIX_STATE_FILE opts in.
self._prefix_state_file: str | None = None
# Compression cache store for token mode (session-scoped). The dict
# itself is mutated under `_compression_caches_lock`; the per-session
@@ -2048,6 +2050,17 @@ class HeadroomProxy(
async def startup(self):
"""Initialize async resources."""
self._get_shutdown_event().clear()
# Opt-in: resume the prefix trackers the previous process saved on its
# graceful shutdown, so warm conversations replay their cached prefix.
from headroom.cache.prefix_tracker import PREFIX_STATE_FILE_ENV, prefix_state_file
self._prefix_state_file = prefix_state_file()
if self._prefix_state_file and _MULTI_WORKER_CONFIG_ENV in os.environ:
# Each worker holds its own trackers; one file cannot hold them all.
logger.warning("%s ignored: not supported with multiple workers", PREFIX_STATE_FILE_ENV)
self._prefix_state_file = None
if self._prefix_state_file:
self.session_tracker_store.load_state(self._prefix_state_file)
self.pipeline_extensions.emit(
PipelineStage.PRE_START,
operation="proxy.startup",
@@ -2395,6 +2408,9 @@ class HeadroomProxy(
async def shutdown(self):
"""Cleanup async resources."""
self._get_shutdown_event().set()
# First, before any await can hit the shutdown step timeout.
if self._prefix_state_file:
self.session_tracker_store.save_state(self._prefix_state_file)
if self.http_client_h1 and self.http_client_h1 is not self.http_client:
await self.http_client_h1.aclose()
self.http_client_h1 = None
+25
View File
@@ -153,6 +153,31 @@ class ReadMaturationManager:
# once per session (content and marker are stable per tool call).
self._replay_token_deltas: dict[str, int] = {}
def export_state(self) -> dict[str, Any]:
"""Session state as JSON, for ``SessionTrackerStore.save_state``."""
return {
"matured": {k: [m.marker, m.ccr_hash] for k, m in self._matured.items()},
"booked": sorted(self._booked),
"deltas": self._replay_token_deltas,
}
@staticmethod
def validate_state(state: Any) -> dict[str, Any]:
"""Normalize :meth:`export_state` output read from disk. Raises on a bad
shape, so a bad file is rejected at load and never inside a request."""
return {
"matured": {str(k): [str(m), str(h)] for k, (m, h) in state["matured"].items()},
"booked": [str(k) for k in state["booked"]],
"deltas": {str(k): int(v) for k, v in state["deltas"].items()},
}
def restore_state(self, state: dict[str, Any]) -> None:
"""Adopt :meth:`validate_state` output from the previous process, so a
matured Read's removal is not booked again after a restart."""
self._matured = {k: MaturedRead(m, h) for k, (m, h) in state["matured"].items()}
self._booked = set(state["booked"])
self._replay_token_deltas = dict(state["deltas"])
# ─── Per-request entry point ────────────────────────────────────────
def apply(
+3 -2
View File
@@ -410,8 +410,9 @@ constraint-dependencies = [
"transformers>=5.5.0",
# PYSEC-2026-3447 — transitive dependency; fix at 83.0.0
"setuptools>=83.0.0",
# PYSEC-2026-3545/3546/3547 — transitive HTTP/WebSocket parser fixes
"aiohttp>=3.14.3",
# PYSEC-2026-3545/3546/3547 — transitive HTTP/WebSocket parser fixes;
# CVE-2026-111063/111064, GHSA-2g87-pp6c-29x5; fix at 3.14.4
"aiohttp>=3.14.4",
# PYSEC-2026-3552/3553/3554 — PKCS#7 and certificate verification fixes
"cryptography>=50.0.0",
]
+9 -3
View File
@@ -156,8 +156,13 @@ class _PromptCache:
return read, write, hit
def _drive(requests: list[list[dict[str, Any]]]) -> list[tuple[list[dict[str, Any]], int]]:
"""Send each history through the real handler; return (forwarded, read_hit)."""
def _drive(
requests: list[list[dict[str, Any]]], cache: _PromptCache | None = None, **config: Any
) -> list[tuple[list[dict[str, Any]], int]]:
"""Send each history through the real handler; return (forwarded, read_hit).
One app per call: a second call with the same ``cache`` is a proxy restart
in front of a provider whose prompt cache stayed warm."""
app = create_app(
ProxyConfig(
optimize=True,
@@ -168,9 +173,10 @@ def _drive(requests: list[list[dict[str, Any]]]) -> list[tuple[list[dict[str, An
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
**config,
)
)
cache = _PromptCache()
cache = cache or _PromptCache()
seen: list[tuple[list[dict[str, Any]], int]] = []
with TestClient(app) as client:
proxy = client.app.state.proxy
+554
View File
@@ -0,0 +1,554 @@
"""A graceful proxy restart must not cost warm conversations their prompt cache.
Production, 2026-10-19: five graceful restarts in one day (app relaunches for
updates) each made the next request of every active Claude Code session read
only the ~25k-token tools + system prefix and re-write the rest, 247k-432k
tokens at a time, about a third of that day's cache writes. After a restart
the proxy has no ``PrefixCacheTracker`` for the conversation, so it compresses
the whole history afresh and the early messages come out as different bytes
than the prefix it forwarded before the restart.
``HEADROOM_PREFIX_STATE_FILE`` (opt-in) saves the tracker store on graceful
shutdown and restores it at startup, so the first request after a restart
replays the exact prefix the provider cached.
"""
from __future__ import annotations
import contextvars
import errno
import json
import os
import stat
from pathlib import Path
from types import SimpleNamespace
from typing import Any
import pytest
pytest.importorskip("fastapi")
import httpx
from fastapi.testclient import TestClient
from headroom.cache import prefix_tracker
from headroom.cache.compression_store import get_compression_store, reset_compression_store
from headroom.cache.prefix_tracker import PREFIX_STATE_FILE_ENV, SessionTrackerStore
from headroom.ccr.tool_injection import CCRToolInjector
from headroom.config import TransformResult
from headroom.proxy.server import ProxyConfig, create_app
from headroom.transforms.pipeline import TransformPipeline
from tests.test_anthropic_fork_lineage_cache import (
_drive,
_main,
_PromptCache,
_send,
_strip,
_summary_fork,
)
def _wire(messages: list[dict[str, Any]]) -> str:
"""The bytes the provider keys its prefix cache on (breakpoints excluded)."""
return json.dumps(_strip(messages))
def _ccr_hashes(messages: list[dict[str, Any]]) -> set[str]:
"""Retrieval references the model sees, found the way the proxy finds them."""
return set(CCRToolInjector().scan_for_markers(messages))
def _with_fake_kompress(monkeypatch: pytest.MonkeyPatch) -> None:
"""A deterministic stand-in for the Kompress ML stage (off in this suite).
A cold start forwards the fast pass, which skips Kompress, while warm
turns compress their new messages with it. So after a restart without
state, the cold fast pass re-emits every message Kompressed after the
first turn in a different form, which re-writes the cache from there."""
real_apply = TransformPipeline.apply
def apply(self, messages, model, **kwargs): # noqa: ANN001, ANN003, ANN202
skip = kwargs.get("skip_kompress", False)
frozen = kwargs.get("frozen_message_count", 0) or 0
result = real_apply(self, messages, model, **kwargs)
if skip:
return result
out = []
for i, message in enumerate(result.messages):
content = message.get("content")
if i >= frozen and isinstance(content, list):
blocks = []
for block in content:
text = block.get("content") if isinstance(block, dict) else None
if block.get("type") == "tool_result" and isinstance(text, str):
if not text.endswith(" [kompressed]"):
block = {**block, "content": text[: len(text) // 2] + " [kompressed]"}
blocks.append(block)
message = {**message, "content": blocks}
out.append(message)
result.messages = out
return result
monkeypatch.setattr(TransformPipeline, "apply", apply)
def _read_turn(i: int) -> list[dict[str, Any]]:
tid = f"toolu_r{i:02d}"
body = "\n".join(f"{n:5d} line {n} of file {i}: some code here" for n in range(300))
return [
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": tid, "name": "Read", "input": {"file_path": f"/f{i}.py"}}
],
},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": tid, "content": body}]},
]
def _reads(turns: int) -> list[dict[str, Any]]:
"""One Read per turn; Claude Code's breakpoint rides the newest block."""
messages: list[dict[str, Any]] = [
{"role": "user", "content": [{"type": "text", "text": "Review the change."}]}
]
for i in range(turns):
messages += _read_turn(i)
messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}
return messages
# (history builder, turns before the restart, proxy config, fake Kompress)
_SCENARIOS = {
# Turns 1-2, restart, turn 3: Kompressed messages come back fast-passed.
"kompress": (_main, 2, {}, True),
# Reads held while active then cached verbatim stay verbatim; a fresh
# process with an empty CompressionCache freezes nothing and matures them.
"read-maturation": (
_reads,
5,
{
"read_maturation": True,
"read_maturation_quiesce_turns": 2,
"read_maturation_min_size_bytes": 100,
},
False,
),
}
def _restart_between_turns(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int, dict[str, str]]:
"""Some turns in one proxy, the next turn in a fresh one, one warm
provider cache. Returns (last forwarded before, first forwarded after,
its read hit, the original behind each retrieval reference forwarded
before the restart)."""
build, turns, config, kompress = _SCENARIOS[scenario]
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
# As the desktop app runs it: a cold start forwards the fast pass.
monkeypatch.setenv("HEADROOM_BACKGROUND_COMPRESSION", "1")
monkeypatch.setenv("HEADROOM_BACKGROUND_COMPRESSION_MIN_TOKENS", "1")
monkeypatch.chdir(tmp_path)
if kompress:
_with_fake_kompress(monkeypatch)
cache = _PromptCache()
reset_compression_store()
before = _drive([build(n) for n in range(1, turns + 1)], cache, **config)
store = get_compression_store()
originals = {h: store.retrieve(h).original_content for h in _ccr_hashes(before[-1][0])} # type: ignore[union-attr]
reset_compression_store() # nothing in process memory survives a restart
(after,) = _drive([build(turns + 1)], cache, **config)
return before[-1][0], after[0], after[1], originals
@pytest.mark.parametrize("scenario", list(_SCENARIOS))
def test_first_request_after_a_restart_replays_the_cached_prefix(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
) -> None:
state = tmp_path / "prefix-state.json"
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(state))
previous, current, hit, _ = _restart_between_turns(monkeypatch, tmp_path, scenario)
assert _wire(current[: len(previous)]) == _wire(previous)
assert hit >= len(previous) - 1, f"read only {hit + 1}/{len(previous)} cached messages"
def test_restored_retrieval_references_resolve_to_their_originals(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Review of #4299: matching bytes are not enough. With the in-process CCR
backend a restart emptied the store, so a matured Read the restored
tracker kept replaying pointed at a hash that no longer resolved."""
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
previous, current, _, originals = _restart_between_turns(
monkeypatch, tmp_path, "read-maturation"
)
emitted = _ccr_hashes(current)
assert emitted & originals.keys(), "no pre-restart reference was replayed"
store = get_compression_store()
for h in emitted:
entry = store.retrieve(h)
assert entry is not None, f"hash={h} does not resolve after the restart"
if h in originals:
assert entry.original_content == originals[h]
@pytest.mark.parametrize("scenario", list(_SCENARIOS))
def test_without_the_state_file_a_restart_rewrites_the_prefix(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
) -> None:
"""Default behavior is unchanged, and it is the bug the opt-in fixes, so
the test above is not vacuous."""
monkeypatch.delenv(PREFIX_STATE_FILE_ENV, raising=False)
previous, current, hit, _ = _restart_between_turns(monkeypatch, tmp_path, scenario)
assert _wire(current[: len(previous)]) != _wire(previous)
assert hit < len(previous) - 1
assert list(tmp_path.iterdir()) == [], "nothing may be written without the opt-in"
def _two_conversations() -> SessionTrackerStore:
store = SessionTrackerStore()
contextvars.Context().run(_send, store, _strip(_main(3)))
contextvars.Context().run(_send, store, [{"role": "user", "content": "unrelated"}])
return store
def test_restored_lineages_resume_and_new_ones_get_fresh_keys(tmp_path: Path) -> None:
"""Store level: a fork answered last still leaves the main turn resumable,
and a new conversation never lands on a restored lineage's key."""
store = _two_conversations()
base = _strip(_main(3))
contextvars.Context().run(_send, store, _strip(_summary_fork(3)))
path = str(tmp_path / "state.json")
assert store.save_state(path) == 2
restored = SessionTrackerStore()
assert restored.load_state(path) == 2
main = contextvars.Context().run(
restored.resolve_tracker, "sid", "anthropic", messages=_strip(_main(4))
)
new = contextvars.Context().run(
restored.resolve_tracker, "sid", "anthropic", messages=[{"role": "user", "content": "new"}]
)
assert main.get_frozen_message_count() == len(base)
assert main.get_last_original_messages() == base
assert restored.active_sessions == 3
assert new is not main and new.get_frozen_message_count() == 0
def test_expired_trackers_are_not_restored(tmp_path: Path) -> None:
path = tmp_path / "state.json"
_two_conversations().save_state(str(path))
data = json.loads(path.read_text())
data["trackers"][0]["meta"]["last_activity"] -= 3600 # past the 600s retention
path.write_text(json.dumps(data))
assert SessionTrackerStore().load_state(str(path)) == 1
def _bad_ref(path: Path) -> None:
data = json.loads(path.read_text())
data["trackers"][0]["meta"]["state"][3] = [-1]
path.write_text(json.dumps(data))
def _bad_maturation(path: Path) -> None:
data = json.loads(path.read_text())
data["trackers"][-1]["meta"]["maturation"] = {"matured": [1], "booked": [], "deltas": {}}
path.write_text(json.dumps(data))
@pytest.mark.parametrize(
"corrupt",
[
lambda p: p.write_text('{"version": 1, "trackers": [{"meta"'),
lambda p: p.write_text('{"version": 99, "trackers": []}'),
lambda p: p.write_text("[1, 2]"),
_bad_ref,
_bad_maturation,
],
ids=["truncated", "version", "shape", "bad-ref", "bad-maturation"],
)
def test_a_bad_state_file_starts_empty(tmp_path: Path, caplog, corrupt) -> None: # noqa: ANN001
path = tmp_path / "state.json"
_two_conversations().save_state(str(path))
corrupt(path)
store = SessionTrackerStore()
assert store.load_state(str(path)) == 0
assert store.active_sessions == 0
assert "ignoring prefix state" in caplog.text
def test_state_file_is_private_and_removed_once_loaded(tmp_path: Path) -> None:
path = tmp_path / "state.json"
path.write_text("stale")
path.chmod(0o644)
assert _two_conversations().save_state(str(path)) == 2
if os.name == "posix":
assert stat.S_IMODE(path.stat().st_mode) == 0o600
assert SessionTrackerStore().load_state(str(path)) == 2
assert not path.exists()
assert SessionTrackerStore().load_state(str(path)) == 0 # missing file: cold start
def test_save_keeps_the_newest_trackers_within_the_cap(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
store = _two_conversations()
newest = max(store._trackers.values(), key=lambda t: t._last_activity)
monkeypatch.setattr(prefix_tracker, "_PREFIX_STATE_MAX_TRACKERS", 1)
path = str(tmp_path / "state.json")
assert store.save_state(path) == 1
restored = SessionTrackerStore()
restored.load_state(path)
(kept,) = restored._trackers.values()
assert kept.get_last_original_messages() == newest.get_last_original_messages()
monkeypatch.setattr(prefix_tracker, "_PREFIX_STATE_MAX_BYTES", 10)
assert store.save_state(path) == 0
def test_stateless_mode_writes_nothing(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
monkeypatch.setenv("HEADROOM_STATELESS", "1")
path = tmp_path / "state.json"
assert _two_conversations().save_state(str(path)) == 0
assert not path.exists()
def test_a_stateless_proxy_leaves_a_saved_snapshot_alone(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Loading deletes the file, so a stateless start must not load it: the
next normal start resumes those conversations."""
path = tmp_path / "state.json"
assert _two_conversations().save_state(str(path)) == 2
saved = path.read_text()
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(path))
monkeypatch.setenv("HEADROOM_STATELESS", "1")
_drive([]) # one stateless proxy start and graceful shutdown
assert path.read_text() == saved
monkeypatch.delenv("HEADROOM_STATELESS")
assert SessionTrackerStore().load_state(str(path)) == 2
def test_a_failed_save_keeps_the_previous_snapshot(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
path = tmp_path / "state.json"
assert _two_conversations().save_state(str(path)) == 2
saved = path.read_text()
def disk_full(*args: Any, **kwargs: Any) -> None:
raise OSError(errno.ENOSPC, "No space left on device")
monkeypatch.setattr("headroom.fsutil.write_text", disk_full)
assert _two_conversations().save_state(str(path)) == 0
assert path.read_text() == saved
def test_saved_ccr_entries_resolve_after_a_restart(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Store level, compressor markers in the replayed prefix: both hash
formats resolve to the same original with the same expiry after the
in-process CCR store is emptied, and an entry that expired does not."""
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
reset_compression_store()
ccr = get_compression_store()
rows = ccr.store(json.dumps(list(range(400))), "[0, 1]", ttl=900)
blob = ccr.store("x" * 5000, "", ttl=900, explicit_hash="abcdef123456")
gone = ccr.store("expired original", "", ttl=900)
ccr._backend.get(gone).created_at -= 1000 # type: ignore[union-attr]
expires = {h: ccr.get_entry_status(h)["expires_at"] for h in (rows, blob)}
text = (
f"[400 items compressed to 2. Retrieve more: hash={rows}] "
f"<<ccr:{blob},base64,4.9KB>> [stale. Retrieve original: hash={gone}]"
)
store = SessionTrackerStore()
contextvars.Context().run(_send, store, [{"role": "user", "content": text}])
path = str(tmp_path / "state.json")
assert store.save_state(path) == 1
reset_compression_store()
assert SessionTrackerStore().load_state(path) == 1
ccr = get_compression_store()
assert ccr.retrieve(rows).original_content == json.dumps(list(range(400))) # type: ignore[union-attr]
assert ccr.retrieve(blob).original_content == "x" * 5000 # type: ignore[union-attr]
assert ccr.retrieve(gone) is None
for h, at in expires.items():
assert abs(ccr.get_entry_status(h)["expires_at"] - at) < 2
reset_compression_store()
def _chat_turns(n: int) -> list[dict[str, Any]]:
"""An OpenAI chat agent: one big tool result, then ``n - 1`` follow-ups."""
messages: list[dict[str, Any]] = [
{"role": "user", "content": "run the build and show me the log"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_1",
"type": "function",
"function": {"name": "bash", "arguments": "{}"},
}
],
},
{"role": "tool", "tool_call_id": "call_1", "content": "build log line OK\n" * 200},
]
for i in range(1, n):
messages += [
{"role": "assistant", "content": f"answer {i}"},
{"role": "user", "content": "and now?"},
]
return messages
def _relevance_pipeline(**kwargs: Any) -> TransformResult:
"""Query-dependent compression, like SmartCrusher's relevance scoring: the
same tool result compresses differently under a different user query."""
frozen = kwargs.get("frozen_message_count") or 0
query = next(m["content"] for m in reversed(kwargs["messages"]) if m["role"] == "user")
messages = [
{**m, "content": f"[log: kept lines relevant to {query!r}]"}
if i >= frozen and m.get("role") == "tool"
else m
for i, m in enumerate(kwargs["messages"])
]
return TransformResult(
messages=messages, tokens_before=2000, tokens_after=40, transforms_applied=["relevance"]
)
def _drive_openai_chat(requests: list[list[dict[str, Any]]]) -> list[list[dict[str, Any]]]:
"""Each history through the real OpenAI chat handler in token mode; one
app per call, so a second call is a proxy restart."""
app = create_app(
ProxyConfig(
optimize=True,
mode="token",
disable_kompress=True,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
)
)
forwarded: list[list[dict[str, Any]]] = []
with TestClient(app) as client:
proxy = client.app.state.proxy
proxy.openai_pipeline.apply = _relevance_pipeline
async def _upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
forwarded.append(json.loads(json.dumps(body["messages"])))
# Automatic prefix caching reports nothing cached on a
# conversation's first request, so there is no confirmed floor.
usage = {"prompt_tokens": 600, "completion_tokens": 3, "total_tokens": 603}
usage["prompt_tokens_details"] = {"cached_tokens": 0}
return httpx.Response(
200,
json={
"id": "chatcmpl_x",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": usage,
},
)
proxy._retry_request = _upstream
for messages in requests:
response = client.post(
"/v1/chat/completions",
headers={"authorization": "Bearer test-key"},
json={"model": "gpt-4o-mini", "messages": messages},
)
assert response.status_code == 200, response.text[:300]
return forwarded
def test_openai_token_mode_replays_its_prefix_after_a_restart(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Review of #4299: only the Anthropic handler seeded its CompressionCache
from a restored tracker. The OpenAI token path froze nothing, recompressed
the tool result under the new query, and with no provider-confirmed floor
the overlay declined the larger previous form, so the prefix changed."""
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
reset_compression_store()
(previous,) = _drive_openai_chat([_chat_turns(1)])
(current,) = _drive_openai_chat([_chat_turns(2)])
assert previous[2]["content"] != _chat_turns(1)[2]["content"], "nothing was compressed"
assert current[: len(previous)] == previous
def test_read_maturation_state_survives_a_restart(tmp_path: Path) -> None:
"""Matured Reads stay booked once: without this a restart books their
removal again on every later request of the conversation."""
from headroom.config import ReadMaturationConfig
from headroom.transforms.read_maturation import MaturedRead, ReadMaturationManager
store = _two_conversations()
manager = ReadMaturationManager(ReadMaturationConfig(enabled=True))
manager._matured["toolu_r00"] = MaturedRead(marker="[Read of /f0.py ...]", ccr_hash="abc")
manager._booked.add("toolu_r00")
manager._replay_token_deltas["toolu_r00"] = 640
newest = max(store._trackers.values(), key=lambda t: t._last_activity)
newest.read_maturation_manager = manager
path = str(tmp_path / "state.json")
store.save_state(path)
restored = SessionTrackerStore()
restored.load_state(path)
states = [t.restored_maturation_state for t in restored._trackers.values()]
assert all(t.restored_from_disk for t in restored._trackers.values())
(state,) = [s for s in states if s is not None]
adopted = ReadMaturationManager(ReadMaturationConfig(enabled=True))
adopted.restore_state(state)
assert adopted._matured == manager._matured
assert adopted._booked == manager._booked
assert adopted._replay_token_deltas == manager._replay_token_deltas
def test_compress_sidecar_session_replays_its_prefix_after_a_restart(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""The same seed on the session-aware /v1/compress path (sidecar gateways),
which replays with the size bound alone (no provider-confirmed floor)."""
from tests.test_compress_session_mode import _compress, _make_client
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
reset_compression_store()
forwarded = []
for history in (_chat_turns(1), _chat_turns(2)): # one proxy each: a restart between
with _make_client() as client:
client.app.state.proxy._no_ccr_pipeline = lambda: SimpleNamespace(
apply=_relevance_pipeline
)
forwarded.append(_compress(client, history, session_id="conv-r")["messages"])
previous, current = forwarded
assert previous[2]["content"] != _chat_turns(1)[2]["content"], "nothing was compressed"
assert current[: len(previous)] == previous
Generated
+1 -1
View File
@@ -23,7 +23,7 @@ resolution-markers = [
[manifest]
constraints = [
{ name = "aiohttp", specifier = ">=3.14.3" },
{ name = "aiohttp", specifier = ">=3.14.4" },
{ name = "cryptography", specifier = ">=50.0.0" },
{ name = "gitpython", specifier = ">=3.1.58" },
{ name = "langsmith", specifier = ">=0.9.0" },