mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): keep prefix-tracker state across a graceful proxy restart (#4299)
## Description Every proxy restart rewrites the prompt cache of every warm conversation. After a restart the proxy has no `PrefixCacheTracker` for the conversation, so it compresses the whole history afresh, the early messages come out as different bytes than the prefix it forwarded before the restart, and the provider re-writes the cache from the first changed message. Measured on 2026-10-19 on one Mac running Claude Code through the proxy: after each graceful restart (app relaunches for updates: proxy SIGTERM'd and respawned), the next request of every active session read only ~25k cached tokens (tools + system) and re-wrote the rest, up to 247k, 310k, 361k and 432k tokens at a time. The restart-aligned rewrites came to about 4.6M tokens of 1h cache writes that day (2x the input price), about a quarter of all cache writes. A client-side integrity probe reported `kind=tracker_reset, stable_messages=861, first_changed_message=3`. This adds an opt-in `HEADROOM_PREFIX_STATE_FILE=<path>`. When set, the proxy saves the `SessionTrackerStore` on graceful shutdown and restores it at startup, before serving, so the first request after a restart replays byte-identical frozen-prefix bytes. Unset, nothing changes. The file holds conversation content, so it is never a default. Related but different: #4156 (away recap replaces the lineage), #4287 (CCR proactive expansion rewrites), #4204 (deferred tool loads in the lineage key), #4284 (mid-history divergence starts a cold lineage). No existing issue; found while auditing one machine's token usage. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `SessionTrackerStore.save_state` / `load_state` (`headroom/cache/prefix_tracker.py`). Per tracker: turn number, cached token and message counts, last original and last forwarded messages, the per-turn states that lineage resolution resumes, provider, last activity and observed cache TTL; per lineage: session id, cache affinity and the recent snapshot history. So `resolve_tracker` matches the next request to the restored tracker, forks included. Each distinct message is stored once per tracker (successive turns, snapshots and the original/forwarded pairs mostly repeat), and new lineage keys and turn ids continue past the restored ones. - CCR entries ride along (review: retrieval integrity). Per tracker, `save_state` collects every CCR reference in its saved messages and read-maturation state (`hash=<hex>` markers and SmartCrusher's `<<ccr:<hex>`) and writes the live `CompressionStore` entries behind them (new `CompressionStore.export_entries`, which reads without counting a retrieval). A tracker and its entries count against the size cap together, so a tracker is never saved without them. `load_state` re-stores them (`import_entries`) with their original expiry, before any tracker is installed, so a replayed marker resolves from the first request even with `HEADROOM_CCR_BACKEND=memory`. An entry a persistent backend still holds is left as it is; a malformed one rejects the file. - Restore keeps only trackers still inside the store's own retention (last activity plus the observed or configured cache TTL and its cleanup margin, at least the session TTL), the same rule the live store uses. NaN or future timestamps are dropped. - Versioned schema (`version: 1`). Any read, parse or validation error logs one warning and leaves the store empty; the whole file is validated before anything is installed. The file is written atomically (`headroom.fsutil.write_text`: temp file, fsync, rename) at mode 0600: an existing file is tightened to 0600 in place, never unlinked first, so a failed write (ENOSPC) keeps the previous snapshot. Deleted after a successful load. Stateless mode (`persistence_allowed`) neither writes nor reads it (a stateless start used to load, and so delete, a snapshot). Ignored with multiple workers (one file cannot hold N per-process stores). - Size cap: the 256 most recently active trackers and at most 64 MB, newest first; whatever does not fit starts cold, as everything did before. - `HeadroomProxy.startup()` loads, `HeadroomProxy.shutdown()` saves as its first step, before any await, so the shutdown step timeout cannot cut it short. No periodic writes. - Token-mode `CompressionCache` seed: a restored tracker has no `CompressionCache` behind it in the new process, so the freeze computed from that cache froze nothing: the pipeline recompressed messages the provider cached, read maturation rewrote Reads inside the cached prefix, and beyond the provider-confirmed floor the overlay declined a larger previous form. One shared `seed_restored_compression_cache` seeds the session's cache from the restored original/forwarded pair (`update_from_result`) on the first request, on all three token-mode paths: Anthropic `/v1/messages`, OpenAI `/v1/chat/completions`, and the session-aware `/v1/compress` (sidecar). It gives the same freeze count as before the restart. - `ReadMaturationManager.export_state` / `validate_state` / `restore_state`: matured markers, booked ids and replay token deltas ride along with the tracker, so a matured Read's removal is not booked as savings again on every request after a restart. The saved state is validated at load with the rest of the file, so a bad entry rejects the file instead of failing a request. - `tests/test_anthropic_fork_lineage_cache.py`: `_drive` accepts a shared provider-cache model and config overrides, so a second call is a proxy restart in front of a warm provider cache. - Documented the variable in `docs/content/docs/configuration.mdx`. - `pyproject.toml` / `uv.lock` (review: Dependency audit): the `aiohttp` constraint floor is now `>=3.14.4` (CVE-2026-111063, CVE-2026-111064 and GHSA-2g87-pp6c-29x5, all fixed in 3.14.4), relocked with `uv lock --upgrade-package aiohttp==3.14.4`. Only aiohttp (3.14.3 -> 3.14.4) and yarl (1.22.0 -> 1.25.1, which 3.14.4 requires) move. Main still resolves 3.14.3, so the same commit is on #4299, #4286, #4285 and #4269. Remaining gaps (unchanged by this PR, none of them changes forwarded message bytes): - Session-sticky `anthropic-beta` union (`HEADROOM_BETA_HEADER_STICKY`): in memory, so after a restart a session's sticky set starts from the client's own header again. Headers only. - Output shaper: stateless per request (byte-stable steering text, effort from the current request's structure), so nothing to persist. - Gemini and Bedrock paths resume the restored tracker and its overlay replay; they keep no token-mode `CompressionCache`, so there is nothing to seed. A crash or SIGKILL still loses everything, by design (no periodic writes). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # New regression tests for the review findings, run against thec141e2ecsource # (headroom/ stashed, new tests kept): all six fail, the 15 earlier tests pass. $ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q FAILED tests/test_prefix_tracker_restart.py::test_restored_retrieval_references_resolve_to_their_originals FAILED tests/test_prefix_tracker_restart.py::test_a_stateless_proxy_leaves_a_saved_snapshot_alone FAILED tests/test_prefix_tracker_restart.py::test_a_failed_save_keeps_the_previous_snapshot FAILED tests/test_prefix_tracker_restart.py::test_saved_ccr_entries_resolve_after_a_restart FAILED tests/test_prefix_tracker_restart.py::test_openai_token_mode_replays_its_prefix_after_a_restart FAILED tests/test_prefix_tracker_restart.py::test_compress_sidecar_session_replays_its_prefix_after_a_restart 6 failed, 15 passed in 8.10s # Why, in order: hash=cc146b21626392f0dee9890c no longer resolves; the stateless start # deleted the snapshot; ENOSPC left no snapshot; retrieve() returns None; the OpenAI chat # and /v1/compress paths re-compressed message 2 under the new query. # Headc379245e: $ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q 21 passed in 9.50s $ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py tests/test_anthropic_fork_lineage_cache.py \ tests/test_cache/test_prefix_tracker.py tests/test_compression_store.py tests/test_ccr_sqlite_backend.py \ tests/test_issue_2671_block_growth_cache.py tests/test_cache_mode_cold_start.py tests/test_cache_prefix_overlay.py \ tests/test_cross_turn_cache_safety.py tests/test_session_engine.py tests/test_read_maturation.py \ tests/test_read_maturation_handler_nobust.py tests/test_graceful_shutdown.py tests/test_proxy_handlers_batch.py \ tests/test_cache_control_move_bust.py tests/test_startup_log_noise.py tests/test_proxy_openai_cache_stability.py \ tests/test_compress_session_mode.py tests/test_proxy_anthropic_cache_stability.py tests/test_ccr_row_drop_store_bridge.py \ tests/test_bedrock_prefix_tracker_wiring.py -q 486 passed, 1 skipped in 46.92s $ uv run --frozen --extra dev pytest tests/test_anthropic_*.py tests/test_proxy_anthropic_*.py tests/test_proxy_openai*.py \ tests/test_openai_chat_*.py tests/test_cache_aligner_prefix_stability.py tests/test_cache_prefix_safety.py tests/test_cold_prefix.py \ tests/test_gemini_prefix_freeze.py tests/test_prefix_tracker_client_rewrote_reply.py tests/test_tool_injection*.py tests/test_ccr_*.py -q 737 passed, 16 skipped in 51.45s $ uvx ruff@0.17.0 check headroom tests All checks passed! $ uvx ruff@0.17.0 format --check headroom tests 1606 files already formatted $ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/cache/compression_store.py \ headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py headroom/proxy/server.py Success: no issues found in 5 source files ``` ## Real Behavior Proof - Environment: macOS 27 (arm64), this branch atc379245e, the real proxy process (`python -m headroom.cli proxy --port 6811 --no-http2 --no-rate-limit --mode token --read-maturation --read-maturation-quiesce-turns 1 --read-maturation-min-size-bytes 200 --disable-kompress --no-subscription-tracking`, `HEADROOM_ROLLOUT_CHANNEL=beta`, clean `HOME` and workspace, `HEADROOM_PREFIX_STATE_FILE` set, `HEADROOM_CCR_BACKEND=memory`, `HEADROOM_BACKGROUND_COMPRESSION=1`) in front of the live Anthropic API, driven by Claude Code 2.1.303 (`claude -p --model haiku`, `ANTHROPIC_BASE_URL=http://127.0.0.1:6811`). Each restart is a SIGTERM (graceful shutdown) and a fresh process. - Exact command / steps: one Claude Code session through the proxy, three graceful restarts, the last one without the state file as the control: 1. Turn 1: `claude -p` reads three files (4.8, 17 and 8.9 KB) one per turn, 5 requests through the proxy. SIGTERM: `SessionTrackerStore: saved 2 of 2 prefix trackers (181266 bytes)`; file mode `-rw-------`; it holds 2 CCR entries, for the two Reads that read maturation had replaced with `Retrieve original: hash=` markers. 2. Restart: `restored 2 of 2 prefix trackers`, file deleted. `GET /v1/retrieve/3a739d6ec5febcd49d0cfc0e` and `/v1/retrieve/2ee4d4d435432b4cca9b72af` on the new process: HTTP 200, `original_content` equal to the pre-restart entries (5,129 and 18,525 chars, the Read outputs of the two files). With the in-process CCR store,c141e2ecloses such entries at a restart (the reviewer's reproduction; `test_restored_retrieval_references_resolve_to_their_originals` fails on it). 3. `claude -p --resume`, then SIGTERM + restart with the state file again, then `claude -p --resume` again (one request each, no tools). 4. Control: SIGTERM, snapshot moved aside, restart without `HEADROOM_PREFIX_STATE_FILE`, `claude -p --resume`. - Observed result: after a restart with the state file the first request read 99% of its prefix from the provider cache and both restored retrieval hashes resolved; the control re-wrote everything after the system prompt (proxy `PERF` lines, which match Claude Code's reported usage): | First request after a restart | cache_read | cache_write | hit | |---|---|---|---| | With the state file, resume after a resume | 40,406 | 514 | 99% | | Without the state file (control), same history | 26,053 | 11,250 | 70% | With the state file the proxy forwarded messages 0-10 byte-identical to what it forwarded before the restart (compared from the saved snapshots, `cache_control` positions excluded since Claude Code moves its breakpoint every turn), and froze 14 of 15 messages. In the control, the forwarded bytes changed from message 1, and the third Read, verbatim inside the cached prefix, matured into a new marker (10,205 -> 368 chars), so everything after the system prompt was re-written. - Not tested: Kompress itself (`--disable-kompress`; the unit suite uses a deterministic stand-in). Windows file modes. The OpenAI chat and `/v1/compress` seeds against a live OpenAI endpoint (covered by the handler-level restart tests above with a query-dependent compressor). The very first `--resume` of a `claude -p` session (step 3's first request) read only 26,053 tokens and wrote 14,845, although the client's messages and the proxy's forwarded messages 0-10 were byte-identical to the pre-restart request; the read stopped at the system-prompt breakpoint, so the change is most likely in the system/tools part of the request, which I did not capture. The resume after it, across another restart, read 99%. ## Runtime Rollout Safety - Rollout-managed feature(s): None; opt-in environment variable `HEADROOM_PREFIX_STATE_FILE`, unset by default. - Minimum rollout channel: Stable. - Stable/default behavior changed: No; with the variable unset nothing is read or written and no code path changes (the new handler branches only fire on a tracker restored from the file, and the CCR export/import only runs inside save/load). - Kill switch / disable path: Unset `HEADROOM_PREFIX_STATE_FILE`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Unset the variable or revert this PR; a leftover state file is inert without the variable. The file format is still `version: 1` (now with a `ccr` list); a file written byc141e2echas no `ccr` list and is rejected at load, which starts cold. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes - Review follow-up (c379245e): CCR entries are saved with their trackers (P1), the seed covers the OpenAI chat and `/v1/compress` token paths, a failed save keeps the previous snapshot, and stateless mode no longer loads (and deletes) a snapshot. Each has a regression test that fails onc141e2ec. - Caller isolation: the restored store has exactly the keys, lineages and retention the in-memory store had at shutdown, and replay still requires the caller to send the same original messages; see the reply on that thread for the full reasoning. - The full `pytest tests` run was not repeated for this head (shared machine); the focused runs above cover every module the change touches. - Size cap choice: 256 trackers covers many concurrent Claude Code sessions plus their subagents; 64 MB bounds shutdown time (encoding is synchronous) and disk use. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4299" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
08b78f45f1
commit
62b3bedaba
@@ -401,6 +401,7 @@ headroom proxy --learn --min-evidence 3
|
||||
| `HEADROOM_THINKING_COMPACT` | Compact plain-text reasoning that models re-send every turn (Kimi/GLM/DeepSeek `reasoning_content` / inline `<think>`): Kompress it on warm turns, drop it on cold turns. No-op for Claude/Codex/OpenAI (encrypted reasoning). See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
|
||||
| `HEADROOM_THINKING_COMPACT_KEEP_LAST` | Most-recent assistant turns whose reasoning is left intact (the active reasoning the model still uses). Only applies with `HEADROOM_THINKING_COMPACT`. | `1` |
|
||||
| `HEADROOM_COLD_RECOMPACT` | On a confirmed-cold turn (idle past the real cache TTL), recompact the whole prefix — cross-turn dedupe + superseded-read drop + lossless folds — instead of forwarding a byte-identical prefix to a dead cache. Warm turns unchanged. Pair with `HEADROOM_DEDUPE`. See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
|
||||
| `HEADROOM_PREFIX_STATE_FILE` | Path where the proxy saves its prefix-cache trackers on a graceful shutdown and restores them on the next start (then deletes the file), so warm conversations replay the exact prefix the provider cached instead of re-writing it. The file holds conversation content: it is written `0600`, capped at the 256 newest conversations and 64 MB, and only trackers still inside their cache lifetime are restored, with the retrieval (CCR) entries their markers point at, so those markers resolve even with `HEADROOM_CCR_BACKEND=memory`. Not supported with multiple workers; ignored in stateless mode (neither read nor written). | -- |
|
||||
| `HEADROOM_DEDUPE` | Whole-conversation verbatim cross-turn dedup in the router (cache-safe, information-preserving via retrieval markers). Superseded-read drop + lossless folds run without it; this adds verbatim dedup. | `off` |
|
||||
| `HEADROOM_CACHE_TTL_LEARN` | Append per-turn cache-outcome observations (provider, model, idle, hit/miss) to `cache_ttl_observations.jsonl` for the offline `headroom-cache-ttl` learner. Observation-only (no request-behavior change); respects `HEADROOM_STATELESS`; the log is size-bounded. | `off` |
|
||||
| `HEADROOM_KOMPRESS_ENDPOINT` / `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | Offload ML compression (Kompress) to a remote endpoint instead of the local ONNX model — used by reasoning compaction and the router when set. | -- |
|
||||
|
||||
Vendored
+43
-1
@@ -40,7 +40,7 @@ import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass, field, replace
|
||||
from dataclasses import asdict, dataclass, field, replace
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ..log_safety import describe_exception, safe_id
|
||||
@@ -737,6 +737,48 @@ class CompressionStore:
|
||||
|
||||
return list(reversed(events_copy[-limit:]))
|
||||
|
||||
def export_entries(self, hashes: list[str]) -> list[dict[str, Any]]:
|
||||
"""The live entries for ``hashes`` as JSON-safe dicts, without counting
|
||||
a retrieval. ``SessionTrackerStore.save_state`` carries them across a
|
||||
restart so the markers it replays still resolve (:meth:`import_entries`)."""
|
||||
with self._lock:
|
||||
found = [self._backend.get(hash_key) for hash_key in hashes]
|
||||
return [asdict(e) for e in found if e is not None and not e.is_expired()]
|
||||
|
||||
def import_entries(self, entries: list[CompressionEntry]) -> int:
|
||||
"""Re-store :meth:`export_entries` output, each with its original expiry.
|
||||
|
||||
An entry the backend still holds (a persistent backend kept it) is left
|
||||
as it is. Raises on a malformed entry. Returns the entries stored.
|
||||
"""
|
||||
now = time.time()
|
||||
stored = 0
|
||||
for e in entries:
|
||||
if not all(
|
||||
isinstance(v, str) for v in (e.hash, e.original_content, e.compressed_content)
|
||||
):
|
||||
raise ValueError(f"malformed CCR entry {e.hash!r}")
|
||||
ttl = int(e.created_at + e.ttl - now)
|
||||
if ttl <= 0 or self.exists(e.hash):
|
||||
continue
|
||||
self.store(
|
||||
e.original_content,
|
||||
e.compressed_content,
|
||||
original_tokens=e.original_tokens,
|
||||
compressed_tokens=e.compressed_tokens,
|
||||
original_item_count=e.original_item_count,
|
||||
compressed_item_count=e.compressed_item_count,
|
||||
tool_name=e.tool_name,
|
||||
tool_call_id=e.tool_call_id,
|
||||
query_context=e.query_context,
|
||||
tool_signature_hash=e.tool_signature_hash,
|
||||
compression_strategy=e.compression_strategy,
|
||||
ttl=ttl,
|
||||
explicit_hash=e.hash,
|
||||
)
|
||||
stored += 1
|
||||
return stored
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Clear all entries. Mainly for testing."""
|
||||
with self._lock:
|
||||
|
||||
Vendored
+277
@@ -16,16 +16,19 @@ pipeline skips them entirely.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import copy
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from collections import OrderedDict, deque
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -71,6 +74,23 @@ _CACHE_TTL_CLEANUP_MARGIN_SECONDS = 60
|
||||
# of forks off one turn leaves that turn resumable.
|
||||
_LINEAGE_HISTORY = 3
|
||||
|
||||
# Opt-in file for SessionTrackerStore state across a graceful restart. It holds
|
||||
# conversation content, so nothing is written unless an operator sets it.
|
||||
PREFIX_STATE_FILE_ENV = "HEADROOM_PREFIX_STATE_FILE"
|
||||
_PREFIX_STATE_VERSION = 1
|
||||
# Newest trackers first; whatever does not fit starts cold, as all did before.
|
||||
_PREFIX_STATE_MAX_TRACKERS = 256
|
||||
_PREFIX_STATE_MAX_BYTES = 64 * 1024 * 1024
|
||||
# Any CCR reference in saved messages: ``hash=<hex>`` markers (Retrieve
|
||||
# more/original, compressed-item counts) and SmartCrusher's ``<<ccr:<hex>``.
|
||||
_CCR_HASH_RE = re.compile(r"(?:hash=|<<ccr:)([0-9a-f]{12,24})")
|
||||
|
||||
|
||||
def prefix_state_file() -> str | None:
|
||||
"""The configured tracker state file, or None when persistence is off."""
|
||||
return os.environ.get(PREFIX_STATE_FILE_ENV, "").strip() or None
|
||||
|
||||
|
||||
# (tracker, turn, resumed state) the current request resolved to. Each proxied
|
||||
# request runs in its own asyncio task, so it reads the state it resumed and
|
||||
# ``update_from_response`` records its response against its own turn, even
|
||||
@@ -1180,6 +1200,12 @@ class PrefixCacheTracker:
|
||||
# lazily by the handler when read maturation is enabled. Rides
|
||||
# here so it shares the session's affinity and TTL cleanup.
|
||||
self.read_maturation_manager: Any = None
|
||||
# Set by SessionTrackerStore.load_state: this process has no
|
||||
# CompressionCache entries behind the restored prefix yet, and the
|
||||
# handler adopts the saved maturation state (validated at load) when it
|
||||
# builds the manager.
|
||||
self.restored_from_disk = False
|
||||
self.restored_maturation_state: dict[str, Any] | None = None
|
||||
|
||||
# Stats
|
||||
self._busts_avoided: int = 0
|
||||
@@ -1609,6 +1635,42 @@ def _lineage_snapshot(obj: Any) -> Any:
|
||||
return obj
|
||||
|
||||
|
||||
def _deref(table: list[Any], refs: list[int]) -> list[Any]:
|
||||
"""Rebuild a saved message list. Restored lists share message objects,
|
||||
which is safe: tracker and lineage lists are never mutated in place."""
|
||||
if not all(type(i) is int and 0 <= i < len(table) for i in refs):
|
||||
raise ValueError("message reference out of range")
|
||||
return [table[i] for i in refs]
|
||||
|
||||
|
||||
def _decode_turn_state(table: list[Any], state: list[Any]) -> tuple[Any, ...]:
|
||||
turn, cached_tokens, cached_messages, original, forwarded = state
|
||||
return (
|
||||
int(turn),
|
||||
int(cached_tokens),
|
||||
int(cached_messages),
|
||||
_deref(table, original),
|
||||
_deref(table, forwarded),
|
||||
)
|
||||
|
||||
|
||||
def seed_restored_compression_cache(tracker: Any, comp_cache: Any) -> None:
|
||||
"""Seed a token-mode CompressionCache on a tracker's first request after
|
||||
:meth:`SessionTrackerStore.load_state`.
|
||||
|
||||
The new process has no cache entries behind the restored prefix, so the
|
||||
freeze computed from that cache freezes nothing: the pipeline recompresses
|
||||
(and read maturation rewrites) messages the provider cached, and beyond the
|
||||
provider-confirmed floor the overlay declines a larger replay. Seeding with
|
||||
what was forwarded gives the freeze the pre-restart count.
|
||||
"""
|
||||
if getattr(tracker, "restored_from_disk", False):
|
||||
tracker.restored_from_disk = False
|
||||
comp_cache.update_from_result(
|
||||
tracker.get_last_original_messages(), tracker.get_last_forwarded_messages()
|
||||
)
|
||||
|
||||
|
||||
class SessionTrackerStore:
|
||||
"""Manages PrefixCacheTracker instances across sessions.
|
||||
|
||||
@@ -2007,3 +2069,218 @@ class SessionTrackerStore:
|
||||
def active_sessions(self) -> int:
|
||||
"""Number of active session trackers (one per conversation lineage)."""
|
||||
return len(self._trackers)
|
||||
|
||||
def save_state(self, path: str) -> int:
|
||||
"""Write the live trackers to ``path`` for the next process to resume.
|
||||
|
||||
Called on graceful shutdown when ``HEADROOM_PREFIX_STATE_FILE`` is set.
|
||||
Without it, a restart leaves every warm conversation with no tracker:
|
||||
its next request recompresses the whole history into bytes that differ
|
||||
from the prefix the provider cached, and the cache is re-written from
|
||||
the first changed message. Newest trackers first, capped at
|
||||
``_PREFIX_STATE_MAX_TRACKERS`` and ``_PREFIX_STATE_MAX_BYTES``; a
|
||||
tracker whose messages are not JSON is skipped. Each tracker carries
|
||||
the CCR entries its messages reference, so its replayed markers still
|
||||
resolve when the CCR backend does not survive the restart (memory).
|
||||
Never raises; returns the number of trackers written.
|
||||
"""
|
||||
from headroom.cache.compression_store import get_compression_store
|
||||
from headroom.fsutil import write_text
|
||||
from headroom.paths import persistence_allowed
|
||||
|
||||
if not persistence_allowed("prefix-cache tracker state"):
|
||||
return 0
|
||||
sessions = {key: sid for sid, family in self._lineages.items() for key in family}
|
||||
live = sorted(
|
||||
(item for item in self._trackers.items() if not item[1].is_expired),
|
||||
key=lambda item: item[1]._last_activity,
|
||||
reverse=True,
|
||||
)
|
||||
entries: list[str] = []
|
||||
ccr: dict[str, str] = {}
|
||||
size = 0
|
||||
for key, tracker in live[:_PREFIX_STATE_MAX_TRACKERS]:
|
||||
try:
|
||||
entry = self._encode_tracker(key, tracker, sessions.get(key))
|
||||
refs = [h for h in dict.fromkeys(_CCR_HASH_RE.findall(entry)) if h not in ccr]
|
||||
records = get_compression_store().export_entries(refs) if refs else []
|
||||
payloads = {r["hash"]: json.dumps(r) for r in records}
|
||||
except Exception: # an unsaveable tracker starts cold, as all did before
|
||||
continue
|
||||
grow = len(entry) + sum(len(p) + 1 for p in payloads.values())
|
||||
if size + grow > _PREFIX_STATE_MAX_BYTES:
|
||||
continue
|
||||
size += grow + 1
|
||||
entries.append(entry)
|
||||
ccr.update(payloads)
|
||||
try:
|
||||
target = Path(path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
# write_text keeps the target's mode, so tighten it in place rather
|
||||
# than unlinking: a failed write must leave the last snapshot.
|
||||
with contextlib.suppress(FileNotFoundError):
|
||||
target.chmod(0o600)
|
||||
write_text(
|
||||
target,
|
||||
f'{{"version":{_PREFIX_STATE_VERSION},"trackers":[{",".join(entries)}],'
|
||||
f'"ccr":[{",".join(ccr.values())}]}}',
|
||||
)
|
||||
except OSError as exc:
|
||||
logger.warning("SessionTrackerStore: prefix state not saved to %s: %s", path, exc)
|
||||
return 0
|
||||
logger.info(
|
||||
"SessionTrackerStore: saved %d of %d prefix trackers (%d bytes) to %s",
|
||||
len(entries),
|
||||
len(self._trackers),
|
||||
size,
|
||||
path,
|
||||
)
|
||||
return len(entries)
|
||||
|
||||
def _encode_tracker(self, key: str, tracker: PrefixCacheTracker, session_id: str | None) -> str:
|
||||
"""One tracker as JSON, each distinct message stored once.
|
||||
|
||||
Successive turns, the lineage snapshots and the original/forwarded
|
||||
pairs mostly repeat the same messages, so this keeps the file near one
|
||||
copy of each conversation instead of up to eleven.
|
||||
"""
|
||||
index: dict[str, int] = {}
|
||||
memo: dict[int, list[int]] = {}
|
||||
|
||||
def ref(messages: list[Any]) -> list[int]:
|
||||
if id(messages) not in memo:
|
||||
refs = []
|
||||
for message in messages:
|
||||
blob = json.dumps(message)
|
||||
refs.append(index.setdefault(blob, len(index)))
|
||||
memo[id(messages)] = refs
|
||||
return memo[id(messages)]
|
||||
|
||||
def state(s: tuple[Any, ...]) -> list[Any]:
|
||||
return [s[0], s[1], s[2], ref(s[3]), ref(s[4])]
|
||||
|
||||
maturation = tracker.read_maturation_manager
|
||||
meta = {
|
||||
"key": key,
|
||||
"session": session_id,
|
||||
"affinity": self._lineage_affinities.get(key),
|
||||
"provider": tracker.provider,
|
||||
"last_activity": tracker._last_activity,
|
||||
"cache_ttl": tracker._cache_ttl_seconds,
|
||||
"state": state(tracker._turn_state()),
|
||||
"turns": [[t, state(s)] for t, s in tracker._turn_states.items() if s is not None],
|
||||
"history": [[ref(snap), t] for snap, t in self._lineage_history.get(key, ())],
|
||||
"maturation": None if maturation is None else maturation.export_state(),
|
||||
}
|
||||
return f'{{"meta":{json.dumps(meta)},"messages":[{",".join(index)}]}}'
|
||||
|
||||
def load_state(self, path: str) -> int:
|
||||
"""Resume the trackers :meth:`save_state` wrote, before serving.
|
||||
|
||||
Trackers past their retention are dropped. The file is removed once
|
||||
loaded, so a later crash cannot bring back state older than what it
|
||||
lost. A missing file is normal; an unreadable or invalid one is logged
|
||||
and leaves the store empty. Never raises; returns trackers restored.
|
||||
"""
|
||||
from headroom.paths import persistence_allowed
|
||||
|
||||
# Loading deletes the file, so stateless mode leaves it for a later run.
|
||||
if not persistence_allowed("prefix-cache tracker state"):
|
||||
return 0
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
restored = self._decode_state(data)
|
||||
except FileNotFoundError:
|
||||
return 0
|
||||
except Exception as exc: # any bad file must leave boot unaffected
|
||||
logger.warning(
|
||||
"SessionTrackerStore: ignoring prefix state %s (%s: %s)",
|
||||
path,
|
||||
type(exc).__name__,
|
||||
exc,
|
||||
)
|
||||
return 0
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError as exc:
|
||||
logger.warning("SessionTrackerStore: could not remove %s: %s", path, exc)
|
||||
logger.info(
|
||||
"SessionTrackerStore: restored %d of %d prefix trackers from %s",
|
||||
restored,
|
||||
len(data["trackers"]),
|
||||
path,
|
||||
)
|
||||
return restored
|
||||
|
||||
def _decode_state(self, data: Any) -> int:
|
||||
"""Validate the whole file first, then install it; a bad file installs nothing."""
|
||||
from headroom.cache.compression_store import CompressionEntry, get_compression_store
|
||||
|
||||
if data.get("version") != _PREFIX_STATE_VERSION:
|
||||
raise ValueError(f"unsupported version {data.get('version')!r}")
|
||||
ccr = [CompressionEntry(**record) for record in data["ccr"]]
|
||||
trackers: dict[str, PrefixCacheTracker] = {}
|
||||
families: dict[str, OrderedDict[str, list[Any]]] = {}
|
||||
affinities: dict[str, str | None] = {}
|
||||
histories: dict[str, deque[tuple[list[Any], int]]] = {}
|
||||
now = time.time()
|
||||
# Oldest first, so each family keeps its lineages in creation order.
|
||||
for entry in reversed(data["trackers"]):
|
||||
meta, table = entry["meta"], entry["messages"]
|
||||
tracker = PrefixCacheTracker(str(meta["provider"]), self._default_config)
|
||||
tracker._last_activity = float(meta["last_activity"])
|
||||
tracker.observe_cache_ttl(meta["cache_ttl"])
|
||||
# NaN or a future stamp would never expire.
|
||||
if tracker.is_expired or not tracker._last_activity <= now:
|
||||
continue
|
||||
(
|
||||
tracker._turn_number,
|
||||
tracker._cached_token_count,
|
||||
tracker._cached_message_count,
|
||||
tracker._last_original_messages,
|
||||
tracker._last_forwarded_messages,
|
||||
) = _decode_turn_state(table, meta["state"])
|
||||
tracker._turn_states = {int(t): _decode_turn_state(table, s) for t, s in meta["turns"]}
|
||||
tracker.restored_from_disk = True
|
||||
if meta["maturation"] is not None:
|
||||
from headroom.transforms.read_maturation import ReadMaturationManager
|
||||
|
||||
tracker.restored_maturation_state = ReadMaturationManager.validate_state(
|
||||
meta["maturation"]
|
||||
)
|
||||
key = str(meta["key"])
|
||||
trackers[key] = tracker
|
||||
history = deque(
|
||||
((_deref(table, snap), int(t)) for snap, t in meta["history"]),
|
||||
maxlen=_LINEAGE_HISTORY,
|
||||
)
|
||||
if not history:
|
||||
continue
|
||||
histories[key] = history
|
||||
if meta["session"] is not None:
|
||||
families.setdefault(str(meta["session"]), OrderedDict())[key] = history[0][0]
|
||||
affinity = meta["affinity"]
|
||||
affinities[key] = None if affinity is None else str(affinity)
|
||||
|
||||
if ccr and trackers:
|
||||
# First, so a marker a restored tracker replays resolves from its
|
||||
# first request; a failure here still installs no tracker.
|
||||
get_compression_store().import_entries(ccr)
|
||||
|
||||
# New turns and lineages must not reuse a restored id.
|
||||
turns = [t for h in histories.values() for _, t in h]
|
||||
turns += [t for tracker in trackers.values() for t in tracker._turn_states]
|
||||
lineage_ns = [
|
||||
int(n)
|
||||
for _, sep, n in (key.rpartition("\x00") for key in trackers)
|
||||
if sep and n.isdecimal()
|
||||
]
|
||||
self._turn_counter = itertools.count(max([next(self._turn_counter), *turns]) + 1)
|
||||
self._lineage_counter = itertools.count(max([next(self._lineage_counter), *lineage_ns]) + 1)
|
||||
self._trackers.update(trackers)
|
||||
self._lineage_history.update(histories)
|
||||
self._lineage_affinities.update(affinities)
|
||||
for session_id, family in families.items():
|
||||
self._lineages.setdefault(session_id, OrderedDict()).update(family)
|
||||
return len(trackers)
|
||||
|
||||
@@ -2029,6 +2029,15 @@ class AnthropicHandlerMixin:
|
||||
compression_policy = resolve_policy(getattr(request.state, "auth_mode", None))
|
||||
if is_token_mode(self.config.mode):
|
||||
comp_cache = self._get_compression_cache(session_id)
|
||||
from headroom.cache.prefix_tracker import (
|
||||
seed_restored_compression_cache,
|
||||
)
|
||||
|
||||
# Seeds the empty cache after a restart
|
||||
# (HEADROOM_PREFIX_STATE_FILE); without it the clamp below
|
||||
# freezes nothing and read maturation rewrites Reads
|
||||
# inside the cached prefix.
|
||||
seed_restored_compression_cache(prefix_tracker, comp_cache)
|
||||
|
||||
# Freeze + stable marking + Zone-1 swap now live in the
|
||||
# shared session engine (PROXY policy: clamp by BOTH the
|
||||
@@ -2609,6 +2618,10 @@ class AnthropicHandlerMixin:
|
||||
compression_store=get_compression_store(),
|
||||
)
|
||||
prefix_tracker.read_maturation_manager = maturation_mgr
|
||||
state = getattr(prefix_tracker, "restored_maturation_state", None)
|
||||
if state is not None:
|
||||
prefix_tracker.restored_maturation_state = None
|
||||
maturation_mgr.restore_state(state)
|
||||
_maturation_mgr = maturation_mgr
|
||||
maturation = maturation_mgr.apply(
|
||||
optimized_messages,
|
||||
|
||||
@@ -3821,6 +3821,12 @@ class OpenAIHandlerMixin:
|
||||
|
||||
if is_token_mode(self.config.mode):
|
||||
comp_cache = self._get_compression_cache(openai_session_id)
|
||||
from headroom.cache.prefix_tracker import seed_restored_compression_cache
|
||||
|
||||
# Seeds the empty cache after a restart
|
||||
# (HEADROOM_PREFIX_STATE_FILE); without it the freeze below
|
||||
# freezes nothing.
|
||||
seed_restored_compression_cache(openai_prefix_tracker, comp_cache)
|
||||
|
||||
if not is_cache_mode(self.config.mode):
|
||||
# Token mode: shared engine, REPLAYABLE policy — its
|
||||
@@ -10695,6 +10701,9 @@ class OpenAIHandlerMixin:
|
||||
session_tracker = self.session_tracker_store.get_or_create(
|
||||
_session_key, _tracker_provider
|
||||
)
|
||||
from headroom.cache.prefix_tracker import seed_restored_compression_cache
|
||||
|
||||
seed_restored_compression_cache(session_tracker, comp_cache)
|
||||
|
||||
# A claimed turn prepares its executor-side work here (tokenizer,
|
||||
# hooks, scope binding) — CPU-bound parts run inside the closures
|
||||
|
||||
@@ -1203,6 +1203,8 @@ class HeadroomProxy(
|
||||
session_ttl_seconds=config.prefix_freeze_session_ttl,
|
||||
)
|
||||
)
|
||||
# Set by startup() when HEADROOM_PREFIX_STATE_FILE opts in.
|
||||
self._prefix_state_file: str | None = None
|
||||
|
||||
# Compression cache store for token mode (session-scoped). The dict
|
||||
# itself is mutated under `_compression_caches_lock`; the per-session
|
||||
@@ -2048,6 +2050,17 @@ class HeadroomProxy(
|
||||
async def startup(self):
|
||||
"""Initialize async resources."""
|
||||
self._get_shutdown_event().clear()
|
||||
# Opt-in: resume the prefix trackers the previous process saved on its
|
||||
# graceful shutdown, so warm conversations replay their cached prefix.
|
||||
from headroom.cache.prefix_tracker import PREFIX_STATE_FILE_ENV, prefix_state_file
|
||||
|
||||
self._prefix_state_file = prefix_state_file()
|
||||
if self._prefix_state_file and _MULTI_WORKER_CONFIG_ENV in os.environ:
|
||||
# Each worker holds its own trackers; one file cannot hold them all.
|
||||
logger.warning("%s ignored: not supported with multiple workers", PREFIX_STATE_FILE_ENV)
|
||||
self._prefix_state_file = None
|
||||
if self._prefix_state_file:
|
||||
self.session_tracker_store.load_state(self._prefix_state_file)
|
||||
self.pipeline_extensions.emit(
|
||||
PipelineStage.PRE_START,
|
||||
operation="proxy.startup",
|
||||
@@ -2395,6 +2408,9 @@ class HeadroomProxy(
|
||||
async def shutdown(self):
|
||||
"""Cleanup async resources."""
|
||||
self._get_shutdown_event().set()
|
||||
# First, before any await can hit the shutdown step timeout.
|
||||
if self._prefix_state_file:
|
||||
self.session_tracker_store.save_state(self._prefix_state_file)
|
||||
if self.http_client_h1 and self.http_client_h1 is not self.http_client:
|
||||
await self.http_client_h1.aclose()
|
||||
self.http_client_h1 = None
|
||||
|
||||
@@ -153,6 +153,31 @@ class ReadMaturationManager:
|
||||
# once per session (content and marker are stable per tool call).
|
||||
self._replay_token_deltas: dict[str, int] = {}
|
||||
|
||||
def export_state(self) -> dict[str, Any]:
|
||||
"""Session state as JSON, for ``SessionTrackerStore.save_state``."""
|
||||
return {
|
||||
"matured": {k: [m.marker, m.ccr_hash] for k, m in self._matured.items()},
|
||||
"booked": sorted(self._booked),
|
||||
"deltas": self._replay_token_deltas,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def validate_state(state: Any) -> dict[str, Any]:
|
||||
"""Normalize :meth:`export_state` output read from disk. Raises on a bad
|
||||
shape, so a bad file is rejected at load and never inside a request."""
|
||||
return {
|
||||
"matured": {str(k): [str(m), str(h)] for k, (m, h) in state["matured"].items()},
|
||||
"booked": [str(k) for k in state["booked"]],
|
||||
"deltas": {str(k): int(v) for k, v in state["deltas"].items()},
|
||||
}
|
||||
|
||||
def restore_state(self, state: dict[str, Any]) -> None:
|
||||
"""Adopt :meth:`validate_state` output from the previous process, so a
|
||||
matured Read's removal is not booked again after a restart."""
|
||||
self._matured = {k: MaturedRead(m, h) for k, (m, h) in state["matured"].items()}
|
||||
self._booked = set(state["booked"])
|
||||
self._replay_token_deltas = dict(state["deltas"])
|
||||
|
||||
# ─── Per-request entry point ────────────────────────────────────────
|
||||
|
||||
def apply(
|
||||
|
||||
+3
-2
@@ -410,8 +410,9 @@ constraint-dependencies = [
|
||||
"transformers>=5.5.0",
|
||||
# PYSEC-2026-3447 — transitive dependency; fix at 83.0.0
|
||||
"setuptools>=83.0.0",
|
||||
# PYSEC-2026-3545/3546/3547 — transitive HTTP/WebSocket parser fixes
|
||||
"aiohttp>=3.14.3",
|
||||
# PYSEC-2026-3545/3546/3547 — transitive HTTP/WebSocket parser fixes;
|
||||
# CVE-2026-111063/111064, GHSA-2g87-pp6c-29x5; fix at 3.14.4
|
||||
"aiohttp>=3.14.4",
|
||||
# PYSEC-2026-3552/3553/3554 — PKCS#7 and certificate verification fixes
|
||||
"cryptography>=50.0.0",
|
||||
]
|
||||
|
||||
@@ -156,8 +156,13 @@ class _PromptCache:
|
||||
return read, write, hit
|
||||
|
||||
|
||||
def _drive(requests: list[list[dict[str, Any]]]) -> list[tuple[list[dict[str, Any]], int]]:
|
||||
"""Send each history through the real handler; return (forwarded, read_hit)."""
|
||||
def _drive(
|
||||
requests: list[list[dict[str, Any]]], cache: _PromptCache | None = None, **config: Any
|
||||
) -> list[tuple[list[dict[str, Any]], int]]:
|
||||
"""Send each history through the real handler; return (forwarded, read_hit).
|
||||
|
||||
One app per call: a second call with the same ``cache`` is a proxy restart
|
||||
in front of a provider whose prompt cache stayed warm."""
|
||||
app = create_app(
|
||||
ProxyConfig(
|
||||
optimize=True,
|
||||
@@ -168,9 +173,10 @@ def _drive(requests: list[list[dict[str, Any]]]) -> list[tuple[list[dict[str, An
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
**config,
|
||||
)
|
||||
)
|
||||
cache = _PromptCache()
|
||||
cache = cache or _PromptCache()
|
||||
seen: list[tuple[list[dict[str, Any]], int]] = []
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
|
||||
@@ -0,0 +1,554 @@
|
||||
"""A graceful proxy restart must not cost warm conversations their prompt cache.
|
||||
|
||||
Production, 2026-10-19: five graceful restarts in one day (app relaunches for
|
||||
updates) each made the next request of every active Claude Code session read
|
||||
only the ~25k-token tools + system prefix and re-write the rest, 247k-432k
|
||||
tokens at a time, about a third of that day's cache writes. After a restart
|
||||
the proxy has no ``PrefixCacheTracker`` for the conversation, so it compresses
|
||||
the whole history afresh and the early messages come out as different bytes
|
||||
than the prefix it forwarded before the restart.
|
||||
|
||||
``HEADROOM_PREFIX_STATE_FILE`` (opt-in) saves the tracker store on graceful
|
||||
shutdown and restores it at startup, so the first request after a restart
|
||||
replays the exact prefix the provider cached.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextvars
|
||||
import errno
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
import httpx
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.cache import prefix_tracker
|
||||
from headroom.cache.compression_store import get_compression_store, reset_compression_store
|
||||
from headroom.cache.prefix_tracker import PREFIX_STATE_FILE_ENV, SessionTrackerStore
|
||||
from headroom.ccr.tool_injection import CCRToolInjector
|
||||
from headroom.config import TransformResult
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
from headroom.transforms.pipeline import TransformPipeline
|
||||
from tests.test_anthropic_fork_lineage_cache import (
|
||||
_drive,
|
||||
_main,
|
||||
_PromptCache,
|
||||
_send,
|
||||
_strip,
|
||||
_summary_fork,
|
||||
)
|
||||
|
||||
|
||||
def _wire(messages: list[dict[str, Any]]) -> str:
|
||||
"""The bytes the provider keys its prefix cache on (breakpoints excluded)."""
|
||||
return json.dumps(_strip(messages))
|
||||
|
||||
|
||||
def _ccr_hashes(messages: list[dict[str, Any]]) -> set[str]:
|
||||
"""Retrieval references the model sees, found the way the proxy finds them."""
|
||||
return set(CCRToolInjector().scan_for_markers(messages))
|
||||
|
||||
|
||||
def _with_fake_kompress(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A deterministic stand-in for the Kompress ML stage (off in this suite).
|
||||
|
||||
A cold start forwards the fast pass, which skips Kompress, while warm
|
||||
turns compress their new messages with it. So after a restart without
|
||||
state, the cold fast pass re-emits every message Kompressed after the
|
||||
first turn in a different form, which re-writes the cache from there."""
|
||||
real_apply = TransformPipeline.apply
|
||||
|
||||
def apply(self, messages, model, **kwargs): # noqa: ANN001, ANN003, ANN202
|
||||
skip = kwargs.get("skip_kompress", False)
|
||||
frozen = kwargs.get("frozen_message_count", 0) or 0
|
||||
result = real_apply(self, messages, model, **kwargs)
|
||||
if skip:
|
||||
return result
|
||||
out = []
|
||||
for i, message in enumerate(result.messages):
|
||||
content = message.get("content")
|
||||
if i >= frozen and isinstance(content, list):
|
||||
blocks = []
|
||||
for block in content:
|
||||
text = block.get("content") if isinstance(block, dict) else None
|
||||
if block.get("type") == "tool_result" and isinstance(text, str):
|
||||
if not text.endswith(" [kompressed]"):
|
||||
block = {**block, "content": text[: len(text) // 2] + " [kompressed]"}
|
||||
blocks.append(block)
|
||||
message = {**message, "content": blocks}
|
||||
out.append(message)
|
||||
result.messages = out
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(TransformPipeline, "apply", apply)
|
||||
|
||||
|
||||
def _read_turn(i: int) -> list[dict[str, Any]]:
|
||||
tid = f"toolu_r{i:02d}"
|
||||
body = "\n".join(f"{n:5d} line {n} of file {i}: some code here" for n in range(300))
|
||||
return [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "tool_use", "id": tid, "name": "Read", "input": {"file_path": f"/f{i}.py"}}
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": tid, "content": body}]},
|
||||
]
|
||||
|
||||
|
||||
def _reads(turns: int) -> list[dict[str, Any]]:
|
||||
"""One Read per turn; Claude Code's breakpoint rides the newest block."""
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "user", "content": [{"type": "text", "text": "Review the change."}]}
|
||||
]
|
||||
for i in range(turns):
|
||||
messages += _read_turn(i)
|
||||
messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}
|
||||
return messages
|
||||
|
||||
|
||||
# (history builder, turns before the restart, proxy config, fake Kompress)
|
||||
_SCENARIOS = {
|
||||
# Turns 1-2, restart, turn 3: Kompressed messages come back fast-passed.
|
||||
"kompress": (_main, 2, {}, True),
|
||||
# Reads held while active then cached verbatim stay verbatim; a fresh
|
||||
# process with an empty CompressionCache freezes nothing and matures them.
|
||||
"read-maturation": (
|
||||
_reads,
|
||||
5,
|
||||
{
|
||||
"read_maturation": True,
|
||||
"read_maturation_quiesce_turns": 2,
|
||||
"read_maturation_min_size_bytes": 100,
|
||||
},
|
||||
False,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _restart_between_turns(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int, dict[str, str]]:
|
||||
"""Some turns in one proxy, the next turn in a fresh one, one warm
|
||||
provider cache. Returns (last forwarded before, first forwarded after,
|
||||
its read hit, the original behind each retrieval reference forwarded
|
||||
before the restart)."""
|
||||
build, turns, config, kompress = _SCENARIOS[scenario]
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
# As the desktop app runs it: a cold start forwards the fast pass.
|
||||
monkeypatch.setenv("HEADROOM_BACKGROUND_COMPRESSION", "1")
|
||||
monkeypatch.setenv("HEADROOM_BACKGROUND_COMPRESSION_MIN_TOKENS", "1")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
if kompress:
|
||||
_with_fake_kompress(monkeypatch)
|
||||
cache = _PromptCache()
|
||||
reset_compression_store()
|
||||
before = _drive([build(n) for n in range(1, turns + 1)], cache, **config)
|
||||
store = get_compression_store()
|
||||
originals = {h: store.retrieve(h).original_content for h in _ccr_hashes(before[-1][0])} # type: ignore[union-attr]
|
||||
reset_compression_store() # nothing in process memory survives a restart
|
||||
(after,) = _drive([build(turns + 1)], cache, **config)
|
||||
return before[-1][0], after[0], after[1], originals
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scenario", list(_SCENARIOS))
|
||||
def test_first_request_after_a_restart_replays_the_cached_prefix(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
|
||||
) -> None:
|
||||
state = tmp_path / "prefix-state.json"
|
||||
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(state))
|
||||
|
||||
previous, current, hit, _ = _restart_between_turns(monkeypatch, tmp_path, scenario)
|
||||
|
||||
assert _wire(current[: len(previous)]) == _wire(previous)
|
||||
assert hit >= len(previous) - 1, f"read only {hit + 1}/{len(previous)} cached messages"
|
||||
|
||||
|
||||
def test_restored_retrieval_references_resolve_to_their_originals(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Review of #4299: matching bytes are not enough. With the in-process CCR
|
||||
backend a restart emptied the store, so a matured Read the restored
|
||||
tracker kept replaying pointed at a hash that no longer resolved."""
|
||||
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
|
||||
|
||||
previous, current, _, originals = _restart_between_turns(
|
||||
monkeypatch, tmp_path, "read-maturation"
|
||||
)
|
||||
|
||||
emitted = _ccr_hashes(current)
|
||||
assert emitted & originals.keys(), "no pre-restart reference was replayed"
|
||||
store = get_compression_store()
|
||||
for h in emitted:
|
||||
entry = store.retrieve(h)
|
||||
assert entry is not None, f"hash={h} does not resolve after the restart"
|
||||
if h in originals:
|
||||
assert entry.original_content == originals[h]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scenario", list(_SCENARIOS))
|
||||
def test_without_the_state_file_a_restart_rewrites_the_prefix(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, scenario: str
|
||||
) -> None:
|
||||
"""Default behavior is unchanged, and it is the bug the opt-in fixes, so
|
||||
the test above is not vacuous."""
|
||||
monkeypatch.delenv(PREFIX_STATE_FILE_ENV, raising=False)
|
||||
|
||||
previous, current, hit, _ = _restart_between_turns(monkeypatch, tmp_path, scenario)
|
||||
|
||||
assert _wire(current[: len(previous)]) != _wire(previous)
|
||||
assert hit < len(previous) - 1
|
||||
assert list(tmp_path.iterdir()) == [], "nothing may be written without the opt-in"
|
||||
|
||||
|
||||
def _two_conversations() -> SessionTrackerStore:
|
||||
store = SessionTrackerStore()
|
||||
contextvars.Context().run(_send, store, _strip(_main(3)))
|
||||
contextvars.Context().run(_send, store, [{"role": "user", "content": "unrelated"}])
|
||||
return store
|
||||
|
||||
|
||||
def test_restored_lineages_resume_and_new_ones_get_fresh_keys(tmp_path: Path) -> None:
|
||||
"""Store level: a fork answered last still leaves the main turn resumable,
|
||||
and a new conversation never lands on a restored lineage's key."""
|
||||
store = _two_conversations()
|
||||
base = _strip(_main(3))
|
||||
contextvars.Context().run(_send, store, _strip(_summary_fork(3)))
|
||||
path = str(tmp_path / "state.json")
|
||||
assert store.save_state(path) == 2
|
||||
|
||||
restored = SessionTrackerStore()
|
||||
assert restored.load_state(path) == 2
|
||||
main = contextvars.Context().run(
|
||||
restored.resolve_tracker, "sid", "anthropic", messages=_strip(_main(4))
|
||||
)
|
||||
new = contextvars.Context().run(
|
||||
restored.resolve_tracker, "sid", "anthropic", messages=[{"role": "user", "content": "new"}]
|
||||
)
|
||||
|
||||
assert main.get_frozen_message_count() == len(base)
|
||||
assert main.get_last_original_messages() == base
|
||||
assert restored.active_sessions == 3
|
||||
assert new is not main and new.get_frozen_message_count() == 0
|
||||
|
||||
|
||||
def test_expired_trackers_are_not_restored(tmp_path: Path) -> None:
|
||||
path = tmp_path / "state.json"
|
||||
_two_conversations().save_state(str(path))
|
||||
data = json.loads(path.read_text())
|
||||
data["trackers"][0]["meta"]["last_activity"] -= 3600 # past the 600s retention
|
||||
path.write_text(json.dumps(data))
|
||||
|
||||
assert SessionTrackerStore().load_state(str(path)) == 1
|
||||
|
||||
|
||||
def _bad_ref(path: Path) -> None:
|
||||
data = json.loads(path.read_text())
|
||||
data["trackers"][0]["meta"]["state"][3] = [-1]
|
||||
path.write_text(json.dumps(data))
|
||||
|
||||
|
||||
def _bad_maturation(path: Path) -> None:
|
||||
data = json.loads(path.read_text())
|
||||
data["trackers"][-1]["meta"]["maturation"] = {"matured": [1], "booked": [], "deltas": {}}
|
||||
path.write_text(json.dumps(data))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"corrupt",
|
||||
[
|
||||
lambda p: p.write_text('{"version": 1, "trackers": [{"meta"'),
|
||||
lambda p: p.write_text('{"version": 99, "trackers": []}'),
|
||||
lambda p: p.write_text("[1, 2]"),
|
||||
_bad_ref,
|
||||
_bad_maturation,
|
||||
],
|
||||
ids=["truncated", "version", "shape", "bad-ref", "bad-maturation"],
|
||||
)
|
||||
def test_a_bad_state_file_starts_empty(tmp_path: Path, caplog, corrupt) -> None: # noqa: ANN001
|
||||
path = tmp_path / "state.json"
|
||||
_two_conversations().save_state(str(path))
|
||||
corrupt(path)
|
||||
|
||||
store = SessionTrackerStore()
|
||||
assert store.load_state(str(path)) == 0
|
||||
assert store.active_sessions == 0
|
||||
assert "ignoring prefix state" in caplog.text
|
||||
|
||||
|
||||
def test_state_file_is_private_and_removed_once_loaded(tmp_path: Path) -> None:
|
||||
path = tmp_path / "state.json"
|
||||
path.write_text("stale")
|
||||
path.chmod(0o644)
|
||||
|
||||
assert _two_conversations().save_state(str(path)) == 2
|
||||
if os.name == "posix":
|
||||
assert stat.S_IMODE(path.stat().st_mode) == 0o600
|
||||
assert SessionTrackerStore().load_state(str(path)) == 2
|
||||
assert not path.exists()
|
||||
assert SessionTrackerStore().load_state(str(path)) == 0 # missing file: cold start
|
||||
|
||||
|
||||
def test_save_keeps_the_newest_trackers_within_the_cap(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
store = _two_conversations()
|
||||
newest = max(store._trackers.values(), key=lambda t: t._last_activity)
|
||||
monkeypatch.setattr(prefix_tracker, "_PREFIX_STATE_MAX_TRACKERS", 1)
|
||||
path = str(tmp_path / "state.json")
|
||||
assert store.save_state(path) == 1
|
||||
|
||||
restored = SessionTrackerStore()
|
||||
restored.load_state(path)
|
||||
(kept,) = restored._trackers.values()
|
||||
assert kept.get_last_original_messages() == newest.get_last_original_messages()
|
||||
|
||||
monkeypatch.setattr(prefix_tracker, "_PREFIX_STATE_MAX_BYTES", 10)
|
||||
assert store.save_state(path) == 0
|
||||
|
||||
|
||||
def test_stateless_mode_writes_nothing(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
monkeypatch.setenv("HEADROOM_STATELESS", "1")
|
||||
path = tmp_path / "state.json"
|
||||
assert _two_conversations().save_state(str(path)) == 0
|
||||
assert not path.exists()
|
||||
|
||||
|
||||
def test_a_stateless_proxy_leaves_a_saved_snapshot_alone(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Loading deletes the file, so a stateless start must not load it: the
|
||||
next normal start resumes those conversations."""
|
||||
path = tmp_path / "state.json"
|
||||
assert _two_conversations().save_state(str(path)) == 2
|
||||
saved = path.read_text()
|
||||
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(path))
|
||||
monkeypatch.setenv("HEADROOM_STATELESS", "1")
|
||||
|
||||
_drive([]) # one stateless proxy start and graceful shutdown
|
||||
|
||||
assert path.read_text() == saved
|
||||
monkeypatch.delenv("HEADROOM_STATELESS")
|
||||
assert SessionTrackerStore().load_state(str(path)) == 2
|
||||
|
||||
|
||||
def test_a_failed_save_keeps_the_previous_snapshot(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
path = tmp_path / "state.json"
|
||||
assert _two_conversations().save_state(str(path)) == 2
|
||||
saved = path.read_text()
|
||||
|
||||
def disk_full(*args: Any, **kwargs: Any) -> None:
|
||||
raise OSError(errno.ENOSPC, "No space left on device")
|
||||
|
||||
monkeypatch.setattr("headroom.fsutil.write_text", disk_full)
|
||||
assert _two_conversations().save_state(str(path)) == 0
|
||||
|
||||
assert path.read_text() == saved
|
||||
|
||||
|
||||
def test_saved_ccr_entries_resolve_after_a_restart(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Store level, compressor markers in the replayed prefix: both hash
|
||||
formats resolve to the same original with the same expiry after the
|
||||
in-process CCR store is emptied, and an entry that expired does not."""
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
reset_compression_store()
|
||||
ccr = get_compression_store()
|
||||
rows = ccr.store(json.dumps(list(range(400))), "[0, 1]", ttl=900)
|
||||
blob = ccr.store("x" * 5000, "", ttl=900, explicit_hash="abcdef123456")
|
||||
gone = ccr.store("expired original", "", ttl=900)
|
||||
ccr._backend.get(gone).created_at -= 1000 # type: ignore[union-attr]
|
||||
expires = {h: ccr.get_entry_status(h)["expires_at"] for h in (rows, blob)}
|
||||
text = (
|
||||
f"[400 items compressed to 2. Retrieve more: hash={rows}] "
|
||||
f"<<ccr:{blob},base64,4.9KB>> [stale. Retrieve original: hash={gone}]"
|
||||
)
|
||||
store = SessionTrackerStore()
|
||||
contextvars.Context().run(_send, store, [{"role": "user", "content": text}])
|
||||
path = str(tmp_path / "state.json")
|
||||
assert store.save_state(path) == 1
|
||||
reset_compression_store()
|
||||
|
||||
assert SessionTrackerStore().load_state(path) == 1
|
||||
|
||||
ccr = get_compression_store()
|
||||
assert ccr.retrieve(rows).original_content == json.dumps(list(range(400))) # type: ignore[union-attr]
|
||||
assert ccr.retrieve(blob).original_content == "x" * 5000 # type: ignore[union-attr]
|
||||
assert ccr.retrieve(gone) is None
|
||||
for h, at in expires.items():
|
||||
assert abs(ccr.get_entry_status(h)["expires_at"] - at) < 2
|
||||
reset_compression_store()
|
||||
|
||||
|
||||
def _chat_turns(n: int) -> list[dict[str, Any]]:
|
||||
"""An OpenAI chat agent: one big tool result, then ``n - 1`` follow-ups."""
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "user", "content": "run the build and show me the log"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "bash", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "call_1", "content": "build log line OK\n" * 200},
|
||||
]
|
||||
for i in range(1, n):
|
||||
messages += [
|
||||
{"role": "assistant", "content": f"answer {i}"},
|
||||
{"role": "user", "content": "and now?"},
|
||||
]
|
||||
return messages
|
||||
|
||||
|
||||
def _relevance_pipeline(**kwargs: Any) -> TransformResult:
|
||||
"""Query-dependent compression, like SmartCrusher's relevance scoring: the
|
||||
same tool result compresses differently under a different user query."""
|
||||
frozen = kwargs.get("frozen_message_count") or 0
|
||||
query = next(m["content"] for m in reversed(kwargs["messages"]) if m["role"] == "user")
|
||||
messages = [
|
||||
{**m, "content": f"[log: kept lines relevant to {query!r}]"}
|
||||
if i >= frozen and m.get("role") == "tool"
|
||||
else m
|
||||
for i, m in enumerate(kwargs["messages"])
|
||||
]
|
||||
return TransformResult(
|
||||
messages=messages, tokens_before=2000, tokens_after=40, transforms_applied=["relevance"]
|
||||
)
|
||||
|
||||
|
||||
def _drive_openai_chat(requests: list[list[dict[str, Any]]]) -> list[list[dict[str, Any]]]:
|
||||
"""Each history through the real OpenAI chat handler in token mode; one
|
||||
app per call, so a second call is a proxy restart."""
|
||||
app = create_app(
|
||||
ProxyConfig(
|
||||
optimize=True,
|
||||
mode="token",
|
||||
disable_kompress=True,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
)
|
||||
forwarded: list[list[dict[str, Any]]] = []
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.openai_pipeline.apply = _relevance_pipeline
|
||||
|
||||
async def _upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
|
||||
forwarded.append(json.loads(json.dumps(body["messages"])))
|
||||
# Automatic prefix caching reports nothing cached on a
|
||||
# conversation's first request, so there is no confirmed floor.
|
||||
usage = {"prompt_tokens": 600, "completion_tokens": 3, "total_tokens": 603}
|
||||
usage["prompt_tokens_details"] = {"cached_tokens": 0}
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"id": "chatcmpl_x",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": "ok"},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": usage,
|
||||
},
|
||||
)
|
||||
|
||||
proxy._retry_request = _upstream
|
||||
for messages in requests:
|
||||
response = client.post(
|
||||
"/v1/chat/completions",
|
||||
headers={"authorization": "Bearer test-key"},
|
||||
json={"model": "gpt-4o-mini", "messages": messages},
|
||||
)
|
||||
assert response.status_code == 200, response.text[:300]
|
||||
return forwarded
|
||||
|
||||
|
||||
def test_openai_token_mode_replays_its_prefix_after_a_restart(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Review of #4299: only the Anthropic handler seeded its CompressionCache
|
||||
from a restored tracker. The OpenAI token path froze nothing, recompressed
|
||||
the tool result under the new query, and with no provider-confirmed floor
|
||||
the overlay declined the larger previous form, so the prefix changed."""
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
|
||||
reset_compression_store()
|
||||
|
||||
(previous,) = _drive_openai_chat([_chat_turns(1)])
|
||||
(current,) = _drive_openai_chat([_chat_turns(2)])
|
||||
|
||||
assert previous[2]["content"] != _chat_turns(1)[2]["content"], "nothing was compressed"
|
||||
assert current[: len(previous)] == previous
|
||||
|
||||
|
||||
def test_read_maturation_state_survives_a_restart(tmp_path: Path) -> None:
|
||||
"""Matured Reads stay booked once: without this a restart books their
|
||||
removal again on every later request of the conversation."""
|
||||
from headroom.config import ReadMaturationConfig
|
||||
from headroom.transforms.read_maturation import MaturedRead, ReadMaturationManager
|
||||
|
||||
store = _two_conversations()
|
||||
manager = ReadMaturationManager(ReadMaturationConfig(enabled=True))
|
||||
manager._matured["toolu_r00"] = MaturedRead(marker="[Read of /f0.py ...]", ccr_hash="abc")
|
||||
manager._booked.add("toolu_r00")
|
||||
manager._replay_token_deltas["toolu_r00"] = 640
|
||||
newest = max(store._trackers.values(), key=lambda t: t._last_activity)
|
||||
newest.read_maturation_manager = manager
|
||||
path = str(tmp_path / "state.json")
|
||||
store.save_state(path)
|
||||
|
||||
restored = SessionTrackerStore()
|
||||
restored.load_state(path)
|
||||
states = [t.restored_maturation_state for t in restored._trackers.values()]
|
||||
assert all(t.restored_from_disk for t in restored._trackers.values())
|
||||
(state,) = [s for s in states if s is not None]
|
||||
adopted = ReadMaturationManager(ReadMaturationConfig(enabled=True))
|
||||
adopted.restore_state(state)
|
||||
|
||||
assert adopted._matured == manager._matured
|
||||
assert adopted._booked == manager._booked
|
||||
assert adopted._replay_token_deltas == manager._replay_token_deltas
|
||||
|
||||
|
||||
def test_compress_sidecar_session_replays_its_prefix_after_a_restart(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""The same seed on the session-aware /v1/compress path (sidecar gateways),
|
||||
which replays with the size bound alone (no provider-confirmed floor)."""
|
||||
from tests.test_compress_session_mode import _compress, _make_client
|
||||
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory")
|
||||
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "prefix-state.json"))
|
||||
reset_compression_store()
|
||||
forwarded = []
|
||||
for history in (_chat_turns(1), _chat_turns(2)): # one proxy each: a restart between
|
||||
with _make_client() as client:
|
||||
client.app.state.proxy._no_ccr_pipeline = lambda: SimpleNamespace(
|
||||
apply=_relevance_pipeline
|
||||
)
|
||||
forwarded.append(_compress(client, history, session_id="conv-r")["messages"])
|
||||
previous, current = forwarded
|
||||
|
||||
assert previous[2]["content"] != _chat_turns(1)[2]["content"], "nothing was compressed"
|
||||
assert current[: len(previous)] == previous
|
||||
@@ -23,7 +23,7 @@ resolution-markers = [
|
||||
|
||||
[manifest]
|
||||
constraints = [
|
||||
{ name = "aiohttp", specifier = ">=3.14.3" },
|
||||
{ name = "aiohttp", specifier = ">=3.14.4" },
|
||||
{ name = "cryptography", specifier = ">=50.0.0" },
|
||||
{ name = "gitpython", specifier = ">=3.1.58" },
|
||||
{ name = "langsmith", specifier = ">=0.9.0" },
|
||||
|
||||
Reference in New Issue
Block a user