fix(cache): save prefix-tracker state when the exit signal arrives (#4715)

## Description

Follow-up to #4299. With `HEADROOM_PREFIX_STATE_FILE` set, the proxy
saves its prefix trackers in `HeadroomProxy.shutdown()`, the last
lifespan step. uvicorn runs the lifespan shutdown only after it has
waited up to `timeout_graceful_shutdown=10` for in-flight requests. So a
supervisor that SIGKILLs sooner loses the save whenever a request is
open at the stop, and the next start re-writes every warm conversation's
cache. Extra Headroom stops the proxy with SIGTERM and SIGKILLs 2s
later; a systemd `TimeoutStopSec` or a container stop timeout under 10s
does the same.

This saves the trackers as the signal arrives. Measured with one request
in flight: main wrote the state file 10.2s after SIGTERM, this branch
0.02s after it.

No existing issue; found while turning the state file on in Extra
Headroom.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_save_prefix_state_on_exit_signal` (`headroom/proxy/server.py`),
called from `HeadroomProxy.startup()` right after `load_state`, only
when the state file is set. It wraps the SIGINT, SIGTERM and SIGBREAK
handlers already installed (uvicorn's: the lifespan runs inside its
`capture_signals`). The wrapper schedules one `save_state` on the event
loop with `call_soon_threadsafe`, then calls the wrapped handler, so
uvicorn's shutdown is unchanged.
- The save never runs inside the signal handler, so it cannot read a
tracker that a paused coroutine was midway through updating. A signal
with no Python handler (`SIG_DFL`) is left alone, and off the main
thread (TestClient's lifespan) nothing is installed.
- `shutdown()` hands the signals back right after its own save: the
helper returns an undo that restores each handler that is still its
wrapper (one another owner installed since is left alone) and turns the
wrapper into a pass-through, so a lifespan run without uvicorn (an
embedding app, tests) never leaves a wrapper bound to a closed loop.
- The shutdown save stays. It runs later and overwrites the snapshot
with the newer state when the process lives that long; `write_text` is
atomic, so a SIGKILL during that second write leaves the first snapshot.
- Docs: the `HEADROOM_PREFIX_STATE_FILE` row in
`docs/content/docs/configuration.mdx` says when it is saved.
- Tests (`tests/test_prefix_tracker_restart.py`): the signal-time save
lands on the loop after uvicorn's handler got the signal, and only once
for two signals; `SIG_DFL` is left alone; no handler is installed off
the main thread; after two `HeadroomProxy.startup()`/`shutdown()`
lifespans on separate loops SIGTERM reaches the app's own handler; a
wrapper left behind by overlapping lifespans passes through after its
loop closed; undo leaves a handler installed later alone; undo off the
main thread only disarms the wrapper. Those four fail on 2fd4854e.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_prefix_tracker_restart.py -q
28 passed in 5.62s
$ uv run --frozen --extra dev pytest tests -q -k "shutdown or lifespan or signal or startup or restart"
292 passed, 14 skipped, 19230 deselected, 2 warnings in 28.63s
$ uv run --frozen --extra dev ruff check .
All checks passed!
$ uv run --frozen --extra dev ruff format --check .
1921 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/server.py --ignore-missing-imports
headroom/memory/adapters/embedders.py:389: error: Unsupported operand types for - ("None" and "int")  [operator]
(no error in server.py; embedders.py is untouched)
$ uv run --frozen --extra dev mypy headroom --ignore-missing-imports
Found 3 errors in 2 files (checked 583 source files)
  headroom/memory/adapters/embedders.py:389, headroom/cache/prefix_tracker.py:1397-1398 (files this PR does not touch)
```

## Real Behavior Proof

- Environment: macOS 27 (arm64), this branch at 4783339b against main at
bb8d18a5, the real proxy process (`python -m headroom.cli proxy --port
6799 --no-http2 --no-rate-limit`, `HEADROOM_MODE=token`,
`HEADROOM_ROLLOUT_CHANNEL=beta`, clean `HOME`,
`HEADROOM_PREFIX_STATE_FILE` set) in front of a local fake Anthropic
upstream that reports a cache read.
- Exact command / steps: one `/v1/messages` turn so the proxy holds a
tracker, then a second request that the fake upstream holds for 60s,
then SIGTERM to the proxy's process group 2s later; record when the
state file appears and when the process exits. Repeated with no request
in flight.
- Observed result: with a request in flight, main wrote the state file
10.2s after SIGTERM and this branch 0.02s after it; both exited after
uvicorn's 10s drain, which is unchanged.

  | Stop | State file written | Process exited |
  |---|---|---|
  | main, request in flight | 10.22s | 10.24s |
  | branch, request in flight | 0.02s | 10.24s |
  | branch, idle | 0.03s | 0.14s |

The branch logged `SessionTrackerStore: saved 2 of 2 prefix trackers`
twice in the in-flight run: once on the signal, once from `shutdown()`.
- Not tested: Windows (SIGBREAK, and SIGTERM delivery there), Linux,
uvicorn before 0.29 (handlers installed with `loop.add_signal_handler`;
the lock pins 0.40.0), and a real Claude Code session across the restart
(#4299's proof covers the restore itself).

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this only changes when the opt-in
`HEADROOM_PREFIX_STATE_FILE` save happens.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No; with the variable unset no
handler is installed and nothing is written.
- Kill switch / disable path: Unset `HEADROOM_PREFIX_STATE_FILE`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR; the shutdown save, the file format and
the restore path are unchanged.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no UI change.

## Additional Notes

- `mypy headroom` is unchecked because of the 3 errors above, which are
in files this PR does not touch; `mypy` on `headroom/proxy/server.py`
reports no error in that file.
- Windows: a supervisor that force-kills the process tree (no console
signal) still loses the save; that needs a periodic save, which this PR
does not attempt.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-11-09 12:34:42 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent c6d708144c
commit 3ba422361e
3 changed files with 299 additions and 1 deletions
+1 -1
View File
@@ -448,7 +448,7 @@ headroom proxy --learn --min-evidence 3
| `HEADROOM_THINKING_COMPACT` | Compact plain-text reasoning that models re-send every turn (Kimi/GLM/DeepSeek `reasoning_content` / inline `<think>`): Kompress it on warm turns, drop it on cold turns. No-op for Claude/Codex/OpenAI (encrypted reasoning). See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
| `HEADROOM_THINKING_COMPACT_KEEP_LAST` | Most-recent assistant turns whose reasoning is left intact (the active reasoning the model still uses). Only applies with `HEADROOM_THINKING_COMPACT`. | `1` |
| `HEADROOM_COLD_RECOMPACT` | On a confirmed-cold turn (idle past the real cache TTL), recompact the whole prefix — cross-turn dedupe + superseded-read drop + lossless folds — instead of forwarding a byte-identical prefix to a dead cache. Warm turns unchanged. Pair with `HEADROOM_DEDUPE`. See [Cold-prefix hook](#cold-prefix-hook--reasoning-compaction). | `off` |
| `HEADROOM_PREFIX_STATE_FILE` | Path where the proxy saves its prefix-cache trackers on a graceful shutdown and restores them on the next start (then deletes the file), so warm conversations replay the exact prefix the provider cached instead of re-writing it. The file holds conversation content: it is written `0600`, capped at the 256 newest conversations and 64 MB, and only trackers still inside their cache lifetime are restored, with the retrieval (CCR) entries their markers point at, so those markers resolve even with `HEADROOM_CCR_BACKEND=memory`. Not supported with multiple workers; ignored in stateless mode (neither read nor written). | -- |
| `HEADROOM_PREFIX_STATE_FILE` | Path where the proxy saves its prefix-cache trackers as soon as it gets SIGINT or SIGTERM (so a stop that kills it before in-flight requests drain still keeps them), saves them again at the end of a graceful shutdown, and restores them on the next start (then deletes the file), so warm conversations replay the exact prefix the provider cached instead of re-writing it. The file holds conversation content: it is written `0600`, capped at the 256 newest conversations and 64 MB, and only trackers still inside their cache lifetime are restored, with the retrieval (CCR) entries their markers point at, so those markers resolve even with `HEADROOM_CCR_BACKEND=memory`. Not supported with multiple workers; ignored in stateless mode (neither read nor written). | -- |
| `HEADROOM_DEDUPE` | Whole-conversation verbatim cross-turn dedup in the router (cache-safe, information-preserving via retrieval markers). Superseded-read drop + lossless folds run without it; this adds verbatim dedup. | `off` |
| `HEADROOM_CACHE_TTL_LEARN` | Append per-turn cache-outcome observations (provider, model, idle, hit/miss) to `cache_ttl_observations.jsonl` for the offline `headroom-cache-ttl` learner. Observation-only (no request-behavior change); respects `HEADROOM_STATELESS`; the log is size-bounded. | `off` |
| `HEADROOM_KOMPRESS_ENDPOINT` / `HEADROOM_KOMPRESS_ENDPOINT_TOKEN` | Offload ML compression (Kompress) to a remote endpoint instead of the local ONNX model — used by reasoning compaction and the router when set. | -- |
+71
View File
@@ -34,6 +34,7 @@ import json
import logging
import math
import os
import signal
import sys
import threading
import time
@@ -600,6 +601,69 @@ class CompressionQuarantinedError(asyncio.TimeoutError):
_MULTI_WORKER_CONFIG_ENV = "HEADROOM_PROXY_CONFIG_JSON"
def _save_prefix_state_on_exit_signal(store: Any, path: str) -> Callable[[], None]:
"""Save prefix-tracker state as soon as an exit signal arrives.
``HeadroomProxy.shutdown`` saves it too, but uvicorn runs the lifespan
shutdown only after waiting up to ``timeout_graceful_shutdown`` for
in-flight requests, so a supervisor that SIGKILLs sooner (a desktop app's
short stop, systemd's ``TimeoutStopSec``) loses that save whenever a request
is open. This wraps the handler uvicorn installed (the lifespan runs inside
its ``capture_signals``), so uvicorn's shutdown is unchanged and the later
save still writes the newer state. The save runs on the event loop, never
inside the signal handler, so it cannot read a tracker that a paused
coroutine was midway through updating.
Returns the undo, for the lifespan's shutdown: it restores each handler
that is still this wrapper (one another owner installed since is left
alone) and turns the wrapper into a pass-through, so a wrapper that could
not be removed never schedules onto this lifespan's closed loop.
"""
if threading.current_thread() is not threading.main_thread():
return lambda: None # signals are only delivered to the main thread
loop = asyncio.get_running_loop()
active = True
saved = False
def save() -> None:
nonlocal saved
if not saved:
saved = True
store.save_state(path)
def chained(previous: Callable[[int, Any], Any]) -> Callable[[int, Any], None]:
def handler(signum: int, frame: Any) -> None:
if active:
with contextlib.suppress(RuntimeError): # loop closed without a shutdown
loop.call_soon_threadsafe(save)
previous(signum, frame)
return handler
installed: list[tuple[int, Any, Callable[[int, Any], None]]] = []
for sig in (signal.SIGINT, signal.SIGTERM, getattr(signal, "SIGBREAK", None)):
if sig is None:
continue
previous = signal.getsignal(sig)
if not callable(previous):
continue # nobody installed a handler: keep the default exit
handler = chained(previous)
signal.signal(sig, handler)
installed.append((sig, previous, handler))
def undo() -> None:
nonlocal active
active = False
if threading.current_thread() is not threading.main_thread():
return
for sig, previous, handler in installed:
if signal.getsignal(sig) is handler:
signal.signal(sig, previous)
return undo
# Env var that opts out of the Rust core deployment smoke test (Hotfix-A0).
# Default behavior: hard-fail at startup if `headroom._core` is unimportable
# (Finding #2 in HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md — production
@@ -1228,6 +1292,7 @@ class HeadroomProxy(
self.shared_compress_sessions = open_shared_sessions()
# Set by startup() when HEADROOM_PREFIX_STATE_FILE opts in.
self._prefix_state_file: str | None = None
self._undo_exit_signal_save: Callable[[], None] | None = None
# Compression cache store for token mode (session-scoped). The dict
# itself is mutated under `_compression_caches_lock`; the per-session
@@ -2126,6 +2191,9 @@ class HeadroomProxy(
self._prefix_state_file = None
if self._prefix_state_file:
self.session_tracker_store.load_state(self._prefix_state_file)
self._undo_exit_signal_save = _save_prefix_state_on_exit_signal(
self.session_tracker_store, self._prefix_state_file
)
self.pipeline_extensions.emit(
PipelineStage.PRE_START,
operation="proxy.startup",
@@ -2476,6 +2544,9 @@ class HeadroomProxy(
# First, before any await can hit the shutdown step timeout.
if self._prefix_state_file:
self.session_tracker_store.save_state(self._prefix_state_file)
if self._undo_exit_signal_save is not None:
self._undo_exit_signal_save()
self._undo_exit_signal_save = None
if self.http_client_h1 and self.http_client_h1 is not self.http_client:
await self.http_client_h1.aclose()
self.http_client_h1 = None
+227
View File
@@ -556,3 +556,230 @@ def test_compress_sidecar_session_replays_its_prefix_after_a_restart(
assert previous[2]["content"] != _chat_turns(1)[2]["content"], "nothing was compressed"
assert current[: len(previous)] == previous
@pytest.fixture
def _restore_exit_handlers():
import signal
sigs = [s for s in (signal.SIGINT, signal.SIGTERM, getattr(signal, "SIGBREAK", None)) if s]
saved = {s: signal.getsignal(s) for s in sigs}
yield
for s, handler in saved.items():
signal.signal(s, handler)
class _RecordingStore:
def __init__(self, calls: list[Any]) -> None:
self.calls = calls
def save_state(self, path: str) -> int:
self.calls.append(("save", path))
return 1
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_an_exit_signal_saves_while_uvicorn_still_drains(tmp_path: Path) -> None:
"""uvicorn waits for in-flight requests before the lifespan shutdown save;
the signal-time save lands first, on the loop, and uvicorn still gets the
signal (so its shutdown is unchanged). A second signal does not save again."""
import asyncio
import signal
from headroom.proxy.server import _save_prefix_state_on_exit_signal
calls: list[Any] = []
path = str(tmp_path / "state.json")
signal.signal(signal.SIGTERM, lambda signum, frame: calls.append(("uvicorn", signum)))
async def serve() -> None:
_save_prefix_state_on_exit_signal(_RecordingStore(calls), path)
signal.raise_signal(signal.SIGTERM)
assert calls == [("uvicorn", signal.SIGTERM)] # never saves inside the handler
await asyncio.sleep(0)
signal.raise_signal(signal.SIGTERM)
await asyncio.sleep(0)
asyncio.run(serve())
assert calls == [("uvicorn", signal.SIGTERM), ("save", path), ("uvicorn", signal.SIGTERM)]
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_the_default_exit_is_left_alone(tmp_path: Path) -> None:
import asyncio
import signal
from headroom.proxy.server import _save_prefix_state_on_exit_signal
signal.signal(signal.SIGTERM, signal.SIG_DFL)
async def serve() -> None:
_save_prefix_state_on_exit_signal(_RecordingStore([]), str(tmp_path / "s.json"))
asyncio.run(serve())
assert signal.getsignal(signal.SIGTERM) is signal.SIG_DFL
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_off_the_main_thread_no_handler_is_installed(tmp_path: Path) -> None:
"""TestClient runs the lifespan in a worker thread, where signal.signal raises."""
import asyncio
import signal
import threading
from headroom.proxy.server import _save_prefix_state_on_exit_signal
before = signal.getsignal(signal.SIGTERM)
errors: list[BaseException] = []
def run() -> None:
async def serve() -> None:
_save_prefix_state_on_exit_signal(_RecordingStore([]), str(tmp_path / "s.json"))
try:
asyncio.run(serve())
except BaseException as exc: # noqa: BLE001
errors.append(exc)
worker = threading.Thread(target=run)
worker.start()
worker.join()
assert errors == []
assert signal.getsignal(signal.SIGTERM) is before
def _lifespan_proxy() -> Any:
from headroom.proxy.server import HeadroomProxy
proxy = HeadroomProxy(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
code_aware_enabled=False,
)
)
proxy.anthropic_pipeline.transforms = []
proxy.openai_pipeline.transforms = []
return proxy
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_repeated_proxy_lifespans_hand_the_signal_back(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""A lifespan run without uvicorn (an embedding app, tests) has no
capture_signals to restore handlers, so shutdown must: after two lifespans
on separate loops, SIGTERM reaches the app's own handler, not a wrapper
bound to a closed loop."""
import asyncio
import signal
monkeypatch.setenv(PREFIX_STATE_FILE_ENV, str(tmp_path / "state.json"))
calls: list[int] = []
def app_handler(signum: int, frame: Any) -> None:
calls.append(signum)
signal.signal(signal.SIGTERM, app_handler)
async def lifespan() -> None:
proxy = _lifespan_proxy()
await proxy.startup()
assert signal.getsignal(signal.SIGTERM) is not app_handler
await proxy.shutdown()
asyncio.run(lifespan())
asyncio.run(lifespan())
assert signal.getsignal(signal.SIGTERM) is app_handler
signal.raise_signal(signal.SIGTERM)
assert calls == [signal.SIGTERM]
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_a_wrapper_left_behind_after_its_loop_closed_passes_through(tmp_path: Path) -> None:
"""Overlapping lifespans: the first one's undo cannot remove its wrapper
while the second wraps it. Once the first is shut down and its loop closed,
a signal still reaches the app handler and saves nothing."""
import asyncio
import signal
from headroom.proxy.server import _save_prefix_state_on_exit_signal
calls: list[Any] = []
signal.signal(signal.SIGTERM, lambda signum, frame: calls.append(("app", signum)))
first = asyncio.new_event_loop()
second = asyncio.new_event_loop()
async def install() -> Any:
return _save_prefix_state_on_exit_signal(_RecordingStore(calls), str(tmp_path / "s.json"))
undo_first = first.run_until_complete(install())
undo_second = second.run_until_complete(install())
undo_first() # the second wrapper is current: leave it in place
first.close()
undo_second() # restores the first wrapper, now inert
second.close()
signal.raise_signal(signal.SIGTERM)
assert calls == [("app", signal.SIGTERM)]
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_undo_leaves_a_handler_installed_later_alone(tmp_path: Path) -> None:
import asyncio
import signal
from headroom.proxy.server import _save_prefix_state_on_exit_signal
signal.signal(signal.SIGTERM, lambda signum, frame: None)
def later(signum: int, frame: Any) -> None:
pass
async def serve() -> None:
undo = _save_prefix_state_on_exit_signal(_RecordingStore([]), str(tmp_path / "s.json"))
signal.signal(signal.SIGTERM, later)
undo()
asyncio.run(serve())
assert signal.getsignal(signal.SIGTERM) is later
@pytest.mark.usefixtures("_restore_exit_handlers")
def test_undo_off_the_main_thread_only_disarms(tmp_path: Path) -> None:
"""signal.signal raises off the main thread, so a shutdown that runs there
leaves the wrapper in place, disarmed: a signal reaches the app, no save."""
import asyncio
import signal
import threading
from headroom.proxy.server import _save_prefix_state_on_exit_signal
calls: list[Any] = []
signal.signal(signal.SIGTERM, lambda signum, frame: calls.append(("app", signum)))
loop = asyncio.new_event_loop()
async def install() -> Any:
return _save_prefix_state_on_exit_signal(_RecordingStore(calls), str(tmp_path / "s.json"))
undo = loop.run_until_complete(install())
wrapper = signal.getsignal(signal.SIGTERM)
errors: list[BaseException] = []
def run() -> None:
try:
undo()
except BaseException as exc: # noqa: BLE001
errors.append(exc)
worker = threading.Thread(target=run)
worker.start()
worker.join()
assert errors == []
assert signal.getsignal(signal.SIGTERM) is wrapper
signal.raise_signal(signal.SIGTERM)
loop.run_until_complete(asyncio.sleep(0))
loop.close()
assert calls == [("app", signal.SIGTERM)]