mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a616232d75 |
feat(compress): share /v1/compress sessions across replicas via Redis (#4369)
Stacked on #4368 (Redis CCR store); merge that first. ## Why A session-aware `/v1/compress` turn must return the prefix it returned last turn byte for byte, or the provider's prompt cache misses. That state (the session's prefix tracker) lived only in the memory of the replica that served the turn, so a gateway with several replicas had to pin sessions (`session_affinity`). Kong's AI Gateway runs several instances without sticky sessions. ## Change With `HEADROOM_COMPRESS_SESSION_STORE=redis://…` (`headroom/cache/shared_sessions.py`), every replica keeps each session in Redis too: - **Messages, stored once by digest** (`…:msgs`). The original and returned transcripts mostly share messages, and each turn mostly repeats the last one, so a turn writes only what Redis lacks: the messages it added, plus any whose bytes changed (a moved cache marker, an edit, a compaction). Unreferenced messages are pruned. - **Metadata** (`…:meta`): the tracker's counters, its transcripts as digest lists, and a random write token. - **Before a turn**, the replica reads the token. If another replica wrote since, it reads the metadata and only the messages it lacks, each checked against its digest. The turn then seeds its compression cache from that tracker, as after a restart. - **After the turn**, it writes the new messages, then the metadata via a Lua compare-and-set on the token. - A random token, not a counter: a lost key or a failover can't leave a replica trusting stale state. - The script also detects a messages hash that `maxmemory` evicted on its own; the writer then stores the session whole. - A replica that finds corrupt or evicted state claims the token and rewrites the session whole. - **Usage relay:** fetches under the session's turn lock, and installs only if no local turn installed a tracker meanwhile. - **Failures:** a Redis error or corrupt value never fails a turn. The replica serves from its own memory, as without the store. Clock skew between replicas is tolerated. - **CCR:** the setting also selects #4368's Redis CCR store unless `HEADROOM_CCR_BACKEND` names another, so markers in a replayed transcript resolve on any replica. - **`SessionTrackerStore`** gains `encode_shared` / `decode_shared` (the restart file's per-tracker encoding, split into metadata and message table) and `install`. ## Results **Real run:** 3 proxy processes on one Redis-protocol server (fakeredis TCP server; no Redis on the test machine), a 20-turn `mode: "ccr"` session sent round-robin: - every turn's returned prefix was byte-identical to the previous turn's; - the CCR marker resolved via `/v1/retrieve` on all three replicas; - Redis received **378 KB** over the 20 turns. Writing the whole transcript every turn would have been about 4.9 MB, growing with the square of the session length. **Tests:** `tests/test_compress_session_shared.py`, each failure mode mutation-checked: - a stale replica reloads newer state; - without the store, the other replica starts cold; - a turn writes only the messages Redis lacks; - a fully lost key is rewritten; - a messages hash evicted on its own is rewritten; - corrupt state is repaired by another replica; - unreferenced messages are pruned; - a writer whose clock runs ahead is trusted; - the usage relay reaches and publishes a session served elsewhere; - an outage or corrupt values never fail a turn; - the CCR store follows the session store; - a URL that isn't Redis stops startup. 818 proxy, gateway, session and restart tests pass. ruff and mypy are clean. ## Notes - Redis holds transcripts and tool output verbatim. Use `rediss://` with authentication on a private network; plain `redis://` to a remote host logs a warning. - Re-drive turns still need affinity: they park a live in-process task between two HTTP calls. - About 540 added lines (excluding `uv.lock`), around 200 of them tests. The `prefix_tracker` refactor can split out if reviewers prefer. ## Real Behavior Proof - **Environment:** macOS 26.7.1 (arm64), Python 3.12.13, redis-py 8.1.0. Redis is a fakeredis 2.40.0 `TcpFakeServer`, which speaks the Redis protocol over TCP; no Redis server was on the test machine. Docker was not available. - **Configuration:** every replica has `HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0` (the CCR store follows it) and its own `HEADROOM_WORKSPACE_DIR`. - **Steps:** 1. Start the fake server: `python -c "from fakeredis import TcpFakeServer; TcpFakeServer(('127.0.0.1', 6391), server_type='redis').serve_forever()"` 2. Start three proxies, one each on ports 8901, 8902 and 8903: `HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0 HEADROOM_WORKSPACE_DIR=<own dir> python -m headroom.cli proxy --port <port>` 3. Drive one session for 20 turns, round-robin across the three: `POST /v1/compress` with `{"messages": …, "session_id": "kong-e2e", "mode": "ccr"}`. The transcript is the 25 KB tool history from `tests/test_compress_session_mode.py`. Every third turn adds a 120-row grep result; the other turns add a user line. 4. After each turn, check: - the returned messages start with the previous turn's returned messages, byte for byte; - how many bytes the turn wrote to the `:msgs` hash. 5. Fetch each `<<ccr:hash>>` marker via `GET /v1/retrieve/{hash}` on all three proxies. - **Result** (after the review fixes, at 95b2d93a): - every turn's prefix is byte-identical to the previous turn's; - Redis received 378 KB in total (63 KB at most in one turn), against about 4.9 MB if the whole transcript were rewritten each turn; - the CCR marker resolved on all three replicas. - **Not tested:** - a real Redis or Redis Cluster (the Lua script and hash-tagged keys are written for it); - TLS (`rediss://`) and AUTH; - Kong's AI Gateway in front of the replicas; - Windows; - real `maxmemory` eviction (simulated in tests by deleting keys); - sustained load. ## Dependencies and scope - **No new runtime dependency:** this uses #4368's `[redis]` extra. - **Dev only:** the `fakeredis` requirement becomes `fakeredis[lua]`, which adds `lupa` so the tests run the compare-and-set Lua script exactly as Redis would. - **Why this feature:** Kong's AI Gateway runs several Headroom instances with no sticky sessions and asked for session state in Redis. It is opt-in: nothing changes unless `HEADROOM_COMPRESS_SESSION_STORE` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4369" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |