Commit Graph
1 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 a616232d75 feat(compress): share /v1/compress sessions across replicas via Redis (#4369)
Stacked on #4368 (Redis CCR store); merge that first.

## Why
A session-aware `/v1/compress` turn must return the prefix it returned
last turn byte for byte, or the provider's prompt cache misses. That
state (the session's prefix tracker) lived only in the memory of the
replica that served the turn, so a gateway with several replicas had to
pin sessions (`session_affinity`). Kong's AI Gateway runs several
instances without sticky sessions.

## Change
With `HEADROOM_COMPRESS_SESSION_STORE=redis://…`
(`headroom/cache/shared_sessions.py`), every replica keeps each session
in Redis too:

- **Messages, stored once by digest** (`…:msgs`). The original and
returned transcripts mostly share messages, and each turn mostly repeats
the last one, so a turn writes only what Redis lacks: the messages it
added, plus any whose bytes changed (a moved cache marker, an edit, a
compaction). Unreferenced messages are pruned.
- **Metadata** (`…:meta`): the tracker's counters, its transcripts as
digest lists, and a random write token.
- **Before a turn**, the replica reads the token. If another replica
wrote since, it reads the metadata and only the messages it lacks, each
checked against its digest. The turn then seeds its compression cache
from that tracker, as after a restart.
- **After the turn**, it writes the new messages, then the metadata via
a Lua compare-and-set on the token.
- A random token, not a counter: a lost key or a failover can't leave a
replica trusting stale state.
- The script also detects a messages hash that `maxmemory` evicted on
its own; the writer then stores the session whole.
- A replica that finds corrupt or evicted state claims the token and
rewrites the session whole.
- **Usage relay:** fetches under the session's turn lock, and installs
only if no local turn installed a tracker meanwhile.
- **Failures:** a Redis error or corrupt value never fails a turn. The
replica serves from its own memory, as without the store. Clock skew
between replicas is tolerated.
- **CCR:** the setting also selects #4368's Redis CCR store unless
`HEADROOM_CCR_BACKEND` names another, so markers in a replayed
transcript resolve on any replica.
- **`SessionTrackerStore`** gains `encode_shared` / `decode_shared` (the
restart file's per-tracker encoding, split into metadata and message
table) and `install`.

## Results
**Real run:** 3 proxy processes on one Redis-protocol server (fakeredis
TCP server; no Redis on the test machine), a 20-turn `mode: "ccr"`
session sent round-robin:
- every turn's returned prefix was byte-identical to the previous
turn's;
- the CCR marker resolved via `/v1/retrieve` on all three replicas;
- Redis received **378 KB** over the 20 turns. Writing the whole
transcript every turn would have been about 4.9 MB, growing with the
square of the session length.

**Tests:** `tests/test_compress_session_shared.py`, each failure mode
mutation-checked:
- a stale replica reloads newer state;
- without the store, the other replica starts cold;
- a turn writes only the messages Redis lacks;
- a fully lost key is rewritten;
- a messages hash evicted on its own is rewritten;
- corrupt state is repaired by another replica;
- unreferenced messages are pruned;
- a writer whose clock runs ahead is trusted;
- the usage relay reaches and publishes a session served elsewhere;
- an outage or corrupt values never fail a turn;
- the CCR store follows the session store;
- a URL that isn't Redis stops startup.

818 proxy, gateway, session and restart tests pass. ruff and mypy are
clean.

## Notes
- Redis holds transcripts and tool output verbatim. Use `rediss://` with
authentication on a private network; plain `redis://` to a remote host
logs a warning.
- Re-drive turns still need affinity: they park a live in-process task
between two HTTP calls.
- About 540 added lines (excluding `uv.lock`), around 200 of them tests.
The `prefix_tracker` refactor can split out if reviewers prefer.

## Real Behavior Proof
- **Environment:** macOS 26.7.1 (arm64), Python 3.12.13, redis-py 8.1.0.
Redis is a fakeredis 2.40.0 `TcpFakeServer`, which speaks the Redis
protocol over TCP; no Redis server was on the test machine. Docker was
not available.
- **Configuration:** every replica has
`HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0` (the CCR
store follows it) and its own `HEADROOM_WORKSPACE_DIR`.
- **Steps:**
  1. Start the fake server:
`python -c "from fakeredis import TcpFakeServer;
TcpFakeServer(('127.0.0.1', 6391),
server_type='redis').serve_forever()"`
  2. Start three proxies, one each on ports 8901, 8902 and 8903:
`HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0
HEADROOM_WORKSPACE_DIR=<own dir> python -m headroom.cli proxy --port
<port>`
3. Drive one session for 20 turns, round-robin across the three: `POST
/v1/compress` with `{"messages": …, "session_id": "kong-e2e", "mode":
"ccr"}`. The transcript is the 25 KB tool history from
`tests/test_compress_session_mode.py`. Every third turn adds a 120-row
grep result; the other turns add a user line.
  4. After each turn, check:
- the returned messages start with the previous turn's returned
messages, byte for byte;
     - how many bytes the turn wrote to the `:msgs` hash.
5. Fetch each `<<ccr:hash>>` marker via `GET /v1/retrieve/{hash}` on all
three proxies.
- **Result** (after the review fixes, at 95b2d93a):
  - every turn's prefix is byte-identical to the previous turn's;
- Redis received 378 KB in total (63 KB at most in one turn), against
about 4.9 MB if the whole transcript were rewritten each turn;
  - the CCR marker resolved on all three replicas.
- **Not tested:**
- a real Redis or Redis Cluster (the Lua script and hash-tagged keys are
written for it);
  - TLS (`rediss://`) and AUTH;
  - Kong's AI Gateway in front of the replicas;
  - Windows;
  - real `maxmemory` eviction (simulated in tests by deleting keys);
  - sustained load.

## Dependencies and scope
- **No new runtime dependency:** this uses #4368's `[redis]` extra.
- **Dev only:** the `fakeredis` requirement becomes `fakeredis[lua]`,
which adds `lupa` so the tests run the compare-and-set Lua script
exactly as Redis would.
- **Why this feature:** Kong's AI Gateway runs several Headroom
instances with no sticky sessions and asked for session state in Redis.
It is opt-in: nothing changes unless `HEADROOM_COMPRESS_SESSION_STORE`
is set.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4369"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 23:52:06 -07:00