mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
Stacked on #4368 (Redis CCR store); merge that first. ## Why A session-aware `/v1/compress` turn must return the prefix it returned last turn byte for byte, or the provider's prompt cache misses. That state (the session's prefix tracker) lived only in the memory of the replica that served the turn, so a gateway with several replicas had to pin sessions (`session_affinity`). Kong's AI Gateway runs several instances without sticky sessions. ## Change With `HEADROOM_COMPRESS_SESSION_STORE=redis://…` (`headroom/cache/shared_sessions.py`), every replica keeps each session in Redis too: - **Messages, stored once by digest** (`…:msgs`). The original and returned transcripts mostly share messages, and each turn mostly repeats the last one, so a turn writes only what Redis lacks: the messages it added, plus any whose bytes changed (a moved cache marker, an edit, a compaction). Unreferenced messages are pruned. - **Metadata** (`…:meta`): the tracker's counters, its transcripts as digest lists, and a random write token. - **Before a turn**, the replica reads the token. If another replica wrote since, it reads the metadata and only the messages it lacks, each checked against its digest. The turn then seeds its compression cache from that tracker, as after a restart. - **After the turn**, it writes the new messages, then the metadata via a Lua compare-and-set on the token. - A random token, not a counter: a lost key or a failover can't leave a replica trusting stale state. - The script also detects a messages hash that `maxmemory` evicted on its own; the writer then stores the session whole. - A replica that finds corrupt or evicted state claims the token and rewrites the session whole. - **Usage relay:** fetches under the session's turn lock, and installs only if no local turn installed a tracker meanwhile. - **Failures:** a Redis error or corrupt value never fails a turn. The replica serves from its own memory, as without the store. Clock skew between replicas is tolerated. - **CCR:** the setting also selects #4368's Redis CCR store unless `HEADROOM_CCR_BACKEND` names another, so markers in a replayed transcript resolve on any replica. - **`SessionTrackerStore`** gains `encode_shared` / `decode_shared` (the restart file's per-tracker encoding, split into metadata and message table) and `install`. ## Results **Real run:** 3 proxy processes on one Redis-protocol server (fakeredis TCP server; no Redis on the test machine), a 20-turn `mode: "ccr"` session sent round-robin: - every turn's returned prefix was byte-identical to the previous turn's; - the CCR marker resolved via `/v1/retrieve` on all three replicas; - Redis received **378 KB** over the 20 turns. Writing the whole transcript every turn would have been about 4.9 MB, growing with the square of the session length. **Tests:** `tests/test_compress_session_shared.py`, each failure mode mutation-checked: - a stale replica reloads newer state; - without the store, the other replica starts cold; - a turn writes only the messages Redis lacks; - a fully lost key is rewritten; - a messages hash evicted on its own is rewritten; - corrupt state is repaired by another replica; - unreferenced messages are pruned; - a writer whose clock runs ahead is trusted; - the usage relay reaches and publishes a session served elsewhere; - an outage or corrupt values never fail a turn; - the CCR store follows the session store; - a URL that isn't Redis stops startup. 818 proxy, gateway, session and restart tests pass. ruff and mypy are clean. ## Notes - Redis holds transcripts and tool output verbatim. Use `rediss://` with authentication on a private network; plain `redis://` to a remote host logs a warning. - Re-drive turns still need affinity: they park a live in-process task between two HTTP calls. - About 540 added lines (excluding `uv.lock`), around 200 of them tests. The `prefix_tracker` refactor can split out if reviewers prefer. ## Real Behavior Proof - **Environment:** macOS 26.7.1 (arm64), Python 3.12.13, redis-py 8.1.0. Redis is a fakeredis 2.40.0 `TcpFakeServer`, which speaks the Redis protocol over TCP; no Redis server was on the test machine. Docker was not available. - **Configuration:** every replica has `HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0` (the CCR store follows it) and its own `HEADROOM_WORKSPACE_DIR`. - **Steps:** 1. Start the fake server: `python -c "from fakeredis import TcpFakeServer; TcpFakeServer(('127.0.0.1', 6391), server_type='redis').serve_forever()"` 2. Start three proxies, one each on ports 8901, 8902 and 8903: `HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0 HEADROOM_WORKSPACE_DIR=<own dir> python -m headroom.cli proxy --port <port>` 3. Drive one session for 20 turns, round-robin across the three: `POST /v1/compress` with `{"messages": …, "session_id": "kong-e2e", "mode": "ccr"}`. The transcript is the 25 KB tool history from `tests/test_compress_session_mode.py`. Every third turn adds a 120-row grep result; the other turns add a user line. 4. After each turn, check: - the returned messages start with the previous turn's returned messages, byte for byte; - how many bytes the turn wrote to the `:msgs` hash. 5. Fetch each `<<ccr:hash>>` marker via `GET /v1/retrieve/{hash}` on all three proxies. - **Result** (after the review fixes, at 95b2d93a): - every turn's prefix is byte-identical to the previous turn's; - Redis received 378 KB in total (63 KB at most in one turn), against about 4.9 MB if the whole transcript were rewritten each turn; - the CCR marker resolved on all three replicas. - **Not tested:** - a real Redis or Redis Cluster (the Lua script and hash-tagged keys are written for it); - TLS (`rediss://`) and AUTH; - Kong's AI Gateway in front of the replicas; - Windows; - real `maxmemory` eviction (simulated in tests by deleting keys); - sustained load. ## Dependencies and scope - **No new runtime dependency:** this uses #4368's `[redis]` extra. - **Dev only:** the `fakeredis` requirement becomes `fakeredis[lua]`, which adds `lupa` so the tests run the compare-and-set Lua script exactly as Redis would. - **Why this feature:** Kong's AI Gateway runs several Headroom instances with no sticky sessions and asked for session state in Redis. It is opt-in: nothing changes unless `HEADROOM_COMPRESS_SESSION_STORE` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4369" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
343 lines
15 KiB
Python
343 lines
15 KiB
Python
"""Session-aware /v1/compress served by several replicas sharing Redis.
|
|
|
|
A gateway without sticky sessions sends each turn to any replica. With
|
|
HEADROOM_COMPRESS_SESSION_STORE set, a replica that did not serve the last turn
|
|
takes the session's state from Redis, freezes the prefix the caller already
|
|
forwarded, and returns it byte for byte.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
fakeredis = pytest.importorskip("fakeredis")
|
|
pytest.importorskip("lupa")
|
|
|
|
from fastapi.testclient import TestClient # noqa: E402
|
|
|
|
from headroom.cache import shared_sessions # noqa: E402
|
|
from headroom.cache.backends.redis import RedisBackend # noqa: E402
|
|
from headroom.cache.compression_store import _create_default_ccr_backend # noqa: E402
|
|
from headroom.exceptions import ConfigurationError # noqa: E402
|
|
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
|
from tests.test_compress_session_mode import _big_tool_history, _compress # noqa: E402
|
|
|
|
|
|
@pytest.fixture
|
|
def redis_server(monkeypatch):
|
|
import redis
|
|
|
|
server = fakeredis.FakeServer()
|
|
monkeypatch.setenv(shared_sessions.STORE_ENV, "redis://127.0.0.1:6399/0")
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory") # CCR sharing is tested on its own
|
|
monkeypatch.setattr(
|
|
redis.Redis,
|
|
"from_url",
|
|
classmethod(lambda cls, *a, **k: fakeredis.FakeRedis(server=server)),
|
|
)
|
|
return fakeredis.FakeRedis(server=server)
|
|
|
|
|
|
def _replica() -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
image_optimize=False,
|
|
)
|
|
return TestClient(create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 1))
|
|
|
|
|
|
def _second_tool_result() -> list[dict]:
|
|
rows = [
|
|
{"id": i, "path": f"src/mod_{i}.py", "line": i * 3, "hit": "x" * 200} for i in range(150)
|
|
]
|
|
return [
|
|
{
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{"id": "c2", "type": "function", "function": {"name": "grep", "arguments": "{}"}}
|
|
],
|
|
},
|
|
{"role": "tool", "tool_call_id": "c2", "content": json.dumps(rows)},
|
|
]
|
|
|
|
|
|
def test_any_replica_serves_the_next_turn_byte_for_byte(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
turn1 = _compress(a, history, session_id="conv")
|
|
assert turn1["messages"][2]["content"] != history[2]["content"] # compressed
|
|
|
|
# Turn 2 on B adds a second tool result, compressed by B.
|
|
history2 = history + _second_tool_result()
|
|
turn2 = _compress(b, history2, session_id="conv")
|
|
assert turn2["messages"][: len(turn1["messages"])] == turn1["messages"]
|
|
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
|
|
assert turn2["messages"][4]["content"] != history2[4]["content"]
|
|
|
|
# Back on A, which last saw turn 1: only B's state has B's bytes.
|
|
history3 = history2 + [{"role": "user", "content": "again"}]
|
|
turn3 = _compress(a, history3, session_id="conv")
|
|
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
def test_without_the_store_a_new_replica_starts_cold(monkeypatch):
|
|
# What the store fixes: B never saw the compressed tool result, so it does
|
|
# not freeze it and compresses it again.
|
|
monkeypatch.delenv(shared_sessions.STORE_ENV, raising=False)
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
turn1 = _compress(a, history, session_id="conv")
|
|
turn2 = _compress(b, history + [{"role": "user", "content": "x"}], session_id="conv")
|
|
assert turn2["session"]["frozen_message_count"] < len(turn1["messages"])
|
|
|
|
|
|
def test_a_turn_writes_only_the_messages_redis_lacks(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a:
|
|
_compress(a, history, session_id="conv-delta")
|
|
(msgs_key,) = redis_server.keys("headroom:session:*:msgs")
|
|
before = set(redis_server.hkeys(msgs_key))
|
|
|
|
import redis
|
|
|
|
written: list[int] = []
|
|
real_hset = redis.client.Pipeline.hset
|
|
|
|
def counting_hset(self, name, *args, mapping=None, **kwargs):
|
|
written.append(len(mapping or {}))
|
|
return real_hset(self, name, *args, mapping=mapping, **kwargs)
|
|
|
|
redis.client.Pipeline.hset = counting_hset
|
|
try:
|
|
_compress(
|
|
a, history + [{"role": "user", "content": "one more"}], session_id="conv-delta"
|
|
)
|
|
finally:
|
|
redis.client.Pipeline.hset = real_hset
|
|
after = set(redis_server.hkeys(msgs_key))
|
|
# The new user message, and at most the few whose cache marker moved.
|
|
assert 1 <= sum(written) <= 3 < len(before)
|
|
assert before <= after
|
|
|
|
|
|
def test_a_lost_key_is_stored_whole_again(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
_compress(a, history, session_id="conv-lost")
|
|
redis_server.flushall()
|
|
turn2 = _compress(a, history + [{"role": "user", "content": "y"}], session_id="conv-lost")
|
|
assert redis_server.keys("headroom:session:*:meta")
|
|
# And another replica is warm from it.
|
|
turn3 = _compress(
|
|
b,
|
|
history + [{"role": "user", "content": "y"}, {"role": "user", "content": "z"}],
|
|
session_id="conv-lost",
|
|
)
|
|
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
def test_the_usage_relay_reaches_and_publishes_a_session_served_elsewhere(redis_server):
|
|
with _replica() as a, _replica() as b:
|
|
_compress(a, _big_tool_history(), session_id="conv-usage")
|
|
(meta_key,) = redis_server.keys("headroom:session:*:meta")
|
|
token = redis_server.hget(meta_key, "t")
|
|
resp = b.post(
|
|
"/v1/usage",
|
|
json={
|
|
"session_id": "conv-usage",
|
|
"usage": {"cache_read_input_tokens": 0, "cache_creation_input_tokens": 50_000},
|
|
},
|
|
)
|
|
assert resp.status_code == 200, resp.text
|
|
assert redis_server.hget(meta_key, "t") != token # B published its update
|
|
|
|
|
|
def test_a_redis_outage_or_corrupt_state_never_fails_a_turn(redis_server, monkeypatch):
|
|
import redis
|
|
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
turn1 = _compress(a, history, session_id="conv-down")
|
|
(meta_key,) = redis_server.keys("headroom:session:*:meta")
|
|
redis_server.hset(meta_key, "m", "{not json")
|
|
turn2 = _compress(b, history + [{"role": "user", "content": "y"}], session_id="conv-down")
|
|
assert turn2["messages"][0] == turn1["messages"][0]
|
|
|
|
def down(*args, **kwargs):
|
|
raise redis.ConnectionError("connection refused")
|
|
|
|
monkeypatch.setattr(fakeredis.FakeRedis, "execute_command", down)
|
|
turn3 = _compress(a, history + [{"role": "user", "content": "z"}], session_id="conv-down")
|
|
assert turn3["messages"][: len(turn1["messages"])] == turn1["messages"]
|
|
|
|
|
|
def test_sharing_sessions_shares_the_ccr_store_too(monkeypatch):
|
|
monkeypatch.setenv(shared_sessions.STORE_ENV, "redis://127.0.0.1:6399/0")
|
|
monkeypatch.delenv("HEADROOM_CCR_BACKEND", raising=False)
|
|
assert isinstance(_create_default_ccr_backend(), RedisBackend)
|
|
|
|
|
|
def test_a_store_url_that_is_not_redis_stops_startup(monkeypatch):
|
|
monkeypatch.setenv(shared_sessions.STORE_ENV, "memcached://cache:11211")
|
|
with pytest.raises(ConfigurationError, match="redis://"):
|
|
shared_sessions.open_shared_sessions()
|
|
|
|
|
|
def _meta_key(redis_server):
|
|
(meta_key,) = redis_server.keys("headroom:session:*:meta")
|
|
return meta_key
|
|
|
|
|
|
def test_messages_evicted_on_their_own_are_stored_whole_again(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
_compress(a, history, session_id="conv-evict")
|
|
meta_key = _meta_key(redis_server)
|
|
redis_server.delete(meta_key.replace(b":meta", b":msgs")) # maxmemory took one key
|
|
history2 = history + [{"role": "user", "content": "y"}]
|
|
turn2 = _compress(a, history2, session_id="conv-evict")
|
|
turn3 = _compress(b, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict")
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
def test_corrupt_state_is_rewritten_by_the_next_replica_to_write(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b, _replica() as c:
|
|
_compress(a, history, session_id="conv-corrupt")
|
|
redis_server.hset(_meta_key(redis_server), "m", "{not json")
|
|
history2 = history + [{"role": "user", "content": "y"}]
|
|
turn2 = _compress(b, history2, session_id="conv-corrupt") # repairs it
|
|
turn3 = _compress(
|
|
c, history2 + [{"role": "user", "content": "z"}], session_id="conv-corrupt"
|
|
)
|
|
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
def test_messages_no_transcript_references_are_pruned(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a:
|
|
_compress(a, history, session_id="conv-prune")
|
|
msgs_key = _meta_key(redis_server).replace(b":meta", b":msgs")
|
|
redis_server.hset(msgs_key, mapping={f"stale{i}": "{}" for i in range(50)})
|
|
_compress(a, history + [{"role": "user", "content": "y"}], session_id="conv-prune")
|
|
assert not any(k.startswith(b"stale") for k in redis_server.hkeys(msgs_key))
|
|
|
|
|
|
def test_a_writer_whose_clock_runs_ahead_is_still_trusted(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
turn1 = _compress(a, history, session_id="conv-skew")
|
|
meta_key = _meta_key(redis_server)
|
|
meta = json.loads(redis_server.hget(meta_key, "m"))
|
|
meta["last_activity"] += 30 # A's clock is 30 s ahead of B's
|
|
redis_server.hset(meta_key, "m", json.dumps(meta))
|
|
turn2 = _compress(b, history + [{"role": "user", "content": "y"}], session_id="conv-skew")
|
|
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
|
|
|
|
|
|
def test_a_lost_message_is_noticed_even_beside_unreferenced_ones(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
_compress(a, history, session_id="conv-masked")
|
|
msgs_key = _meta_key(redis_server).replace(b":meta", b":msgs")
|
|
# Old variants left behind outnumber the one referenced message lost.
|
|
redis_server.hset(msgs_key, mapping={f"old{i}": "{}" for i in range(5)})
|
|
redis_server.hdel(msgs_key, redis_server.hkeys(msgs_key)[0])
|
|
history2 = history + [{"role": "user", "content": "y"}]
|
|
turn2 = _compress(a, history2, session_id="conv-masked")
|
|
turn3 = _compress(
|
|
b, history2 + [{"role": "user", "content": "z"}], session_id="conv-masked"
|
|
)
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
def test_a_corrupt_token_never_fails_a_turn_and_is_replaced(redis_server):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
_compress(a, history, session_id="conv-token")
|
|
redis_server.hset(_meta_key(redis_server), "t", b"\xff")
|
|
history2 = history + [{"role": "user", "content": "y"}]
|
|
turn2 = _compress(b, history2, session_id="conv-token")
|
|
turn3 = _compress(a, history2 + [{"role": "user", "content": "z"}], session_id="conv-token")
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
@pytest.fixture
|
|
def evict_before_cas(monkeypatch):
|
|
"""Delete the messages hash just before the next ``times`` compare-and-set
|
|
scripts run, as maxmemory might between the message writes and the script."""
|
|
budget = {"times": 0}
|
|
real_init = shared_sessions.SharedSessions.__init__
|
|
|
|
def init(self, client):
|
|
real_init(self, client)
|
|
cas = self._cas
|
|
|
|
def evicting_cas(keys, args):
|
|
if budget["times"]:
|
|
budget["times"] -= 1
|
|
client.delete(keys[1])
|
|
return cas(keys=keys, args=args)
|
|
|
|
self._cas = evicting_cas
|
|
|
|
monkeypatch.setattr(shared_sessions.SharedSessions, "__init__", init)
|
|
return budget
|
|
|
|
|
|
def test_eviction_before_the_first_publish_is_never_acknowledged(redis_server, evict_before_cas):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
evict_before_cas["times"] = 1
|
|
turn1 = _compress(a, history, session_id="conv-evict-first")
|
|
turn2 = _compress(
|
|
b, history + [{"role": "user", "content": "y"}], session_id="conv-evict-first"
|
|
)
|
|
assert turn2["messages"][: len(turn1["messages"])] == turn1["messages"]
|
|
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
|
|
|
|
|
|
@pytest.mark.parametrize("corrupt", [False, True], ids=["delta", "whole-rewrite"])
|
|
def test_eviction_before_a_later_publish_is_never_acknowledged(
|
|
redis_server, evict_before_cas, corrupt
|
|
):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b, _replica() as c:
|
|
_compress(a, history, session_id="conv-evict-later")
|
|
if corrupt: # B then rewrites the session whole, assuming no message stored
|
|
redis_server.hset(_meta_key(redis_server), "m", "{not json")
|
|
history2 = history + [{"role": "user", "content": "y"}]
|
|
evict_before_cas["times"] = 1
|
|
turn2 = _compress(b, history2, session_id="conv-evict-later")
|
|
turn3 = _compress(
|
|
c, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict-later"
|
|
)
|
|
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
|
|
|
|
|
def test_a_session_whose_messages_keep_vanishing_is_not_published(redis_server, evict_before_cas):
|
|
history = _big_tool_history()
|
|
with _replica() as a, _replica() as b:
|
|
evict_before_cas["times"] = 2 # the write and its whole rewrite
|
|
_compress(a, history, session_id="conv-evict-twice")
|
|
assert not redis_server.keys("headroom:session:*:meta")
|
|
# Once Redis keeps the messages again, the next turn publishes it whole.
|
|
history2 = history + [{"role": "user", "content": "y"}]
|
|
turn2 = _compress(a, history2, session_id="conv-evict-twice")
|
|
turn3 = _compress(
|
|
b, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict-twice"
|
|
)
|
|
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|