feat(compress): share /v1/compress sessions across replicas via Redis (#4369)

Stacked on #4368 (Redis CCR store); merge that first.

## Why
A session-aware `/v1/compress` turn must return the prefix it returned
last turn byte for byte, or the provider's prompt cache misses. That
state (the session's prefix tracker) lived only in the memory of the
replica that served the turn, so a gateway with several replicas had to
pin sessions (`session_affinity`). Kong's AI Gateway runs several
instances without sticky sessions.

## Change
With `HEADROOM_COMPRESS_SESSION_STORE=redis://…`
(`headroom/cache/shared_sessions.py`), every replica keeps each session
in Redis too:

- **Messages, stored once by digest** (`…:msgs`). The original and
returned transcripts mostly share messages, and each turn mostly repeats
the last one, so a turn writes only what Redis lacks: the messages it
added, plus any whose bytes changed (a moved cache marker, an edit, a
compaction). Unreferenced messages are pruned.
- **Metadata** (`…:meta`): the tracker's counters, its transcripts as
digest lists, and a random write token.
- **Before a turn**, the replica reads the token. If another replica
wrote since, it reads the metadata and only the messages it lacks, each
checked against its digest. The turn then seeds its compression cache
from that tracker, as after a restart.
- **After the turn**, it writes the new messages, then the metadata via
a Lua compare-and-set on the token.
- A random token, not a counter: a lost key or a failover can't leave a
replica trusting stale state.
- The script also detects a messages hash that `maxmemory` evicted on
its own; the writer then stores the session whole.
- A replica that finds corrupt or evicted state claims the token and
rewrites the session whole.
- **Usage relay:** fetches under the session's turn lock, and installs
only if no local turn installed a tracker meanwhile.
- **Failures:** a Redis error or corrupt value never fails a turn. The
replica serves from its own memory, as without the store. Clock skew
between replicas is tolerated.
- **CCR:** the setting also selects #4368's Redis CCR store unless
`HEADROOM_CCR_BACKEND` names another, so markers in a replayed
transcript resolve on any replica.
- **`SessionTrackerStore`** gains `encode_shared` / `decode_shared` (the
restart file's per-tracker encoding, split into metadata and message
table) and `install`.

## Results
**Real run:** 3 proxy processes on one Redis-protocol server (fakeredis
TCP server; no Redis on the test machine), a 20-turn `mode: "ccr"`
session sent round-robin:
- every turn's returned prefix was byte-identical to the previous
turn's;
- the CCR marker resolved via `/v1/retrieve` on all three replicas;
- Redis received **378 KB** over the 20 turns. Writing the whole
transcript every turn would have been about 4.9 MB, growing with the
square of the session length.

**Tests:** `tests/test_compress_session_shared.py`, each failure mode
mutation-checked:
- a stale replica reloads newer state;
- without the store, the other replica starts cold;
- a turn writes only the messages Redis lacks;
- a fully lost key is rewritten;
- a messages hash evicted on its own is rewritten;
- corrupt state is repaired by another replica;
- unreferenced messages are pruned;
- a writer whose clock runs ahead is trusted;
- the usage relay reaches and publishes a session served elsewhere;
- an outage or corrupt values never fail a turn;
- the CCR store follows the session store;
- a URL that isn't Redis stops startup.

818 proxy, gateway, session and restart tests pass. ruff and mypy are
clean.

## Notes
- Redis holds transcripts and tool output verbatim. Use `rediss://` with
authentication on a private network; plain `redis://` to a remote host
logs a warning.
- Re-drive turns still need affinity: they park a live in-process task
between two HTTP calls.
- About 540 added lines (excluding `uv.lock`), around 200 of them tests.
The `prefix_tracker` refactor can split out if reviewers prefer.

## Real Behavior Proof
- **Environment:** macOS 26.7.1 (arm64), Python 3.12.13, redis-py 8.1.0.
Redis is a fakeredis 2.40.0 `TcpFakeServer`, which speaks the Redis
protocol over TCP; no Redis server was on the test machine. Docker was
not available.
- **Configuration:** every replica has
`HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0` (the CCR
store follows it) and its own `HEADROOM_WORKSPACE_DIR`.
- **Steps:**
  1. Start the fake server:
`python -c "from fakeredis import TcpFakeServer;
TcpFakeServer(('127.0.0.1', 6391),
server_type='redis').serve_forever()"`
  2. Start three proxies, one each on ports 8901, 8902 and 8903:
`HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0
HEADROOM_WORKSPACE_DIR=<own dir> python -m headroom.cli proxy --port
<port>`
3. Drive one session for 20 turns, round-robin across the three: `POST
/v1/compress` with `{"messages": …, "session_id": "kong-e2e", "mode":
"ccr"}`. The transcript is the 25 KB tool history from
`tests/test_compress_session_mode.py`. Every third turn adds a 120-row
grep result; the other turns add a user line.
  4. After each turn, check:
- the returned messages start with the previous turn's returned
messages, byte for byte;
     - how many bytes the turn wrote to the `:msgs` hash.
5. Fetch each `<<ccr:hash>>` marker via `GET /v1/retrieve/{hash}` on all
three proxies.
- **Result** (after the review fixes, at 95b2d93a):
  - every turn's prefix is byte-identical to the previous turn's;
- Redis received 378 KB in total (63 KB at most in one turn), against
about 4.9 MB if the whole transcript were rewritten each turn;
  - the CCR marker resolved on all three replicas.
- **Not tested:**
- a real Redis or Redis Cluster (the Lua script and hash-tagged keys are
written for it);
  - TLS (`rediss://`) and AUTH;
  - Kong's AI Gateway in front of the replicas;
  - Windows;
  - real `maxmemory` eviction (simulated in tests by deleting keys);
  - sustained load.

## Dependencies and scope
- **No new runtime dependency:** this uses #4368's `[redis]` extra.
- **Dev only:** the `fakeredis` requirement becomes `fakeredis[lua]`,
which adds `lupa` so the tests run the compare-and-set Lua script
exactly as Redis would.
- **Why this feature:** Kong's AI Gateway runs several Headroom
instances with no sticky sessions and asked for session state in Redis.
It is opt-in: nothing changes unless `HEADROOM_COMPRESS_SESSION_STORE`
is set.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4369"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-23 23:52:06 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent b196424e35
commit a616232d75
8 changed files with 831 additions and 31 deletions
+8 -1
View File
@@ -1232,6 +1232,13 @@ def _create_default_ccr_backend() -> CompressionStoreBackend | None:
from ..paths import persistence_allowed
backend_type = (os.environ.get("HEADROOM_CCR_BACKEND") or "").strip().lower()
session_store = os.environ.get("HEADROOM_COMPRESS_SESSION_STORE", "").strip()
if not backend_type and session_store:
# Replicas sharing compress sessions share the CCR store too: a marker
# in a replayed transcript must resolve on whichever replica serves it.
from .backends.redis import RedisBackend
return RedisBackend(session_store)
if backend_type == "memory":
return None
if not backend_type or backend_type == "sqlite":
@@ -1259,7 +1266,7 @@ def _create_default_ccr_backend() -> CompressionStoreBackend | None:
# registered as "redis" still wins, above.
from .backends.redis import RedisBackend
url = os.environ.get("HEADROOM_REDIS_URL", "").strip()
url = os.environ.get("HEADROOM_REDIS_URL", "").strip() or session_store
if not url:
logger.error(
"HEADROOM_CCR_BACKEND=redis needs HEADROOM_REDIS_URL; using the "
+74 -23
View File
@@ -1206,6 +1206,11 @@ class PrefixCacheTracker:
# builds the manager.
self.restored_from_disk = False
self.restored_maturation_state: dict[str, Any] | None = None
# Shared-store state (headroom.cache.shared_sessions): the write token
# this tracker matches ("" when never shared) and the digests of the
# messages known to be stored with it.
self.shared_token = ""
self.shared_digests: frozenset[str] = frozenset()
# Stats
self._busts_avoided: int = 0
@@ -1509,16 +1514,20 @@ class PrefixCacheTracker:
def is_expired_for_cache_ttl(self, cache_ttl_seconds: int | None) -> bool:
"""Check expiry including the longest configured or observed cache lifetime."""
return (time.time() - self._last_activity) > self.retention_seconds(cache_ttl_seconds)
def retention_seconds(self, cache_ttl_seconds: int | None = None) -> float:
"""How long this tracker is kept after its last activity: the session
TTL, or the longest configured or observed cache lifetime plus a margin."""
observed_ttl = max(
self._cache_ttl_seconds or 0,
self.config.cache_ttl_seconds or 0,
cache_ttl_seconds or 0,
)
retention = max(
return max(
self.config.session_ttl_seconds,
observed_ttl + _CACHE_TTL_CLEANUP_MARGIN_SECONDS if observed_ttl else 0,
)
return (time.time() - self._last_activity) > retention
def observe_cache_ttl(self, cache_ttl_seconds: int | None) -> None:
"""Remember the longest provider cache lifetime observed for this lineage."""
@@ -2144,6 +2153,14 @@ class SessionTrackerStore:
pairs mostly repeat the same messages, so this keeps the file near one
copy of each conversation instead of up to eleven.
"""
meta, table = self._tracker_parts(key, tracker, session_id)
return f'{{"meta":{json.dumps(meta)},"messages":[{",".join(table)}]}}'
def _tracker_parts(
self, key: str, tracker: PrefixCacheTracker, session_id: str | None
) -> tuple[dict[str, Any], list[str]]:
"""A tracker as its metadata, whose message lists are indexes into the
table, and the table: each distinct message once, as JSON."""
index: dict[str, int] = {}
memo: dict[int, list[int]] = {}
@@ -2172,7 +2189,7 @@ class SessionTrackerStore:
"history": [[ref(snap), t] for snap, t in self._lineage_history.get(key, ())],
"maturation": None if maturation is None else maturation.export_state(),
}
return f'{{"meta":{json.dumps(meta)},"messages":[{",".join(index)}]}}'
return meta, list(index)
def load_state(self, path: str) -> int:
"""Resume the trackers :meth:`save_state` wrote, before serving.
@@ -2213,6 +2230,58 @@ class SessionTrackerStore:
)
return restored
def _decode_tracker(self, entry: Any, now: float) -> PrefixCacheTracker | None:
"""One :meth:`_encode_tracker` entry as a tracker marked restored, or
``None`` when it has expired. Raises on a malformed entry."""
meta, table = entry["meta"], entry["messages"]
tracker = PrefixCacheTracker(str(meta["provider"]), self._default_config)
tracker._last_activity = float(meta["last_activity"])
tracker.observe_cache_ttl(meta["cache_ttl"])
# NaN or a future stamp would never expire.
if tracker.is_expired or not tracker._last_activity <= now:
return None
(
tracker._turn_number,
tracker._cached_token_count,
tracker._cached_message_count,
tracker._last_original_messages,
tracker._last_forwarded_messages,
) = _decode_turn_state(table, meta["state"])
tracker._turn_states = {int(t): _decode_turn_state(table, s) for t, s in meta["turns"]}
tracker.restored_from_disk = True
if meta["maturation"] is not None:
from headroom.transforms.read_maturation import ReadMaturationManager
tracker.restored_maturation_state = ReadMaturationManager.validate_state(
meta["maturation"]
)
return tracker
def encode_shared(
self, key: str, tracker: PrefixCacheTracker
) -> tuple[dict[str, Any], list[str]]:
"""``tracker`` for another process (headroom.cache.shared_sessions): its
metadata and its message table, each distinct message once, as JSON.
Raises when its messages are not JSON."""
return self._tracker_parts(key, tracker, None)
def decode_shared(self, meta: Any, messages: list[Any]) -> PrefixCacheTracker | None:
"""The tracker :meth:`encode_shared` described, marked restored so its
next turn seeds the compression cache from it; ``None`` when it has
expired. Raises on malformed input. Does not install it."""
if not isinstance(meta, dict):
raise ValueError("tracker metadata is not an object")
now = time.time()
# Written on another host: a clock ahead of ours is not a future stamp.
meta = {**meta, "last_activity": min(float(meta["last_activity"]), now)}
return self._decode_tracker({"meta": meta, "messages": messages}, now)
def install(self, key: str, tracker: PrefixCacheTracker) -> None:
"""Make ``tracker`` the one under ``key``. Call on the event loop, or
from a worker thread only for a key already present (replacing a value
does not resize the dict the loop may be iterating)."""
self._trackers[key] = tracker
def _decode_state(self, data: Any) -> int:
"""Validate the whole file first, then install it; a bad file installs nothing."""
from headroom.cache.compression_store import CompressionEntry, get_compression_store
@@ -2228,27 +2297,9 @@ class SessionTrackerStore:
# Oldest first, so each family keeps its lineages in creation order.
for entry in reversed(data["trackers"]):
meta, table = entry["meta"], entry["messages"]
tracker = PrefixCacheTracker(str(meta["provider"]), self._default_config)
tracker._last_activity = float(meta["last_activity"])
tracker.observe_cache_ttl(meta["cache_ttl"])
# NaN or a future stamp would never expire.
if tracker.is_expired or not tracker._last_activity <= now:
tracker = self._decode_tracker(entry, now)
if tracker is None:
continue
(
tracker._turn_number,
tracker._cached_token_count,
tracker._cached_message_count,
tracker._last_original_messages,
tracker._last_forwarded_messages,
) = _decode_turn_state(table, meta["state"])
tracker._turn_states = {int(t): _decode_turn_state(table, s) for t, s in meta["turns"]}
tracker.restored_from_disk = True
if meta["maturation"] is not None:
from headroom.transforms.read_maturation import ReadMaturationManager
tracker.restored_maturation_state = ReadMaturationManager.validate_state(
meta["maturation"]
)
key = str(meta["key"])
trackers[key] = tracker
history = deque(
+276
View File
@@ -0,0 +1,276 @@
"""Session-aware ``/v1/compress`` state shared by several Headroom replicas.
A compress session's state is its prefix tracker: the transcript it was last
given and the one it returned, which every later turn must replay byte for byte
so the provider's prompt cache holds. In memory, only the replica that served
the last turn has it, so a gateway running several replicas must pin each
session to one (``session_affinity``).
With ``HEADROOM_COMPRESS_SESSION_STORE=redis://...`` every replica also keeps
each session in Redis, and any replica can serve its next turn. Per session:
* ``<prefix>:session:{<digest>}:msgs``, a hash of message digest to message
JSON. The original and returned transcripts mostly share messages, and each
turn mostly repeats the last one's, so each message is stored once and a turn
writes only the messages Redis does not have yet: what the turn added, and
any message whose bytes changed (a moved cache marker, an edit, a compaction);
* ``<prefix>:session:{<digest>}:meta``, the tracker's counters and its
transcripts as lists of digests (``m``), and a write token (``t``).
Before a turn a replica reads the token (one small ``HGET``). When another
replica has written since, it reads the metadata and only the messages it does
not already hold, and the turn seeds its compression cache from that tracker as
after a restart (:func:`~headroom.cache.prefix_tracker.seed_restored_compression_cache`).
After the turn it writes the new messages, then the metadata if the token is
still the one it read and every message it references is stored, checked in
one script (compare-and-set): a random token, so a lost key or a failover
cannot leave a replica believing stale state is current. A missing key
resets the replica, whose next write stores the session whole again. Fetched
messages are checked against their digests. Both keys expire with the tracker.
A Redis error never fails a turn (:mod:`headroom.cache.redis_client`): the
replica serves from its own memory, as without the store. CCR originals behind
the markers in these transcripts live in the shared CCR store, which this
setting selects too unless ``HEADROOM_CCR_BACKEND`` names another. Requires the
``redis`` extra.
"""
from __future__ import annotations
import functools
import hashlib
import json
import logging
import os
import uuid
from typing import TYPE_CHECKING, Any
from .redis_client import Guard, connect, key_prefix
if TYPE_CHECKING:
from .prefix_tracker import PrefixCacheTracker, SessionTrackerStore
logger = logging.getLogger(__name__)
STORE_ENV = "HEADROOM_COMPRESS_SESSION_STORE"
#: Messages per HSET/HMGET, so no single command carries a whole transcript.
_CHUNK = 256
# Install ARGV[2] as the token and ARGV[3] as the metadata if the token is still
# ARGV[1] ("" when the key must not exist) and the messages hash (KEYS[2]) holds
# every message the metadata references (ARGV[5:], including those this write
# just sent: Redis can evict the hash between those writes and this script).
# 1 on success, 0 when another replica wrote first, 2 when the messages hash
# lost one.
_CAS = """
local t = redis.call('HGET', KEYS[1], 't') or ''
if t ~= ARGV[1] then return 0 end
for i = 5, #ARGV do
if redis.call('HEXISTS', KEYS[2], ARGV[i]) == 0 then return 2 end
end
redis.call('HSET', KEYS[1], 't', ARGV[2], 'm', ARGV[3])
redis.call('PEXPIRE', KEYS[1], ARGV[4])
return 1
"""
_FAILED = object()
class SharedSessions:
"""Compress-session trackers in Redis, beside the in-memory store."""
def __init__(self, client: Any) -> None:
self._client = client
self._cas = client.register_script(_CAS)
self._guard = Guard("compress sessions")
self._prefix = f"{key_prefix()}:session:"
def fetch(
self, store: SessionTrackerStore, key: str, local: PrefixCacheTracker | None
) -> PrefixCacheTracker | None:
"""The session's tracker when another replica wrote a newer one than
``local``, decoded but not installed; else ``None``."""
meta_key, msgs_key = self._keys(key)
token = self._guard.call("read", lambda: self._client.hget(meta_key, "t"), _FAILED)
if token is _FAILED:
return None
if token is None:
if local is not None: # lost or expired: the next write stores it whole
local.shared_token, local.shared_digests = "", frozenset()
return None
try:
if local is not None and _text(token) == local.shared_token:
return None
except UnicodeDecodeError:
return self._drop(local, meta_key, msgs_key)
row = self._guard.call("read", lambda: self._client.hmget(meta_key, "t", "m"), None)
if row is None:
return None
if row[0] is None: # expired between the two reads
return None
try:
token = _text(row[0])
except UnicodeDecodeError:
return self._drop(local, meta_key, msgs_key)
try:
meta = json.loads(_text(row[1]))
digests = meta.pop("table")
if not isinstance(digests, list) or not all(isinstance(d, str) for d in digests):
raise ValueError("table is not a list of digests")
except (TypeError, ValueError, KeyError, AttributeError, IndexError):
return self._unusable(local, token, "metadata", meta_key)
held = self._held(store, key, local)
missing = [d for d in dict.fromkeys(digests) if d not in held]
for start in range(0, len(missing), _CHUNK):
part = missing[start : start + _CHUNK]
values = self._guard.call(
"read", functools.partial(self._client.hmget, msgs_key, part), None
)
if values is None:
return None
for digest, value in zip(part, values, strict=True):
try:
text = None if value is None else _text(value)
except UnicodeDecodeError:
text = None
if text is None or _digest(text) != digest:
return self._unusable(local, token, "messages", msgs_key)
held[digest] = text
try:
tracker = store.decode_shared(meta, [json.loads(held[d]) for d in digests])
except (TypeError, ValueError, KeyError, AttributeError, OverflowError):
return self._unusable(local, token, "metadata", meta_key)
if tracker is None:
logger.info("compress sessions: shared state for %s has expired", meta_key)
return None
tracker.shared_token, tracker.shared_digests = token, frozenset(digests)
return tracker
@staticmethod
def _unusable(
local: PrefixCacheTracker | None, token: str, what: str, name: str
) -> PrefixCacheTracker | None:
"""Shared state this replica cannot use (evicted, corrupt or altered):
keep its own, and claim the current token with no messages known to be
stored, so its next write replaces the session whole."""
logger.warning("compress sessions: unusable %s in %s; this replica rewrites it", what, name)
if local is not None:
local.shared_token, local.shared_digests = token, frozenset()
return None
def _drop(
self, local: PrefixCacheTracker | None, meta_key: str, msgs_key: str
) -> PrefixCacheTracker | None:
"""A token that is not text cannot be claimed: delete the session, so
the next write stores it whole."""
logger.warning("compress sessions: corrupt token in %s; deleting the session", meta_key)
self._guard.call("write", lambda: self._client.delete(meta_key, msgs_key), 0)
if local is not None:
local.shared_token, local.shared_digests = "", frozenset()
return None
def publish(self, store: SessionTrackerStore, key: str, tracker: PrefixCacheTracker) -> None:
"""Write the messages Redis lacks, then the metadata unless another
replica wrote since ``tracker`` was read."""
try:
meta, table = store.encode_shared(key, tracker)
except (TypeError, ValueError) as exc: # messages that are not JSON
logger.debug("compress sessions: tracker not shareable (%s)", exc)
return
digests = [_digest(m) for m in table]
meta["table"] = digests
meta_json = json.dumps(meta)
ttl_ms = max(1000, int(tracker.retention_seconds() * 1000))
meta_key, msgs_key = self._keys(key)
referenced = sorted(set(digests))
def write(known: frozenset[str]) -> tuple[int, str]:
new = [(d, m) for d, m in zip(digests, table, strict=True) if d not in known]
token = uuid.uuid4().hex
pipe = self._client.pipeline(transaction=False)
for start in range(0, len(new), _CHUNK):
pipe.hset(msgs_key, mapping=dict(new[start : start + _CHUNK]))
pipe.pexpire(msgs_key, ttl_ms)
pipe.execute()
args = [tracker.shared_token, token, meta_json, ttl_ms, *referenced]
return int(self._cas(keys=[meta_key, msgs_key], args=args)), token
result = self._guard.call("write", lambda: write(tracker.shared_digests), None)
if result is not None and result[0] == 2:
# The messages hash lost entries (evicted on its own): store it whole.
result = self._guard.call("write", lambda: write(frozenset()), None)
if result is None:
return
written, token = result
if written == 1:
tracker.shared_token, tracker.shared_digests = token, frozenset(digests)
self._prune(msgs_key, set(digests))
elif written == 2:
# Evicted again during the whole rewrite: publish nothing, so no
# replica takes a session it cannot replay; the next turn retries.
logger.warning("compress sessions: messages of %s evicted while writing", meta_key)
elif written == 0:
# Two replicas ran turns of one session at once, which the caller's
# contract excludes; the next turn takes the state that won.
logger.warning("compress sessions: another replica wrote %s first", meta_key)
def _prune(self, msgs_key: str, referenced: set[str]) -> None:
"""Drop messages no transcript references (moved markers, edits,
compactions) once they outnumber the referenced ones."""
size = self._guard.call("write", lambda: int(self._client.hlen(msgs_key)), 0)
if size <= 2 * len(referenced):
return
def prune() -> None:
stale = [k for k in map(_text, self._client.hkeys(msgs_key)) if k not in referenced]
for start in range(0, len(stale), _CHUNK):
self._client.hdel(msgs_key, *stale[start : start + _CHUNK])
self._guard.call("write", prune, None)
def _keys(self, key: str) -> tuple[str, str]:
# One hash tag: the compare-and-set script names both keys, which Redis
# Cluster allows only within one slot.
base = f"{self._prefix}{{{hashlib.blake2b(key.encode(), digest_size=16).hexdigest()}}}"
return f"{base}:meta", f"{base}:msgs"
@staticmethod
def _held(
store: SessionTrackerStore, key: str, local: PrefixCacheTracker | None
) -> dict[str, str]:
"""The messages this replica already holds for the session, by digest."""
if local is None:
return {}
try:
_, table = store.encode_shared(key, local)
except (TypeError, ValueError):
return {}
return {_digest(m): m for m in table}
def _digest(message_json: str) -> str:
return hashlib.blake2b(message_json.encode(), digest_size=16).hexdigest()
def _text(value: Any) -> str:
return value.decode() if isinstance(value, bytes) else str(value)
def open_shared_sessions() -> SharedSessions | None:
"""The shared store ``HEADROOM_COMPRESS_SESSION_STORE`` names, or ``None``.
Raises :class:`~headroom.exceptions.ConfigurationError` when it is set but
unusable, so a replica meant to share sessions never starts without it."""
url = os.environ.get(STORE_ENV, "").strip()
if not url:
return None
client = connect(url, setting=STORE_ENV)
ccr = (os.environ.get("HEADROOM_CCR_BACKEND") or "").strip().lower()
if ccr not in ("", "redis"):
logger.warning(
"%s is set but HEADROOM_CCR_BACKEND=%s: CCR markers in a session's "
"transcript resolve only on the replica that made them",
STORE_ENV,
ccr,
)
return SharedSessions(client)
+53 -3
View File
@@ -10701,9 +10701,6 @@ class OpenAIHandlerMixin:
session_tracker = self.session_tracker_store.get_or_create(
_session_key, _tracker_provider
)
from headroom.cache.prefix_tracker import seed_restored_compression_cache
seed_restored_compression_cache(session_tracker, comp_cache)
# A claimed turn prepares its executor-side work here (tokenizer,
# hooks, scope binding) — CPU-bound parts run inside the closures
@@ -10764,6 +10761,21 @@ class OpenAIHandlerMixin:
"(a previous turn for this session is still running)"
)
try:
nonlocal session_tracker
from headroom.cache.prefix_tracker import seed_restored_compression_cache
shared = self.shared_compress_sessions
if shared is not None:
# Another replica may have served this session's last
# turn: take its tracker when it is newer than ours.
newer = shared.fetch(
self.session_tracker_store, _session_key, session_tracker
)
if newer is not None:
# The key is present (get_or_create above).
self.session_tracker_store.install(_session_key, newer)
session_tracker = newer
seed_restored_compression_cache(session_tracker, comp_cache)
prev_original = session_tracker.get_last_original_messages()
prev_returned = session_tracker.get_last_forwarded_messages()
prep = prepare_turn(
@@ -10838,6 +10850,8 @@ class OpenAIHandlerMixin:
# return time because whatever we hand back IS what the
# caller sends upstream.
session_tracker.record_returned(messages, final)
if shared is not None:
shared.publish(self.session_tracker_store, _session_key, session_tracker)
info = {
"id": session_id,
"frozen_message_count": session_frozen,
@@ -11149,6 +11163,40 @@ class OpenAIHandlerMixin:
# from here.
_session_key = f"compress\x00{session_id}"
tracker = self.session_tracker_store.peek(_session_key)
shared = self.shared_compress_sessions
if shared is not None:
# The session's last turn may have run on another replica. Fetched
# under the turn lock (a local turn in flight must finish first),
# installed here on the loop: the key may be new to this store.
_local = tracker
_lock_cache = self._peek_compression_cache(_session_key)
def _fetch():
lock = _lock_cache.session_turn_lock if _lock_cache is not None else None
if lock is not None and not lock.acquire(
timeout=_SESSION_TURN_LOCK_TIMEOUT_SECONDS
):
return None
try:
return shared.fetch(self.session_tracker_store, _session_key, _local)
finally:
if lock is not None:
lock.release()
try:
newer = await self._run_compression_in_executor(
_fetch, timeout=COMPRESSION_TIMEOUT_SECONDS
)
except TimeoutError:
newer = None # answer from this replica's state, as without the store
if newer is not None:
current = self.session_tracker_store.peek(_session_key)
if current is _local:
self.session_tracker_store.install(_session_key, newer)
tracker = newer
else:
# A local turn installed a tracker meanwhile: it is newer.
tracker = current
if tracker is None:
return self._compress_usage_unknown_session(session_id)
# No create, no LRU bump: the cache is only needed for its turn lock.
@@ -11195,6 +11243,8 @@ class OpenAIHandlerMixin:
messages=last_returned,
original_messages=tracker.get_last_original_messages(),
)
if shared is not None:
shared.publish(self.session_tracker_store, _session_key, tracker)
return tracker.get_frozen_message_count()
finally:
if lock is not None:
+5
View File
@@ -1207,6 +1207,11 @@ class HeadroomProxy(
session_ttl_seconds=config.prefix_freeze_session_ttl,
)
)
# Session-aware /v1/compress state in Redis, so any replica serves any
# turn (HEADROOM_COMPRESS_SESSION_STORE); None keeps it in memory.
from headroom.cache.shared_sessions import open_shared_sessions
self.shared_compress_sessions = open_shared_sessions()
# Set by startup() when HEADROOM_PREFIX_STATE_FILE opts in.
self._prefix_state_file: str | None = None
+3 -2
View File
@@ -300,7 +300,8 @@ bedrock = [
"boto3>=1.41.0",
"botocore[crt]>=1.41.0",
]
# State shared across replicas: CCR store (HEADROOM_CCR_BACKEND=redis).
# State shared across replicas: CCR store (HEADROOM_CCR_BACKEND=redis) and
# session-aware /v1/compress (HEADROOM_COMPRESS_SESSION_STORE=redis://...).
redis = [
"redis>=5.0.0",
]
@@ -336,7 +337,7 @@ dev = [
"sentence-transformers>=2.2.0,<7.0",
"numpy>=1.24.0",
"openpyxl>=3.1.0", # exercises spreadsheet_ingest (.xlsx) in the test suite
"fakeredis>=2.20.0", # Redis-backed shared state, without a Redis server
"fakeredis[lua]>=2.20.0", # Redis-backed shared state, without a Redis server
"xlrd>=2.0.1", # exercises spreadsheet_ingest (.xls) in the test suite
"xlwt>=1.3.0", # writes the legacy .xls fixtures for spreadsheet_ingest tests
# test_upstream_guard builds a self-signed cert to prove the pinned connection
+342
View File
@@ -0,0 +1,342 @@
"""Session-aware /v1/compress served by several replicas sharing Redis.
A gateway without sticky sessions sends each turn to any replica. With
HEADROOM_COMPRESS_SESSION_STORE set, a replica that did not serve the last turn
takes the session's state from Redis, freezes the prefix the caller already
forwarded, and returns it byte for byte.
"""
from __future__ import annotations
import json
import pytest
pytest.importorskip("fastapi")
fakeredis = pytest.importorskip("fakeredis")
pytest.importorskip("lupa")
from fastapi.testclient import TestClient # noqa: E402
from headroom.cache import shared_sessions # noqa: E402
from headroom.cache.backends.redis import RedisBackend # noqa: E402
from headroom.cache.compression_store import _create_default_ccr_backend # noqa: E402
from headroom.exceptions import ConfigurationError # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
from tests.test_compress_session_mode import _big_tool_history, _compress # noqa: E402
@pytest.fixture
def redis_server(monkeypatch):
import redis
server = fakeredis.FakeServer()
monkeypatch.setenv(shared_sessions.STORE_ENV, "redis://127.0.0.1:6399/0")
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory") # CCR sharing is tested on its own
monkeypatch.setattr(
redis.Redis,
"from_url",
classmethod(lambda cls, *a, **k: fakeredis.FakeRedis(server=server)),
)
return fakeredis.FakeRedis(server=server)
def _replica() -> TestClient:
config = ProxyConfig(
optimize=True,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
image_optimize=False,
)
return TestClient(create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 1))
def _second_tool_result() -> list[dict]:
rows = [
{"id": i, "path": f"src/mod_{i}.py", "line": i * 3, "hit": "x" * 200} for i in range(150)
]
return [
{
"role": "assistant",
"content": None,
"tool_calls": [
{"id": "c2", "type": "function", "function": {"name": "grep", "arguments": "{}"}}
],
},
{"role": "tool", "tool_call_id": "c2", "content": json.dumps(rows)},
]
def test_any_replica_serves_the_next_turn_byte_for_byte(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b:
turn1 = _compress(a, history, session_id="conv")
assert turn1["messages"][2]["content"] != history[2]["content"] # compressed
# Turn 2 on B adds a second tool result, compressed by B.
history2 = history + _second_tool_result()
turn2 = _compress(b, history2, session_id="conv")
assert turn2["messages"][: len(turn1["messages"])] == turn1["messages"]
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
assert turn2["messages"][4]["content"] != history2[4]["content"]
# Back on A, which last saw turn 1: only B's state has B's bytes.
history3 = history2 + [{"role": "user", "content": "again"}]
turn3 = _compress(a, history3, session_id="conv")
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
def test_without_the_store_a_new_replica_starts_cold(monkeypatch):
# What the store fixes: B never saw the compressed tool result, so it does
# not freeze it and compresses it again.
monkeypatch.delenv(shared_sessions.STORE_ENV, raising=False)
history = _big_tool_history()
with _replica() as a, _replica() as b:
turn1 = _compress(a, history, session_id="conv")
turn2 = _compress(b, history + [{"role": "user", "content": "x"}], session_id="conv")
assert turn2["session"]["frozen_message_count"] < len(turn1["messages"])
def test_a_turn_writes_only_the_messages_redis_lacks(redis_server):
history = _big_tool_history()
with _replica() as a:
_compress(a, history, session_id="conv-delta")
(msgs_key,) = redis_server.keys("headroom:session:*:msgs")
before = set(redis_server.hkeys(msgs_key))
import redis
written: list[int] = []
real_hset = redis.client.Pipeline.hset
def counting_hset(self, name, *args, mapping=None, **kwargs):
written.append(len(mapping or {}))
return real_hset(self, name, *args, mapping=mapping, **kwargs)
redis.client.Pipeline.hset = counting_hset
try:
_compress(
a, history + [{"role": "user", "content": "one more"}], session_id="conv-delta"
)
finally:
redis.client.Pipeline.hset = real_hset
after = set(redis_server.hkeys(msgs_key))
# The new user message, and at most the few whose cache marker moved.
assert 1 <= sum(written) <= 3 < len(before)
assert before <= after
def test_a_lost_key_is_stored_whole_again(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b:
_compress(a, history, session_id="conv-lost")
redis_server.flushall()
turn2 = _compress(a, history + [{"role": "user", "content": "y"}], session_id="conv-lost")
assert redis_server.keys("headroom:session:*:meta")
# And another replica is warm from it.
turn3 = _compress(
b,
history + [{"role": "user", "content": "y"}, {"role": "user", "content": "z"}],
session_id="conv-lost",
)
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
def test_the_usage_relay_reaches_and_publishes_a_session_served_elsewhere(redis_server):
with _replica() as a, _replica() as b:
_compress(a, _big_tool_history(), session_id="conv-usage")
(meta_key,) = redis_server.keys("headroom:session:*:meta")
token = redis_server.hget(meta_key, "t")
resp = b.post(
"/v1/usage",
json={
"session_id": "conv-usage",
"usage": {"cache_read_input_tokens": 0, "cache_creation_input_tokens": 50_000},
},
)
assert resp.status_code == 200, resp.text
assert redis_server.hget(meta_key, "t") != token # B published its update
def test_a_redis_outage_or_corrupt_state_never_fails_a_turn(redis_server, monkeypatch):
import redis
history = _big_tool_history()
with _replica() as a, _replica() as b:
turn1 = _compress(a, history, session_id="conv-down")
(meta_key,) = redis_server.keys("headroom:session:*:meta")
redis_server.hset(meta_key, "m", "{not json")
turn2 = _compress(b, history + [{"role": "user", "content": "y"}], session_id="conv-down")
assert turn2["messages"][0] == turn1["messages"][0]
def down(*args, **kwargs):
raise redis.ConnectionError("connection refused")
monkeypatch.setattr(fakeredis.FakeRedis, "execute_command", down)
turn3 = _compress(a, history + [{"role": "user", "content": "z"}], session_id="conv-down")
assert turn3["messages"][: len(turn1["messages"])] == turn1["messages"]
def test_sharing_sessions_shares_the_ccr_store_too(monkeypatch):
monkeypatch.setenv(shared_sessions.STORE_ENV, "redis://127.0.0.1:6399/0")
monkeypatch.delenv("HEADROOM_CCR_BACKEND", raising=False)
assert isinstance(_create_default_ccr_backend(), RedisBackend)
def test_a_store_url_that_is_not_redis_stops_startup(monkeypatch):
monkeypatch.setenv(shared_sessions.STORE_ENV, "memcached://cache:11211")
with pytest.raises(ConfigurationError, match="redis://"):
shared_sessions.open_shared_sessions()
def _meta_key(redis_server):
(meta_key,) = redis_server.keys("headroom:session:*:meta")
return meta_key
def test_messages_evicted_on_their_own_are_stored_whole_again(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b:
_compress(a, history, session_id="conv-evict")
meta_key = _meta_key(redis_server)
redis_server.delete(meta_key.replace(b":meta", b":msgs")) # maxmemory took one key
history2 = history + [{"role": "user", "content": "y"}]
turn2 = _compress(a, history2, session_id="conv-evict")
turn3 = _compress(b, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict")
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
def test_corrupt_state_is_rewritten_by_the_next_replica_to_write(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b, _replica() as c:
_compress(a, history, session_id="conv-corrupt")
redis_server.hset(_meta_key(redis_server), "m", "{not json")
history2 = history + [{"role": "user", "content": "y"}]
turn2 = _compress(b, history2, session_id="conv-corrupt") # repairs it
turn3 = _compress(
c, history2 + [{"role": "user", "content": "z"}], session_id="conv-corrupt"
)
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
def test_messages_no_transcript_references_are_pruned(redis_server):
history = _big_tool_history()
with _replica() as a:
_compress(a, history, session_id="conv-prune")
msgs_key = _meta_key(redis_server).replace(b":meta", b":msgs")
redis_server.hset(msgs_key, mapping={f"stale{i}": "{}" for i in range(50)})
_compress(a, history + [{"role": "user", "content": "y"}], session_id="conv-prune")
assert not any(k.startswith(b"stale") for k in redis_server.hkeys(msgs_key))
def test_a_writer_whose_clock_runs_ahead_is_still_trusted(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b:
turn1 = _compress(a, history, session_id="conv-skew")
meta_key = _meta_key(redis_server)
meta = json.loads(redis_server.hget(meta_key, "m"))
meta["last_activity"] += 30 # A's clock is 30 s ahead of B's
redis_server.hset(meta_key, "m", json.dumps(meta))
turn2 = _compress(b, history + [{"role": "user", "content": "y"}], session_id="conv-skew")
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
def test_a_lost_message_is_noticed_even_beside_unreferenced_ones(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b:
_compress(a, history, session_id="conv-masked")
msgs_key = _meta_key(redis_server).replace(b":meta", b":msgs")
# Old variants left behind outnumber the one referenced message lost.
redis_server.hset(msgs_key, mapping={f"old{i}": "{}" for i in range(5)})
redis_server.hdel(msgs_key, redis_server.hkeys(msgs_key)[0])
history2 = history + [{"role": "user", "content": "y"}]
turn2 = _compress(a, history2, session_id="conv-masked")
turn3 = _compress(
b, history2 + [{"role": "user", "content": "z"}], session_id="conv-masked"
)
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
def test_a_corrupt_token_never_fails_a_turn_and_is_replaced(redis_server):
history = _big_tool_history()
with _replica() as a, _replica() as b:
_compress(a, history, session_id="conv-token")
redis_server.hset(_meta_key(redis_server), "t", b"\xff")
history2 = history + [{"role": "user", "content": "y"}]
turn2 = _compress(b, history2, session_id="conv-token")
turn3 = _compress(a, history2 + [{"role": "user", "content": "z"}], session_id="conv-token")
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
@pytest.fixture
def evict_before_cas(monkeypatch):
"""Delete the messages hash just before the next ``times`` compare-and-set
scripts run, as maxmemory might between the message writes and the script."""
budget = {"times": 0}
real_init = shared_sessions.SharedSessions.__init__
def init(self, client):
real_init(self, client)
cas = self._cas
def evicting_cas(keys, args):
if budget["times"]:
budget["times"] -= 1
client.delete(keys[1])
return cas(keys=keys, args=args)
self._cas = evicting_cas
monkeypatch.setattr(shared_sessions.SharedSessions, "__init__", init)
return budget
def test_eviction_before_the_first_publish_is_never_acknowledged(redis_server, evict_before_cas):
history = _big_tool_history()
with _replica() as a, _replica() as b:
evict_before_cas["times"] = 1
turn1 = _compress(a, history, session_id="conv-evict-first")
turn2 = _compress(
b, history + [{"role": "user", "content": "y"}], session_id="conv-evict-first"
)
assert turn2["messages"][: len(turn1["messages"])] == turn1["messages"]
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
@pytest.mark.parametrize("corrupt", [False, True], ids=["delta", "whole-rewrite"])
def test_eviction_before_a_later_publish_is_never_acknowledged(
redis_server, evict_before_cas, corrupt
):
history = _big_tool_history()
with _replica() as a, _replica() as b, _replica() as c:
_compress(a, history, session_id="conv-evict-later")
if corrupt: # B then rewrites the session whole, assuming no message stored
redis_server.hset(_meta_key(redis_server), "m", "{not json")
history2 = history + [{"role": "user", "content": "y"}]
evict_before_cas["times"] = 1
turn2 = _compress(b, history2, session_id="conv-evict-later")
turn3 = _compress(
c, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict-later"
)
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
def test_a_session_whose_messages_keep_vanishing_is_not_published(redis_server, evict_before_cas):
history = _big_tool_history()
with _replica() as a, _replica() as b:
evict_before_cas["times"] = 2 # the write and its whole rewrite
_compress(a, history, session_id="conv-evict-twice")
assert not redis_server.keys("headroom:session:*:meta")
# Once Redis keeps the messages again, the next turn publishes it whole.
history2 = history + [{"role": "user", "content": "y"}]
turn2 = _compress(a, history2, session_id="conv-evict-twice")
turn3 = _compress(
b, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict-twice"
)
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
Generated
+70 -2
View File
@@ -1153,6 +1153,11 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c7/e4/6919d3653d72c53d1fb22c97ceb6fa3664cad302994e90ee52279f7eb394/fakeredis-2.40.0-py3-none-any.whl", hash = "sha256:b155ef2442134372eb1cc5664cf5638ccbe0a6dde9d1942153708e2782f315c9", size = 204148, upload-time = "2026-10-14T12:46:00.014Z" },
]
[package.optional-dependencies]
lua = [
{ name = "lupa" },
]
[[package]]
name = "fastapi"
version = "0.136.3"
@@ -1705,7 +1710,7 @@ code = [
dev = [
{ name = "anthropic" },
{ name = "cryptography" },
{ name = "fakeredis" },
{ name = "fakeredis", extra = ["lua"] },
{ name = "fastapi" },
{ name = "hnswlib" },
{ name = "httpx", extra = ["http2", "socks"] },
@@ -1903,7 +1908,7 @@ requires-dist = [
{ name = "cryptography", marker = "extra == 'dev'", specifier = ">=50.0.0" },
{ name = "datasets", marker = "extra == 'evals'", specifier = ">=5.0.1" },
{ name = "datasets", marker = "extra == 'voice-train'", specifier = ">=5.0.1" },
{ name = "fakeredis", marker = "extra == 'dev'", specifier = ">=2.20.0" },
{ name = "fakeredis", extras = ["lua"], marker = "extra == 'dev'", specifier = ">=2.20.0" },
{ name = "fastapi", marker = "extra == 'dev'", specifier = ">=0.100.0" },
{ name = "fastapi", marker = "extra == 'proxy'", specifier = ">=0.100.0" },
{ name = "fastembed", marker = "(python_full_version < '3.14' and extra == 'relevance') or (platform_machine != 'x86_64' and extra == 'relevance') or (sys_platform != 'darwin' and extra == 'relevance')", specifier = ">=0.6.0" },
@@ -2651,6 +2656,69 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },
]
[[package]]
name = "lupa"
version = "2.8"
source = { registry = "https://pypi.org/simple/" }
sdist = { url = "https://files.pythonhosted.org/packages/c3/a6/0f869fbb07c393f15473b1eefefb7b5bec162fb7481803d040ed4dc46002/lupa-2.8.tar.gz", hash = "sha256:d8022641b9ec8ecf2c5ecbe9f47e5a70e0b87c4b5ae921b92cb02a638e0acd08", size = 6156370, upload-time = "2026-04-15T20:08:30.534Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/09/21/9be4516ddd22f8eadba336d9ba065d17d79108465ae1b7f71424ab99b9d0/lupa-2.8-cp310-abi3-win32.whl", hash = "sha256:c2a5fd15dc62374e1661a55f01744c9ec1c56f291ba4a0749d3af2174556e78f", size = 1594887, upload-time = "2026-04-15T20:05:23.377Z" },
{ url = "https://files.pythonhosted.org/packages/2d/99/1557c9685d7034d9ce8dd2b54c40a26d6deb7c67c1fdb5c801abd1a02c3f/lupa-2.8-cp310-abi3-win_arm64.whl", hash = "sha256:9e304fb1c50cf23fd8882afbe1aa87525ef8a72667bcab3b37b2bbb2bc542269", size = 1371742, upload-time = "2026-04-15T20:05:27.417Z" },
{ url = "https://files.pythonhosted.org/packages/1c/34/05ce4745b191633f90ff1ab50f1a19a37da282bb0a41fb500d9157fc9b8f/lupa-2.8-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:97bd01e90b8031e56a5fd5bb70605aea09f1dba675c1140308a52780f93d06f1", size = 1202714, upload-time = "2026-04-15T20:05:31.088Z" },
{ url = "https://files.pythonhosted.org/packages/7d/d2/f70fdbeec2d4c69ee6a469e6cddde9635fff4af4e13fb652e6a1229eef51/lupa-2.8-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b5ebe1a13c45767919c86750b84fe2da9f6288b6f3cea4ce7660bb2abc9d921", size = 1857453, upload-time = "2026-04-15T20:05:34.611Z" },
{ url = "https://files.pythonhosted.org/packages/97/dc/6fcda0e36e75eb6cb98dc9190fa4737d727eeae29e58f892980b2c96b656/lupa-2.8-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:097e7d0f1719a88020b67c82e05d53d7973c166952393afcecfd8434c7e19a15", size = 2408890, upload-time = "2026-04-15T20:05:37.994Z" },
{ url = "https://files.pythonhosted.org/packages/58/29/7ea176eac3c1dac83d059762daa875ad1390decc0bf2c3b4c7bbfc1f1665/lupa-2.8-cp310-cp310-win_amd64.whl", hash = "sha256:7bb223ee8f72d0dc076b0d65296ee72f1c69450f9d2fed5315f7707d98c4a03d", size = 1910396, upload-time = "2026-04-15T20:05:41.163Z" },
{ url = "https://files.pythonhosted.org/packages/b7/0a/5a740717f27aa77481e6a61b97cf79d1e0c1ede729b1268caacded915326/lupa-2.8-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:b12e43c1fb787189dfc28cd604aef0baa2cb95e27da19498d520361d0ace070a", size = 1202376, upload-time = "2026-04-15T20:05:44.049Z" },
{ url = "https://files.pythonhosted.org/packages/1b/75/6b64d0098c64275a801896cb7a6a30e7e653d25fa102c64e747292afcdbb/lupa-2.8-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6f603391dffb256e36a79fd2044084d5f4b8a0a4c0e5ad291cd3ab3aaf1fd0a", size = 1839271, upload-time = "2026-04-15T20:05:47.399Z" },
{ url = "https://files.pythonhosted.org/packages/7b/2f/0d4f00563046ff616ef6a421f8b776a5ffb327f7b32ed69e856d52b917a8/lupa-2.8-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f6f41c91366e7d0d474f87d81c1274af861f40812bf729c9f97ab4c8f3c7ac8", size = 2376251, upload-time = "2026-04-15T20:05:49.891Z" },
{ url = "https://files.pythonhosted.org/packages/4c/8e/caa83237f427d9e85b7f02c816e7270c9c9571dec1673e06b0180402f70e/lupa-2.8-cp311-cp311-win_amd64.whl", hash = "sha256:f5a6af145b0ea818f01d27bfe2583a4b538570bef61d22c8773e0eccf011234c", size = 1923488, upload-time = "2026-04-15T20:05:52.954Z" },
{ url = "https://files.pythonhosted.org/packages/ad/0b/368f2f0bc750b25c69d4563e44f677925ab5dd3d2887f9b0c15465d21a2a/lupa-2.8-cp312-abi3-macosx_10_13_x86_64.whl", hash = "sha256:f4342f4de76ae7ce2ab0672d36003bdb7e1a33252f293b569298ddd792e70e33", size = 1194056, upload-time = "2026-04-15T20:05:55.794Z" },
{ url = "https://files.pythonhosted.org/packages/5b/0f/c89eb8dd36fdea4e50ae3f7f5275bea3b0cc5d4057b8ee7b3bbc78010422/lupa-2.8-cp312-abi3-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:4203fa1659315e939a5304e75001b8cc14234fb3cbb3ed86c049b0cc5d90fcee", size = 1434278, upload-time = "2026-04-15T20:05:57.94Z" },
{ url = "https://files.pythonhosted.org/packages/47/30/c3b4d2cd8733621b404b8a4214e5f852955c4ba632546dc84123bea9ee89/lupa-2.8-cp312-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:81f2d843ce668b653146c007467570210ae44be51dac6926666c51d49536f307", size = 1150068, upload-time = "2026-04-15T20:06:01.04Z" },
{ url = "https://files.pythonhosted.org/packages/8d/d2/bac12c398519efafc6af84be1974edd0d7a4895fb4735b5c8d615d298595/lupa-2.8-cp312-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d3d0cde2c77588d1c60875a4f34f059513476c6e1775351897195b51e0f3df08", size = 1409532, upload-time = "2026-04-15T20:06:03.592Z" },
{ url = "https://files.pythonhosted.org/packages/9c/6a/18b52e11962014026e07813530b0b108ee8bc0a2a13ef0eaea5d41dce023/lupa-2.8-cp312-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9e0d11b8f3a8dac6413f704fef7161d048bb10c58bdac6cbffa5e60efa56e9a3", size = 1242687, upload-time = "2026-04-15T20:06:06.863Z" },
{ url = "https://files.pythonhosted.org/packages/b3/8e/7fd4eb049875f61429b96780d2eae4700f0e78fe0a52db8edb231b1cd09f/lupa-2.8-cp312-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:54cff414f21f8cd8c6be4aae52541f3b9cd39602b59e3a3db9b5c9f9f674ff18", size = 1856038, upload-time = "2026-04-15T20:06:09.358Z" },
{ url = "https://files.pythonhosted.org/packages/e9/f9/37ad9d2773d30f2931890d310a4bdce28d45484206e6f48bc18b0325eabd/lupa-2.8-cp312-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:24b4d8af5558e549b70daf1547f5c1c1d664ecea9fc790f83efe5d75e9a93797", size = 1128982, upload-time = "2026-04-15T20:06:12.312Z" },
{ url = "https://files.pythonhosted.org/packages/57/31/c0fd7984c24844ea79caa45c0235f61a06b38fd69a839f6c62770f8d684a/lupa-2.8-cp312-abi3-musllinux_1_2_i686.whl", hash = "sha256:ce86dff1ee7f7cf45f5622065ae991949dd7bb1703581cbc58a630137bb7ccf9", size = 1457594, upload-time = "2026-04-15T20:06:15.881Z" },
{ url = "https://files.pythonhosted.org/packages/11/f5/a28e411be30ec1bf0db1eb0c087eebc73be9e7a1adcfe6ac209861ccc446/lupa-2.8-cp312-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:f4d01b2a08c70bbb883a9e082b6b36b89121ed5910b710f1ba11c73295ff4fba", size = 1425721, upload-time = "2026-04-15T20:06:18.009Z" },
{ url = "https://files.pythonhosted.org/packages/ed/c1/359f767c4ae024be30d909fe8a9f0e9af266bad47ce2bd2ed248fb986fcf/lupa-2.8-cp312-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:7f210d5a8353e510ea1199c42cf3cbdd630553bf2bc8fb4c00fea06fdec7c798", size = 1253258, upload-time = "2026-04-15T20:06:21.17Z" },
{ url = "https://files.pythonhosted.org/packages/17/52/473f11790c261fd02bbf318a546fe040e9ec9f677181272fa78d3b4112a4/lupa-2.8-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4f81a02806e7c7ad26d8c6fa222c8bef1b0c1b124347c879be880b41339d41e4", size = 2395272, upload-time = "2026-04-15T20:06:24.137Z" },
{ url = "https://files.pythonhosted.org/packages/94/bf/75c8795655a8836eab6a11a630352c4b7c5dc5c54d075077bc9bffdeee45/lupa-2.8-cp312-abi3-win32.whl", hash = "sha256:360056453a7a4eaa4ac5a204c31a5a014b1eb2ee5490603234d2ba831684f1f2", size = 1606136, upload-time = "2026-04-15T20:06:27.815Z" },
{ url = "https://files.pythonhosted.org/packages/d8/29/11a2cdd612b6f55e506292dfb6ba343216e80a693e7fe3f876ef204ce9c6/lupa-2.8-cp312-abi3-win_arm64.whl", hash = "sha256:1628371c6592a6d5650497a9e31fb2bb3a7e9883c1f301d1111265e484045af9", size = 1364495, upload-time = "2026-04-15T20:06:30.254Z" },
{ url = "https://files.pythonhosted.org/packages/4d/17/fa834b6b09ad17e7df5d0f7715d64877a125a3776ada689751a1f9dc2959/lupa-2.8-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:450650f91c48c2415b0d59ab3abfcfda3b6efb5b858205f4d4bda8ad141fa529", size = 1190111, upload-time = "2026-04-15T20:06:32.84Z" },
{ url = "https://files.pythonhosted.org/packages/ab/43/45589901b7d1a0e3a9d91d19a311fb6a56924e8571536c3f2212160fd953/lupa-2.8-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:27044f3363047f946b3d3aab9157cbd172b3538ada9ec1baef43432bf7d03a78", size = 1812999, upload-time = "2026-04-15T20:06:35.664Z" },
{ url = "https://files.pythonhosted.org/packages/a1/ac/4ade7d15ff5c61758d7943ac6f0a496bf1cc65b6c09f842b52a0702e664c/lupa-2.8-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8cf4f064a0e5531afce2d7d750120c10c10f9529139af6ca6150d13151034398", size = 2368731, upload-time = "2026-04-15T20:06:37.959Z" },
{ url = "https://files.pythonhosted.org/packages/0c/27/05f950d15b8ab120b39c43588b438ff3ace70c1b1b0225a960393a497483/lupa-2.8-cp312-cp312-win_amd64.whl", hash = "sha256:281bedc5deb92d31e649a3552edd662449365a635904fa4d5cb4509c7245e34e", size = 1941809, upload-time = "2026-04-15T20:06:40.302Z" },
{ url = "https://files.pythonhosted.org/packages/a6/3f/19f83c3a0c84dc8bea8a58e7416dca6a3ede662c33c8d1ec758e5afc754a/lupa-2.8-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:45fc9da0145ecb0083ef5ff9975116cc784bd0258bdc2bd131ba15483ce18398", size = 1201203, upload-time = "2026-04-15T20:06:42.169Z" },
{ url = "https://files.pythonhosted.org/packages/89/0f/a14f0073f09610158038582e230618a48c14da6bd88185289461aa4cb854/lupa-2.8-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:58e18afed57955b41130e269c78f53d4123ab86e236b53816f4cbffa25cb5d30", size = 1806210, upload-time = "2026-04-15T20:06:45.486Z" },
{ url = "https://files.pythonhosted.org/packages/2f/14/48fff156c63a136001a7620878af7d31aa07e66b495ed621e3eddd73c294/lupa-2.8-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fc47f536ac13a79cef47d29a2b205576a22841f042a2bcec1676b95806e7706a", size = 2359005, upload-time = "2026-04-15T20:06:47.819Z" },
{ url = "https://files.pythonhosted.org/packages/fe/18/3ac638ec90edf178242b8a2b2f00f8adae694248c03a26341ef941bb746e/lupa-2.8-cp313-cp313-win_amd64.whl", hash = "sha256:ce9404c661dbac65cc9bed351ad45e797af93d30d70be309a3fa8209ac86d93b", size = 1936754, upload-time = "2026-04-15T20:06:50.448Z" },
{ url = "https://files.pythonhosted.org/packages/b0/ef/5ee5fed6ea7459a671196359ce04bfeeaf26be1dac8ff24bf28e5c7a6e81/lupa-2.8-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:348c3f8ecabb6324dcbc05c2740d762ef8fcec7b06c79e45262ab97a217684e3", size = 1209388, upload-time = "2026-04-15T20:06:53.022Z" },
{ url = "https://files.pythonhosted.org/packages/6e/b1/67a940d5542cb0384b443fe951b5a83ea9340d1333a733a258fdd1c619ba/lupa-2.8-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:951496471056061598a7d1729a6cdf48d662fec777a9f2d8aa5a1e62fd30e5a5", size = 1826821, upload-time = "2026-04-15T20:06:55.699Z" },
{ url = "https://files.pythonhosted.org/packages/a1/a2/b354e5ba3b911ec50686003dc8897e892b9e8c5c036b33219b03d54c4daf/lupa-2.8-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a591b9947ca347b41a63370e121d6e2b1458fe6dde9ae065029ec10a37f25ff4", size = 2366893, upload-time = "2026-04-15T20:06:58.9Z" },
{ url = "https://files.pythonhosted.org/packages/8e/52/d76066401f29539df5352f70ecded66576f32933b6045cd0bfc56cb770b9/lupa-2.8-cp314-cp314-win_amd64.whl", hash = "sha256:3903c9cf628dae2f56405503247b77a61a3a61bd2dda470e336950c74776d55d", size = 1994716, upload-time = "2026-04-15T20:07:19.194Z" },
{ url = "https://files.pythonhosted.org/packages/c3/bd/3efc437a4361c16d25e66478c50357c9a8e8ecfb718fe749eb9ca3176ef6/lupa-2.8-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f711a8ab0486b9ac6fdda94a22ddcfbc9f0d4a27e3a8cf1bf79c6e48b33017c1", size = 1251217, upload-time = "2026-04-15T20:07:01.64Z" },
{ url = "https://files.pythonhosted.org/packages/ea/f4/2e9f8ecbaca854bfdf14af8a9b505ec0cbc640377b3b218921594b7563cd/lupa-2.8-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dc51250e76367a3e27fcd01dc769b9bfcbbc34f48df48dde53d6af6e75b7eaa5", size = 1814701, upload-time = "2026-04-15T20:07:04.149Z" },
{ url = "https://files.pythonhosted.org/packages/ba/53/4000b1acaa8b1f3827fcff0cfcdff44d3befddda42cab7e685a49689b5a1/lupa-2.8-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8a22088a552828958603323f0a5c4b3e11e03b75d0bf4c965ef879de9b60a8d", size = 2348414, upload-time = "2026-04-15T20:07:07.285Z" },
{ url = "https://files.pythonhosted.org/packages/d5/78/26ee48d3890cddf03cefb65f433e3492759c0b3c0582180755bddbaab7bd/lupa-2.8-cp314-cp314t-win32.whl", hash = "sha256:4f7c553c1d8cfffbe85d81daef730d12cae4b6002d457542914da0ac8a1145b3", size = 1831611, upload-time = "2026-04-15T20:07:09.752Z" },
{ url = "https://files.pythonhosted.org/packages/3c/d1/4a5cc64a3cad22821ae4c3f7a90456a08ca19457d8354f4abf46ad03c7e8/lupa-2.8-cp314-cp314t-win_amd64.whl", hash = "sha256:d8766aff03a78c80ad2d188a8bdb216de5ec838359cd87e05bbdfa56394a6105", size = 2209250, upload-time = "2026-04-15T20:07:11.906Z" },
{ url = "https://files.pythonhosted.org/packages/37/7c/cdcb654daf668192aaf36b0aeb94f2281dad092aaa5003688691131736ea/lupa-2.8-cp314-cp314t-win_arm64.whl", hash = "sha256:91d622777febda3ab1bed1d45295f2f32a4680c7b3d7caf8c669998ed5c44118", size = 1126735, upload-time = "2026-04-15T20:07:15.434Z" },
{ url = "https://files.pythonhosted.org/packages/1d/44/de1961ad38e17cd326a53c246c7e3b91178ed578f4cf22ffcd5e7e11b041/lupa-2.8-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:b036738282a5acd2e71fdddb317c9df8b87c1673aa57f403d05fcc2be8abc4ba", size = 1186020, upload-time = "2026-04-15T20:07:35.017Z" },
{ url = "https://files.pythonhosted.org/packages/13/c2/276f0b9dc8bcc5a8a58af5316dfa0e6f56be3613dd6dbcc8d3d2cb6559ba/lupa-2.8-cp39-abi3-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:ac6b6e8d0e617e26a98cbb44880bcd75de5d32b3ad7b3b3793583909292b47ed", size = 1468944, upload-time = "2026-04-15T20:07:37.782Z" },
{ url = "https://files.pythonhosted.org/packages/63/38/52934e52a5180dc6425d20284d004fe4b27a4f9171a82dc99fb67af250bf/lupa-2.8-cp39-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:ba3a7dd839f90c3d2e53bebe3c192b1f3f9fd720a6781256405123211fd0dce6", size = 1172998, upload-time = "2026-04-15T20:07:40.812Z" },
{ url = "https://files.pythonhosted.org/packages/c7/82/76b3809bd0839d9b3b4ec58d06591e08f17337b6d9576877cb9d48b34e94/lupa-2.8-cp39-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d7edb13a7a5250b5c6c22d1495d9e842b5c9fc5081c8fe6b5efe2112fe3e41f9", size = 1449975, upload-time = "2026-04-15T20:07:44.262Z" },
{ url = "https://files.pythonhosted.org/packages/16/07/2f89d54f747c67c23b4b9ae4aa8c8dd06bb409155dedcf406157f2736b66/lupa-2.8-cp39-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:891f72e0bffbed1e4175f975aeb2a083956586a100066525e1be485f617f7b25", size = 1281944, upload-time = "2026-04-15T20:07:46.458Z" },
{ url = "https://files.pythonhosted.org/packages/e7/bd/7375d2b0fcae79d806baf52a76f26c96964593f58e1372d13ae5ac09c676/lupa-2.8-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:a295f87b5b7ebbfd5191932e8cb0e51df3c7769101ac6b6c7d7c9fb27bfd1307", size = 1910455, upload-time = "2026-04-15T20:07:49.75Z" },
{ url = "https://files.pythonhosted.org/packages/8b/0c/8abb3bc0e08b311fc01db05b6e9f9ff31a8f65e4fc3f0aeb05cfef75c8ac/lupa-2.8-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4fe5d7a810b64ea8511eb885fc8cdde042ee5ff7b7d08ae78f32449756acb177", size = 1155548, upload-time = "2026-04-15T20:07:52.657Z" },
{ url = "https://files.pythonhosted.org/packages/80/2e/9eeecd3f493099721c1d3f31beeca23a4237db1a54223684df4dc96aa1bd/lupa-2.8-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:bfc470012ef66ad064c7bd77416af03a3452ef630b04b9012595ea13f2e54518", size = 1489232, upload-time = "2026-04-15T20:07:54.92Z" },
{ url = "https://files.pythonhosted.org/packages/c3/13/731c99dc2e7652ae818a6de45bdf0142049f7cb566049061c898355f1891/lupa-2.8-cp39-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:250e035fdaffe8c87093e3ebc206ac29a26131b1568ea711d780c26001ce96e7", size = 1466321, upload-time = "2026-04-15T20:07:57.627Z" },
{ url = "https://files.pythonhosted.org/packages/de/71/3ad8cc4fc05a77dc0d3f7079348bd1cad4675a0d14c24f8e6a3ce5f008f7/lupa-2.8-cp39-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:b9bddb09acfffb4f828f790f444b11dc0cca591afea1a244d9329eea2d20c003", size = 1288577, upload-time = "2026-04-15T20:07:59.913Z" },
{ url = "https://files.pythonhosted.org/packages/d8/b2/1175f6d0aa7b68627fbe2f58bd1e8bea36a89d10dfd67671d2b024c96162/lupa-2.8-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2e64acbbd47e9b82a64405a39e0d2b36a5a7dad8ab41c0f3437f572f7d282ba3", size = 2444866, upload-time = "2026-04-15T20:08:02.753Z" },
{ url = "https://files.pythonhosted.org/packages/92/f7/e78df680c7a0ea452daac07467ca188d63c2c00ca1c884c0a50e27eb83b5/lupa-2.8-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:32e4e5103bbddcdd2458fb2ccae6c8ba11c9997c711d7e379e0d45551d109c76", size = 1778509, upload-time = "2026-04-15T20:08:21.784Z" },
{ url = "https://files.pythonhosted.org/packages/e6/23/0e53cabb16b2a8aa9cf1fde499c097d8942c5dab709fc8e921f3b824b18b/lupa-2.8-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7667001804657496dee9feced2daae5000b4604a3218dd8e6b7b754982ba88b8", size = 2300480, upload-time = "2026-04-15T20:08:24.394Z" },
{ url = "https://files.pythonhosted.org/packages/7e/85/0271227eab939921a12ebba5d17aa4cd18346aa534ca7f5da09cd0b63dd4/lupa-2.8-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:86f6f668966965b15247dc32d064cfe7be67b71e584ccfacbe2f637575296878", size = 1847445, upload-time = "2026-04-15T20:08:27.031Z" },
]
[[package]]
name = "lxml"
version = "6.1.1"