mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(compress): share /v1/compress sessions across replicas via Redis (#4369)
Stacked on #4368 (Redis CCR store); merge that first. ## Why A session-aware `/v1/compress` turn must return the prefix it returned last turn byte for byte, or the provider's prompt cache misses. That state (the session's prefix tracker) lived only in the memory of the replica that served the turn, so a gateway with several replicas had to pin sessions (`session_affinity`). Kong's AI Gateway runs several instances without sticky sessions. ## Change With `HEADROOM_COMPRESS_SESSION_STORE=redis://…` (`headroom/cache/shared_sessions.py`), every replica keeps each session in Redis too: - **Messages, stored once by digest** (`…:msgs`). The original and returned transcripts mostly share messages, and each turn mostly repeats the last one, so a turn writes only what Redis lacks: the messages it added, plus any whose bytes changed (a moved cache marker, an edit, a compaction). Unreferenced messages are pruned. - **Metadata** (`…:meta`): the tracker's counters, its transcripts as digest lists, and a random write token. - **Before a turn**, the replica reads the token. If another replica wrote since, it reads the metadata and only the messages it lacks, each checked against its digest. The turn then seeds its compression cache from that tracker, as after a restart. - **After the turn**, it writes the new messages, then the metadata via a Lua compare-and-set on the token. - A random token, not a counter: a lost key or a failover can't leave a replica trusting stale state. - The script also detects a messages hash that `maxmemory` evicted on its own; the writer then stores the session whole. - A replica that finds corrupt or evicted state claims the token and rewrites the session whole. - **Usage relay:** fetches under the session's turn lock, and installs only if no local turn installed a tracker meanwhile. - **Failures:** a Redis error or corrupt value never fails a turn. The replica serves from its own memory, as without the store. Clock skew between replicas is tolerated. - **CCR:** the setting also selects #4368's Redis CCR store unless `HEADROOM_CCR_BACKEND` names another, so markers in a replayed transcript resolve on any replica. - **`SessionTrackerStore`** gains `encode_shared` / `decode_shared` (the restart file's per-tracker encoding, split into metadata and message table) and `install`. ## Results **Real run:** 3 proxy processes on one Redis-protocol server (fakeredis TCP server; no Redis on the test machine), a 20-turn `mode: "ccr"` session sent round-robin: - every turn's returned prefix was byte-identical to the previous turn's; - the CCR marker resolved via `/v1/retrieve` on all three replicas; - Redis received **378 KB** over the 20 turns. Writing the whole transcript every turn would have been about 4.9 MB, growing with the square of the session length. **Tests:** `tests/test_compress_session_shared.py`, each failure mode mutation-checked: - a stale replica reloads newer state; - without the store, the other replica starts cold; - a turn writes only the messages Redis lacks; - a fully lost key is rewritten; - a messages hash evicted on its own is rewritten; - corrupt state is repaired by another replica; - unreferenced messages are pruned; - a writer whose clock runs ahead is trusted; - the usage relay reaches and publishes a session served elsewhere; - an outage or corrupt values never fail a turn; - the CCR store follows the session store; - a URL that isn't Redis stops startup. 818 proxy, gateway, session and restart tests pass. ruff and mypy are clean. ## Notes - Redis holds transcripts and tool output verbatim. Use `rediss://` with authentication on a private network; plain `redis://` to a remote host logs a warning. - Re-drive turns still need affinity: they park a live in-process task between two HTTP calls. - About 540 added lines (excluding `uv.lock`), around 200 of them tests. The `prefix_tracker` refactor can split out if reviewers prefer. ## Real Behavior Proof - **Environment:** macOS 26.7.1 (arm64), Python 3.12.13, redis-py 8.1.0. Redis is a fakeredis 2.40.0 `TcpFakeServer`, which speaks the Redis protocol over TCP; no Redis server was on the test machine. Docker was not available. - **Configuration:** every replica has `HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0` (the CCR store follows it) and its own `HEADROOM_WORKSPACE_DIR`. - **Steps:** 1. Start the fake server: `python -c "from fakeredis import TcpFakeServer; TcpFakeServer(('127.0.0.1', 6391), server_type='redis').serve_forever()"` 2. Start three proxies, one each on ports 8901, 8902 and 8903: `HEADROOM_COMPRESS_SESSION_STORE=redis://127.0.0.1:6391/0 HEADROOM_WORKSPACE_DIR=<own dir> python -m headroom.cli proxy --port <port>` 3. Drive one session for 20 turns, round-robin across the three: `POST /v1/compress` with `{"messages": …, "session_id": "kong-e2e", "mode": "ccr"}`. The transcript is the 25 KB tool history from `tests/test_compress_session_mode.py`. Every third turn adds a 120-row grep result; the other turns add a user line. 4. After each turn, check: - the returned messages start with the previous turn's returned messages, byte for byte; - how many bytes the turn wrote to the `:msgs` hash. 5. Fetch each `<<ccr:hash>>` marker via `GET /v1/retrieve/{hash}` on all three proxies. - **Result** (after the review fixes, at 95b2d93a): - every turn's prefix is byte-identical to the previous turn's; - Redis received 378 KB in total (63 KB at most in one turn), against about 4.9 MB if the whole transcript were rewritten each turn; - the CCR marker resolved on all three replicas. - **Not tested:** - a real Redis or Redis Cluster (the Lua script and hash-tagged keys are written for it); - TLS (`rediss://`) and AUTH; - Kong's AI Gateway in front of the replicas; - Windows; - real `maxmemory` eviction (simulated in tests by deleting keys); - sustained load. ## Dependencies and scope - **No new runtime dependency:** this uses #4368's `[redis]` extra. - **Dev only:** the `fakeredis` requirement becomes `fakeredis[lua]`, which adds `lupa` so the tests run the compare-and-set Lua script exactly as Redis would. - **Why this feature:** Kong's AI Gateway runs several Headroom instances with no sticky sessions and asked for session state in Redis. It is opt-in: nothing changes unless `HEADROOM_COMPRESS_SESSION_STORE` is set. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4369" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
b196424e35
commit
a616232d75
Vendored
+8
-1
@@ -1232,6 +1232,13 @@ def _create_default_ccr_backend() -> CompressionStoreBackend | None:
|
||||
from ..paths import persistence_allowed
|
||||
|
||||
backend_type = (os.environ.get("HEADROOM_CCR_BACKEND") or "").strip().lower()
|
||||
session_store = os.environ.get("HEADROOM_COMPRESS_SESSION_STORE", "").strip()
|
||||
if not backend_type and session_store:
|
||||
# Replicas sharing compress sessions share the CCR store too: a marker
|
||||
# in a replayed transcript must resolve on whichever replica serves it.
|
||||
from .backends.redis import RedisBackend
|
||||
|
||||
return RedisBackend(session_store)
|
||||
if backend_type == "memory":
|
||||
return None
|
||||
if not backend_type or backend_type == "sqlite":
|
||||
@@ -1259,7 +1266,7 @@ def _create_default_ccr_backend() -> CompressionStoreBackend | None:
|
||||
# registered as "redis" still wins, above.
|
||||
from .backends.redis import RedisBackend
|
||||
|
||||
url = os.environ.get("HEADROOM_REDIS_URL", "").strip()
|
||||
url = os.environ.get("HEADROOM_REDIS_URL", "").strip() or session_store
|
||||
if not url:
|
||||
logger.error(
|
||||
"HEADROOM_CCR_BACKEND=redis needs HEADROOM_REDIS_URL; using the "
|
||||
|
||||
Vendored
+74
-23
@@ -1206,6 +1206,11 @@ class PrefixCacheTracker:
|
||||
# builds the manager.
|
||||
self.restored_from_disk = False
|
||||
self.restored_maturation_state: dict[str, Any] | None = None
|
||||
# Shared-store state (headroom.cache.shared_sessions): the write token
|
||||
# this tracker matches ("" when never shared) and the digests of the
|
||||
# messages known to be stored with it.
|
||||
self.shared_token = ""
|
||||
self.shared_digests: frozenset[str] = frozenset()
|
||||
|
||||
# Stats
|
||||
self._busts_avoided: int = 0
|
||||
@@ -1509,16 +1514,20 @@ class PrefixCacheTracker:
|
||||
|
||||
def is_expired_for_cache_ttl(self, cache_ttl_seconds: int | None) -> bool:
|
||||
"""Check expiry including the longest configured or observed cache lifetime."""
|
||||
return (time.time() - self._last_activity) > self.retention_seconds(cache_ttl_seconds)
|
||||
|
||||
def retention_seconds(self, cache_ttl_seconds: int | None = None) -> float:
|
||||
"""How long this tracker is kept after its last activity: the session
|
||||
TTL, or the longest configured or observed cache lifetime plus a margin."""
|
||||
observed_ttl = max(
|
||||
self._cache_ttl_seconds or 0,
|
||||
self.config.cache_ttl_seconds or 0,
|
||||
cache_ttl_seconds or 0,
|
||||
)
|
||||
retention = max(
|
||||
return max(
|
||||
self.config.session_ttl_seconds,
|
||||
observed_ttl + _CACHE_TTL_CLEANUP_MARGIN_SECONDS if observed_ttl else 0,
|
||||
)
|
||||
return (time.time() - self._last_activity) > retention
|
||||
|
||||
def observe_cache_ttl(self, cache_ttl_seconds: int | None) -> None:
|
||||
"""Remember the longest provider cache lifetime observed for this lineage."""
|
||||
@@ -2144,6 +2153,14 @@ class SessionTrackerStore:
|
||||
pairs mostly repeat the same messages, so this keeps the file near one
|
||||
copy of each conversation instead of up to eleven.
|
||||
"""
|
||||
meta, table = self._tracker_parts(key, tracker, session_id)
|
||||
return f'{{"meta":{json.dumps(meta)},"messages":[{",".join(table)}]}}'
|
||||
|
||||
def _tracker_parts(
|
||||
self, key: str, tracker: PrefixCacheTracker, session_id: str | None
|
||||
) -> tuple[dict[str, Any], list[str]]:
|
||||
"""A tracker as its metadata, whose message lists are indexes into the
|
||||
table, and the table: each distinct message once, as JSON."""
|
||||
index: dict[str, int] = {}
|
||||
memo: dict[int, list[int]] = {}
|
||||
|
||||
@@ -2172,7 +2189,7 @@ class SessionTrackerStore:
|
||||
"history": [[ref(snap), t] for snap, t in self._lineage_history.get(key, ())],
|
||||
"maturation": None if maturation is None else maturation.export_state(),
|
||||
}
|
||||
return f'{{"meta":{json.dumps(meta)},"messages":[{",".join(index)}]}}'
|
||||
return meta, list(index)
|
||||
|
||||
def load_state(self, path: str) -> int:
|
||||
"""Resume the trackers :meth:`save_state` wrote, before serving.
|
||||
@@ -2213,6 +2230,58 @@ class SessionTrackerStore:
|
||||
)
|
||||
return restored
|
||||
|
||||
def _decode_tracker(self, entry: Any, now: float) -> PrefixCacheTracker | None:
|
||||
"""One :meth:`_encode_tracker` entry as a tracker marked restored, or
|
||||
``None`` when it has expired. Raises on a malformed entry."""
|
||||
meta, table = entry["meta"], entry["messages"]
|
||||
tracker = PrefixCacheTracker(str(meta["provider"]), self._default_config)
|
||||
tracker._last_activity = float(meta["last_activity"])
|
||||
tracker.observe_cache_ttl(meta["cache_ttl"])
|
||||
# NaN or a future stamp would never expire.
|
||||
if tracker.is_expired or not tracker._last_activity <= now:
|
||||
return None
|
||||
(
|
||||
tracker._turn_number,
|
||||
tracker._cached_token_count,
|
||||
tracker._cached_message_count,
|
||||
tracker._last_original_messages,
|
||||
tracker._last_forwarded_messages,
|
||||
) = _decode_turn_state(table, meta["state"])
|
||||
tracker._turn_states = {int(t): _decode_turn_state(table, s) for t, s in meta["turns"]}
|
||||
tracker.restored_from_disk = True
|
||||
if meta["maturation"] is not None:
|
||||
from headroom.transforms.read_maturation import ReadMaturationManager
|
||||
|
||||
tracker.restored_maturation_state = ReadMaturationManager.validate_state(
|
||||
meta["maturation"]
|
||||
)
|
||||
return tracker
|
||||
|
||||
def encode_shared(
|
||||
self, key: str, tracker: PrefixCacheTracker
|
||||
) -> tuple[dict[str, Any], list[str]]:
|
||||
"""``tracker`` for another process (headroom.cache.shared_sessions): its
|
||||
metadata and its message table, each distinct message once, as JSON.
|
||||
Raises when its messages are not JSON."""
|
||||
return self._tracker_parts(key, tracker, None)
|
||||
|
||||
def decode_shared(self, meta: Any, messages: list[Any]) -> PrefixCacheTracker | None:
|
||||
"""The tracker :meth:`encode_shared` described, marked restored so its
|
||||
next turn seeds the compression cache from it; ``None`` when it has
|
||||
expired. Raises on malformed input. Does not install it."""
|
||||
if not isinstance(meta, dict):
|
||||
raise ValueError("tracker metadata is not an object")
|
||||
now = time.time()
|
||||
# Written on another host: a clock ahead of ours is not a future stamp.
|
||||
meta = {**meta, "last_activity": min(float(meta["last_activity"]), now)}
|
||||
return self._decode_tracker({"meta": meta, "messages": messages}, now)
|
||||
|
||||
def install(self, key: str, tracker: PrefixCacheTracker) -> None:
|
||||
"""Make ``tracker`` the one under ``key``. Call on the event loop, or
|
||||
from a worker thread only for a key already present (replacing a value
|
||||
does not resize the dict the loop may be iterating)."""
|
||||
self._trackers[key] = tracker
|
||||
|
||||
def _decode_state(self, data: Any) -> int:
|
||||
"""Validate the whole file first, then install it; a bad file installs nothing."""
|
||||
from headroom.cache.compression_store import CompressionEntry, get_compression_store
|
||||
@@ -2228,27 +2297,9 @@ class SessionTrackerStore:
|
||||
# Oldest first, so each family keeps its lineages in creation order.
|
||||
for entry in reversed(data["trackers"]):
|
||||
meta, table = entry["meta"], entry["messages"]
|
||||
tracker = PrefixCacheTracker(str(meta["provider"]), self._default_config)
|
||||
tracker._last_activity = float(meta["last_activity"])
|
||||
tracker.observe_cache_ttl(meta["cache_ttl"])
|
||||
# NaN or a future stamp would never expire.
|
||||
if tracker.is_expired or not tracker._last_activity <= now:
|
||||
tracker = self._decode_tracker(entry, now)
|
||||
if tracker is None:
|
||||
continue
|
||||
(
|
||||
tracker._turn_number,
|
||||
tracker._cached_token_count,
|
||||
tracker._cached_message_count,
|
||||
tracker._last_original_messages,
|
||||
tracker._last_forwarded_messages,
|
||||
) = _decode_turn_state(table, meta["state"])
|
||||
tracker._turn_states = {int(t): _decode_turn_state(table, s) for t, s in meta["turns"]}
|
||||
tracker.restored_from_disk = True
|
||||
if meta["maturation"] is not None:
|
||||
from headroom.transforms.read_maturation import ReadMaturationManager
|
||||
|
||||
tracker.restored_maturation_state = ReadMaturationManager.validate_state(
|
||||
meta["maturation"]
|
||||
)
|
||||
key = str(meta["key"])
|
||||
trackers[key] = tracker
|
||||
history = deque(
|
||||
|
||||
Vendored
+276
@@ -0,0 +1,276 @@
|
||||
"""Session-aware ``/v1/compress`` state shared by several Headroom replicas.
|
||||
|
||||
A compress session's state is its prefix tracker: the transcript it was last
|
||||
given and the one it returned, which every later turn must replay byte for byte
|
||||
so the provider's prompt cache holds. In memory, only the replica that served
|
||||
the last turn has it, so a gateway running several replicas must pin each
|
||||
session to one (``session_affinity``).
|
||||
|
||||
With ``HEADROOM_COMPRESS_SESSION_STORE=redis://...`` every replica also keeps
|
||||
each session in Redis, and any replica can serve its next turn. Per session:
|
||||
|
||||
* ``<prefix>:session:{<digest>}:msgs``, a hash of message digest to message
|
||||
JSON. The original and returned transcripts mostly share messages, and each
|
||||
turn mostly repeats the last one's, so each message is stored once and a turn
|
||||
writes only the messages Redis does not have yet: what the turn added, and
|
||||
any message whose bytes changed (a moved cache marker, an edit, a compaction);
|
||||
* ``<prefix>:session:{<digest>}:meta``, the tracker's counters and its
|
||||
transcripts as lists of digests (``m``), and a write token (``t``).
|
||||
|
||||
Before a turn a replica reads the token (one small ``HGET``). When another
|
||||
replica has written since, it reads the metadata and only the messages it does
|
||||
not already hold, and the turn seeds its compression cache from that tracker as
|
||||
after a restart (:func:`~headroom.cache.prefix_tracker.seed_restored_compression_cache`).
|
||||
After the turn it writes the new messages, then the metadata if the token is
|
||||
still the one it read and every message it references is stored, checked in
|
||||
one script (compare-and-set): a random token, so a lost key or a failover
|
||||
cannot leave a replica believing stale state is current. A missing key
|
||||
resets the replica, whose next write stores the session whole again. Fetched
|
||||
messages are checked against their digests. Both keys expire with the tracker.
|
||||
|
||||
A Redis error never fails a turn (:mod:`headroom.cache.redis_client`): the
|
||||
replica serves from its own memory, as without the store. CCR originals behind
|
||||
the markers in these transcripts live in the shared CCR store, which this
|
||||
setting selects too unless ``HEADROOM_CCR_BACKEND`` names another. Requires the
|
||||
``redis`` extra.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from .redis_client import Guard, connect, key_prefix
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .prefix_tracker import PrefixCacheTracker, SessionTrackerStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
STORE_ENV = "HEADROOM_COMPRESS_SESSION_STORE"
|
||||
#: Messages per HSET/HMGET, so no single command carries a whole transcript.
|
||||
_CHUNK = 256
|
||||
|
||||
# Install ARGV[2] as the token and ARGV[3] as the metadata if the token is still
|
||||
# ARGV[1] ("" when the key must not exist) and the messages hash (KEYS[2]) holds
|
||||
# every message the metadata references (ARGV[5:], including those this write
|
||||
# just sent: Redis can evict the hash between those writes and this script).
|
||||
# 1 on success, 0 when another replica wrote first, 2 when the messages hash
|
||||
# lost one.
|
||||
_CAS = """
|
||||
local t = redis.call('HGET', KEYS[1], 't') or ''
|
||||
if t ~= ARGV[1] then return 0 end
|
||||
for i = 5, #ARGV do
|
||||
if redis.call('HEXISTS', KEYS[2], ARGV[i]) == 0 then return 2 end
|
||||
end
|
||||
redis.call('HSET', KEYS[1], 't', ARGV[2], 'm', ARGV[3])
|
||||
redis.call('PEXPIRE', KEYS[1], ARGV[4])
|
||||
return 1
|
||||
"""
|
||||
|
||||
_FAILED = object()
|
||||
|
||||
|
||||
class SharedSessions:
|
||||
"""Compress-session trackers in Redis, beside the in-memory store."""
|
||||
|
||||
def __init__(self, client: Any) -> None:
|
||||
self._client = client
|
||||
self._cas = client.register_script(_CAS)
|
||||
self._guard = Guard("compress sessions")
|
||||
self._prefix = f"{key_prefix()}:session:"
|
||||
|
||||
def fetch(
|
||||
self, store: SessionTrackerStore, key: str, local: PrefixCacheTracker | None
|
||||
) -> PrefixCacheTracker | None:
|
||||
"""The session's tracker when another replica wrote a newer one than
|
||||
``local``, decoded but not installed; else ``None``."""
|
||||
meta_key, msgs_key = self._keys(key)
|
||||
token = self._guard.call("read", lambda: self._client.hget(meta_key, "t"), _FAILED)
|
||||
if token is _FAILED:
|
||||
return None
|
||||
if token is None:
|
||||
if local is not None: # lost or expired: the next write stores it whole
|
||||
local.shared_token, local.shared_digests = "", frozenset()
|
||||
return None
|
||||
try:
|
||||
if local is not None and _text(token) == local.shared_token:
|
||||
return None
|
||||
except UnicodeDecodeError:
|
||||
return self._drop(local, meta_key, msgs_key)
|
||||
row = self._guard.call("read", lambda: self._client.hmget(meta_key, "t", "m"), None)
|
||||
if row is None:
|
||||
return None
|
||||
if row[0] is None: # expired between the two reads
|
||||
return None
|
||||
try:
|
||||
token = _text(row[0])
|
||||
except UnicodeDecodeError:
|
||||
return self._drop(local, meta_key, msgs_key)
|
||||
try:
|
||||
meta = json.loads(_text(row[1]))
|
||||
digests = meta.pop("table")
|
||||
if not isinstance(digests, list) or not all(isinstance(d, str) for d in digests):
|
||||
raise ValueError("table is not a list of digests")
|
||||
except (TypeError, ValueError, KeyError, AttributeError, IndexError):
|
||||
return self._unusable(local, token, "metadata", meta_key)
|
||||
held = self._held(store, key, local)
|
||||
missing = [d for d in dict.fromkeys(digests) if d not in held]
|
||||
for start in range(0, len(missing), _CHUNK):
|
||||
part = missing[start : start + _CHUNK]
|
||||
values = self._guard.call(
|
||||
"read", functools.partial(self._client.hmget, msgs_key, part), None
|
||||
)
|
||||
if values is None:
|
||||
return None
|
||||
for digest, value in zip(part, values, strict=True):
|
||||
try:
|
||||
text = None if value is None else _text(value)
|
||||
except UnicodeDecodeError:
|
||||
text = None
|
||||
if text is None or _digest(text) != digest:
|
||||
return self._unusable(local, token, "messages", msgs_key)
|
||||
held[digest] = text
|
||||
try:
|
||||
tracker = store.decode_shared(meta, [json.loads(held[d]) for d in digests])
|
||||
except (TypeError, ValueError, KeyError, AttributeError, OverflowError):
|
||||
return self._unusable(local, token, "metadata", meta_key)
|
||||
if tracker is None:
|
||||
logger.info("compress sessions: shared state for %s has expired", meta_key)
|
||||
return None
|
||||
tracker.shared_token, tracker.shared_digests = token, frozenset(digests)
|
||||
return tracker
|
||||
|
||||
@staticmethod
|
||||
def _unusable(
|
||||
local: PrefixCacheTracker | None, token: str, what: str, name: str
|
||||
) -> PrefixCacheTracker | None:
|
||||
"""Shared state this replica cannot use (evicted, corrupt or altered):
|
||||
keep its own, and claim the current token with no messages known to be
|
||||
stored, so its next write replaces the session whole."""
|
||||
logger.warning("compress sessions: unusable %s in %s; this replica rewrites it", what, name)
|
||||
if local is not None:
|
||||
local.shared_token, local.shared_digests = token, frozenset()
|
||||
return None
|
||||
|
||||
def _drop(
|
||||
self, local: PrefixCacheTracker | None, meta_key: str, msgs_key: str
|
||||
) -> PrefixCacheTracker | None:
|
||||
"""A token that is not text cannot be claimed: delete the session, so
|
||||
the next write stores it whole."""
|
||||
logger.warning("compress sessions: corrupt token in %s; deleting the session", meta_key)
|
||||
self._guard.call("write", lambda: self._client.delete(meta_key, msgs_key), 0)
|
||||
if local is not None:
|
||||
local.shared_token, local.shared_digests = "", frozenset()
|
||||
return None
|
||||
|
||||
def publish(self, store: SessionTrackerStore, key: str, tracker: PrefixCacheTracker) -> None:
|
||||
"""Write the messages Redis lacks, then the metadata unless another
|
||||
replica wrote since ``tracker`` was read."""
|
||||
try:
|
||||
meta, table = store.encode_shared(key, tracker)
|
||||
except (TypeError, ValueError) as exc: # messages that are not JSON
|
||||
logger.debug("compress sessions: tracker not shareable (%s)", exc)
|
||||
return
|
||||
digests = [_digest(m) for m in table]
|
||||
meta["table"] = digests
|
||||
meta_json = json.dumps(meta)
|
||||
ttl_ms = max(1000, int(tracker.retention_seconds() * 1000))
|
||||
meta_key, msgs_key = self._keys(key)
|
||||
referenced = sorted(set(digests))
|
||||
|
||||
def write(known: frozenset[str]) -> tuple[int, str]:
|
||||
new = [(d, m) for d, m in zip(digests, table, strict=True) if d not in known]
|
||||
token = uuid.uuid4().hex
|
||||
pipe = self._client.pipeline(transaction=False)
|
||||
for start in range(0, len(new), _CHUNK):
|
||||
pipe.hset(msgs_key, mapping=dict(new[start : start + _CHUNK]))
|
||||
pipe.pexpire(msgs_key, ttl_ms)
|
||||
pipe.execute()
|
||||
args = [tracker.shared_token, token, meta_json, ttl_ms, *referenced]
|
||||
return int(self._cas(keys=[meta_key, msgs_key], args=args)), token
|
||||
|
||||
result = self._guard.call("write", lambda: write(tracker.shared_digests), None)
|
||||
if result is not None and result[0] == 2:
|
||||
# The messages hash lost entries (evicted on its own): store it whole.
|
||||
result = self._guard.call("write", lambda: write(frozenset()), None)
|
||||
if result is None:
|
||||
return
|
||||
written, token = result
|
||||
if written == 1:
|
||||
tracker.shared_token, tracker.shared_digests = token, frozenset(digests)
|
||||
self._prune(msgs_key, set(digests))
|
||||
elif written == 2:
|
||||
# Evicted again during the whole rewrite: publish nothing, so no
|
||||
# replica takes a session it cannot replay; the next turn retries.
|
||||
logger.warning("compress sessions: messages of %s evicted while writing", meta_key)
|
||||
elif written == 0:
|
||||
# Two replicas ran turns of one session at once, which the caller's
|
||||
# contract excludes; the next turn takes the state that won.
|
||||
logger.warning("compress sessions: another replica wrote %s first", meta_key)
|
||||
|
||||
def _prune(self, msgs_key: str, referenced: set[str]) -> None:
|
||||
"""Drop messages no transcript references (moved markers, edits,
|
||||
compactions) once they outnumber the referenced ones."""
|
||||
size = self._guard.call("write", lambda: int(self._client.hlen(msgs_key)), 0)
|
||||
if size <= 2 * len(referenced):
|
||||
return
|
||||
|
||||
def prune() -> None:
|
||||
stale = [k for k in map(_text, self._client.hkeys(msgs_key)) if k not in referenced]
|
||||
for start in range(0, len(stale), _CHUNK):
|
||||
self._client.hdel(msgs_key, *stale[start : start + _CHUNK])
|
||||
|
||||
self._guard.call("write", prune, None)
|
||||
|
||||
def _keys(self, key: str) -> tuple[str, str]:
|
||||
# One hash tag: the compare-and-set script names both keys, which Redis
|
||||
# Cluster allows only within one slot.
|
||||
base = f"{self._prefix}{{{hashlib.blake2b(key.encode(), digest_size=16).hexdigest()}}}"
|
||||
return f"{base}:meta", f"{base}:msgs"
|
||||
|
||||
@staticmethod
|
||||
def _held(
|
||||
store: SessionTrackerStore, key: str, local: PrefixCacheTracker | None
|
||||
) -> dict[str, str]:
|
||||
"""The messages this replica already holds for the session, by digest."""
|
||||
if local is None:
|
||||
return {}
|
||||
try:
|
||||
_, table = store.encode_shared(key, local)
|
||||
except (TypeError, ValueError):
|
||||
return {}
|
||||
return {_digest(m): m for m in table}
|
||||
|
||||
|
||||
def _digest(message_json: str) -> str:
|
||||
return hashlib.blake2b(message_json.encode(), digest_size=16).hexdigest()
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value.decode() if isinstance(value, bytes) else str(value)
|
||||
|
||||
|
||||
def open_shared_sessions() -> SharedSessions | None:
|
||||
"""The shared store ``HEADROOM_COMPRESS_SESSION_STORE`` names, or ``None``.
|
||||
|
||||
Raises :class:`~headroom.exceptions.ConfigurationError` when it is set but
|
||||
unusable, so a replica meant to share sessions never starts without it."""
|
||||
url = os.environ.get(STORE_ENV, "").strip()
|
||||
if not url:
|
||||
return None
|
||||
client = connect(url, setting=STORE_ENV)
|
||||
ccr = (os.environ.get("HEADROOM_CCR_BACKEND") or "").strip().lower()
|
||||
if ccr not in ("", "redis"):
|
||||
logger.warning(
|
||||
"%s is set but HEADROOM_CCR_BACKEND=%s: CCR markers in a session's "
|
||||
"transcript resolve only on the replica that made them",
|
||||
STORE_ENV,
|
||||
ccr,
|
||||
)
|
||||
return SharedSessions(client)
|
||||
@@ -10701,9 +10701,6 @@ class OpenAIHandlerMixin:
|
||||
session_tracker = self.session_tracker_store.get_or_create(
|
||||
_session_key, _tracker_provider
|
||||
)
|
||||
from headroom.cache.prefix_tracker import seed_restored_compression_cache
|
||||
|
||||
seed_restored_compression_cache(session_tracker, comp_cache)
|
||||
|
||||
# A claimed turn prepares its executor-side work here (tokenizer,
|
||||
# hooks, scope binding) — CPU-bound parts run inside the closures
|
||||
@@ -10764,6 +10761,21 @@ class OpenAIHandlerMixin:
|
||||
"(a previous turn for this session is still running)"
|
||||
)
|
||||
try:
|
||||
nonlocal session_tracker
|
||||
from headroom.cache.prefix_tracker import seed_restored_compression_cache
|
||||
|
||||
shared = self.shared_compress_sessions
|
||||
if shared is not None:
|
||||
# Another replica may have served this session's last
|
||||
# turn: take its tracker when it is newer than ours.
|
||||
newer = shared.fetch(
|
||||
self.session_tracker_store, _session_key, session_tracker
|
||||
)
|
||||
if newer is not None:
|
||||
# The key is present (get_or_create above).
|
||||
self.session_tracker_store.install(_session_key, newer)
|
||||
session_tracker = newer
|
||||
seed_restored_compression_cache(session_tracker, comp_cache)
|
||||
prev_original = session_tracker.get_last_original_messages()
|
||||
prev_returned = session_tracker.get_last_forwarded_messages()
|
||||
prep = prepare_turn(
|
||||
@@ -10838,6 +10850,8 @@ class OpenAIHandlerMixin:
|
||||
# return time because whatever we hand back IS what the
|
||||
# caller sends upstream.
|
||||
session_tracker.record_returned(messages, final)
|
||||
if shared is not None:
|
||||
shared.publish(self.session_tracker_store, _session_key, session_tracker)
|
||||
info = {
|
||||
"id": session_id,
|
||||
"frozen_message_count": session_frozen,
|
||||
@@ -11149,6 +11163,40 @@ class OpenAIHandlerMixin:
|
||||
# from here.
|
||||
_session_key = f"compress\x00{session_id}"
|
||||
tracker = self.session_tracker_store.peek(_session_key)
|
||||
shared = self.shared_compress_sessions
|
||||
if shared is not None:
|
||||
# The session's last turn may have run on another replica. Fetched
|
||||
# under the turn lock (a local turn in flight must finish first),
|
||||
# installed here on the loop: the key may be new to this store.
|
||||
_local = tracker
|
||||
_lock_cache = self._peek_compression_cache(_session_key)
|
||||
|
||||
def _fetch():
|
||||
lock = _lock_cache.session_turn_lock if _lock_cache is not None else None
|
||||
if lock is not None and not lock.acquire(
|
||||
timeout=_SESSION_TURN_LOCK_TIMEOUT_SECONDS
|
||||
):
|
||||
return None
|
||||
try:
|
||||
return shared.fetch(self.session_tracker_store, _session_key, _local)
|
||||
finally:
|
||||
if lock is not None:
|
||||
lock.release()
|
||||
|
||||
try:
|
||||
newer = await self._run_compression_in_executor(
|
||||
_fetch, timeout=COMPRESSION_TIMEOUT_SECONDS
|
||||
)
|
||||
except TimeoutError:
|
||||
newer = None # answer from this replica's state, as without the store
|
||||
if newer is not None:
|
||||
current = self.session_tracker_store.peek(_session_key)
|
||||
if current is _local:
|
||||
self.session_tracker_store.install(_session_key, newer)
|
||||
tracker = newer
|
||||
else:
|
||||
# A local turn installed a tracker meanwhile: it is newer.
|
||||
tracker = current
|
||||
if tracker is None:
|
||||
return self._compress_usage_unknown_session(session_id)
|
||||
# No create, no LRU bump: the cache is only needed for its turn lock.
|
||||
@@ -11195,6 +11243,8 @@ class OpenAIHandlerMixin:
|
||||
messages=last_returned,
|
||||
original_messages=tracker.get_last_original_messages(),
|
||||
)
|
||||
if shared is not None:
|
||||
shared.publish(self.session_tracker_store, _session_key, tracker)
|
||||
return tracker.get_frozen_message_count()
|
||||
finally:
|
||||
if lock is not None:
|
||||
|
||||
@@ -1207,6 +1207,11 @@ class HeadroomProxy(
|
||||
session_ttl_seconds=config.prefix_freeze_session_ttl,
|
||||
)
|
||||
)
|
||||
# Session-aware /v1/compress state in Redis, so any replica serves any
|
||||
# turn (HEADROOM_COMPRESS_SESSION_STORE); None keeps it in memory.
|
||||
from headroom.cache.shared_sessions import open_shared_sessions
|
||||
|
||||
self.shared_compress_sessions = open_shared_sessions()
|
||||
# Set by startup() when HEADROOM_PREFIX_STATE_FILE opts in.
|
||||
self._prefix_state_file: str | None = None
|
||||
|
||||
|
||||
+3
-2
@@ -300,7 +300,8 @@ bedrock = [
|
||||
"boto3>=1.41.0",
|
||||
"botocore[crt]>=1.41.0",
|
||||
]
|
||||
# State shared across replicas: CCR store (HEADROOM_CCR_BACKEND=redis).
|
||||
# State shared across replicas: CCR store (HEADROOM_CCR_BACKEND=redis) and
|
||||
# session-aware /v1/compress (HEADROOM_COMPRESS_SESSION_STORE=redis://...).
|
||||
redis = [
|
||||
"redis>=5.0.0",
|
||||
]
|
||||
@@ -336,7 +337,7 @@ dev = [
|
||||
"sentence-transformers>=2.2.0,<7.0",
|
||||
"numpy>=1.24.0",
|
||||
"openpyxl>=3.1.0", # exercises spreadsheet_ingest (.xlsx) in the test suite
|
||||
"fakeredis>=2.20.0", # Redis-backed shared state, without a Redis server
|
||||
"fakeredis[lua]>=2.20.0", # Redis-backed shared state, without a Redis server
|
||||
"xlrd>=2.0.1", # exercises spreadsheet_ingest (.xls) in the test suite
|
||||
"xlwt>=1.3.0", # writes the legacy .xls fixtures for spreadsheet_ingest tests
|
||||
# test_upstream_guard builds a self-signed cert to prove the pinned connection
|
||||
|
||||
@@ -0,0 +1,342 @@
|
||||
"""Session-aware /v1/compress served by several replicas sharing Redis.
|
||||
|
||||
A gateway without sticky sessions sends each turn to any replica. With
|
||||
HEADROOM_COMPRESS_SESSION_STORE set, a replica that did not serve the last turn
|
||||
takes the session's state from Redis, freezes the prefix the caller already
|
||||
forwarded, and returns it byte for byte.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
fakeredis = pytest.importorskip("fakeredis")
|
||||
pytest.importorskip("lupa")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.cache import shared_sessions # noqa: E402
|
||||
from headroom.cache.backends.redis import RedisBackend # noqa: E402
|
||||
from headroom.cache.compression_store import _create_default_ccr_backend # noqa: E402
|
||||
from headroom.exceptions import ConfigurationError # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
from tests.test_compress_session_mode import _big_tool_history, _compress # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def redis_server(monkeypatch):
|
||||
import redis
|
||||
|
||||
server = fakeredis.FakeServer()
|
||||
monkeypatch.setenv(shared_sessions.STORE_ENV, "redis://127.0.0.1:6399/0")
|
||||
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "memory") # CCR sharing is tested on its own
|
||||
monkeypatch.setattr(
|
||||
redis.Redis,
|
||||
"from_url",
|
||||
classmethod(lambda cls, *a, **k: fakeredis.FakeRedis(server=server)),
|
||||
)
|
||||
return fakeredis.FakeRedis(server=server)
|
||||
|
||||
|
||||
def _replica() -> TestClient:
|
||||
config = ProxyConfig(
|
||||
optimize=True,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
image_optimize=False,
|
||||
)
|
||||
return TestClient(create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 1))
|
||||
|
||||
|
||||
def _second_tool_result() -> list[dict]:
|
||||
rows = [
|
||||
{"id": i, "path": f"src/mod_{i}.py", "line": i * 3, "hit": "x" * 200} for i in range(150)
|
||||
]
|
||||
return [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{"id": "c2", "type": "function", "function": {"name": "grep", "arguments": "{}"}}
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c2", "content": json.dumps(rows)},
|
||||
]
|
||||
|
||||
|
||||
def test_any_replica_serves_the_next_turn_byte_for_byte(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
turn1 = _compress(a, history, session_id="conv")
|
||||
assert turn1["messages"][2]["content"] != history[2]["content"] # compressed
|
||||
|
||||
# Turn 2 on B adds a second tool result, compressed by B.
|
||||
history2 = history + _second_tool_result()
|
||||
turn2 = _compress(b, history2, session_id="conv")
|
||||
assert turn2["messages"][: len(turn1["messages"])] == turn1["messages"]
|
||||
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
|
||||
assert turn2["messages"][4]["content"] != history2[4]["content"]
|
||||
|
||||
# Back on A, which last saw turn 1: only B's state has B's bytes.
|
||||
history3 = history2 + [{"role": "user", "content": "again"}]
|
||||
turn3 = _compress(a, history3, session_id="conv")
|
||||
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
def test_without_the_store_a_new_replica_starts_cold(monkeypatch):
|
||||
# What the store fixes: B never saw the compressed tool result, so it does
|
||||
# not freeze it and compresses it again.
|
||||
monkeypatch.delenv(shared_sessions.STORE_ENV, raising=False)
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
turn1 = _compress(a, history, session_id="conv")
|
||||
turn2 = _compress(b, history + [{"role": "user", "content": "x"}], session_id="conv")
|
||||
assert turn2["session"]["frozen_message_count"] < len(turn1["messages"])
|
||||
|
||||
|
||||
def test_a_turn_writes_only_the_messages_redis_lacks(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a:
|
||||
_compress(a, history, session_id="conv-delta")
|
||||
(msgs_key,) = redis_server.keys("headroom:session:*:msgs")
|
||||
before = set(redis_server.hkeys(msgs_key))
|
||||
|
||||
import redis
|
||||
|
||||
written: list[int] = []
|
||||
real_hset = redis.client.Pipeline.hset
|
||||
|
||||
def counting_hset(self, name, *args, mapping=None, **kwargs):
|
||||
written.append(len(mapping or {}))
|
||||
return real_hset(self, name, *args, mapping=mapping, **kwargs)
|
||||
|
||||
redis.client.Pipeline.hset = counting_hset
|
||||
try:
|
||||
_compress(
|
||||
a, history + [{"role": "user", "content": "one more"}], session_id="conv-delta"
|
||||
)
|
||||
finally:
|
||||
redis.client.Pipeline.hset = real_hset
|
||||
after = set(redis_server.hkeys(msgs_key))
|
||||
# The new user message, and at most the few whose cache marker moved.
|
||||
assert 1 <= sum(written) <= 3 < len(before)
|
||||
assert before <= after
|
||||
|
||||
|
||||
def test_a_lost_key_is_stored_whole_again(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
_compress(a, history, session_id="conv-lost")
|
||||
redis_server.flushall()
|
||||
turn2 = _compress(a, history + [{"role": "user", "content": "y"}], session_id="conv-lost")
|
||||
assert redis_server.keys("headroom:session:*:meta")
|
||||
# And another replica is warm from it.
|
||||
turn3 = _compress(
|
||||
b,
|
||||
history + [{"role": "user", "content": "y"}, {"role": "user", "content": "z"}],
|
||||
session_id="conv-lost",
|
||||
)
|
||||
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
def test_the_usage_relay_reaches_and_publishes_a_session_served_elsewhere(redis_server):
|
||||
with _replica() as a, _replica() as b:
|
||||
_compress(a, _big_tool_history(), session_id="conv-usage")
|
||||
(meta_key,) = redis_server.keys("headroom:session:*:meta")
|
||||
token = redis_server.hget(meta_key, "t")
|
||||
resp = b.post(
|
||||
"/v1/usage",
|
||||
json={
|
||||
"session_id": "conv-usage",
|
||||
"usage": {"cache_read_input_tokens": 0, "cache_creation_input_tokens": 50_000},
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert redis_server.hget(meta_key, "t") != token # B published its update
|
||||
|
||||
|
||||
def test_a_redis_outage_or_corrupt_state_never_fails_a_turn(redis_server, monkeypatch):
|
||||
import redis
|
||||
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
turn1 = _compress(a, history, session_id="conv-down")
|
||||
(meta_key,) = redis_server.keys("headroom:session:*:meta")
|
||||
redis_server.hset(meta_key, "m", "{not json")
|
||||
turn2 = _compress(b, history + [{"role": "user", "content": "y"}], session_id="conv-down")
|
||||
assert turn2["messages"][0] == turn1["messages"][0]
|
||||
|
||||
def down(*args, **kwargs):
|
||||
raise redis.ConnectionError("connection refused")
|
||||
|
||||
monkeypatch.setattr(fakeredis.FakeRedis, "execute_command", down)
|
||||
turn3 = _compress(a, history + [{"role": "user", "content": "z"}], session_id="conv-down")
|
||||
assert turn3["messages"][: len(turn1["messages"])] == turn1["messages"]
|
||||
|
||||
|
||||
def test_sharing_sessions_shares_the_ccr_store_too(monkeypatch):
|
||||
monkeypatch.setenv(shared_sessions.STORE_ENV, "redis://127.0.0.1:6399/0")
|
||||
monkeypatch.delenv("HEADROOM_CCR_BACKEND", raising=False)
|
||||
assert isinstance(_create_default_ccr_backend(), RedisBackend)
|
||||
|
||||
|
||||
def test_a_store_url_that_is_not_redis_stops_startup(monkeypatch):
|
||||
monkeypatch.setenv(shared_sessions.STORE_ENV, "memcached://cache:11211")
|
||||
with pytest.raises(ConfigurationError, match="redis://"):
|
||||
shared_sessions.open_shared_sessions()
|
||||
|
||||
|
||||
def _meta_key(redis_server):
|
||||
(meta_key,) = redis_server.keys("headroom:session:*:meta")
|
||||
return meta_key
|
||||
|
||||
|
||||
def test_messages_evicted_on_their_own_are_stored_whole_again(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
_compress(a, history, session_id="conv-evict")
|
||||
meta_key = _meta_key(redis_server)
|
||||
redis_server.delete(meta_key.replace(b":meta", b":msgs")) # maxmemory took one key
|
||||
history2 = history + [{"role": "user", "content": "y"}]
|
||||
turn2 = _compress(a, history2, session_id="conv-evict")
|
||||
turn3 = _compress(b, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict")
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
def test_corrupt_state_is_rewritten_by_the_next_replica_to_write(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b, _replica() as c:
|
||||
_compress(a, history, session_id="conv-corrupt")
|
||||
redis_server.hset(_meta_key(redis_server), "m", "{not json")
|
||||
history2 = history + [{"role": "user", "content": "y"}]
|
||||
turn2 = _compress(b, history2, session_id="conv-corrupt") # repairs it
|
||||
turn3 = _compress(
|
||||
c, history2 + [{"role": "user", "content": "z"}], session_id="conv-corrupt"
|
||||
)
|
||||
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
def test_messages_no_transcript_references_are_pruned(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a:
|
||||
_compress(a, history, session_id="conv-prune")
|
||||
msgs_key = _meta_key(redis_server).replace(b":meta", b":msgs")
|
||||
redis_server.hset(msgs_key, mapping={f"stale{i}": "{}" for i in range(50)})
|
||||
_compress(a, history + [{"role": "user", "content": "y"}], session_id="conv-prune")
|
||||
assert not any(k.startswith(b"stale") for k in redis_server.hkeys(msgs_key))
|
||||
|
||||
|
||||
def test_a_writer_whose_clock_runs_ahead_is_still_trusted(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
turn1 = _compress(a, history, session_id="conv-skew")
|
||||
meta_key = _meta_key(redis_server)
|
||||
meta = json.loads(redis_server.hget(meta_key, "m"))
|
||||
meta["last_activity"] += 30 # A's clock is 30 s ahead of B's
|
||||
redis_server.hset(meta_key, "m", json.dumps(meta))
|
||||
turn2 = _compress(b, history + [{"role": "user", "content": "y"}], session_id="conv-skew")
|
||||
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
|
||||
|
||||
|
||||
def test_a_lost_message_is_noticed_even_beside_unreferenced_ones(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
_compress(a, history, session_id="conv-masked")
|
||||
msgs_key = _meta_key(redis_server).replace(b":meta", b":msgs")
|
||||
# Old variants left behind outnumber the one referenced message lost.
|
||||
redis_server.hset(msgs_key, mapping={f"old{i}": "{}" for i in range(5)})
|
||||
redis_server.hdel(msgs_key, redis_server.hkeys(msgs_key)[0])
|
||||
history2 = history + [{"role": "user", "content": "y"}]
|
||||
turn2 = _compress(a, history2, session_id="conv-masked")
|
||||
turn3 = _compress(
|
||||
b, history2 + [{"role": "user", "content": "z"}], session_id="conv-masked"
|
||||
)
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
def test_a_corrupt_token_never_fails_a_turn_and_is_replaced(redis_server):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
_compress(a, history, session_id="conv-token")
|
||||
redis_server.hset(_meta_key(redis_server), "t", b"\xff")
|
||||
history2 = history + [{"role": "user", "content": "y"}]
|
||||
turn2 = _compress(b, history2, session_id="conv-token")
|
||||
turn3 = _compress(a, history2 + [{"role": "user", "content": "z"}], session_id="conv-token")
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def evict_before_cas(monkeypatch):
|
||||
"""Delete the messages hash just before the next ``times`` compare-and-set
|
||||
scripts run, as maxmemory might between the message writes and the script."""
|
||||
budget = {"times": 0}
|
||||
real_init = shared_sessions.SharedSessions.__init__
|
||||
|
||||
def init(self, client):
|
||||
real_init(self, client)
|
||||
cas = self._cas
|
||||
|
||||
def evicting_cas(keys, args):
|
||||
if budget["times"]:
|
||||
budget["times"] -= 1
|
||||
client.delete(keys[1])
|
||||
return cas(keys=keys, args=args)
|
||||
|
||||
self._cas = evicting_cas
|
||||
|
||||
monkeypatch.setattr(shared_sessions.SharedSessions, "__init__", init)
|
||||
return budget
|
||||
|
||||
|
||||
def test_eviction_before_the_first_publish_is_never_acknowledged(redis_server, evict_before_cas):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
evict_before_cas["times"] = 1
|
||||
turn1 = _compress(a, history, session_id="conv-evict-first")
|
||||
turn2 = _compress(
|
||||
b, history + [{"role": "user", "content": "y"}], session_id="conv-evict-first"
|
||||
)
|
||||
assert turn2["messages"][: len(turn1["messages"])] == turn1["messages"]
|
||||
assert turn2["session"]["frozen_message_count"] >= len(turn1["messages"])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("corrupt", [False, True], ids=["delta", "whole-rewrite"])
|
||||
def test_eviction_before_a_later_publish_is_never_acknowledged(
|
||||
redis_server, evict_before_cas, corrupt
|
||||
):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b, _replica() as c:
|
||||
_compress(a, history, session_id="conv-evict-later")
|
||||
if corrupt: # B then rewrites the session whole, assuming no message stored
|
||||
redis_server.hset(_meta_key(redis_server), "m", "{not json")
|
||||
history2 = history + [{"role": "user", "content": "y"}]
|
||||
evict_before_cas["times"] = 1
|
||||
turn2 = _compress(b, history2, session_id="conv-evict-later")
|
||||
turn3 = _compress(
|
||||
c, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict-later"
|
||||
)
|
||||
assert turn3["messages"][: len(turn2["messages"])] == turn2["messages"]
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
|
||||
|
||||
def test_a_session_whose_messages_keep_vanishing_is_not_published(redis_server, evict_before_cas):
|
||||
history = _big_tool_history()
|
||||
with _replica() as a, _replica() as b:
|
||||
evict_before_cas["times"] = 2 # the write and its whole rewrite
|
||||
_compress(a, history, session_id="conv-evict-twice")
|
||||
assert not redis_server.keys("headroom:session:*:meta")
|
||||
# Once Redis keeps the messages again, the next turn publishes it whole.
|
||||
history2 = history + [{"role": "user", "content": "y"}]
|
||||
turn2 = _compress(a, history2, session_id="conv-evict-twice")
|
||||
turn3 = _compress(
|
||||
b, history2 + [{"role": "user", "content": "z"}], session_id="conv-evict-twice"
|
||||
)
|
||||
assert turn3["session"]["frozen_message_count"] >= len(turn2["messages"])
|
||||
@@ -1153,6 +1153,11 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/e4/6919d3653d72c53d1fb22c97ceb6fa3664cad302994e90ee52279f7eb394/fakeredis-2.40.0-py3-none-any.whl", hash = "sha256:b155ef2442134372eb1cc5664cf5638ccbe0a6dde9d1942153708e2782f315c9", size = 204148, upload-time = "2026-10-14T12:46:00.014Z" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
lua = [
|
||||
{ name = "lupa" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fastapi"
|
||||
version = "0.136.3"
|
||||
@@ -1705,7 +1710,7 @@ code = [
|
||||
dev = [
|
||||
{ name = "anthropic" },
|
||||
{ name = "cryptography" },
|
||||
{ name = "fakeredis" },
|
||||
{ name = "fakeredis", extra = ["lua"] },
|
||||
{ name = "fastapi" },
|
||||
{ name = "hnswlib" },
|
||||
{ name = "httpx", extra = ["http2", "socks"] },
|
||||
@@ -1903,7 +1908,7 @@ requires-dist = [
|
||||
{ name = "cryptography", marker = "extra == 'dev'", specifier = ">=50.0.0" },
|
||||
{ name = "datasets", marker = "extra == 'evals'", specifier = ">=5.0.1" },
|
||||
{ name = "datasets", marker = "extra == 'voice-train'", specifier = ">=5.0.1" },
|
||||
{ name = "fakeredis", marker = "extra == 'dev'", specifier = ">=2.20.0" },
|
||||
{ name = "fakeredis", extras = ["lua"], marker = "extra == 'dev'", specifier = ">=2.20.0" },
|
||||
{ name = "fastapi", marker = "extra == 'dev'", specifier = ">=0.100.0" },
|
||||
{ name = "fastapi", marker = "extra == 'proxy'", specifier = ">=0.100.0" },
|
||||
{ name = "fastembed", marker = "(python_full_version < '3.14' and extra == 'relevance') or (platform_machine != 'x86_64' and extra == 'relevance') or (sys_platform != 'darwin' and extra == 'relevance')", specifier = ">=0.6.0" },
|
||||
@@ -2651,6 +2656,69 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lupa"
|
||||
version = "2.8"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c3/a6/0f869fbb07c393f15473b1eefefb7b5bec162fb7481803d040ed4dc46002/lupa-2.8.tar.gz", hash = "sha256:d8022641b9ec8ecf2c5ecbe9f47e5a70e0b87c4b5ae921b92cb02a638e0acd08", size = 6156370, upload-time = "2026-04-15T20:08:30.534Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/09/21/9be4516ddd22f8eadba336d9ba065d17d79108465ae1b7f71424ab99b9d0/lupa-2.8-cp310-abi3-win32.whl", hash = "sha256:c2a5fd15dc62374e1661a55f01744c9ec1c56f291ba4a0749d3af2174556e78f", size = 1594887, upload-time = "2026-04-15T20:05:23.377Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2d/99/1557c9685d7034d9ce8dd2b54c40a26d6deb7c67c1fdb5c801abd1a02c3f/lupa-2.8-cp310-abi3-win_arm64.whl", hash = "sha256:9e304fb1c50cf23fd8882afbe1aa87525ef8a72667bcab3b37b2bbb2bc542269", size = 1371742, upload-time = "2026-04-15T20:05:27.417Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1c/34/05ce4745b191633f90ff1ab50f1a19a37da282bb0a41fb500d9157fc9b8f/lupa-2.8-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:97bd01e90b8031e56a5fd5bb70605aea09f1dba675c1140308a52780f93d06f1", size = 1202714, upload-time = "2026-04-15T20:05:31.088Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7d/d2/f70fdbeec2d4c69ee6a469e6cddde9635fff4af4e13fb652e6a1229eef51/lupa-2.8-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b5ebe1a13c45767919c86750b84fe2da9f6288b6f3cea4ce7660bb2abc9d921", size = 1857453, upload-time = "2026-04-15T20:05:34.611Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/97/dc/6fcda0e36e75eb6cb98dc9190fa4737d727eeae29e58f892980b2c96b656/lupa-2.8-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:097e7d0f1719a88020b67c82e05d53d7973c166952393afcecfd8434c7e19a15", size = 2408890, upload-time = "2026-04-15T20:05:37.994Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/58/29/7ea176eac3c1dac83d059762daa875ad1390decc0bf2c3b4c7bbfc1f1665/lupa-2.8-cp310-cp310-win_amd64.whl", hash = "sha256:7bb223ee8f72d0dc076b0d65296ee72f1c69450f9d2fed5315f7707d98c4a03d", size = 1910396, upload-time = "2026-04-15T20:05:41.163Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b7/0a/5a740717f27aa77481e6a61b97cf79d1e0c1ede729b1268caacded915326/lupa-2.8-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:b12e43c1fb787189dfc28cd604aef0baa2cb95e27da19498d520361d0ace070a", size = 1202376, upload-time = "2026-04-15T20:05:44.049Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1b/75/6b64d0098c64275a801896cb7a6a30e7e653d25fa102c64e747292afcdbb/lupa-2.8-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6f603391dffb256e36a79fd2044084d5f4b8a0a4c0e5ad291cd3ab3aaf1fd0a", size = 1839271, upload-time = "2026-04-15T20:05:47.399Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/2f/0d4f00563046ff616ef6a421f8b776a5ffb327f7b32ed69e856d52b917a8/lupa-2.8-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f6f41c91366e7d0d474f87d81c1274af861f40812bf729c9f97ab4c8f3c7ac8", size = 2376251, upload-time = "2026-04-15T20:05:49.891Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4c/8e/caa83237f427d9e85b7f02c816e7270c9c9571dec1673e06b0180402f70e/lupa-2.8-cp311-cp311-win_amd64.whl", hash = "sha256:f5a6af145b0ea818f01d27bfe2583a4b538570bef61d22c8773e0eccf011234c", size = 1923488, upload-time = "2026-04-15T20:05:52.954Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ad/0b/368f2f0bc750b25c69d4563e44f677925ab5dd3d2887f9b0c15465d21a2a/lupa-2.8-cp312-abi3-macosx_10_13_x86_64.whl", hash = "sha256:f4342f4de76ae7ce2ab0672d36003bdb7e1a33252f293b569298ddd792e70e33", size = 1194056, upload-time = "2026-04-15T20:05:55.794Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5b/0f/c89eb8dd36fdea4e50ae3f7f5275bea3b0cc5d4057b8ee7b3bbc78010422/lupa-2.8-cp312-abi3-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:4203fa1659315e939a5304e75001b8cc14234fb3cbb3ed86c049b0cc5d90fcee", size = 1434278, upload-time = "2026-04-15T20:05:57.94Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/30/c3b4d2cd8733621b404b8a4214e5f852955c4ba632546dc84123bea9ee89/lupa-2.8-cp312-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:81f2d843ce668b653146c007467570210ae44be51dac6926666c51d49536f307", size = 1150068, upload-time = "2026-04-15T20:06:01.04Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8d/d2/bac12c398519efafc6af84be1974edd0d7a4895fb4735b5c8d615d298595/lupa-2.8-cp312-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d3d0cde2c77588d1c60875a4f34f059513476c6e1775351897195b51e0f3df08", size = 1409532, upload-time = "2026-04-15T20:06:03.592Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9c/6a/18b52e11962014026e07813530b0b108ee8bc0a2a13ef0eaea5d41dce023/lupa-2.8-cp312-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9e0d11b8f3a8dac6413f704fef7161d048bb10c58bdac6cbffa5e60efa56e9a3", size = 1242687, upload-time = "2026-04-15T20:06:06.863Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/8e/7fd4eb049875f61429b96780d2eae4700f0e78fe0a52db8edb231b1cd09f/lupa-2.8-cp312-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:54cff414f21f8cd8c6be4aae52541f3b9cd39602b59e3a3db9b5c9f9f674ff18", size = 1856038, upload-time = "2026-04-15T20:06:09.358Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e9/f9/37ad9d2773d30f2931890d310a4bdce28d45484206e6f48bc18b0325eabd/lupa-2.8-cp312-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:24b4d8af5558e549b70daf1547f5c1c1d664ecea9fc790f83efe5d75e9a93797", size = 1128982, upload-time = "2026-04-15T20:06:12.312Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/57/31/c0fd7984c24844ea79caa45c0235f61a06b38fd69a839f6c62770f8d684a/lupa-2.8-cp312-abi3-musllinux_1_2_i686.whl", hash = "sha256:ce86dff1ee7f7cf45f5622065ae991949dd7bb1703581cbc58a630137bb7ccf9", size = 1457594, upload-time = "2026-04-15T20:06:15.881Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/11/f5/a28e411be30ec1bf0db1eb0c087eebc73be9e7a1adcfe6ac209861ccc446/lupa-2.8-cp312-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:f4d01b2a08c70bbb883a9e082b6b36b89121ed5910b710f1ba11c73295ff4fba", size = 1425721, upload-time = "2026-04-15T20:06:18.009Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/c1/359f767c4ae024be30d909fe8a9f0e9af266bad47ce2bd2ed248fb986fcf/lupa-2.8-cp312-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:7f210d5a8353e510ea1199c42cf3cbdd630553bf2bc8fb4c00fea06fdec7c798", size = 1253258, upload-time = "2026-04-15T20:06:21.17Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/17/52/473f11790c261fd02bbf318a546fe040e9ec9f677181272fa78d3b4112a4/lupa-2.8-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4f81a02806e7c7ad26d8c6fa222c8bef1b0c1b124347c879be880b41339d41e4", size = 2395272, upload-time = "2026-04-15T20:06:24.137Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/94/bf/75c8795655a8836eab6a11a630352c4b7c5dc5c54d075077bc9bffdeee45/lupa-2.8-cp312-abi3-win32.whl", hash = "sha256:360056453a7a4eaa4ac5a204c31a5a014b1eb2ee5490603234d2ba831684f1f2", size = 1606136, upload-time = "2026-04-15T20:06:27.815Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d8/29/11a2cdd612b6f55e506292dfb6ba343216e80a693e7fe3f876ef204ce9c6/lupa-2.8-cp312-abi3-win_arm64.whl", hash = "sha256:1628371c6592a6d5650497a9e31fb2bb3a7e9883c1f301d1111265e484045af9", size = 1364495, upload-time = "2026-04-15T20:06:30.254Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4d/17/fa834b6b09ad17e7df5d0f7715d64877a125a3776ada689751a1f9dc2959/lupa-2.8-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:450650f91c48c2415b0d59ab3abfcfda3b6efb5b858205f4d4bda8ad141fa529", size = 1190111, upload-time = "2026-04-15T20:06:32.84Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ab/43/45589901b7d1a0e3a9d91d19a311fb6a56924e8571536c3f2212160fd953/lupa-2.8-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:27044f3363047f946b3d3aab9157cbd172b3538ada9ec1baef43432bf7d03a78", size = 1812999, upload-time = "2026-04-15T20:06:35.664Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a1/ac/4ade7d15ff5c61758d7943ac6f0a496bf1cc65b6c09f842b52a0702e664c/lupa-2.8-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8cf4f064a0e5531afce2d7d750120c10c10f9529139af6ca6150d13151034398", size = 2368731, upload-time = "2026-04-15T20:06:37.959Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0c/27/05f950d15b8ab120b39c43588b438ff3ace70c1b1b0225a960393a497483/lupa-2.8-cp312-cp312-win_amd64.whl", hash = "sha256:281bedc5deb92d31e649a3552edd662449365a635904fa4d5cb4509c7245e34e", size = 1941809, upload-time = "2026-04-15T20:06:40.302Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a6/3f/19f83c3a0c84dc8bea8a58e7416dca6a3ede662c33c8d1ec758e5afc754a/lupa-2.8-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:45fc9da0145ecb0083ef5ff9975116cc784bd0258bdc2bd131ba15483ce18398", size = 1201203, upload-time = "2026-04-15T20:06:42.169Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/89/0f/a14f0073f09610158038582e230618a48c14da6bd88185289461aa4cb854/lupa-2.8-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:58e18afed57955b41130e269c78f53d4123ab86e236b53816f4cbffa25cb5d30", size = 1806210, upload-time = "2026-04-15T20:06:45.486Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2f/14/48fff156c63a136001a7620878af7d31aa07e66b495ed621e3eddd73c294/lupa-2.8-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fc47f536ac13a79cef47d29a2b205576a22841f042a2bcec1676b95806e7706a", size = 2359005, upload-time = "2026-04-15T20:06:47.819Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fe/18/3ac638ec90edf178242b8a2b2f00f8adae694248c03a26341ef941bb746e/lupa-2.8-cp313-cp313-win_amd64.whl", hash = "sha256:ce9404c661dbac65cc9bed351ad45e797af93d30d70be309a3fa8209ac86d93b", size = 1936754, upload-time = "2026-04-15T20:06:50.448Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/ef/5ee5fed6ea7459a671196359ce04bfeeaf26be1dac8ff24bf28e5c7a6e81/lupa-2.8-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:348c3f8ecabb6324dcbc05c2740d762ef8fcec7b06c79e45262ab97a217684e3", size = 1209388, upload-time = "2026-04-15T20:06:53.022Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6e/b1/67a940d5542cb0384b443fe951b5a83ea9340d1333a733a258fdd1c619ba/lupa-2.8-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:951496471056061598a7d1729a6cdf48d662fec777a9f2d8aa5a1e62fd30e5a5", size = 1826821, upload-time = "2026-04-15T20:06:55.699Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a1/a2/b354e5ba3b911ec50686003dc8897e892b9e8c5c036b33219b03d54c4daf/lupa-2.8-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a591b9947ca347b41a63370e121d6e2b1458fe6dde9ae065029ec10a37f25ff4", size = 2366893, upload-time = "2026-04-15T20:06:58.9Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8e/52/d76066401f29539df5352f70ecded66576f32933b6045cd0bfc56cb770b9/lupa-2.8-cp314-cp314-win_amd64.whl", hash = "sha256:3903c9cf628dae2f56405503247b77a61a3a61bd2dda470e336950c74776d55d", size = 1994716, upload-time = "2026-04-15T20:07:19.194Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/bd/3efc437a4361c16d25e66478c50357c9a8e8ecfb718fe749eb9ca3176ef6/lupa-2.8-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f711a8ab0486b9ac6fdda94a22ddcfbc9f0d4a27e3a8cf1bf79c6e48b33017c1", size = 1251217, upload-time = "2026-04-15T20:07:01.64Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ea/f4/2e9f8ecbaca854bfdf14af8a9b505ec0cbc640377b3b218921594b7563cd/lupa-2.8-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dc51250e76367a3e27fcd01dc769b9bfcbbc34f48df48dde53d6af6e75b7eaa5", size = 1814701, upload-time = "2026-04-15T20:07:04.149Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ba/53/4000b1acaa8b1f3827fcff0cfcdff44d3befddda42cab7e685a49689b5a1/lupa-2.8-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8a22088a552828958603323f0a5c4b3e11e03b75d0bf4c965ef879de9b60a8d", size = 2348414, upload-time = "2026-04-15T20:07:07.285Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d5/78/26ee48d3890cddf03cefb65f433e3492759c0b3c0582180755bddbaab7bd/lupa-2.8-cp314-cp314t-win32.whl", hash = "sha256:4f7c553c1d8cfffbe85d81daef730d12cae4b6002d457542914da0ac8a1145b3", size = 1831611, upload-time = "2026-04-15T20:07:09.752Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3c/d1/4a5cc64a3cad22821ae4c3f7a90456a08ca19457d8354f4abf46ad03c7e8/lupa-2.8-cp314-cp314t-win_amd64.whl", hash = "sha256:d8766aff03a78c80ad2d188a8bdb216de5ec838359cd87e05bbdfa56394a6105", size = 2209250, upload-time = "2026-04-15T20:07:11.906Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/37/7c/cdcb654daf668192aaf36b0aeb94f2281dad092aaa5003688691131736ea/lupa-2.8-cp314-cp314t-win_arm64.whl", hash = "sha256:91d622777febda3ab1bed1d45295f2f32a4680c7b3d7caf8c669998ed5c44118", size = 1126735, upload-time = "2026-04-15T20:07:15.434Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/1d/44/de1961ad38e17cd326a53c246c7e3b91178ed578f4cf22ffcd5e7e11b041/lupa-2.8-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:b036738282a5acd2e71fdddb317c9df8b87c1673aa57f403d05fcc2be8abc4ba", size = 1186020, upload-time = "2026-04-15T20:07:35.017Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/13/c2/276f0b9dc8bcc5a8a58af5316dfa0e6f56be3613dd6dbcc8d3d2cb6559ba/lupa-2.8-cp39-abi3-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:ac6b6e8d0e617e26a98cbb44880bcd75de5d32b3ad7b3b3793583909292b47ed", size = 1468944, upload-time = "2026-04-15T20:07:37.782Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/63/38/52934e52a5180dc6425d20284d004fe4b27a4f9171a82dc99fb67af250bf/lupa-2.8-cp39-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:ba3a7dd839f90c3d2e53bebe3c192b1f3f9fd720a6781256405123211fd0dce6", size = 1172998, upload-time = "2026-04-15T20:07:40.812Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/82/76b3809bd0839d9b3b4ec58d06591e08f17337b6d9576877cb9d48b34e94/lupa-2.8-cp39-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d7edb13a7a5250b5c6c22d1495d9e842b5c9fc5081c8fe6b5efe2112fe3e41f9", size = 1449975, upload-time = "2026-04-15T20:07:44.262Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/16/07/2f89d54f747c67c23b4b9ae4aa8c8dd06bb409155dedcf406157f2736b66/lupa-2.8-cp39-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:891f72e0bffbed1e4175f975aeb2a083956586a100066525e1be485f617f7b25", size = 1281944, upload-time = "2026-04-15T20:07:46.458Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/bd/7375d2b0fcae79d806baf52a76f26c96964593f58e1372d13ae5ac09c676/lupa-2.8-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:a295f87b5b7ebbfd5191932e8cb0e51df3c7769101ac6b6c7d7c9fb27bfd1307", size = 1910455, upload-time = "2026-04-15T20:07:49.75Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/8b/0c/8abb3bc0e08b311fc01db05b6e9f9ff31a8f65e4fc3f0aeb05cfef75c8ac/lupa-2.8-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4fe5d7a810b64ea8511eb885fc8cdde042ee5ff7b7d08ae78f32449756acb177", size = 1155548, upload-time = "2026-04-15T20:07:52.657Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/80/2e/9eeecd3f493099721c1d3f31beeca23a4237db1a54223684df4dc96aa1bd/lupa-2.8-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:bfc470012ef66ad064c7bd77416af03a3452ef630b04b9012595ea13f2e54518", size = 1489232, upload-time = "2026-04-15T20:07:54.92Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/13/731c99dc2e7652ae818a6de45bdf0142049f7cb566049061c898355f1891/lupa-2.8-cp39-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:250e035fdaffe8c87093e3ebc206ac29a26131b1568ea711d780c26001ce96e7", size = 1466321, upload-time = "2026-04-15T20:07:57.627Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/de/71/3ad8cc4fc05a77dc0d3f7079348bd1cad4675a0d14c24f8e6a3ce5f008f7/lupa-2.8-cp39-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:b9bddb09acfffb4f828f790f444b11dc0cca591afea1a244d9329eea2d20c003", size = 1288577, upload-time = "2026-04-15T20:07:59.913Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d8/b2/1175f6d0aa7b68627fbe2f58bd1e8bea36a89d10dfd67671d2b024c96162/lupa-2.8-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:2e64acbbd47e9b82a64405a39e0d2b36a5a7dad8ab41c0f3437f572f7d282ba3", size = 2444866, upload-time = "2026-04-15T20:08:02.753Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/92/f7/e78df680c7a0ea452daac07467ca188d63c2c00ca1c884c0a50e27eb83b5/lupa-2.8-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:32e4e5103bbddcdd2458fb2ccae6c8ba11c9997c711d7e379e0d45551d109c76", size = 1778509, upload-time = "2026-04-15T20:08:21.784Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e6/23/0e53cabb16b2a8aa9cf1fde499c097d8942c5dab709fc8e921f3b824b18b/lupa-2.8-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7667001804657496dee9feced2daae5000b4604a3218dd8e6b7b754982ba88b8", size = 2300480, upload-time = "2026-04-15T20:08:24.394Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/7e/85/0271227eab939921a12ebba5d17aa4cd18346aa534ca7f5da09cd0b63dd4/lupa-2.8-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:86f6f668966965b15247dc32d064cfe7be67b71e584ccfacbe2f637575296878", size = 1847445, upload-time = "2026-04-15T20:08:27.031Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lxml"
|
||||
version = "6.1.1"
|
||||
|
||||
Reference in New Issue
Block a user