mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description With `HEADROOM_COMPRESS_SESSION_STORE=redis://...` or the Redis CCR store, a Redis outage is silent: `Guard` fails open (by design) but `/health` stays `healthy` and `/metrics` has no series. This exposes the state `Guard` already tracks (`_skip_until`, `_failing`) without changing the fail-open behaviour. Closes #4512. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Breaking change - [ ] Documentation ## Changes Made - `headroom/cache/redis_client.py`: `Guard(feature, label)` registers itself under a short label; new `Guard.available` (false during backoff or while any call kind is failing) and `guards()` snapshot. - CCR backend uses label `ccr`; shared `/v1/compress` sessions use `compress_session`. - `/metrics`: `headroom_redis_available{feature="..."}` gauge (1/0), emitted only when a Redis store is configured. - `/health`: optional `session_store` component built with `_component_health(optional=True)`, with per-feature availability. Shows `degraded` during an outage and is excluded from the top-level `ready`, like `kompress`, so a shared-Redis outage never drains every replica. - `tests/test_redis_availability.py`: availability transitions, gauge output, health payload. ## Testing - [x] Unit tests added/updated - [x] Existing tests pass locally (failures below are pre-existing on `upstream/main`) ``` $ pytest tests/test_redis_availability.py tests/test_ccr_redis_backend.py tests/test_proxy_health.py -q 5 failed, 52 passed, 4 skipped # same 5 failures on clean upstream/main (5 failed, 40 passed): local env ModuleNotFoundError / rust-core SystemExit $ ruff check <touched files> && ruff format --check <touched files> All checks passed! / 6 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, fakeredis, branch feat/4512-redis-availability - Exact command / steps: pytest tests/test_redis_availability.py (forces redis.ConnectionError through Guard, then renders /metrics and GET /health via TestClient) - Observed result: during backoff `headroom_redis_available{feature="compress_session"} 0` and `session_store.status == "degraded"` with top-level `ready: true`; after a successful call the gauge returns to 1 and the component to `healthy` - Not tested: a live multi-replica deployment against a real Redis server ## Runtime Rollout Safety - Rollout-managed feature(s): none; observability only - Minimum rollout channel: stable - Stable/default behavior changed: no; without Redis configured the gauge is absent and `session_store` reports `disabled` - Kill switch / disable path: unset the Redis store settings - Unsafe override required: no - Qualification impact: none; new optional health component does not affect readiness - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4578" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `c54bb1a33210f5f20c5a135bc6b3608f5429716d`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Mechanically updated from current upstream main without rewriting contributor Redis/lifecycle logic. All 167 focused availability, Redis-backend, proxy-health and compression-store tests pass with isolated redis 8.1.0 / fakeredis 2.40.0 qualification packages; ten shared Gemini request-fixture cases also pass after the test-only repair for independently verified main failures. Changed Python Ruff/format and main-relative whitespace checks pass. A bounded 100,000-registration/snapshot guard stress probe reports no error; this is not a universal concurrency guarantee. Independent real-service proof uses an owned local redis:7-alpine container, the actual Python Redis backend/shared-session store and a real Uvicorn HTTP listener. /readyz and /metrics report both features unknown before a command, available after successful Redis operations, unavailable during Docker pause, and available after unpause plus the unchanged five-second backoff. Redis is optional, so top-level readiness stays true throughout. No Redis client/service mocks are used. Proxy shutdown preserves the process-wide in-memory CCR store's identity and original, removes its owned session guard, and explicit Redis-store close removes that guard. No multi-replica, TLS, cluster or Redis-server restart qualification is claimed. An existing native binary was used only for proxy startup, not a fresh native-build claim. Reproduction: on Windows/Python 3.12 with Docker and the proxy dependencies, install redis==8.1.0 into an isolated dependency directory. Save the first program as pr4578-runtime.py and the cleanup driver as run-pr4578-runtime.py, adapt their scratch/checkouts/dependency paths, and run the driver. Its child operates only on an owned container and scratch directory; the parent removes the remaining cache database after the child exits and releases Windows handles. Pause/unpause retains the same published Redis endpoint; Docker stop/start with an ephemeral port was independently observed to reassign it and is not used for the qualification below. <details><summary>Reproduction program</summary> ```python import json import os from pathlib import Path import re import socket import subprocess import sys import tempfile import threading import time import uuid sys.stdout.reconfigure(encoding='utf-8') sys.path.insert(0,sys.argv[1]);sys.path.insert(0,sys.argv[2]) owned=Path('G:/git/headroom/.codex').resolve() def docker(*args): p=subprocess.run(['docker',*args],capture_output=True,text=True,encoding='utf-8',timeout=60,creationflags=subprocess.CREATE_NO_WINDOW) assert p.returncode==0,(args,p.stderr) return p.stdout.strip() cid=None;server=None;thread=None;store=None with tempfile.TemporaryDirectory(prefix='pr4578-runtime-',dir=owned,ignore_cleanup_errors=True) as directory: root=Path(directory).resolve();assert root.is_relative_to(owned) (owned/'pr4578-owned-root.txt').write_text(str(root),encoding='utf-8') os.environ.update(HEADROOM_OFFLINE='1',DO_NOT_TRACK='1',HEADROOM_BEACON_ENABLED='0',HEADROOM_SKIP_UPSTREAM_CHECK='1',HEADROOM_WORKSPACE_DIR=str(root),HEADROOM_CCR_BACKEND='memory') try: cid=docker('run','--detach','--name','headroom-review-4578-'+uuid.uuid4().hex[:10],'--publish','127.0.0.1::6379','redis:7-alpine','redis-server','--save','','--appendonly','no') assert re.fullmatch(r'[a-f0-9]{64}',cid) port=int(docker('port',cid,'6379/tcp').rsplit(':',1)[1]);url=f'redis://127.0.0.1:{port}' os.environ['HEADROOM_COMPRESS_SESSION_STORE']=url import redis admin=redis.Redis.from_url(url,socket_connect_timeout=1,socket_timeout=1) for _ in range(50): try: if admin.ping():break except redis.RedisError:time.sleep(.1) else:raise AssertionError('Redis did not start') admin.close() import httpx import uvicorn from headroom.cache.backends.redis import RedisBackend from headroom.cache.compression_store import CompressionStore,get_compression_store from headroom.cache.prefix_tracker import SessionTrackerStore from headroom.cache.redis_client import BACKOFF_SECONDS,guards from headroom.proxy.server import ProxyConfig,create_app app=create_app(ProxyConfig(optimize=False,cache_enabled=False,rate_limit_enabled=False,cost_tracking_enabled=False,subscription_tracking_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False)) listener=socket.socket();listener.bind(('127.0.0.1',0));http_port=listener.getsockname()[1] server=uvicorn.Server(uvicorn.Config(app,log_level='error')) thread=threading.Thread(target=lambda:server.run(sockets=[listener]),daemon=True);thread.start() deadline=time.monotonic()+30 while not server.started: assert thread.is_alive() and time.monotonic()<deadline,'proxy startup failed' time.sleep(.05) sessions=app.state.proxy.shared_compress_sessions backend=RedisBackend(url);store=CompressionStore(backend=backend,enable_feedback=False) global_store=get_compression_store();global_key=global_store.store('process-wide original','x') results=[] def snapshot(stage,expected): base=f'http://127.0.0.1:{http_port}' response=httpx.get(base+'/readyz',timeout=10);payload=response.json() metrics=httpx.get(base+'/metrics',timeout=10);assert metrics.status_code==200 states=payload['checks']['session_store']['features'] assert states=={'ccr':expected,'compress_session':expected},(stage,payload) assert response.status_code==200 and payload['ready'] is True,(stage,payload) for label in states: assert f'headroom_redis_checked{{feature="{label}"}} {int(expected!="unknown")}' in metrics.text gauge=f'headroom_redis_available{{feature="{label}"}}' if expected=='unknown':assert gauge not in metrics.text else:assert gauge+' '+str(int(expected=='available')) in metrics.text results.append({'stage':stage,'ready':payload['ready'],'states':states}) snapshot('configured_before_any_command','unknown') key=store.store('live Redis original','x');assert store.retrieve(key).original_content=='live Redis original' sessions.fetch(SessionTrackerStore(),'review-missing',None) snapshot('observed_success','available') docker('pause',cid) assert store.retrieve(key) is None sessions.fetch(SessionTrackerStore(),'review-missing',None) snapshot('redis_paused_fail_open','unavailable') docker('unpause',cid) time.sleep(BACKOFF_SECONDS+.2) recovered=store.store('recovered Redis original','x');assert store.retrieve(recovered).original_content=='recovered Redis original' sessions.fetch(SessionTrackerStore(),'review-missing',None) snapshot('redis_unpaused_after_backoff','available') server.should_exit=True;thread.join(timeout=15);assert not thread.is_alive() assert get_compression_store() is global_store assert global_store.retrieve(global_key).original_content=='process-wide original' assert 'compress_session' not in guards() and 'ccr' in guards() store.close();store=None;assert 'ccr' not in guards() results.append({'stage':'proxy_shutdown_and_store_close','global_store_identity_and_original_preserved':True,'owned_session_guard_removed':True,'ccr_guard_removed_on_store_close':True}) print(json.dumps({'source':sys.argv[1],'redis_python':redis.__version__,'container_image':'redis:7-alpine','results':results},indent=2)) finally: if server is not None:server.should_exit=True if thread is not None:thread.join(timeout=15) if store is not None:store.close() if cid is not None:docker('rm','--force',cid) ``` </details> <details><summary>Observed output: pr4578-e4a-runtime.log</summary> ```json { "source": "G:/git/headroom-review4578", "redis_python": "8.1.0", "container_image": "redis:7-alpine", "results": [ { "stage": "configured_before_any_command", "ready": true, "states": { "ccr": "unknown", "compress_session": "unknown" } }, { "stage": "observed_success", "ready": true, "states": { "ccr": "available", "compress_session": "available" } }, { "stage": "redis_paused_fail_open", "ready": true, "states": { "ccr": "unavailable", "compress_session": "unavailable" } }, { "stage": "redis_unpaused_after_backoff", "ready": true, "states": { "ccr": "available", "compress_session": "available" } }, { "stage": "proxy_shutdown_and_store_close", "global_store_identity_and_original_preserved": true, "owned_session_guard_removed": true, "ccr_guard_removed_on_store_close": true } ] } ``` </details> <details><summary>Reproduction program</summary> ```python import os from pathlib import Path import shutil import subprocess import sys root=Path('G:/git/headroom/.codex').resolve() env=os.environ.copy();env['PYTHONIOENCODING']='utf-8' with (root/'pr4578-e4a-runtime.log').open('w',encoding='utf-8') as out,(root/'pr4578-e4a-runtime-stderr.log').open('w',encoding='utf-8') as err: result=subprocess.run([sys.executable,str(root/'pr4578-runtime.py'),'G:/git/headroom-review4578',str(root/'pr4578-redis-deps')],stdout=out,stderr=err,env=env) manifest=root/'pr4578-owned-root.txt' if manifest.exists(): target=Path(manifest.read_text(encoding='utf-8')).resolve() assert target.parent==root and target.name.startswith('pr4578-runtime-'),target if target.exists():shutil.rmtree(target) print('Runtime exit',result.returncode,'owned scratch cleaned',flush=True) sys.exit(result.returncode) ``` </details> Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `a06923dc6cab75e2ed0d518922d6dc90003a3fd8`, upstream main `de1a1e0f478d1eb9257dd8dd0d4ac87c9e6fc639`. Clean integration with upstream/mainde1a1e0f47and re-review against main compression-store/lifecycle changes.148 Redis-availability/backend/compression-store/Gemini tests pass with isolated redis8.1.0/fakeredis dependencies; Ruff and whitespace pass. Initial local failures were only missing optional redis in the shared environment, resolved by loading the existing isolated test overlay; no production fix or weakened assertion. Availability remains tri-state, optional Redis does not flip overall readiness, owned shared sessions close on proxy shutdown while process-wide CCR storage remains attached, and detach unregisters the former guard without closing storage still referenced elsewhere. Existing actual Redis+TCP pause/recovery/shutdown evidence remains scoped to unchanged behavior; no fresh live-network/native-build claim. No new source finding; final approval awaits exact-head hosted qualification. Nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `6a0368b3c436477390e8adc370a343da370a986b`, upstream main `c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`. Final post-Dependabot requalification against upstream/mainc01cf07d94. All six Dependabot updates are now merged. Relative to this branch's freshly tested prepared tree, the final main sync changes only the three intended Cargo.lock records; Python/source/test inputs are unchanged. Re-reviewed the integration and confirmed complete native source/lock/toolchain objects match the tested combined build and exact fresh native DLL used for this branch's local tests. Focused fresh-native qualification: 148 passed, 1 warning in 18.84s. Selected real dependency overlays and native DLL paths are recorded in final4578-fresh-native-tests.log; the DLL provenance/input hashes are in final-native-artifact-proof.json. No redundant rerun is implied after this Cargo-only merge. Whitespace passes. Prior source review and scoped real-behavior evidence remain applicable to unchanged contributor inputs. No new source finding in the reviewed scope. Final formal approval awaits exact-head hosted qualification. Maintainers handle hosted checks; no external-provider claim, contributor semantic rewrite or non-Dependabot merge. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `a83a2c5a0e378cdd8e3850e4cc5256902dd47daf`, upstream main `2b0315b2d4b0d54947050bbdad2de9e89df06378`. Requalified against upstream main2b0315b2d4. All 71 integration patches match the main delta after diff-location normalization. The Redis client/guard, backend, shared-session, compression-store and metric implementations are byte-identical to the preceding reviewed head; current server integration retains their ownership and health contracts. The fresh Redis availability/backend/store/Gemini/doctor suite passes 279 tests with Redis 8.1.0 and one deprecation warning. Ruff lint/format passes for all 33 changed Python files; offline lock and whitespace checks pass. The existing native DLL matches its complete recorded input hashes and artifact checksum; no fresh build is claimed. Current author and Devin discussions were considered, including unknown-versus-available reporting, inactive guards and process-wide store ownership. The previously recorded real Redis pause/recovery and shutdown-ownership evidence remains scoped to unchanged feature inputs. It was not repeated here because the Docker Linux daemon is unavailable; no fresh live-Redis run is claimed. No new source blocker found. Exact-head hosted qualification is pending before approval. Maintainers handle CI; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `712a91416a93fc090adcde9154ddd17c3d9628b5`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean integration with upstream/main0ddaa56c21. All 17 integration patches exactly match the prior-reviewed-main delta after diff-location normalization; Redis availability, lifecycle and metrics logic is unchanged. Fresh Redis availability/backend/compression-store and incoming Vertex/backend/CLI suite: 296 passed, 11 skipped, 1 warning using the previously qualified Redis 8.1.0 overlay. Initial execution without that optional client had 11 missing-redis failures; those were environment setup errors, corrected without modifying PR code. Twelve changed Python files pass Ruff lint/format; offline lock and whitespace pass. Author/Devin discussion reviewed, including explicit unknown state, configured lazy CCR, inactive guard ownership and detach lifecycle. Prior real Redis pause/recovery and shutdown evidence applies to unchanged feature inputs; no fresh live Redis or cloud-provider run claimed. Static source/test review found no new credential/URL disclosure or unrelated execution/workflow/dependency changes; health exposes feature state, not Redis URL or credentials. Flagged inherited workflow-validation job is excluded by unchanged workflow paths. Native DLL reused after complete tracked input-set/hash and artifact checksum verification; no fresh build claimed. Formal approval awaits exact published-head hosted qualification. Maintainers handle checks; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
359 lines
13 KiB
Python
359 lines
13 KiB
Python
"""Guard availability tracking, /metrics gauge and /health session_store (#4512).
|
|
|
|
Covers:
|
|
* ``Guard.state`` reports "unknown" before any command runs, "available" on
|
|
success, and "unavailable" on backoff/failure.
|
|
* ``headroom_redis_checked{feature=...}`` gauge appears in /metrics output for
|
|
every configured guard (1 if checked, 0 if unknown).
|
|
* ``headroom_redis_available{feature=...}`` gauge appears in /metrics output
|
|
only for observed guards (1 if available, 0 during outage).
|
|
* ``/health`` reports ``session_store`` feature state strings ("unknown" |
|
|
"available" | "unavailable") and stays degraded when unobserved or unavailable
|
|
while top-level ``ready`` stays True.
|
|
* Unregistering/closing guards on store close/detach.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.cache import redis_client # noqa: E402
|
|
from headroom.cache.redis_client import Guard, guards # noqa: E402
|
|
|
|
# --- Guard.state & Guard.available --------------------------------------------
|
|
|
|
|
|
class TestGuardAvailable:
|
|
def test_unknown_initially_before_any_call(self):
|
|
g = Guard("test feature", "test")
|
|
assert g.state == "unknown"
|
|
assert g.available is False
|
|
g.close()
|
|
|
|
def test_available_after_successful_call(self):
|
|
g = Guard("test feature", "test")
|
|
result = g.call("read", lambda: "ok", "default")
|
|
assert result == "ok"
|
|
assert g.state == "available"
|
|
assert g.available is True
|
|
g.close()
|
|
|
|
def test_unavailable_after_connection_error(self, monkeypatch):
|
|
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
|
|
g = Guard("test feature", "test")
|
|
import redis
|
|
|
|
def failing():
|
|
raise redis.ConnectionError("connection refused")
|
|
|
|
result = g.call("read", failing, "default")
|
|
assert result == "default"
|
|
assert g.state == "unavailable"
|
|
assert g.available is False
|
|
g.close()
|
|
|
|
def test_available_again_after_successful_call(self, monkeypatch):
|
|
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 0.0)
|
|
g = Guard("test feature", "test")
|
|
import redis
|
|
|
|
def failing():
|
|
raise redis.RedisError("some error")
|
|
|
|
g.call("read", failing, "default")
|
|
assert g.state == "unavailable"
|
|
assert g.available is False
|
|
|
|
result = g.call("read", lambda: "ok", "default")
|
|
assert result == "ok"
|
|
assert g.state == "available"
|
|
assert g.available is True
|
|
g.close()
|
|
|
|
def test_untested_shared_session_store_is_unknown(self, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_COMPRESS_SESSION_STORE", "redis://127.0.0.1:1")
|
|
from headroom.cache.shared_sessions import open_shared_sessions
|
|
|
|
store = open_shared_sessions()
|
|
assert store is not None
|
|
try:
|
|
assert store._guard.state == "unknown"
|
|
assert store._guard.available is False
|
|
assert guards()["compress_session"].state == "unknown"
|
|
assert guards()["compress_session"].available is False
|
|
finally:
|
|
store.close()
|
|
|
|
def test_lifecycle_close_unregisters_guard(self, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_COMPRESS_SESSION_STORE", "redis://127.0.0.1:1")
|
|
from headroom.cache.shared_sessions import open_shared_sessions
|
|
|
|
store = open_shared_sessions()
|
|
assert store is not None
|
|
assert "compress_session" in guards()
|
|
|
|
store.close()
|
|
monkeypatch.delenv("HEADROOM_COMPRESS_SESSION_STORE", raising=False)
|
|
assert open_shared_sessions() is None
|
|
assert "compress_session" not in guards()
|
|
|
|
|
|
# --- /metrics gauge ----------------------------------------------------------
|
|
|
|
|
|
class TestMetricsGauge:
|
|
@pytest.mark.asyncio
|
|
async def test_gauge_checked_and_available_when_observed(self):
|
|
g = Guard("test feature", "test_label")
|
|
g.call("read", lambda: "ok", None)
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
metrics = PrometheusMetrics(stateless=True)
|
|
text = await metrics.export()
|
|
assert "# HELP headroom_redis_checked" in text
|
|
assert "# TYPE headroom_redis_checked gauge" in text
|
|
assert 'headroom_redis_checked{feature="test_label"} 1' in text
|
|
assert "# HELP headroom_redis_available" in text
|
|
assert "# TYPE headroom_redis_available gauge" in text
|
|
assert 'headroom_redis_available{feature="test_label"} 1' in text
|
|
g.close()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_gauge_unobserved_shows_checked_0_and_no_available(self):
|
|
g = Guard("test feature", "unobserved_label")
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
metrics = PrometheusMetrics(stateless=True)
|
|
text = await metrics.export()
|
|
assert 'headroom_redis_checked{feature="unobserved_label"} 0' in text
|
|
assert "headroom_redis_available" not in text
|
|
g.close()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_gauge_zero_during_outage(self, monkeypatch):
|
|
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
|
|
import redis
|
|
|
|
g = Guard("test feature", "outage_label")
|
|
|
|
def failing():
|
|
raise redis.ConnectionError("down")
|
|
|
|
g.call("read", failing, None)
|
|
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
metrics = PrometheusMetrics(stateless=True)
|
|
text = await metrics.export()
|
|
assert 'headroom_redis_checked{feature="outage_label"} 1' in text
|
|
assert 'headroom_redis_available{feature="outage_label"} 0' in text
|
|
g.close()
|
|
|
|
|
|
# --- /health session_store ---------------------------------------------------
|
|
|
|
|
|
class TestHealthSessionStore:
|
|
def test_session_store_degraded_does_not_flip_ready(self, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
|
|
import redis
|
|
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import create_app
|
|
|
|
g = Guard("compress sessions", "compress_session")
|
|
|
|
def failing():
|
|
raise redis.ConnectionError("down")
|
|
|
|
g.call("read", failing, None)
|
|
assert g.state == "unavailable"
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
)
|
|
)
|
|
app.state.ready = True
|
|
proxy = app.state.proxy
|
|
proxy.http_client = object()
|
|
|
|
client = TestClient(app)
|
|
response = client.get("/readyz")
|
|
assert response.status_code == 200
|
|
payload = response.json()
|
|
|
|
assert payload["ready"] is True
|
|
assert payload["status"] == "healthy"
|
|
|
|
ss = payload["checks"]["session_store"]
|
|
assert ss["enabled"] is True
|
|
assert ss["ready"] is False
|
|
assert ss["status"] == "degraded"
|
|
assert ss["optional"] is True
|
|
assert ss["features"]["compress_session"] == "unavailable"
|
|
g.close()
|
|
|
|
def test_session_store_disabled_when_no_guards(self, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import create_app
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
)
|
|
)
|
|
app.state.ready = True
|
|
proxy = app.state.proxy
|
|
proxy.http_client = object()
|
|
|
|
client = TestClient(app)
|
|
response = client.get("/readyz")
|
|
assert response.status_code == 200
|
|
payload = response.json()
|
|
|
|
ss = payload["checks"]["session_store"]
|
|
assert ss["enabled"] is False
|
|
assert ss["status"] == "disabled"
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_lazy_ccr_redis_visible_on_first_scrape(self, monkeypatch):
|
|
"""Configured-but-lazy CCR Redis shows ``checked 0`` on the first
|
|
scrape, before any traffic has built the store; ``available`` stays
|
|
absent and /health reports the same unknown state. No connection is
|
|
attempted for the observation."""
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
monkeypatch.setenv("HEADROOM_STATELESS", "0")
|
|
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redis")
|
|
monkeypatch.setenv("HEADROOM_REDIS_URL", "redis://127.0.0.1:1")
|
|
|
|
from headroom.cache import compression_store, redis_client
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
from headroom.proxy.server import create_app
|
|
|
|
compression_store.reset_compression_store()
|
|
try:
|
|
# Full startup, no store created yet: no guard registered.
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
)
|
|
)
|
|
app.state.ready = True
|
|
proxy = app.state.proxy
|
|
proxy.http_client = object()
|
|
|
|
assert redis_client.guards() == {} # nothing connected
|
|
|
|
text = await PrometheusMetrics(stateless=False).export()
|
|
assert 'headroom_redis_checked{feature="ccr"} 0' in text
|
|
assert "headroom_redis_available" not in text
|
|
|
|
response = TestClient(app).get("/readyz")
|
|
assert response.status_code == 200
|
|
payload = response.json()
|
|
ss = payload["checks"]["session_store"]
|
|
assert ss["enabled"] is True
|
|
assert ss["ready"] is False
|
|
assert ss["status"] == "degraded"
|
|
assert ss["optional"] is True
|
|
assert ss["features"]["ccr"] == "unknown"
|
|
assert payload["ready"] is True
|
|
finally:
|
|
compression_store.reset_compression_store()
|
|
|
|
def test_session_store_healthy_when_available(self, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import create_app
|
|
|
|
g = Guard("CCR store", "ccr")
|
|
g.call("read", lambda: "ok", None)
|
|
assert g.state == "available"
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
)
|
|
)
|
|
app.state.ready = True
|
|
proxy = app.state.proxy
|
|
proxy.http_client = object()
|
|
|
|
client = TestClient(app)
|
|
response = client.get("/readyz")
|
|
assert response.status_code == 200
|
|
payload = response.json()
|
|
|
|
ss = payload["checks"]["session_store"]
|
|
assert ss["enabled"] is True
|
|
assert ss["ready"] is True
|
|
assert ss["status"] == "healthy"
|
|
assert ss["features"]["ccr"] == "available"
|
|
g.close()
|
|
|
|
|
|
# --- CCR store lifecycle across proxy shutdown -------------------------------
|
|
|
|
|
|
class TestCCRStoreSurvivesProxyShutdown:
|
|
@pytest.mark.asyncio
|
|
async def test_ccr_store_persists_across_proxy_shutdown(self, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
from headroom.cache.compression_store import (
|
|
get_compression_store,
|
|
reset_compression_store,
|
|
)
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import create_app
|
|
|
|
reset_compression_store()
|
|
try:
|
|
app1 = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
)
|
|
)
|
|
app1.state.ready = True
|
|
proxy1 = app1.state.proxy
|
|
proxy1.http_client = None
|
|
proxy1.http_client_h1 = None
|
|
|
|
store = get_compression_store()
|
|
original_hash = store.store(
|
|
original='{"data": "hello"}',
|
|
compressed='{"data": "[COMPRESSED:abc123]"}',
|
|
original_tokens=10,
|
|
compressed_tokens=2,
|
|
tool_name="test_tool",
|
|
)
|
|
entry_before = store.retrieve(original_hash)
|
|
assert entry_before is not None
|
|
assert entry_before.original_content == '{"data": "hello"}'
|
|
|
|
await proxy1.shutdown()
|
|
|
|
store_after = get_compression_store()
|
|
assert store_after is store
|
|
entry_after = store_after.retrieve(original_hash)
|
|
assert entry_after is not None
|
|
assert entry_after.original_content == '{"data": "hello"}'
|
|
finally:
|
|
reset_compression_store()
|