mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
feat/https-compose-caddy
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
602ca65126 |
feat(proxy): expose shared Redis store availability in /health and /metrics (#4578)
## Description With `HEADROOM_COMPRESS_SESSION_STORE=redis://...` or the Redis CCR store, a Redis outage is silent: `Guard` fails open (by design) but `/health` stays `healthy` and `/metrics` has no series. This exposes the state `Guard` already tracks (`_skip_until`, `_failing`) without changing the fail-open behaviour. Closes #4512. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Breaking change - [ ] Documentation ## Changes Made - `headroom/cache/redis_client.py`: `Guard(feature, label)` registers itself under a short label; new `Guard.available` (false during backoff or while any call kind is failing) and `guards()` snapshot. - CCR backend uses label `ccr`; shared `/v1/compress` sessions use `compress_session`. - `/metrics`: `headroom_redis_available{feature="..."}` gauge (1/0), emitted only when a Redis store is configured. - `/health`: optional `session_store` component built with `_component_health(optional=True)`, with per-feature availability. Shows `degraded` during an outage and is excluded from the top-level `ready`, like `kompress`, so a shared-Redis outage never drains every replica. - `tests/test_redis_availability.py`: availability transitions, gauge output, health payload. ## Testing - [x] Unit tests added/updated - [x] Existing tests pass locally (failures below are pre-existing on `upstream/main`) ``` $ pytest tests/test_redis_availability.py tests/test_ccr_redis_backend.py tests/test_proxy_health.py -q 5 failed, 52 passed, 4 skipped # same 5 failures on clean upstream/main (5 failed, 40 passed): local env ModuleNotFoundError / rust-core SystemExit $ ruff check <touched files> && ruff format --check <touched files> All checks passed! / 6 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, fakeredis, branch feat/4512-redis-availability - Exact command / steps: pytest tests/test_redis_availability.py (forces redis.ConnectionError through Guard, then renders /metrics and GET /health via TestClient) - Observed result: during backoff `headroom_redis_available{feature="compress_session"} 0` and `session_store.status == "degraded"` with top-level `ready: true`; after a successful call the gauge returns to 1 and the component to `healthy` - Not tested: a live multi-replica deployment against a real Redis server ## Runtime Rollout Safety - Rollout-managed feature(s): none; observability only - Minimum rollout channel: stable - Stable/default behavior changed: no; without Redis configured the gauge is absent and `session_store` reports `disabled` - Kill switch / disable path: unset the Redis store settings - Unsafe override required: no - Qualification impact: none; new optional health component does not affect readiness - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4578" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `c54bb1a33210f5f20c5a135bc6b3608f5429716d`, upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. Mechanically updated from current upstream main without rewriting contributor Redis/lifecycle logic. All 167 focused availability, Redis-backend, proxy-health and compression-store tests pass with isolated redis 8.1.0 / fakeredis 2.40.0 qualification packages; ten shared Gemini request-fixture cases also pass after the test-only repair for independently verified main failures. Changed Python Ruff/format and main-relative whitespace checks pass. A bounded 100,000-registration/snapshot guard stress probe reports no error; this is not a universal concurrency guarantee. Independent real-service proof uses an owned local redis:7-alpine container, the actual Python Redis backend/shared-session store and a real Uvicorn HTTP listener. /readyz and /metrics report both features unknown before a command, available after successful Redis operations, unavailable during Docker pause, and available after unpause plus the unchanged five-second backoff. Redis is optional, so top-level readiness stays true throughout. No Redis client/service mocks are used. Proxy shutdown preserves the process-wide in-memory CCR store's identity and original, removes its owned session guard, and explicit Redis-store close removes that guard. No multi-replica, TLS, cluster or Redis-server restart qualification is claimed. An existing native binary was used only for proxy startup, not a fresh native-build claim. Reproduction: on Windows/Python 3.12 with Docker and the proxy dependencies, install redis==8.1.0 into an isolated dependency directory. Save the first program as pr4578-runtime.py and the cleanup driver as run-pr4578-runtime.py, adapt their scratch/checkouts/dependency paths, and run the driver. Its child operates only on an owned container and scratch directory; the parent removes the remaining cache database after the child exits and releases Windows handles. Pause/unpause retains the same published Redis endpoint; Docker stop/start with an ephemeral port was independently observed to reassign it and is not used for the qualification below. <details><summary>Reproduction program</summary> ```python import json import os from pathlib import Path import re import socket import subprocess import sys import tempfile import threading import time import uuid sys.stdout.reconfigure(encoding='utf-8') sys.path.insert(0,sys.argv[1]);sys.path.insert(0,sys.argv[2]) owned=Path('G:/git/headroom/.codex').resolve() def docker(*args): p=subprocess.run(['docker',*args],capture_output=True,text=True,encoding='utf-8',timeout=60,creationflags=subprocess.CREATE_NO_WINDOW) assert p.returncode==0,(args,p.stderr) return p.stdout.strip() cid=None;server=None;thread=None;store=None with tempfile.TemporaryDirectory(prefix='pr4578-runtime-',dir=owned,ignore_cleanup_errors=True) as directory: root=Path(directory).resolve();assert root.is_relative_to(owned) (owned/'pr4578-owned-root.txt').write_text(str(root),encoding='utf-8') os.environ.update(HEADROOM_OFFLINE='1',DO_NOT_TRACK='1',HEADROOM_BEACON_ENABLED='0',HEADROOM_SKIP_UPSTREAM_CHECK='1',HEADROOM_WORKSPACE_DIR=str(root),HEADROOM_CCR_BACKEND='memory') try: cid=docker('run','--detach','--name','headroom-review-4578-'+uuid.uuid4().hex[:10],'--publish','127.0.0.1::6379','redis:7-alpine','redis-server','--save','','--appendonly','no') assert re.fullmatch(r'[a-f0-9]{64}',cid) port=int(docker('port',cid,'6379/tcp').rsplit(':',1)[1]);url=f'redis://127.0.0.1:{port}' os.environ['HEADROOM_COMPRESS_SESSION_STORE']=url import redis admin=redis.Redis.from_url(url,socket_connect_timeout=1,socket_timeout=1) for _ in range(50): try: if admin.ping():break except redis.RedisError:time.sleep(.1) else:raise AssertionError('Redis did not start') admin.close() import httpx import uvicorn from headroom.cache.backends.redis import RedisBackend from headroom.cache.compression_store import CompressionStore,get_compression_store from headroom.cache.prefix_tracker import SessionTrackerStore from headroom.cache.redis_client import BACKOFF_SECONDS,guards from headroom.proxy.server import ProxyConfig,create_app app=create_app(ProxyConfig(optimize=False,cache_enabled=False,rate_limit_enabled=False,cost_tracking_enabled=False,subscription_tracking_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False)) listener=socket.socket();listener.bind(('127.0.0.1',0));http_port=listener.getsockname()[1] server=uvicorn.Server(uvicorn.Config(app,log_level='error')) thread=threading.Thread(target=lambda:server.run(sockets=[listener]),daemon=True);thread.start() deadline=time.monotonic()+30 while not server.started: assert thread.is_alive() and time.monotonic()<deadline,'proxy startup failed' time.sleep(.05) sessions=app.state.proxy.shared_compress_sessions backend=RedisBackend(url);store=CompressionStore(backend=backend,enable_feedback=False) global_store=get_compression_store();global_key=global_store.store('process-wide original','x') results=[] def snapshot(stage,expected): base=f'http://127.0.0.1:{http_port}' response=httpx.get(base+'/readyz',timeout=10);payload=response.json() metrics=httpx.get(base+'/metrics',timeout=10);assert metrics.status_code==200 states=payload['checks']['session_store']['features'] assert states=={'ccr':expected,'compress_session':expected},(stage,payload) assert response.status_code==200 and payload['ready'] is True,(stage,payload) for label in states: assert f'headroom_redis_checked{{feature="{label}"}} {int(expected!="unknown")}' in metrics.text gauge=f'headroom_redis_available{{feature="{label}"}}' if expected=='unknown':assert gauge not in metrics.text else:assert gauge+' '+str(int(expected=='available')) in metrics.text results.append({'stage':stage,'ready':payload['ready'],'states':states}) snapshot('configured_before_any_command','unknown') key=store.store('live Redis original','x');assert store.retrieve(key).original_content=='live Redis original' sessions.fetch(SessionTrackerStore(),'review-missing',None) snapshot('observed_success','available') docker('pause',cid) assert store.retrieve(key) is None sessions.fetch(SessionTrackerStore(),'review-missing',None) snapshot('redis_paused_fail_open','unavailable') docker('unpause',cid) time.sleep(BACKOFF_SECONDS+.2) recovered=store.store('recovered Redis original','x');assert store.retrieve(recovered).original_content=='recovered Redis original' sessions.fetch(SessionTrackerStore(),'review-missing',None) snapshot('redis_unpaused_after_backoff','available') server.should_exit=True;thread.join(timeout=15);assert not thread.is_alive() assert get_compression_store() is global_store assert global_store.retrieve(global_key).original_content=='process-wide original' assert 'compress_session' not in guards() and 'ccr' in guards() store.close();store=None;assert 'ccr' not in guards() results.append({'stage':'proxy_shutdown_and_store_close','global_store_identity_and_original_preserved':True,'owned_session_guard_removed':True,'ccr_guard_removed_on_store_close':True}) print(json.dumps({'source':sys.argv[1],'redis_python':redis.__version__,'container_image':'redis:7-alpine','results':results},indent=2)) finally: if server is not None:server.should_exit=True if thread is not None:thread.join(timeout=15) if store is not None:store.close() if cid is not None:docker('rm','--force',cid) ``` </details> <details><summary>Observed output: pr4578-e4a-runtime.log</summary> ```json { "source": "G:/git/headroom-review4578", "redis_python": "8.1.0", "container_image": "redis:7-alpine", "results": [ { "stage": "configured_before_any_command", "ready": true, "states": { "ccr": "unknown", "compress_session": "unknown" } }, { "stage": "observed_success", "ready": true, "states": { "ccr": "available", "compress_session": "available" } }, { "stage": "redis_paused_fail_open", "ready": true, "states": { "ccr": "unavailable", "compress_session": "unavailable" } }, { "stage": "redis_unpaused_after_backoff", "ready": true, "states": { "ccr": "available", "compress_session": "available" } }, { "stage": "proxy_shutdown_and_store_close", "global_store_identity_and_original_preserved": true, "owned_session_guard_removed": true, "ccr_guard_removed_on_store_close": true } ] } ``` </details> <details><summary>Reproduction program</summary> ```python import os from pathlib import Path import shutil import subprocess import sys root=Path('G:/git/headroom/.codex').resolve() env=os.environ.copy();env['PYTHONIOENCODING']='utf-8' with (root/'pr4578-e4a-runtime.log').open('w',encoding='utf-8') as out,(root/'pr4578-e4a-runtime-stderr.log').open('w',encoding='utf-8') as err: result=subprocess.run([sys.executable,str(root/'pr4578-runtime.py'),'G:/git/headroom-review4578',str(root/'pr4578-redis-deps')],stdout=out,stderr=err,env=env) manifest=root/'pr4578-owned-root.txt' if manifest.exists(): target=Path(manifest.read_text(encoding='utf-8')).resolve() assert target.parent==root and target.name.startswith('pr4578-runtime-'),target if target.exists():shutil.rmtree(target) print('Runtime exit',result.returncode,'owned scratch cleaned',flush=True) sys.exit(result.returncode) ``` </details> Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. ### Maintainer current-main requalification Exact head `a06923dc6cab75e2ed0d518922d6dc90003a3fd8`, upstream main `de1a1e0f478d1eb9257dd8dd0d4ac87c9e6fc639`. Clean integration with upstream/main |