feat(proxy): expose shared Redis store availability in /health and /metrics (#4578)

## Description

With `HEADROOM_COMPRESS_SESSION_STORE=redis://...` or the Redis CCR
store, a Redis outage is silent: `Guard` fails open (by design) but
`/health` stays `healthy` and `/metrics` has no series. This exposes the
state `Guard` already tracks (`_skip_until`, `_failing`) without
changing the fail-open behaviour.

Closes #4512.

## Type of Change

- [ ] Bug fix
- [x] New feature
- [ ] Breaking change
- [ ] Documentation

## Changes Made

- `headroom/cache/redis_client.py`: `Guard(feature, label)` registers
itself under a short label; new `Guard.available` (false during backoff
or while any call kind is failing) and `guards()` snapshot.
- CCR backend uses label `ccr`; shared `/v1/compress` sessions use
`compress_session`.
- `/metrics`: `headroom_redis_available{feature="..."}` gauge (1/0),
emitted only when a Redis store is configured.
- `/health`: optional `session_store` component built with
`_component_health(optional=True)`, with per-feature availability. Shows
`degraded` during an outage and is excluded from the top-level `ready`,
like `kompress`, so a shared-Redis outage never drains every replica.
- `tests/test_redis_availability.py`: availability transitions, gauge
output, health payload.

## Testing

- [x] Unit tests added/updated
- [x] Existing tests pass locally (failures below are pre-existing on
`upstream/main`)

```
$ pytest tests/test_redis_availability.py tests/test_ccr_redis_backend.py tests/test_proxy_health.py -q
5 failed, 52 passed, 4 skipped
# same 5 failures on clean upstream/main (5 failed, 40 passed): local env ModuleNotFoundError / rust-core SystemExit
$ ruff check <touched files> && ruff format --check <touched files>
All checks passed! / 6 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, fakeredis, branch
feat/4512-redis-availability
- Exact command / steps: pytest tests/test_redis_availability.py (forces
redis.ConnectionError through Guard, then renders /metrics and GET
/health via TestClient)
- Observed result: during backoff
`headroom_redis_available{feature="compress_session"} 0` and
`session_store.status == "degraded"` with top-level `ready: true`; after
a successful call the gauge returns to 1 and the component to `healthy`
- Not tested: a live multi-replica deployment against a real Redis
server

## Runtime Rollout Safety

- Rollout-managed feature(s): none; observability only
- Minimum rollout channel: stable
- Stable/default behavior changed: no; without Redis configured the
gauge is absent and `session_store` reports `disabled`
- Kill switch / disable path: unset the Redis store settings
- Unsafe override required: no
- Qualification impact: none; new optional health component does not
affect readiness
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4578"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `c54bb1a33210f5f20c5a135bc6b3608f5429716d`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically updated from current upstream main without rewriting
contributor Redis/lifecycle logic. All 167 focused availability,
Redis-backend, proxy-health and compression-store tests pass with
isolated redis 8.1.0 / fakeredis 2.40.0 qualification packages; ten
shared Gemini request-fixture cases also pass after the test-only repair
for independently verified main failures. Changed Python Ruff/format and
main-relative whitespace checks pass. A bounded
100,000-registration/snapshot guard stress probe reports no error; this
is not a universal concurrency guarantee.

Independent real-service proof uses an owned local redis:7-alpine
container, the actual Python Redis backend/shared-session store and a
real Uvicorn HTTP listener. /readyz and /metrics report both features
unknown before a command, available after successful Redis operations,
unavailable during Docker pause, and available after unpause plus the
unchanged five-second backoff. Redis is optional, so top-level readiness
stays true throughout. No Redis client/service mocks are used. Proxy
shutdown preserves the process-wide in-memory CCR store's identity and
original, removes its owned session guard, and explicit Redis-store
close removes that guard. No multi-replica, TLS, cluster or Redis-server
restart qualification is claimed. An existing native binary was used
only for proxy startup, not a fresh native-build claim.

Reproduction: on Windows/Python 3.12 with Docker and the proxy
dependencies, install redis==8.1.0 into an isolated dependency
directory. Save the first program as pr4578-runtime.py and the cleanup
driver as run-pr4578-runtime.py, adapt their
scratch/checkouts/dependency paths, and run the driver. Its child
operates only on an owned container and scratch directory; the parent
removes the remaining cache database after the child exits and releases
Windows handles. Pause/unpause retains the same published Redis
endpoint; Docker stop/start with an ephemeral port was independently
observed to reassign it and is not used for the qualification below.


<details><summary>Reproduction program</summary>

```python
import json
import os
from pathlib import Path
import re
import socket
import subprocess
import sys
import tempfile
import threading
import time
import uuid

sys.stdout.reconfigure(encoding='utf-8')
sys.path.insert(0,sys.argv[1]);sys.path.insert(0,sys.argv[2])
owned=Path('G:/git/headroom/.codex').resolve()
def docker(*args):
    p=subprocess.run(['docker',*args],capture_output=True,text=True,encoding='utf-8',timeout=60,creationflags=subprocess.CREATE_NO_WINDOW)
    assert p.returncode==0,(args,p.stderr)
    return p.stdout.strip()

cid=None;server=None;thread=None;store=None
with tempfile.TemporaryDirectory(prefix='pr4578-runtime-',dir=owned,ignore_cleanup_errors=True) as directory:
    root=Path(directory).resolve();assert root.is_relative_to(owned)
    (owned/'pr4578-owned-root.txt').write_text(str(root),encoding='utf-8')
    os.environ.update(HEADROOM_OFFLINE='1',DO_NOT_TRACK='1',HEADROOM_BEACON_ENABLED='0',HEADROOM_SKIP_UPSTREAM_CHECK='1',HEADROOM_WORKSPACE_DIR=str(root),HEADROOM_CCR_BACKEND='memory')
    try:
        cid=docker('run','--detach','--name','headroom-review-4578-'+uuid.uuid4().hex[:10],'--publish','127.0.0.1::6379','redis:7-alpine','redis-server','--save','','--appendonly','no')
        assert re.fullmatch(r'[a-f0-9]{64}',cid)
        port=int(docker('port',cid,'6379/tcp').rsplit(':',1)[1]);url=f'redis://127.0.0.1:{port}'
        os.environ['HEADROOM_COMPRESS_SESSION_STORE']=url
        import redis
        admin=redis.Redis.from_url(url,socket_connect_timeout=1,socket_timeout=1)
        for _ in range(50):
            try:
                if admin.ping():break
            except redis.RedisError:time.sleep(.1)
        else:raise AssertionError('Redis did not start')
        admin.close()
        import httpx
        import uvicorn
        from headroom.cache.backends.redis import RedisBackend
        from headroom.cache.compression_store import CompressionStore,get_compression_store
        from headroom.cache.prefix_tracker import SessionTrackerStore
        from headroom.cache.redis_client import BACKOFF_SECONDS,guards
        from headroom.proxy.server import ProxyConfig,create_app
        app=create_app(ProxyConfig(optimize=False,cache_enabled=False,rate_limit_enabled=False,cost_tracking_enabled=False,subscription_tracking_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False))
        listener=socket.socket();listener.bind(('127.0.0.1',0));http_port=listener.getsockname()[1]
        server=uvicorn.Server(uvicorn.Config(app,log_level='error'))
        thread=threading.Thread(target=lambda:server.run(sockets=[listener]),daemon=True);thread.start()
        deadline=time.monotonic()+30
        while not server.started:
            assert thread.is_alive() and time.monotonic()<deadline,'proxy startup failed'
            time.sleep(.05)
        sessions=app.state.proxy.shared_compress_sessions
        backend=RedisBackend(url);store=CompressionStore(backend=backend,enable_feedback=False)
        global_store=get_compression_store();global_key=global_store.store('process-wide original','x')
        results=[]
        def snapshot(stage,expected):
            base=f'http://127.0.0.1:{http_port}'
            response=httpx.get(base+'/readyz',timeout=10);payload=response.json()
            metrics=httpx.get(base+'/metrics',timeout=10);assert metrics.status_code==200
            states=payload['checks']['session_store']['features']
            assert states=={'ccr':expected,'compress_session':expected},(stage,payload)
            assert response.status_code==200 and payload['ready'] is True,(stage,payload)
            for label in states:
                assert f'headroom_redis_checked{{feature="{label}"}} {int(expected!="unknown")}' in metrics.text
                gauge=f'headroom_redis_available{{feature="{label}"}}'
                if expected=='unknown':assert gauge not in metrics.text
                else:assert gauge+' '+str(int(expected=='available')) in metrics.text
            results.append({'stage':stage,'ready':payload['ready'],'states':states})
        snapshot('configured_before_any_command','unknown')
        key=store.store('live Redis original','x');assert store.retrieve(key).original_content=='live Redis original'
        sessions.fetch(SessionTrackerStore(),'review-missing',None)
        snapshot('observed_success','available')
        docker('pause',cid)
        assert store.retrieve(key) is None
        sessions.fetch(SessionTrackerStore(),'review-missing',None)
        snapshot('redis_paused_fail_open','unavailable')
        docker('unpause',cid)
        time.sleep(BACKOFF_SECONDS+.2)
        recovered=store.store('recovered Redis original','x');assert store.retrieve(recovered).original_content=='recovered Redis original'
        sessions.fetch(SessionTrackerStore(),'review-missing',None)
        snapshot('redis_unpaused_after_backoff','available')
        server.should_exit=True;thread.join(timeout=15);assert not thread.is_alive()
        assert get_compression_store() is global_store
        assert global_store.retrieve(global_key).original_content=='process-wide original'
        assert 'compress_session' not in guards() and 'ccr' in guards()
        store.close();store=None;assert 'ccr' not in guards()
        results.append({'stage':'proxy_shutdown_and_store_close','global_store_identity_and_original_preserved':True,'owned_session_guard_removed':True,'ccr_guard_removed_on_store_close':True})
        print(json.dumps({'source':sys.argv[1],'redis_python':redis.__version__,'container_image':'redis:7-alpine','results':results},indent=2))
    finally:
        if server is not None:server.should_exit=True
        if thread is not None:thread.join(timeout=15)
        if store is not None:store.close()
        if cid is not None:docker('rm','--force',cid)

```
</details>

<details><summary>Observed output: pr4578-e4a-runtime.log</summary>

```json
{
  "source": "G:/git/headroom-review4578",
  "redis_python": "8.1.0",
  "container_image": "redis:7-alpine",
  "results": [
    {
      "stage": "configured_before_any_command",
      "ready": true,
      "states": {
        "ccr": "unknown",
        "compress_session": "unknown"
      }
    },
    {
      "stage": "observed_success",
      "ready": true,
      "states": {
        "ccr": "available",
        "compress_session": "available"
      }
    },
    {
      "stage": "redis_paused_fail_open",
      "ready": true,
      "states": {
        "ccr": "unavailable",
        "compress_session": "unavailable"
      }
    },
    {
      "stage": "redis_unpaused_after_backoff",
      "ready": true,
      "states": {
        "ccr": "available",
        "compress_session": "available"
      }
    },
    {
      "stage": "proxy_shutdown_and_store_close",
      "global_store_identity_and_original_preserved": true,
      "owned_session_guard_removed": true,
      "ccr_guard_removed_on_store_close": true
    }
  ]
}

```
</details>


<details><summary>Reproduction program</summary>

```python
import os
from pathlib import Path
import shutil
import subprocess
import sys

root=Path('G:/git/headroom/.codex').resolve()
env=os.environ.copy();env['PYTHONIOENCODING']='utf-8'
with (root/'pr4578-e4a-runtime.log').open('w',encoding='utf-8') as out,(root/'pr4578-e4a-runtime-stderr.log').open('w',encoding='utf-8') as err:
    result=subprocess.run([sys.executable,str(root/'pr4578-runtime.py'),'G:/git/headroom-review4578',str(root/'pr4578-redis-deps')],stdout=out,stderr=err,env=env)
manifest=root/'pr4578-owned-root.txt'
if manifest.exists():
    target=Path(manifest.read_text(encoding='utf-8')).resolve()
    assert target.parent==root and target.name.startswith('pr4578-runtime-'),target
    if target.exists():shutil.rmtree(target)
print('Runtime exit',result.returncode,'owned scratch cleaned',flush=True)
sys.exit(result.returncode)

```
</details>


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `a06923dc6cab75e2ed0d518922d6dc90003a3fd8`, upstream main
`de1a1e0f478d1eb9257dd8dd0d4ac87c9e6fc639`.

Clean integration with upstream/main
de1a1e0f47 and re-review against main
compression-store/lifecycle changes.148
Redis-availability/backend/compression-store/Gemini tests pass with
isolated redis8.1.0/fakeredis dependencies; Ruff and whitespace pass.
Initial local failures were only missing optional redis in the shared
environment, resolved by loading the existing isolated test overlay; no
production fix or weakened assertion. Availability remains tri-state,
optional Redis does not flip overall readiness, owned shared sessions
close on proxy shutdown while process-wide CCR storage remains attached,
and detach unregisters the former guard without closing storage still
referenced elsewhere. Existing actual Redis+TCP pause/recovery/shutdown
evidence remains scoped to unchanged behavior; no fresh
live-network/native-build claim. No new source finding; final approval
awaits exact-head hosted qualification. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `6a0368b3c436477390e8adc370a343da370a986b`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 148 passed, 1 warning in 18.84s.
Selected real dependency overlays and native DLL paths are recorded in
final4578-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `a83a2c5a0e378cdd8e3850e4cc5256902dd47daf`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against upstream main
2b0315b2d4. All 71 integration patches
match the main delta after diff-location normalization. The Redis
client/guard, backend, shared-session, compression-store and metric
implementations are byte-identical to the preceding reviewed head;
current server integration retains their ownership and health contracts.

The fresh Redis availability/backend/store/Gemini/doctor suite passes
279 tests with Redis 8.1.0 and one deprecation warning. Ruff lint/format
passes for all 33 changed Python files; offline lock and whitespace
checks pass. The existing native DLL matches its complete recorded input
hashes and artifact checksum; no fresh build is claimed.

Current author and Devin discussions were considered, including
unknown-versus-available reporting, inactive guards and process-wide
store ownership. The previously recorded real Redis pause/recovery and
shutdown-ownership evidence remains scoped to unchanged feature inputs.
It was not repeated here because the Docker Linux daemon is unavailable;
no fresh live-Redis run is claimed. No new source blocker found.
Exact-head hosted qualification is pending before approval. Maintainers
handle CI; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `712a91416a93fc090adcde9154ddd17c3d9628b5`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization; Redis availability, lifecycle and metrics logic is
unchanged. Fresh Redis availability/backend/compression-store and
incoming Vertex/backend/CLI suite: 296 passed, 11 skipped, 1 warning
using the previously qualified Redis 8.1.0 overlay. Initial execution
without that optional client had 11 missing-redis failures; those were
environment setup errors, corrected without modifying PR code. Twelve
changed Python files pass Ruff lint/format; offline lock and whitespace
pass.

Author/Devin discussion reviewed, including explicit unknown state,
configured lazy CCR, inactive guard ownership and detach lifecycle.
Prior real Redis pause/recovery and shutdown evidence applies to
unchanged feature inputs; no fresh live Redis or cloud-provider run
claimed. Static source/test review found no new credential/URL
disclosure or unrelated execution/workflow/dependency changes; health
exposes feature state, not Redis URL or credentials. Flagged inherited
workflow-validation job is excluded by unchanged workflow paths. Native
DLL reused after complete tracked input-set/hash and artifact checksum
verification; no fresh build claimed. Formal approval awaits exact
published-head hosted qualification. Maintainers handle checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Parideboy
2026-11-07 20:48:04 -06:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 84cedbf2b4
commit 602ca65126
7 changed files with 562 additions and 8 deletions
+12 -1
View File
@@ -34,7 +34,18 @@ class RedisBackend:
def __init__(self, url: str, client: Any = None) -> None:
self._client = client if client is not None else connect(url, setting="HEADROOM_REDIS_URL")
self._prefix = f"{key_prefix()}:ccr:"
self._guard = Guard("CCR store")
self._guard = Guard("CCR store", "ccr")
def close(self) -> None:
"""Unregister the guard and close the Redis client."""
if hasattr(self, "_guard"):
self._guard.close()
client = getattr(self, "_client", None)
if client is not None and hasattr(client, "close"):
try:
client.close()
except Exception:
pass
def _key(self, hash_key: str) -> str:
return self._prefix + hash_key
+25 -1
View File
@@ -849,6 +849,21 @@ class CompressionStore:
self._stale_heap_entries = 0 # CRITICAL FIX: Reset stale counter
self._stale_expiration_heap_entries = 0
def close(self) -> None:
"""Close the storage backend and release its resources.
A shared backend (Redis) registers its guard in the module-level
registry; closing the store unregisters it so availability reports
reflect the live store lifecycle instead of a detached one.
"""
with self._lock:
backend = getattr(self, "_backend", None)
if backend is not None and hasattr(backend, "close"):
try:
backend.close()
except Exception:
pass
def _evict_if_needed(self) -> None:
"""Evict expired, then oldest live entries until below capacity.
@@ -1356,11 +1371,15 @@ def detach_compression_store() -> None:
entries are left as they are: nothing is deleted from its backend, so a
SQLite file is neither written nor emptied. Used when the proxy switches to
stateless mode at runtime. The old backend is not closed, because code
that already holds the old store may still be using it.
that already holds the old store may still be using it; only its Redis
guard is unregistered, so /health and /metrics stop reporting it.
"""
global _compression_store
with _store_lock:
guard = getattr(getattr(_compression_store, "_backend", None), "_guard", None)
if guard is not None:
guard.close()
_compression_store = None
@@ -1371,4 +1390,9 @@ def reset_compression_store() -> None:
with _store_lock:
if _compression_store is not None:
_compression_store.clear()
if hasattr(_compression_store, "close"):
try:
_compression_store.close()
except Exception:
pass
_compression_store = None
+91 -2
View File
@@ -12,6 +12,9 @@ Shared state is an optimisation of a request, never a condition for it: a Redis
error is a miss, logged when it starts and when Redis is back, and Redis is not
asked again for :data:`BACKOFF_SECONDS` after a connection failure, so an outage
costs requests nothing but the sharing.
Each :class:`Guard` registers itself in a module-level registry keyed by a short
label; :func:`guards` exposes a snapshot for ``/health`` and ``/metrics``.
"""
from __future__ import annotations
@@ -21,6 +24,7 @@ import os
import re
import threading
import time
import weakref
from collections.abc import Callable
from typing import Any, TypeVar
from urllib.parse import urlsplit
@@ -38,6 +42,55 @@ TIMEOUT_SECONDS = 1.0
T = TypeVar("T")
#: Registry of live guards, keyed by their short label (last wins). A
#: :class:`weakref.WeakValueDictionary` so a guard is dropped when its owning
#: store is closed or detached: registrations follow the store lifecycle
#: instead of outliving it.
_GUARDS: weakref.WeakValueDictionary[str, Guard] = weakref.WeakValueDictionary()
def guards() -> dict[str, Guard]:
"""Snapshot of every registered :class:`Guard`, keyed by label."""
return dict(_GUARDS)
def states() -> dict[str, str]:
"""Label -> state for ``/health`` and ``/metrics``.
A ``"ccr"`` entry in ``"unknown"`` is added when the lazy CCR store is
configured for Redis but has not been built yet: ``get_compression_store()``
constructs it on first use, so no guard exists to report before that.
"""
snapshot = {label: guard.state for label, guard in _GUARDS.items()}
if "ccr" not in snapshot and _lazy_ccr_uses_redis():
snapshot["ccr"] = "unknown"
return snapshot
def _lazy_ccr_uses_redis() -> bool:
"""True when the not-yet-built CCR store would use Redis. Mirrors
:func:`_create_default_ccr_backend`."""
from . import compression_store
if compression_store._compression_store is not None:
# Built already: a Redis backend would have registered its guard, so
# nothing here is Redis (e.g. it fell back to the in-process store).
return False
backend = os.environ.get("HEADROOM_CCR_BACKEND", "").strip().lower()
session_store = os.environ.get("HEADROOM_COMPRESS_SESSION_STORE", "").strip()
if backend == "redis":
return bool(os.environ.get("HEADROOM_REDIS_URL", "").strip() or session_store)
return not backend and bool(session_store)
def unregister(guard: Guard) -> None:
"""Remove ``guard`` from the module-level registry if it is registered
under its label. A no-op when another guard already holds that label or
when ``guard`` was never registered."""
label = getattr(guard, "_label", None)
if label and _GUARDS.get(label) is guard:
_GUARDS.pop(label, None)
def key_prefix() -> str:
return os.environ.get(PREFIX_ENV, "").strip() or "headroom"
@@ -77,17 +130,51 @@ class Guard:
kind of call (``what``: read, write, ...) starts failing and when it works
again; after a connection error Redis is skipped for :data:`BACKOFF_SECONDS`.
Tracking each kind apart keeps a Redis that serves reads but refuses writes
(``maxmemory`` with ``noeviction``) from logging on every request."""
(``maxmemory`` with ``noeviction``) from logging on every request.
def __init__(self, feature: str) -> None:
``label`` is a short stable key (``"ccr"``, ``"compress_session"``) used in
the module-level registry so ``/health`` and ``/metrics`` can report
availability without importing the ``redis`` package at module level.
"""
def __init__(self, feature: str, label: str) -> None:
import redis
self._feature = feature
self._label = label
self._errors: tuple[type[BaseException], ...] = (redis.RedisError,)
self._connection_errors = (redis.ConnectionError, redis.TimeoutError)
self._lock = threading.Lock()
self._skip_until = 0.0
self._failing: set[str] = set()
# ``connect`` never connects, so a configured-but-unchecked store is
# "unknown" until ``call`` actually runs a command (success or error).
self._observed = False
_GUARDS[label] = self
@property
def state(self) -> str:
"""``"unknown"`` before any command ran, ``"available"`` when Redis
answered its last call and no backoff is active, else
``"unavailable"``."""
if not self._observed:
return "unknown"
if time.monotonic() >= self._skip_until and not self._failing:
return "available"
return "unavailable"
@property
def available(self) -> bool:
"""True when Redis answered its last call and no backoff is active.
Kept as ``state == "available"``: an unobserved store reports
``False`` rather than advertising connectivity it never tested.
"""
return self.state == "available"
def close(self) -> None:
"""Unregister this guard from the module-level registry."""
unregister(self)
def call(self, what: str, fn: Callable[[], T], default: T) -> T:
if time.monotonic() < self._skip_until:
@@ -96,6 +183,7 @@ class Guard:
result = fn()
except self._errors as exc:
with self._lock:
self._observed = True
if isinstance(exc, self._connection_errors):
self._skip_until = time.monotonic() + BACKOFF_SECONDS
if what not in self._failing:
@@ -113,6 +201,7 @@ class Guard:
if what in self._failing:
self._failing.discard(what)
logger.info("%s: Redis %s works again", self._feature, what)
self._observed = True
return result
+12 -1
View File
@@ -82,9 +82,20 @@ class SharedSessions:
def __init__(self, client: Any) -> None:
self._client = client
self._cas = client.register_script(_CAS)
self._guard = Guard("compress sessions")
self._guard = Guard("compress sessions", "compress_session")
self._prefix = f"{key_prefix()}:session:"
def close(self) -> None:
"""Close the shared session store and unregister its guard."""
if hasattr(self, "_guard"):
self._guard.close()
client = getattr(self, "_client", None)
if client is not None and hasattr(client, "close"):
try:
client.close()
except Exception:
pass
def fetch(
self, store: SessionTrackerStore, key: str, local: PrefixCacheTracker | None
) -> PrefixCacheTracker | None:
+39
View File
@@ -1883,6 +1883,45 @@ class PrometheusMetrics:
)
lines.append("")
# Shared Redis store availability (lazy import: module must not
# require the redis package at import time). ``headroom_redis_checked``
# is emitted for every configured feature so a store that has not
# been called yet (or not built yet) stays visible;
# ``headroom_redis_available`` is emitted only for observed ones (1
# when available, 0 during an outage or backoff), so an unknown
# store does not advertise connectivity.
from headroom.cache.redis_client import states as _redis_states
_states = _redis_states()
if _states:
lines.extend(
[
"# HELP headroom_redis_checked 1 when the shared Redis store has been called at least once, 0 when unobserved",
"# TYPE headroom_redis_checked gauge",
]
)
for _label in sorted(_states):
lines.append(
f'headroom_redis_checked{{feature="{_escape_label_value(_label)}"}}'
f" {int(_states[_label] != 'unknown')}"
)
lines.append("")
_observed = [_label for _label in sorted(_states) if _states[_label] != "unknown"]
if _observed:
lines.extend(
[
"# HELP headroom_redis_available 1 when the shared Redis store answered its last call, 0 during an outage or backoff",
"# TYPE headroom_redis_available gauge",
]
)
for _label in _observed:
lines.append(
f'headroom_redis_available{{feature="{_escape_label_value(_label)}"}}'
f" {int(_states[_label] == 'available')}"
)
lines.append("")
# Unit 3: WS session lifecycle gauges + duration histogram.
lines.extend(
[
+25 -3
View File
@@ -2486,6 +2486,11 @@ class HeadroomProxy(
if self.memory_handler and hasattr(self.memory_handler, "close"):
await self.memory_handler.close()
if self.shared_compress_sessions is not None:
with contextlib.suppress(Exception):
self.shared_compress_sessions.close()
self.shared_compress_sessions = None
with contextlib.suppress(Exception):
from headroom.models.ml_models import MLModelRegistry
@@ -3814,6 +3819,9 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
return KOMPRESS_DETAIL_NOT_INSTALLED
def _health_checks() -> dict[str, dict[str, Any]]:
from headroom.cache.redis_client import states as _redis_states
features = _redis_states()
kompress_enabled = _reconcile_kompress_health()
memory_status = (
proxy.memory_handler.health_status()
@@ -3828,6 +3836,10 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
)
memory_enabled = bool(memory_status.get("enabled", False))
memory_initialized = bool(memory_status.get("initialized", False))
session_store_enabled = bool(features)
session_store_ready = session_store_enabled and all(
v == "available" for v in features.values()
)
return {
"startup": _component_health(
enabled=True,
@@ -3867,6 +3879,12 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
backend=proxy.warmup.kompress.info.get("backend", None),
detail=_kompress_detail(),
),
"session_store": _component_health(
enabled=session_store_enabled,
ready=session_store_ready,
optional=True,
features=features,
),
}
def _runtime_payload() -> dict[str, Any]:
@@ -3983,9 +4001,13 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
def _health_payload(*, include_config: bool) -> dict[str, Any]:
checks = _health_checks()
# Kompress is an optional soft component: model downloads lazily on
# first use, so "not ready" (cold cache) must not degrade overall health.
ready = all(check["ready"] for name, check in checks.items() if name != "kompress")
# Optional soft components (kompress, session_store) must not degrade
# overall health: a cold model cache or a Redis backoff is not a failure.
ready = all(
check["ready"]
for name, check in checks.items()
if name not in ("kompress", "session_store")
)
payload: dict[str, Any] = {
"service": "headroom-proxy",
"status": "healthy" if ready else "unhealthy",
+358
View File
@@ -0,0 +1,358 @@
"""Guard availability tracking, /metrics gauge and /health session_store (#4512).
Covers:
* ``Guard.state`` reports "unknown" before any command runs, "available" on
success, and "unavailable" on backoff/failure.
* ``headroom_redis_checked{feature=...}`` gauge appears in /metrics output for
every configured guard (1 if checked, 0 if unknown).
* ``headroom_redis_available{feature=...}`` gauge appears in /metrics output
only for observed guards (1 if available, 0 during outage).
* ``/health`` reports ``session_store`` feature state strings ("unknown" |
"available" | "unavailable") and stays degraded when unobserved or unavailable
while top-level ``ready`` stays True.
* Unregistering/closing guards on store close/detach.
"""
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from headroom.cache import redis_client # noqa: E402
from headroom.cache.redis_client import Guard, guards # noqa: E402
# --- Guard.state & Guard.available --------------------------------------------
class TestGuardAvailable:
def test_unknown_initially_before_any_call(self):
g = Guard("test feature", "test")
assert g.state == "unknown"
assert g.available is False
g.close()
def test_available_after_successful_call(self):
g = Guard("test feature", "test")
result = g.call("read", lambda: "ok", "default")
assert result == "ok"
assert g.state == "available"
assert g.available is True
g.close()
def test_unavailable_after_connection_error(self, monkeypatch):
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
g = Guard("test feature", "test")
import redis
def failing():
raise redis.ConnectionError("connection refused")
result = g.call("read", failing, "default")
assert result == "default"
assert g.state == "unavailable"
assert g.available is False
g.close()
def test_available_again_after_successful_call(self, monkeypatch):
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 0.0)
g = Guard("test feature", "test")
import redis
def failing():
raise redis.RedisError("some error")
g.call("read", failing, "default")
assert g.state == "unavailable"
assert g.available is False
result = g.call("read", lambda: "ok", "default")
assert result == "ok"
assert g.state == "available"
assert g.available is True
g.close()
def test_untested_shared_session_store_is_unknown(self, monkeypatch):
monkeypatch.setenv("HEADROOM_COMPRESS_SESSION_STORE", "redis://127.0.0.1:1")
from headroom.cache.shared_sessions import open_shared_sessions
store = open_shared_sessions()
assert store is not None
try:
assert store._guard.state == "unknown"
assert store._guard.available is False
assert guards()["compress_session"].state == "unknown"
assert guards()["compress_session"].available is False
finally:
store.close()
def test_lifecycle_close_unregisters_guard(self, monkeypatch):
monkeypatch.setenv("HEADROOM_COMPRESS_SESSION_STORE", "redis://127.0.0.1:1")
from headroom.cache.shared_sessions import open_shared_sessions
store = open_shared_sessions()
assert store is not None
assert "compress_session" in guards()
store.close()
monkeypatch.delenv("HEADROOM_COMPRESS_SESSION_STORE", raising=False)
assert open_shared_sessions() is None
assert "compress_session" not in guards()
# --- /metrics gauge ----------------------------------------------------------
class TestMetricsGauge:
@pytest.mark.asyncio
async def test_gauge_checked_and_available_when_observed(self):
g = Guard("test feature", "test_label")
g.call("read", lambda: "ok", None)
from headroom.proxy.prometheus_metrics import PrometheusMetrics
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert "# HELP headroom_redis_checked" in text
assert "# TYPE headroom_redis_checked gauge" in text
assert 'headroom_redis_checked{feature="test_label"} 1' in text
assert "# HELP headroom_redis_available" in text
assert "# TYPE headroom_redis_available gauge" in text
assert 'headroom_redis_available{feature="test_label"} 1' in text
g.close()
@pytest.mark.asyncio
async def test_gauge_unobserved_shows_checked_0_and_no_available(self):
g = Guard("test feature", "unobserved_label")
from headroom.proxy.prometheus_metrics import PrometheusMetrics
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert 'headroom_redis_checked{feature="unobserved_label"} 0' in text
assert "headroom_redis_available" not in text
g.close()
@pytest.mark.asyncio
async def test_gauge_zero_during_outage(self, monkeypatch):
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
import redis
g = Guard("test feature", "outage_label")
def failing():
raise redis.ConnectionError("down")
g.call("read", failing, None)
from headroom.proxy.prometheus_metrics import PrometheusMetrics
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert 'headroom_redis_checked{feature="outage_label"} 1' in text
assert 'headroom_redis_available{feature="outage_label"} 0' in text
g.close()
# --- /health session_store ---------------------------------------------------
class TestHealthSessionStore:
def test_session_store_degraded_does_not_flip_ready(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
import redis
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
g = Guard("compress sessions", "compress_session")
def failing():
raise redis.ConnectionError("down")
g.call("read", failing, None)
assert g.state == "unavailable"
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
assert payload["ready"] is True
assert payload["status"] == "healthy"
ss = payload["checks"]["session_store"]
assert ss["enabled"] is True
assert ss["ready"] is False
assert ss["status"] == "degraded"
assert ss["optional"] is True
assert ss["features"]["compress_session"] == "unavailable"
g.close()
def test_session_store_disabled_when_no_guards(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
ss = payload["checks"]["session_store"]
assert ss["enabled"] is False
assert ss["status"] == "disabled"
@pytest.mark.asyncio
async def test_lazy_ccr_redis_visible_on_first_scrape(self, monkeypatch):
"""Configured-but-lazy CCR Redis shows ``checked 0`` on the first
scrape, before any traffic has built the store; ``available`` stays
absent and /health reports the same unknown state. No connection is
attempted for the observation."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_STATELESS", "0")
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redis")
monkeypatch.setenv("HEADROOM_REDIS_URL", "redis://127.0.0.1:1")
from headroom.cache import compression_store, redis_client
from headroom.proxy.models import ProxyConfig
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.server import create_app
compression_store.reset_compression_store()
try:
# Full startup, no store created yet: no guard registered.
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
assert redis_client.guards() == {} # nothing connected
text = await PrometheusMetrics(stateless=False).export()
assert 'headroom_redis_checked{feature="ccr"} 0' in text
assert "headroom_redis_available" not in text
response = TestClient(app).get("/readyz")
assert response.status_code == 200
payload = response.json()
ss = payload["checks"]["session_store"]
assert ss["enabled"] is True
assert ss["ready"] is False
assert ss["status"] == "degraded"
assert ss["optional"] is True
assert ss["features"]["ccr"] == "unknown"
assert payload["ready"] is True
finally:
compression_store.reset_compression_store()
def test_session_store_healthy_when_available(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
g = Guard("CCR store", "ccr")
g.call("read", lambda: "ok", None)
assert g.state == "available"
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
ss = payload["checks"]["session_store"]
assert ss["enabled"] is True
assert ss["ready"] is True
assert ss["status"] == "healthy"
assert ss["features"]["ccr"] == "available"
g.close()
# --- CCR store lifecycle across proxy shutdown -------------------------------
class TestCCRStoreSurvivesProxyShutdown:
@pytest.mark.asyncio
async def test_ccr_store_persists_across_proxy_shutdown(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
from headroom.cache.compression_store import (
get_compression_store,
reset_compression_store,
)
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
reset_compression_store()
try:
app1 = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app1.state.ready = True
proxy1 = app1.state.proxy
proxy1.http_client = None
proxy1.http_client_h1 = None
store = get_compression_store()
original_hash = store.store(
original='{"data": "hello"}',
compressed='{"data": "[COMPRESSED:abc123]"}',
original_tokens=10,
compressed_tokens=2,
tool_name="test_tool",
)
entry_before = store.retrieve(original_hash)
assert entry_before is not None
assert entry_before.original_content == '{"data": "hello"}'
await proxy1.shutdown()
store_after = get_compression_store()
assert store_after is store
entry_after = store_after.retrieve(original_hash)
assert entry_after is not None
assert entry_after.original_content == '{"data": "hello"}'
finally:
reset_compression_store()