Files
headroom/tests/test_redis_availability.py
602ca65126 feat(proxy): expose shared Redis store availability in /health and /metrics (#4578)
## Description

With `HEADROOM_COMPRESS_SESSION_STORE=redis://...` or the Redis CCR
store, a Redis outage is silent: `Guard` fails open (by design) but
`/health` stays `healthy` and `/metrics` has no series. This exposes the
state `Guard` already tracks (`_skip_until`, `_failing`) without
changing the fail-open behaviour.

Closes #4512.

## Type of Change

- [ ] Bug fix
- [x] New feature
- [ ] Breaking change
- [ ] Documentation

## Changes Made

- `headroom/cache/redis_client.py`: `Guard(feature, label)` registers
itself under a short label; new `Guard.available` (false during backoff
or while any call kind is failing) and `guards()` snapshot.
- CCR backend uses label `ccr`; shared `/v1/compress` sessions use
`compress_session`.
- `/metrics`: `headroom_redis_available{feature="..."}` gauge (1/0),
emitted only when a Redis store is configured.
- `/health`: optional `session_store` component built with
`_component_health(optional=True)`, with per-feature availability. Shows
`degraded` during an outage and is excluded from the top-level `ready`,
like `kompress`, so a shared-Redis outage never drains every replica.
- `tests/test_redis_availability.py`: availability transitions, gauge
output, health payload.

## Testing

- [x] Unit tests added/updated
- [x] Existing tests pass locally (failures below are pre-existing on
`upstream/main`)

```
$ pytest tests/test_redis_availability.py tests/test_ccr_redis_backend.py tests/test_proxy_health.py -q
5 failed, 52 passed, 4 skipped
# same 5 failures on clean upstream/main (5 failed, 40 passed): local env ModuleNotFoundError / rust-core SystemExit
$ ruff check <touched files> && ruff format --check <touched files>
All checks passed! / 6 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, fakeredis, branch
feat/4512-redis-availability
- Exact command / steps: pytest tests/test_redis_availability.py (forces
redis.ConnectionError through Guard, then renders /metrics and GET
/health via TestClient)
- Observed result: during backoff
`headroom_redis_available{feature="compress_session"} 0` and
`session_store.status == "degraded"` with top-level `ready: true`; after
a successful call the gauge returns to 1 and the component to `healthy`
- Not tested: a live multi-replica deployment against a real Redis
server

## Runtime Rollout Safety

- Rollout-managed feature(s): none; observability only
- Minimum rollout channel: stable
- Stable/default behavior changed: no; without Redis configured the
gauge is absent and `session_store` reports `disabled`
- Kill switch / disable path: unset the Redis store settings
- Unsafe override required: no
- Qualification impact: none; new optional health component does not
affect readiness
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4578"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `c54bb1a33210f5f20c5a135bc6b3608f5429716d`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.

Mechanically updated from current upstream main without rewriting
contributor Redis/lifecycle logic. All 167 focused availability,
Redis-backend, proxy-health and compression-store tests pass with
isolated redis 8.1.0 / fakeredis 2.40.0 qualification packages; ten
shared Gemini request-fixture cases also pass after the test-only repair
for independently verified main failures. Changed Python Ruff/format and
main-relative whitespace checks pass. A bounded
100,000-registration/snapshot guard stress probe reports no error; this
is not a universal concurrency guarantee.

Independent real-service proof uses an owned local redis:7-alpine
container, the actual Python Redis backend/shared-session store and a
real Uvicorn HTTP listener. /readyz and /metrics report both features
unknown before a command, available after successful Redis operations,
unavailable during Docker pause, and available after unpause plus the
unchanged five-second backoff. Redis is optional, so top-level readiness
stays true throughout. No Redis client/service mocks are used. Proxy
shutdown preserves the process-wide in-memory CCR store's identity and
original, removes its owned session guard, and explicit Redis-store
close removes that guard. No multi-replica, TLS, cluster or Redis-server
restart qualification is claimed. An existing native binary was used
only for proxy startup, not a fresh native-build claim.

Reproduction: on Windows/Python 3.12 with Docker and the proxy
dependencies, install redis==8.1.0 into an isolated dependency
directory. Save the first program as pr4578-runtime.py and the cleanup
driver as run-pr4578-runtime.py, adapt their
scratch/checkouts/dependency paths, and run the driver. Its child
operates only on an owned container and scratch directory; the parent
removes the remaining cache database after the child exits and releases
Windows handles. Pause/unpause retains the same published Redis
endpoint; Docker stop/start with an ephemeral port was independently
observed to reassign it and is not used for the qualification below.


<details><summary>Reproduction program</summary>

```python
import json
import os
from pathlib import Path
import re
import socket
import subprocess
import sys
import tempfile
import threading
import time
import uuid

sys.stdout.reconfigure(encoding='utf-8')
sys.path.insert(0,sys.argv[1]);sys.path.insert(0,sys.argv[2])
owned=Path('G:/git/headroom/.codex').resolve()
def docker(*args):
    p=subprocess.run(['docker',*args],capture_output=True,text=True,encoding='utf-8',timeout=60,creationflags=subprocess.CREATE_NO_WINDOW)
    assert p.returncode==0,(args,p.stderr)
    return p.stdout.strip()

cid=None;server=None;thread=None;store=None
with tempfile.TemporaryDirectory(prefix='pr4578-runtime-',dir=owned,ignore_cleanup_errors=True) as directory:
    root=Path(directory).resolve();assert root.is_relative_to(owned)
    (owned/'pr4578-owned-root.txt').write_text(str(root),encoding='utf-8')
    os.environ.update(HEADROOM_OFFLINE='1',DO_NOT_TRACK='1',HEADROOM_BEACON_ENABLED='0',HEADROOM_SKIP_UPSTREAM_CHECK='1',HEADROOM_WORKSPACE_DIR=str(root),HEADROOM_CCR_BACKEND='memory')
    try:
        cid=docker('run','--detach','--name','headroom-review-4578-'+uuid.uuid4().hex[:10],'--publish','127.0.0.1::6379','redis:7-alpine','redis-server','--save','','--appendonly','no')
        assert re.fullmatch(r'[a-f0-9]{64}',cid)
        port=int(docker('port',cid,'6379/tcp').rsplit(':',1)[1]);url=f'redis://127.0.0.1:{port}'
        os.environ['HEADROOM_COMPRESS_SESSION_STORE']=url
        import redis
        admin=redis.Redis.from_url(url,socket_connect_timeout=1,socket_timeout=1)
        for _ in range(50):
            try:
                if admin.ping():break
            except redis.RedisError:time.sleep(.1)
        else:raise AssertionError('Redis did not start')
        admin.close()
        import httpx
        import uvicorn
        from headroom.cache.backends.redis import RedisBackend
        from headroom.cache.compression_store import CompressionStore,get_compression_store
        from headroom.cache.prefix_tracker import SessionTrackerStore
        from headroom.cache.redis_client import BACKOFF_SECONDS,guards
        from headroom.proxy.server import ProxyConfig,create_app
        app=create_app(ProxyConfig(optimize=False,cache_enabled=False,rate_limit_enabled=False,cost_tracking_enabled=False,subscription_tracking_enabled=False,log_requests=False,image_optimize=False,ccr_inject_tool=False,ccr_handle_responses=False,ccr_context_tracking=False))
        listener=socket.socket();listener.bind(('127.0.0.1',0));http_port=listener.getsockname()[1]
        server=uvicorn.Server(uvicorn.Config(app,log_level='error'))
        thread=threading.Thread(target=lambda:server.run(sockets=[listener]),daemon=True);thread.start()
        deadline=time.monotonic()+30
        while not server.started:
            assert thread.is_alive() and time.monotonic()<deadline,'proxy startup failed'
            time.sleep(.05)
        sessions=app.state.proxy.shared_compress_sessions
        backend=RedisBackend(url);store=CompressionStore(backend=backend,enable_feedback=False)
        global_store=get_compression_store();global_key=global_store.store('process-wide original','x')
        results=[]
        def snapshot(stage,expected):
            base=f'http://127.0.0.1:{http_port}'
            response=httpx.get(base+'/readyz',timeout=10);payload=response.json()
            metrics=httpx.get(base+'/metrics',timeout=10);assert metrics.status_code==200
            states=payload['checks']['session_store']['features']
            assert states=={'ccr':expected,'compress_session':expected},(stage,payload)
            assert response.status_code==200 and payload['ready'] is True,(stage,payload)
            for label in states:
                assert f'headroom_redis_checked{{feature="{label}"}} {int(expected!="unknown")}' in metrics.text
                gauge=f'headroom_redis_available{{feature="{label}"}}'
                if expected=='unknown':assert gauge not in metrics.text
                else:assert gauge+' '+str(int(expected=='available')) in metrics.text
            results.append({'stage':stage,'ready':payload['ready'],'states':states})
        snapshot('configured_before_any_command','unknown')
        key=store.store('live Redis original','x');assert store.retrieve(key).original_content=='live Redis original'
        sessions.fetch(SessionTrackerStore(),'review-missing',None)
        snapshot('observed_success','available')
        docker('pause',cid)
        assert store.retrieve(key) is None
        sessions.fetch(SessionTrackerStore(),'review-missing',None)
        snapshot('redis_paused_fail_open','unavailable')
        docker('unpause',cid)
        time.sleep(BACKOFF_SECONDS+.2)
        recovered=store.store('recovered Redis original','x');assert store.retrieve(recovered).original_content=='recovered Redis original'
        sessions.fetch(SessionTrackerStore(),'review-missing',None)
        snapshot('redis_unpaused_after_backoff','available')
        server.should_exit=True;thread.join(timeout=15);assert not thread.is_alive()
        assert get_compression_store() is global_store
        assert global_store.retrieve(global_key).original_content=='process-wide original'
        assert 'compress_session' not in guards() and 'ccr' in guards()
        store.close();store=None;assert 'ccr' not in guards()
        results.append({'stage':'proxy_shutdown_and_store_close','global_store_identity_and_original_preserved':True,'owned_session_guard_removed':True,'ccr_guard_removed_on_store_close':True})
        print(json.dumps({'source':sys.argv[1],'redis_python':redis.__version__,'container_image':'redis:7-alpine','results':results},indent=2))
    finally:
        if server is not None:server.should_exit=True
        if thread is not None:thread.join(timeout=15)
        if store is not None:store.close()
        if cid is not None:docker('rm','--force',cid)

```
</details>

<details><summary>Observed output: pr4578-e4a-runtime.log</summary>

```json
{
  "source": "G:/git/headroom-review4578",
  "redis_python": "8.1.0",
  "container_image": "redis:7-alpine",
  "results": [
    {
      "stage": "configured_before_any_command",
      "ready": true,
      "states": {
        "ccr": "unknown",
        "compress_session": "unknown"
      }
    },
    {
      "stage": "observed_success",
      "ready": true,
      "states": {
        "ccr": "available",
        "compress_session": "available"
      }
    },
    {
      "stage": "redis_paused_fail_open",
      "ready": true,
      "states": {
        "ccr": "unavailable",
        "compress_session": "unavailable"
      }
    },
    {
      "stage": "redis_unpaused_after_backoff",
      "ready": true,
      "states": {
        "ccr": "available",
        "compress_session": "available"
      }
    },
    {
      "stage": "proxy_shutdown_and_store_close",
      "global_store_identity_and_original_preserved": true,
      "owned_session_guard_removed": true,
      "ccr_guard_removed_on_store_close": true
    }
  ]
}

```
</details>


<details><summary>Reproduction program</summary>

```python
import os
from pathlib import Path
import shutil
import subprocess
import sys

root=Path('G:/git/headroom/.codex').resolve()
env=os.environ.copy();env['PYTHONIOENCODING']='utf-8'
with (root/'pr4578-e4a-runtime.log').open('w',encoding='utf-8') as out,(root/'pr4578-e4a-runtime-stderr.log').open('w',encoding='utf-8') as err:
    result=subprocess.run([sys.executable,str(root/'pr4578-runtime.py'),'G:/git/headroom-review4578',str(root/'pr4578-redis-deps')],stdout=out,stderr=err,env=env)
manifest=root/'pr4578-owned-root.txt'
if manifest.exists():
    target=Path(manifest.read_text(encoding='utf-8')).resolve()
    assert target.parent==root and target.name.startswith('pr4578-runtime-'),target
    if target.exists():shutil.rmtree(target)
print('Runtime exit',result.returncode,'owned scratch cleaned',flush=True)
sys.exit(result.returncode)

```
</details>


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `a06923dc6cab75e2ed0d518922d6dc90003a3fd8`, upstream main
`de1a1e0f478d1eb9257dd8dd0d4ac87c9e6fc639`.

Clean integration with upstream/main
de1a1e0f47 and re-review against main
compression-store/lifecycle changes.148
Redis-availability/backend/compression-store/Gemini tests pass with
isolated redis8.1.0/fakeredis dependencies; Ruff and whitespace pass.
Initial local failures were only missing optional redis in the shared
environment, resolved by loading the existing isolated test overlay; no
production fix or weakened assertion. Availability remains tri-state,
optional Redis does not flip overall readiness, owned shared sessions
close on proxy shutdown while process-wide CCR storage remains attached,
and detach unregisters the former guard without closing storage still
referenced elsewhere. Existing actual Redis+TCP pause/recovery/shutdown
evidence remains scoped to unchanged behavior; no fresh
live-network/native-build claim. No new source finding; final approval
awaits exact-head hosted qualification. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `6a0368b3c436477390e8adc370a343da370a986b`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 148 passed, 1 warning in 18.84s.
Selected real dependency overlays and native DLL paths are recorded in
final4578-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `a83a2c5a0e378cdd8e3850e4cc5256902dd47daf`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Requalified against upstream main
2b0315b2d4. All 71 integration patches
match the main delta after diff-location normalization. The Redis
client/guard, backend, shared-session, compression-store and metric
implementations are byte-identical to the preceding reviewed head;
current server integration retains their ownership and health contracts.

The fresh Redis availability/backend/store/Gemini/doctor suite passes
279 tests with Redis 8.1.0 and one deprecation warning. Ruff lint/format
passes for all 33 changed Python files; offline lock and whitespace
checks pass. The existing native DLL matches its complete recorded input
hashes and artifact checksum; no fresh build is claimed.

Current author and Devin discussions were considered, including
unknown-versus-available reporting, inactive guards and process-wide
store ownership. The previously recorded real Redis pause/recovery and
shutdown-ownership evidence remains scoped to unchanged feature inputs.
It was not repeated here because the Docker Linux daemon is unavailable;
no fresh live-Redis run is claimed. No new source blocker found.
Exact-head hosted qualification is pending before approval. Maintainers
handle CI; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `712a91416a93fc090adcde9154ddd17c3d9628b5`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization; Redis availability, lifecycle and metrics logic is
unchanged. Fresh Redis availability/backend/compression-store and
incoming Vertex/backend/CLI suite: 296 passed, 11 skipped, 1 warning
using the previously qualified Redis 8.1.0 overlay. Initial execution
without that optional client had 11 missing-redis failures; those were
environment setup errors, corrected without modifying PR code. Twelve
changed Python files pass Ruff lint/format; offline lock and whitespace
pass.

Author/Devin discussion reviewed, including explicit unknown state,
configured lazy CCR, inactive guard ownership and detach lifecycle.
Prior real Redis pause/recovery and shutdown evidence applies to
unchanged feature inputs; no fresh live Redis or cloud-provider run
claimed. Static source/test review found no new credential/URL
disclosure or unrelated execution/workflow/dependency changes; health
exposes feature state, not Redis URL or credentials. Flagged inherited
workflow-validation job is excluded by unchanged workflow paths. Native
DLL reused after complete tracked input-set/hash and artifact checksum
verification; no fresh build claimed. Formal approval awaits exact
published-head hosted qualification. Maintainers handle checks; nothing
merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-07 20:48:04 -06:00

359 lines
13 KiB
Python

"""Guard availability tracking, /metrics gauge and /health session_store (#4512).
Covers:
* ``Guard.state`` reports "unknown" before any command runs, "available" on
success, and "unavailable" on backoff/failure.
* ``headroom_redis_checked{feature=...}`` gauge appears in /metrics output for
every configured guard (1 if checked, 0 if unknown).
* ``headroom_redis_available{feature=...}`` gauge appears in /metrics output
only for observed guards (1 if available, 0 during outage).
* ``/health`` reports ``session_store`` feature state strings ("unknown" |
"available" | "unavailable") and stays degraded when unobserved or unavailable
while top-level ``ready`` stays True.
* Unregistering/closing guards on store close/detach.
"""
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from headroom.cache import redis_client # noqa: E402
from headroom.cache.redis_client import Guard, guards # noqa: E402
# --- Guard.state & Guard.available --------------------------------------------
class TestGuardAvailable:
def test_unknown_initially_before_any_call(self):
g = Guard("test feature", "test")
assert g.state == "unknown"
assert g.available is False
g.close()
def test_available_after_successful_call(self):
g = Guard("test feature", "test")
result = g.call("read", lambda: "ok", "default")
assert result == "ok"
assert g.state == "available"
assert g.available is True
g.close()
def test_unavailable_after_connection_error(self, monkeypatch):
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
g = Guard("test feature", "test")
import redis
def failing():
raise redis.ConnectionError("connection refused")
result = g.call("read", failing, "default")
assert result == "default"
assert g.state == "unavailable"
assert g.available is False
g.close()
def test_available_again_after_successful_call(self, monkeypatch):
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 0.0)
g = Guard("test feature", "test")
import redis
def failing():
raise redis.RedisError("some error")
g.call("read", failing, "default")
assert g.state == "unavailable"
assert g.available is False
result = g.call("read", lambda: "ok", "default")
assert result == "ok"
assert g.state == "available"
assert g.available is True
g.close()
def test_untested_shared_session_store_is_unknown(self, monkeypatch):
monkeypatch.setenv("HEADROOM_COMPRESS_SESSION_STORE", "redis://127.0.0.1:1")
from headroom.cache.shared_sessions import open_shared_sessions
store = open_shared_sessions()
assert store is not None
try:
assert store._guard.state == "unknown"
assert store._guard.available is False
assert guards()["compress_session"].state == "unknown"
assert guards()["compress_session"].available is False
finally:
store.close()
def test_lifecycle_close_unregisters_guard(self, monkeypatch):
monkeypatch.setenv("HEADROOM_COMPRESS_SESSION_STORE", "redis://127.0.0.1:1")
from headroom.cache.shared_sessions import open_shared_sessions
store = open_shared_sessions()
assert store is not None
assert "compress_session" in guards()
store.close()
monkeypatch.delenv("HEADROOM_COMPRESS_SESSION_STORE", raising=False)
assert open_shared_sessions() is None
assert "compress_session" not in guards()
# --- /metrics gauge ----------------------------------------------------------
class TestMetricsGauge:
@pytest.mark.asyncio
async def test_gauge_checked_and_available_when_observed(self):
g = Guard("test feature", "test_label")
g.call("read", lambda: "ok", None)
from headroom.proxy.prometheus_metrics import PrometheusMetrics
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert "# HELP headroom_redis_checked" in text
assert "# TYPE headroom_redis_checked gauge" in text
assert 'headroom_redis_checked{feature="test_label"} 1' in text
assert "# HELP headroom_redis_available" in text
assert "# TYPE headroom_redis_available gauge" in text
assert 'headroom_redis_available{feature="test_label"} 1' in text
g.close()
@pytest.mark.asyncio
async def test_gauge_unobserved_shows_checked_0_and_no_available(self):
g = Guard("test feature", "unobserved_label")
from headroom.proxy.prometheus_metrics import PrometheusMetrics
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert 'headroom_redis_checked{feature="unobserved_label"} 0' in text
assert "headroom_redis_available" not in text
g.close()
@pytest.mark.asyncio
async def test_gauge_zero_during_outage(self, monkeypatch):
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
import redis
g = Guard("test feature", "outage_label")
def failing():
raise redis.ConnectionError("down")
g.call("read", failing, None)
from headroom.proxy.prometheus_metrics import PrometheusMetrics
metrics = PrometheusMetrics(stateless=True)
text = await metrics.export()
assert 'headroom_redis_checked{feature="outage_label"} 1' in text
assert 'headroom_redis_available{feature="outage_label"} 0' in text
g.close()
# --- /health session_store ---------------------------------------------------
class TestHealthSessionStore:
def test_session_store_degraded_does_not_flip_ready(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setattr(redis_client, "BACKOFF_SECONDS", 60.0)
import redis
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
g = Guard("compress sessions", "compress_session")
def failing():
raise redis.ConnectionError("down")
g.call("read", failing, None)
assert g.state == "unavailable"
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
assert payload["ready"] is True
assert payload["status"] == "healthy"
ss = payload["checks"]["session_store"]
assert ss["enabled"] is True
assert ss["ready"] is False
assert ss["status"] == "degraded"
assert ss["optional"] is True
assert ss["features"]["compress_session"] == "unavailable"
g.close()
def test_session_store_disabled_when_no_guards(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
ss = payload["checks"]["session_store"]
assert ss["enabled"] is False
assert ss["status"] == "disabled"
@pytest.mark.asyncio
async def test_lazy_ccr_redis_visible_on_first_scrape(self, monkeypatch):
"""Configured-but-lazy CCR Redis shows ``checked 0`` on the first
scrape, before any traffic has built the store; ``available`` stays
absent and /health reports the same unknown state. No connection is
attempted for the observation."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_STATELESS", "0")
monkeypatch.setenv("HEADROOM_CCR_BACKEND", "redis")
monkeypatch.setenv("HEADROOM_REDIS_URL", "redis://127.0.0.1:1")
from headroom.cache import compression_store, redis_client
from headroom.proxy.models import ProxyConfig
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.server import create_app
compression_store.reset_compression_store()
try:
# Full startup, no store created yet: no guard registered.
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
assert redis_client.guards() == {} # nothing connected
text = await PrometheusMetrics(stateless=False).export()
assert 'headroom_redis_checked{feature="ccr"} 0' in text
assert "headroom_redis_available" not in text
response = TestClient(app).get("/readyz")
assert response.status_code == 200
payload = response.json()
ss = payload["checks"]["session_store"]
assert ss["enabled"] is True
assert ss["ready"] is False
assert ss["status"] == "degraded"
assert ss["optional"] is True
assert ss["features"]["ccr"] == "unknown"
assert payload["ready"] is True
finally:
compression_store.reset_compression_store()
def test_session_store_healthy_when_available(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
g = Guard("CCR store", "ccr")
g.call("read", lambda: "ok", None)
assert g.state == "available"
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
ss = payload["checks"]["session_store"]
assert ss["enabled"] is True
assert ss["ready"] is True
assert ss["status"] == "healthy"
assert ss["features"]["ccr"] == "available"
g.close()
# --- CCR store lifecycle across proxy shutdown -------------------------------
class TestCCRStoreSurvivesProxyShutdown:
@pytest.mark.asyncio
async def test_ccr_store_persists_across_proxy_shutdown(self, monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
from headroom.cache.compression_store import (
get_compression_store,
reset_compression_store,
)
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
reset_compression_store()
try:
app1 = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app1.state.ready = True
proxy1 = app1.state.proxy
proxy1.http_client = None
proxy1.http_client_h1 = None
store = get_compression_store()
original_hash = store.store(
original='{"data": "hello"}',
compressed='{"data": "[COMPRESSED:abc123]"}',
original_tokens=10,
compressed_tokens=2,
tool_name="test_tool",
)
entry_before = store.retrieve(original_hash)
assert entry_before is not None
assert entry_before.original_content == '{"data": "hello"}'
await proxy1.shutdown()
store_after = get_compression_store()
assert store_after is store
entry_after = store_after.retrieve(original_hash)
assert entry_after is not None
assert entry_after.original_content == '{"data": "hello"}'
finally:
reset_compression_store()