Files
headroom/tests/test_proxy_health.py
e2350bc8fd fix(proxy): warm deferred kompress model so /health stops reporting backend=null (#2799)
## Description

On an idle proxy with the ML extras installed and the Kompress model
cached, `/health` and `/readyz` reported `kompress: {"ready": false,
"status": "unhealthy", "backend": null}` forever (#2730).

The cause was not missing wiring: `enable_kompress` defaults to `True`
and `ContentRouter` sits in both pipelines. The chain was:

1. `ContentRouter.eager_load_compressors()` deliberately skips
`preload()`, so a native load never runs before the port binds (#790
hang on Windows, #1908 segfault on RHEL/CentOS 7). It returns `kompress
= "deferred"`.
2. `WarmupRegistry.merge_transform_status` leaves a deferred slot at
`status="null"`.
3. `_reconcile_kompress_health()` promotes the slot only once a
compressor reports `is_ready()`, and nothing loaded the model without a
kompress-routed request.

#3619 (merged after this PR was opened) added
`_start_kompress_background_warmup()`, which loads the model on a daemon
thread shortly after startup. That fixes the load itself. Rebased onto
it, this PR covers what is left:

1. **`/health` says why `ready` is false.** The kompress check gains a
`detail` field. `ready: false` alone cannot tell "still warming" from
"extras not installed" from "warm failed", which is the confusion in
#2730.
2. **The warm thread records its outcome on `warmup.kompress`.** Success
promotes the slot immediately with `source_status="background"`, instead
of leaving `source_status="deferred"` on a loaded slot until the next
reconcile. Failure leaves a bounded reason.
3. **The detail is a closed vocabulary.** `/health` and `/readyz` are
auth-exempt, and loader exceptions carry paths, model ids and URLs, so
anything outside the allowlist collapses to `unavailable`.

Closes #2730

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/kompress_health_detail_policy.py` (new, pure, matching
the existing `*_policy.py` modules): `KOMPRESS_HEALTH_DETAILS` lists the
values the field may report (`loaded`, `warming`, `model not cached`,
`warm failed`, `not installed`, plus the `eager_load_compressors()`
statuses `enabled` / `deferred` / `unavailable`). `public_detail()`
passes those through and maps anything else to `unavailable`.
- `headroom/proxy/server.py`:
- `_start_kompress_background_warmup()` (from #3619, still
`preload(allow_download=True)` on thread `kompress-warmup`) now writes
its result to the slot. On success it calls `mark_loaded(...,
source_status="background")`. On any exception it sets detail `warm
failed` and logs the real cause with `exc_info=True`; the exception text
never reaches the slot. A compressor that raises
`KompressModelNotCached` gets `model not cached`. The local loader only
raises that with downloads off, so on a cold cache it downloads and
`/health` reports `warming` in the meantime.
- `_kompress_detail()` feeds the new field. Every candidate, including
raw `slot.error` and `source_status`, goes through `public_detail()`, so
the allowlist has a single choke point.
- Router/compressor lookup moves to module-level `_kompress_routers()` /
`_kompress_compressors()`, shared by startup and `/health`.
- `enabled`, `ready` and `backend` keep their meaning, so the `/readyz`
aggregate exclusion of optional Kompress still holds.
- Tests:
- `tests/test_kompress_preload_deferral.py`: the warm thread promotes a
cached model with no request (`source_status == "background"`), a
compressor-reported cache miss stays deferred with `model not cached`,
and a failure carrying a path and a token leaves exactly `warm failed`.
- `tests/test_proxy_health.py`: sentinel tests over `/health` and
`/readyz` for both leak paths (`slot.info["detail"]` and `slot.error`).
- `tests/test_kompress_health_detail_policy.py`: covers the policy
directly.
- `tests/test_proxy/test_kompress_background_warmup.py` and
`tests/test_proxy_eager_preload_bind.py`: expectations updated for the
slot promotion.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod` does not
exist on Windows). That file is not touched by this PR, and there are no
errors in `server.py` or the new module.

### Test Output

```text
$ pytest tests/test_kompress_preload_deferral.py tests/test_proxy_health.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_health_detail_policy.py tests/test_proxy_eager_preload_bind.py tests/test_proxy_warmup.py -q
73 passed in 22.36s

$ ruff check headroom tests
All checks passed!

$ ruff format --check headroom tests
1519 files already formatted

$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"; maybe "chmod"?  [attr-defined]
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `chopratejas/kompress-v2-base` in the local HF
cache, `HF_HUB_OFFLINE=1`, branch rebased on `upstream/main` 540f4da1
- Exact command / steps: built the real app with
`create_app(ProxyConfig(optimize=True, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False))`, started it
through the FastAPI lifespan with
`HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`, sent no compression request,
and polled `GET /health` every 250 ms until Kompress reported ready,
then read `/readyz` and the warmup slot
- Observed result: at t=0.25 s `/health` returned `{"ready": false,
"status": "degraded", "backend": null, "detail": "warming"}`. The log
showed `Kompress: warmed in the background in 7618 ms (backend onnx)`,
and at t=6.44 s `/health` returned `{"ready": true, "status": "healthy",
"backend": "onnx", "detail": "loaded"}`. `/readyz` matched, and the slot
was `{"status": "loaded", "info": {"source_status": "background",
"backend": "onnx"}}`. No request was needed.
- Not tested: macOS arm64 (the reporter's platform), a real cold-cache
download through the warm thread, and glibc < 2.28 hosts, where #3619
skips the warm-up and `/health` keeps reporting `deferred`.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Health reporting for the existing
background warm-up from #3619.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes, additively. The kompress health
check gains a `detail` field, and a successful background warm marks the
slot `loaded` straight away instead of at the next `/health` call.
`enabled`, `ready`, `status` and `backend` are unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` disables the
background warm-up (from #3619). The `detail` field has no separate
switch; it only reads state.
- Unsafe override required: No
- Qualification impact: None expected. Health payloads gain one field,
and every value comes from a fixed allowlist.
- Rollback path: Revert the PR. There is no stored state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A. No user-facing UI change.

## Additional Notes

- No documentation change: the `detail` field is diagnostic.
`HEADROOM_KOMPRESS_WARMUP` (from #3619) is not in the env-var reference
yet either; happy to add it here if maintainers prefer.
- An earlier revision of this PR added its own
`HEADROOM_KOMPRESS_BACKGROUND_WARM` switch. It was removed after the
rebase because it did the same job as `HEADROOM_KOMPRESS_WARMUP` and was
never released.
- Follow-up, out of scope: the #3619 warm thread is not told about
`shutdown()`. It is a daemon thread and production shutdown ends the
process, but an in-process restart could still run one load after stop.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:30:37 -06:00

395 lines
12 KiB
Python

import pytest
from fastapi.testclient import TestClient
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import create_app
from headroom.transforms import kompress_compressor
class _ReadyCompressor:
def __init__(self, backend="onnx", ready=True, error=None):
self.backend = backend
self.ready = ready
self.error = error
self.calls = []
def is_ready(self):
self.calls.append("is_ready")
if self.error:
raise self.error
return self.ready
def ready_backend(self):
self.calls.append("ready_backend")
return self.backend
def _health_app(monkeypatch, compressor=None, *, disabled=False, **config_kwargs):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
disable_kompress=disabled,
**config_kwargs,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
router = proxy.anthropic_pipeline.transforms[-1]
if compressor is not None:
router._kompress = compressor
return app, proxy
def test_readyz_excludes_kompress_from_aggregate_readiness(monkeypatch):
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
)
)
app.state.ready = True
proxy = app.state.proxy
proxy.http_client = object()
proxy.warmup.kompress.mark_error("model not cached")
client = TestClient(app)
response = client.get("/readyz")
assert response.status_code == 200
payload = response.json()
assert payload["ready"] is True
assert payload["status"] == "healthy"
assert payload["checks"]["kompress"] == {
"enabled": True,
"ready": False,
"status": "degraded",
"optional": True,
"backend": None,
"detail": "model not cached",
}
def test_readyz_promotes_deferred_kompress_after_runtime_load(monkeypatch):
compressor = _ReadyCompressor()
app, proxy = _health_app(monkeypatch, compressor)
proxy.warmup.kompress.info["source_status"] = "deferred"
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"] == {
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "onnx",
"detail": "loaded",
}
# The promotion must also clear the startup marker, otherwise the slot
# serializes as loaded-but-deferred in /debug/warmup.
assert proxy.warmup.kompress.info["source_status"] == "runtime"
@pytest.mark.parametrize("attached", [False, True])
def test_readyz_promotes_kompress_from_module_cache(monkeypatch, attached):
model = object()
monkeypatch.setattr(
kompress_compressor,
"_kompress_cache",
{kompress_compressor.HF_MODEL_ID: (model, object(), "onnx")},
)
compressor = _ReadyCompressor(ready=False) if attached else None
app, proxy = _health_app(monkeypatch, compressor)
proxy.warmup.kompress.info["source_status"] = "deferred"
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"] == {
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "onnx",
"detail": "loaded",
}
assert proxy.warmup.kompress.handle is model
if compressor is not None:
assert compressor.calls == ["is_ready"]
def test_readyz_promotes_remote_kompress_backend(monkeypatch):
compressor = _ReadyCompressor(backend="remote")
app, proxy = _health_app(monkeypatch)
router = proxy.anthropic_pipeline.transforms[-1]
router._kompress = None
router._kompress_remote = compressor
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"]["backend"] == "remote"
assert payload["checks"]["kompress"]["ready"] is True
def test_readyz_keeps_pending_kompress_unloaded(monkeypatch):
compressor = _ReadyCompressor(backend="onnx", ready=False)
app, proxy = _health_app(monkeypatch, compressor)
router = proxy.anthropic_pipeline.transforms[-1]
router._kompress = compressor
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"] == {
"enabled": True,
"ready": False,
"status": "degraded",
"optional": True,
"backend": None,
"detail": "not installed",
}
assert compressor.calls == ["is_ready"]
def test_readyz_never_starts_kompress_loading(monkeypatch):
compressor = _ReadyCompressor()
app, proxy = _health_app(monkeypatch, compressor)
router = proxy.anthropic_pipeline.transforms[-1]
router._kompress = compressor
TestClient(app).get("/readyz")
assert compressor.calls == ["is_ready", "ready_backend"]
def test_readyz_kompress_inspection_failure_fails_open(monkeypatch):
compressor = _ReadyCompressor(error=RuntimeError("inspection failed"))
app, proxy = _health_app(monkeypatch, compressor)
proxy.warmup.kompress.mark_loaded(handle=object(), backend="onnx")
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"]["ready"] is True
assert payload["checks"]["kompress"]["backend"] == "onnx"
def test_readyz_disabled_kompress_skips_inspection(monkeypatch):
compressor = _ReadyCompressor()
app, proxy = _health_app(monkeypatch, compressor, disabled=True)
router = proxy.anthropic_pipeline.transforms[-1]
router._kompress = compressor
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"] == {
"enabled": False,
"ready": True,
"status": "disabled",
"optional": True,
"backend": None,
"detail": "not installed",
}
assert compressor.calls == []
def test_readyz_per_provider_kompress_override_reenables_health(monkeypatch):
compressor = _ReadyCompressor()
app, proxy = _health_app(
monkeypatch,
disabled=True,
disable_kompress_anthropic=False,
)
router = proxy.anthropic_pipeline.transforms[-1]
router._kompress = compressor
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"] == {
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "onnx",
"detail": "loaded",
}
assert compressor.calls == ["is_ready", "ready_backend"]
def test_readyz_never_calls_lazy_kompress_getters(monkeypatch):
app, proxy = _health_app(monkeypatch)
router = proxy.anthropic_pipeline.transforms[-1]
def _boom():
raise AssertionError("health should not instantiate kompress")
router._get_kompress = _boom
router._get_remote_kompress = _boom
payload = TestClient(app).get("/readyz").json()
assert payload["checks"]["kompress"] == {
"enabled": True,
"ready": False,
"status": "degraded",
"optional": True,
"backend": None,
"detail": "not installed",
}
@pytest.mark.parametrize(
("slot_status", "compressor", "disabled", "expected"),
[
(
"null",
None,
False,
{
"enabled": True,
"ready": False,
"status": "degraded",
"optional": True,
"backend": None,
"detail": "not installed",
},
),
(
"null",
_ReadyCompressor(),
False,
{
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "onnx",
"detail": "loaded",
},
),
(
"null",
_ReadyCompressor(backend="remote"),
False,
{
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "remote",
"detail": "loaded",
},
),
(
"error",
_ReadyCompressor(),
False,
{
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "onnx",
"detail": "loaded",
},
),
(
"loaded",
_ReadyCompressor(),
False,
{
"enabled": True,
"ready": True,
"status": "healthy",
"optional": True,
"backend": "existing",
"detail": "loaded",
},
),
(
"null",
_ReadyCompressor(),
True,
{
"enabled": False,
"ready": True,
"status": "disabled",
"optional": True,
"backend": None,
"detail": "not installed",
},
),
],
)
def test_readyz_kompress_state_matrix(monkeypatch, slot_status, compressor, disabled, expected):
app, proxy = _health_app(monkeypatch, compressor, disabled=disabled)
if slot_status == "error":
proxy.warmup.kompress.mark_error("not cached")
elif slot_status == "loaded":
proxy.warmup.kompress.mark_loaded(handle=object(), backend=expected["backend"])
if compressor is not None and expected["backend"] == "existing":
compressor.backend = "new"
payload = TestClient(app).get("/readyz").json()["checks"]["kompress"]
assert payload == expected
# /health and /readyz are auth-exempt, so nothing they serialize may carry raw
# exception text: loader failures embed absolute paths, model ids and URLs.
_SENTINEL_DETAIL = "/opt/secret-path/model.onnx token=SENTINEL-LEAK-9f3a"
@pytest.mark.parametrize("endpoint", ["/health", "/readyz"])
def test_public_health_does_not_leak_warm_failure_text(monkeypatch, endpoint):
app, proxy = _health_app(monkeypatch)
# What the background warm thread would have written pre-#2799 review.
proxy.warmup.kompress.info["detail"] = f"warm failed: {_SENTINEL_DETAIL}"
response = TestClient(app).get(endpoint)
assert response.json()["checks"]["kompress"]["detail"] == "unavailable"
assert "SENTINEL-LEAK-9f3a" not in response.text
assert "secret-path" not in response.text
@pytest.mark.parametrize("endpoint", ["/health", "/readyz"])
def test_public_health_does_not_leak_slot_error_text(monkeypatch, endpoint):
app, proxy = _health_app(monkeypatch)
proxy.warmup.kompress.mark_error(f"native load failed: {_SENTINEL_DETAIL}")
response = TestClient(app).get(endpoint)
assert response.json()["checks"]["kompress"]["detail"] == "unavailable"
assert "SENTINEL-LEAK-9f3a" not in response.text
assert "secret-path" not in response.text
def test_remote_kompress_endpoint_skips_local_model_at_startup(monkeypatch):
"""With HEADROOM_KOMPRESS_ENDPOINT set, startup neither fetches nor warms the
local model, and /readyz still names the remote backend."""
import threading
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "https://kompress.example.test")
monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP", "1")
monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS", "3600")
prefetched: list[str] = []
monkeypatch.setattr(
kompress_compressor,
"ensure_background_prefetch",
lambda model_id: (prefetched.append(model_id), True)[1],
)
app = create_app(ProxyConfig(optimize=True, cache_enabled=False, rate_limit_enabled=False))
with TestClient(app) as client:
app.state.proxy.http_client = object()
payload = client.get("/readyz").json()
assert prefetched == []
assert not any(t.name == "kompress-warmup" for t in threading.enumerate())
assert payload["checks"]["kompress"]["backend"] == "remote"