mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
On an idle proxy with the ML extras installed and the Kompress model
cached, `/health` and `/readyz` reported `kompress: {"ready": false,
"status": "unhealthy", "backend": null}` forever (#2730).
The cause was not missing wiring: `enable_kompress` defaults to `True`
and `ContentRouter` sits in both pipelines. The chain was:
1. `ContentRouter.eager_load_compressors()` deliberately skips
`preload()`, so a native load never runs before the port binds (#790
hang on Windows, #1908 segfault on RHEL/CentOS 7). It returns `kompress
= "deferred"`.
2. `WarmupRegistry.merge_transform_status` leaves a deferred slot at
`status="null"`.
3. `_reconcile_kompress_health()` promotes the slot only once a
compressor reports `is_ready()`, and nothing loaded the model without a
kompress-routed request.
#3619 (merged after this PR was opened) added
`_start_kompress_background_warmup()`, which loads the model on a daemon
thread shortly after startup. That fixes the load itself. Rebased onto
it, this PR covers what is left:
1. **`/health` says why `ready` is false.** The kompress check gains a
`detail` field. `ready: false` alone cannot tell "still warming" from
"extras not installed" from "warm failed", which is the confusion in
#2730.
2. **The warm thread records its outcome on `warmup.kompress`.** Success
promotes the slot immediately with `source_status="background"`, instead
of leaving `source_status="deferred"` on a loaded slot until the next
reconcile. Failure leaves a bounded reason.
3. **The detail is a closed vocabulary.** `/health` and `/readyz` are
auth-exempt, and loader exceptions carry paths, model ids and URLs, so
anything outside the allowlist collapses to `unavailable`.
Closes #2730
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/kompress_health_detail_policy.py` (new, pure, matching
the existing `*_policy.py` modules): `KOMPRESS_HEALTH_DETAILS` lists the
values the field may report (`loaded`, `warming`, `model not cached`,
`warm failed`, `not installed`, plus the `eager_load_compressors()`
statuses `enabled` / `deferred` / `unavailable`). `public_detail()`
passes those through and maps anything else to `unavailable`.
- `headroom/proxy/server.py`:
- `_start_kompress_background_warmup()` (from #3619, still
`preload(allow_download=True)` on thread `kompress-warmup`) now writes
its result to the slot. On success it calls `mark_loaded(...,
source_status="background")`. On any exception it sets detail `warm
failed` and logs the real cause with `exc_info=True`; the exception text
never reaches the slot. A compressor that raises
`KompressModelNotCached` gets `model not cached`. The local loader only
raises that with downloads off, so on a cold cache it downloads and
`/health` reports `warming` in the meantime.
- `_kompress_detail()` feeds the new field. Every candidate, including
raw `slot.error` and `source_status`, goes through `public_detail()`, so
the allowlist has a single choke point.
- Router/compressor lookup moves to module-level `_kompress_routers()` /
`_kompress_compressors()`, shared by startup and `/health`.
- `enabled`, `ready` and `backend` keep their meaning, so the `/readyz`
aggregate exclusion of optional Kompress still holds.
- Tests:
- `tests/test_kompress_preload_deferral.py`: the warm thread promotes a
cached model with no request (`source_status == "background"`), a
compressor-reported cache miss stays deferred with `model not cached`,
and a failure carrying a path and a token leaves exactly `warm failed`.
- `tests/test_proxy_health.py`: sentinel tests over `/health` and
`/readyz` for both leak paths (`slot.info["detail"]` and `slot.error`).
- `tests/test_kompress_health_detail_policy.py`: covers the policy
directly.
- `tests/test_proxy/test_kompress_background_warmup.py` and
`tests/test_proxy_eager_preload_bind.py`: expectations updated for the
slot promotion.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod` does not
exist on Windows). That file is not touched by this PR, and there are no
errors in `server.py` or the new module.
### Test Output
```text
$ pytest tests/test_kompress_preload_deferral.py tests/test_proxy_health.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_health_detail_policy.py tests/test_proxy_eager_preload_bind.py tests/test_proxy_warmup.py -q
73 passed in 22.36s
$ ruff check headroom tests
All checks passed!
$ ruff format --check headroom tests
1519 files already formatted
$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"; maybe "chmod"? [attr-defined]
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `chopratejas/kompress-v2-base` in the local HF
cache, `HF_HUB_OFFLINE=1`, branch rebased on `upstream/main` 540f4da1
- Exact command / steps: built the real app with
`create_app(ProxyConfig(optimize=True, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False))`, started it
through the FastAPI lifespan with
`HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`, sent no compression request,
and polled `GET /health` every 250 ms until Kompress reported ready,
then read `/readyz` and the warmup slot
- Observed result: at t=0.25 s `/health` returned `{"ready": false,
"status": "degraded", "backend": null, "detail": "warming"}`. The log
showed `Kompress: warmed in the background in 7618 ms (backend onnx)`,
and at t=6.44 s `/health` returned `{"ready": true, "status": "healthy",
"backend": "onnx", "detail": "loaded"}`. `/readyz` matched, and the slot
was `{"status": "loaded", "info": {"source_status": "background",
"backend": "onnx"}}`. No request was needed.
- Not tested: macOS arm64 (the reporter's platform), a real cold-cache
download through the warm thread, and glibc < 2.28 hosts, where #3619
skips the warm-up and `/health` keeps reporting `deferred`.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. Health reporting for the existing
background warm-up from #3619.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes, additively. The kompress health
check gains a `detail` field, and a successful background warm marks the
slot `loaded` straight away instead of at the next `/health` call.
`enabled`, `ready`, `status` and `backend` are unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` disables the
background warm-up (from #3619). The `detail` field has no separate
switch; it only reads state.
- Unsafe override required: No
- Qualification impact: None expected. Health payloads gain one field,
and every value comes from a fixed allowlist.
- Rollback path: Revert the PR. There is no stored state or migration.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A. No user-facing UI change.
## Additional Notes
- No documentation change: the `detail` field is diagnostic.
`HEADROOM_KOMPRESS_WARMUP` (from #3619) is not in the env-var reference
yet either; happy to add it here if maintainers prefer.
- An earlier revision of this PR added its own
`HEADROOM_KOMPRESS_BACKGROUND_WARM` switch. It was removed after the
rebase because it did the same job as `HEADROOM_KOMPRESS_WARMUP` and was
never released.
- Follow-up, out of scope: the #3619 warm thread is not told about
`shutdown()`. It is a daemon thread and production shutdown ends the
process, but an in-process restart could still run one load after stop.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
395 lines
12 KiB
Python
395 lines
12 KiB
Python
import pytest
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.models import ProxyConfig
|
|
from headroom.proxy.server import create_app
|
|
from headroom.transforms import kompress_compressor
|
|
|
|
|
|
class _ReadyCompressor:
|
|
def __init__(self, backend="onnx", ready=True, error=None):
|
|
self.backend = backend
|
|
self.ready = ready
|
|
self.error = error
|
|
self.calls = []
|
|
|
|
def is_ready(self):
|
|
self.calls.append("is_ready")
|
|
if self.error:
|
|
raise self.error
|
|
return self.ready
|
|
|
|
def ready_backend(self):
|
|
self.calls.append("ready_backend")
|
|
return self.backend
|
|
|
|
|
|
def _health_app(monkeypatch, compressor=None, *, disabled=False, **config_kwargs):
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
disable_kompress=disabled,
|
|
**config_kwargs,
|
|
)
|
|
)
|
|
app.state.ready = True
|
|
proxy = app.state.proxy
|
|
proxy.http_client = object()
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
if compressor is not None:
|
|
router._kompress = compressor
|
|
return app, proxy
|
|
|
|
|
|
def test_readyz_excludes_kompress_from_aggregate_readiness(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
)
|
|
)
|
|
app.state.ready = True
|
|
proxy = app.state.proxy
|
|
proxy.http_client = object()
|
|
proxy.warmup.kompress.mark_error("model not cached")
|
|
|
|
client = TestClient(app)
|
|
response = client.get("/readyz")
|
|
|
|
assert response.status_code == 200
|
|
payload = response.json()
|
|
assert payload["ready"] is True
|
|
assert payload["status"] == "healthy"
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": True,
|
|
"ready": False,
|
|
"status": "degraded",
|
|
"optional": True,
|
|
"backend": None,
|
|
"detail": "model not cached",
|
|
}
|
|
|
|
|
|
def test_readyz_promotes_deferred_kompress_after_runtime_load(monkeypatch):
|
|
compressor = _ReadyCompressor()
|
|
app, proxy = _health_app(monkeypatch, compressor)
|
|
proxy.warmup.kompress.info["source_status"] = "deferred"
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "onnx",
|
|
"detail": "loaded",
|
|
}
|
|
# The promotion must also clear the startup marker, otherwise the slot
|
|
# serializes as loaded-but-deferred in /debug/warmup.
|
|
assert proxy.warmup.kompress.info["source_status"] == "runtime"
|
|
|
|
|
|
@pytest.mark.parametrize("attached", [False, True])
|
|
def test_readyz_promotes_kompress_from_module_cache(monkeypatch, attached):
|
|
model = object()
|
|
monkeypatch.setattr(
|
|
kompress_compressor,
|
|
"_kompress_cache",
|
|
{kompress_compressor.HF_MODEL_ID: (model, object(), "onnx")},
|
|
)
|
|
compressor = _ReadyCompressor(ready=False) if attached else None
|
|
app, proxy = _health_app(monkeypatch, compressor)
|
|
proxy.warmup.kompress.info["source_status"] = "deferred"
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "onnx",
|
|
"detail": "loaded",
|
|
}
|
|
assert proxy.warmup.kompress.handle is model
|
|
if compressor is not None:
|
|
assert compressor.calls == ["is_ready"]
|
|
|
|
|
|
def test_readyz_promotes_remote_kompress_backend(monkeypatch):
|
|
compressor = _ReadyCompressor(backend="remote")
|
|
app, proxy = _health_app(monkeypatch)
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
router._kompress = None
|
|
router._kompress_remote = compressor
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"]["backend"] == "remote"
|
|
assert payload["checks"]["kompress"]["ready"] is True
|
|
|
|
|
|
def test_readyz_keeps_pending_kompress_unloaded(monkeypatch):
|
|
compressor = _ReadyCompressor(backend="onnx", ready=False)
|
|
app, proxy = _health_app(monkeypatch, compressor)
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
router._kompress = compressor
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": True,
|
|
"ready": False,
|
|
"status": "degraded",
|
|
"optional": True,
|
|
"backend": None,
|
|
"detail": "not installed",
|
|
}
|
|
assert compressor.calls == ["is_ready"]
|
|
|
|
|
|
def test_readyz_never_starts_kompress_loading(monkeypatch):
|
|
compressor = _ReadyCompressor()
|
|
app, proxy = _health_app(monkeypatch, compressor)
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
router._kompress = compressor
|
|
|
|
TestClient(app).get("/readyz")
|
|
|
|
assert compressor.calls == ["is_ready", "ready_backend"]
|
|
|
|
|
|
def test_readyz_kompress_inspection_failure_fails_open(monkeypatch):
|
|
compressor = _ReadyCompressor(error=RuntimeError("inspection failed"))
|
|
app, proxy = _health_app(monkeypatch, compressor)
|
|
proxy.warmup.kompress.mark_loaded(handle=object(), backend="onnx")
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"]["ready"] is True
|
|
assert payload["checks"]["kompress"]["backend"] == "onnx"
|
|
|
|
|
|
def test_readyz_disabled_kompress_skips_inspection(monkeypatch):
|
|
compressor = _ReadyCompressor()
|
|
app, proxy = _health_app(monkeypatch, compressor, disabled=True)
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
router._kompress = compressor
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": False,
|
|
"ready": True,
|
|
"status": "disabled",
|
|
"optional": True,
|
|
"backend": None,
|
|
"detail": "not installed",
|
|
}
|
|
assert compressor.calls == []
|
|
|
|
|
|
def test_readyz_per_provider_kompress_override_reenables_health(monkeypatch):
|
|
compressor = _ReadyCompressor()
|
|
app, proxy = _health_app(
|
|
monkeypatch,
|
|
disabled=True,
|
|
disable_kompress_anthropic=False,
|
|
)
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
router._kompress = compressor
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "onnx",
|
|
"detail": "loaded",
|
|
}
|
|
assert compressor.calls == ["is_ready", "ready_backend"]
|
|
|
|
|
|
def test_readyz_never_calls_lazy_kompress_getters(monkeypatch):
|
|
app, proxy = _health_app(monkeypatch)
|
|
router = proxy.anthropic_pipeline.transforms[-1]
|
|
|
|
def _boom():
|
|
raise AssertionError("health should not instantiate kompress")
|
|
|
|
router._get_kompress = _boom
|
|
router._get_remote_kompress = _boom
|
|
|
|
payload = TestClient(app).get("/readyz").json()
|
|
|
|
assert payload["checks"]["kompress"] == {
|
|
"enabled": True,
|
|
"ready": False,
|
|
"status": "degraded",
|
|
"optional": True,
|
|
"backend": None,
|
|
"detail": "not installed",
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("slot_status", "compressor", "disabled", "expected"),
|
|
[
|
|
(
|
|
"null",
|
|
None,
|
|
False,
|
|
{
|
|
"enabled": True,
|
|
"ready": False,
|
|
"status": "degraded",
|
|
"optional": True,
|
|
"backend": None,
|
|
"detail": "not installed",
|
|
},
|
|
),
|
|
(
|
|
"null",
|
|
_ReadyCompressor(),
|
|
False,
|
|
{
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "onnx",
|
|
"detail": "loaded",
|
|
},
|
|
),
|
|
(
|
|
"null",
|
|
_ReadyCompressor(backend="remote"),
|
|
False,
|
|
{
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "remote",
|
|
"detail": "loaded",
|
|
},
|
|
),
|
|
(
|
|
"error",
|
|
_ReadyCompressor(),
|
|
False,
|
|
{
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "onnx",
|
|
"detail": "loaded",
|
|
},
|
|
),
|
|
(
|
|
"loaded",
|
|
_ReadyCompressor(),
|
|
False,
|
|
{
|
|
"enabled": True,
|
|
"ready": True,
|
|
"status": "healthy",
|
|
"optional": True,
|
|
"backend": "existing",
|
|
"detail": "loaded",
|
|
},
|
|
),
|
|
(
|
|
"null",
|
|
_ReadyCompressor(),
|
|
True,
|
|
{
|
|
"enabled": False,
|
|
"ready": True,
|
|
"status": "disabled",
|
|
"optional": True,
|
|
"backend": None,
|
|
"detail": "not installed",
|
|
},
|
|
),
|
|
],
|
|
)
|
|
def test_readyz_kompress_state_matrix(monkeypatch, slot_status, compressor, disabled, expected):
|
|
app, proxy = _health_app(monkeypatch, compressor, disabled=disabled)
|
|
if slot_status == "error":
|
|
proxy.warmup.kompress.mark_error("not cached")
|
|
elif slot_status == "loaded":
|
|
proxy.warmup.kompress.mark_loaded(handle=object(), backend=expected["backend"])
|
|
if compressor is not None and expected["backend"] == "existing":
|
|
compressor.backend = "new"
|
|
|
|
payload = TestClient(app).get("/readyz").json()["checks"]["kompress"]
|
|
|
|
assert payload == expected
|
|
|
|
|
|
# /health and /readyz are auth-exempt, so nothing they serialize may carry raw
|
|
# exception text: loader failures embed absolute paths, model ids and URLs.
|
|
_SENTINEL_DETAIL = "/opt/secret-path/model.onnx token=SENTINEL-LEAK-9f3a"
|
|
|
|
|
|
@pytest.mark.parametrize("endpoint", ["/health", "/readyz"])
|
|
def test_public_health_does_not_leak_warm_failure_text(monkeypatch, endpoint):
|
|
app, proxy = _health_app(monkeypatch)
|
|
# What the background warm thread would have written pre-#2799 review.
|
|
proxy.warmup.kompress.info["detail"] = f"warm failed: {_SENTINEL_DETAIL}"
|
|
|
|
response = TestClient(app).get(endpoint)
|
|
|
|
assert response.json()["checks"]["kompress"]["detail"] == "unavailable"
|
|
assert "SENTINEL-LEAK-9f3a" not in response.text
|
|
assert "secret-path" not in response.text
|
|
|
|
|
|
@pytest.mark.parametrize("endpoint", ["/health", "/readyz"])
|
|
def test_public_health_does_not_leak_slot_error_text(monkeypatch, endpoint):
|
|
app, proxy = _health_app(monkeypatch)
|
|
proxy.warmup.kompress.mark_error(f"native load failed: {_SENTINEL_DETAIL}")
|
|
|
|
response = TestClient(app).get(endpoint)
|
|
|
|
assert response.json()["checks"]["kompress"]["detail"] == "unavailable"
|
|
assert "SENTINEL-LEAK-9f3a" not in response.text
|
|
assert "secret-path" not in response.text
|
|
|
|
|
|
def test_remote_kompress_endpoint_skips_local_model_at_startup(monkeypatch):
|
|
"""With HEADROOM_KOMPRESS_ENDPOINT set, startup neither fetches nor warms the
|
|
local model, and /readyz still names the remote backend."""
|
|
import threading
|
|
|
|
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
|
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "https://kompress.example.test")
|
|
monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP", "1")
|
|
monkeypatch.setenv("HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS", "3600")
|
|
prefetched: list[str] = []
|
|
monkeypatch.setattr(
|
|
kompress_compressor,
|
|
"ensure_background_prefetch",
|
|
lambda model_id: (prefetched.append(model_id), True)[1],
|
|
)
|
|
app = create_app(ProxyConfig(optimize=True, cache_enabled=False, rate_limit_enabled=False))
|
|
|
|
with TestClient(app) as client:
|
|
app.state.proxy.http_client = object()
|
|
payload = client.get("/readyz").json()
|
|
|
|
assert prefetched == []
|
|
assert not any(t.name == "kompress-warmup" for t in threading.enumerate())
|
|
assert payload["checks"]["kompress"]["backend"] == "remote"
|