mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description
On an idle proxy with the ML extras installed and the Kompress model
cached, `/health` and `/readyz` reported `kompress: {"ready": false,
"status": "unhealthy", "backend": null}` forever (#2730).
The cause was not missing wiring: `enable_kompress` defaults to `True`
and `ContentRouter` sits in both pipelines. The chain was:
1. `ContentRouter.eager_load_compressors()` deliberately skips
`preload()`, so a native load never runs before the port binds (#790
hang on Windows, #1908 segfault on RHEL/CentOS 7). It returns `kompress
= "deferred"`.
2. `WarmupRegistry.merge_transform_status` leaves a deferred slot at
`status="null"`.
3. `_reconcile_kompress_health()` promotes the slot only once a
compressor reports `is_ready()`, and nothing loaded the model without a
kompress-routed request.
#3619 (merged after this PR was opened) added
`_start_kompress_background_warmup()`, which loads the model on a daemon
thread shortly after startup. That fixes the load itself. Rebased onto
it, this PR covers what is left:
1. **`/health` says why `ready` is false.** The kompress check gains a
`detail` field. `ready: false` alone cannot tell "still warming" from
"extras not installed" from "warm failed", which is the confusion in
#2730.
2. **The warm thread records its outcome on `warmup.kompress`.** Success
promotes the slot immediately with `source_status="background"`, instead
of leaving `source_status="deferred"` on a loaded slot until the next
reconcile. Failure leaves a bounded reason.
3. **The detail is a closed vocabulary.** `/health` and `/readyz` are
auth-exempt, and loader exceptions carry paths, model ids and URLs, so
anything outside the allowlist collapses to `unavailable`.
Closes #2730
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/kompress_health_detail_policy.py` (new, pure, matching
the existing `*_policy.py` modules): `KOMPRESS_HEALTH_DETAILS` lists the
values the field may report (`loaded`, `warming`, `model not cached`,
`warm failed`, `not installed`, plus the `eager_load_compressors()`
statuses `enabled` / `deferred` / `unavailable`). `public_detail()`
passes those through and maps anything else to `unavailable`.
- `headroom/proxy/server.py`:
- `_start_kompress_background_warmup()` (from #3619, still
`preload(allow_download=True)` on thread `kompress-warmup`) now writes
its result to the slot. On success it calls `mark_loaded(...,
source_status="background")`. On any exception it sets detail `warm
failed` and logs the real cause with `exc_info=True`; the exception text
never reaches the slot. A compressor that raises
`KompressModelNotCached` gets `model not cached`. The local loader only
raises that with downloads off, so on a cold cache it downloads and
`/health` reports `warming` in the meantime.
- `_kompress_detail()` feeds the new field. Every candidate, including
raw `slot.error` and `source_status`, goes through `public_detail()`, so
the allowlist has a single choke point.
- Router/compressor lookup moves to module-level `_kompress_routers()` /
`_kompress_compressors()`, shared by startup and `/health`.
- `enabled`, `ready` and `backend` keep their meaning, so the `/readyz`
aggregate exclusion of optional Kompress still holds.
- Tests:
- `tests/test_kompress_preload_deferral.py`: the warm thread promotes a
cached model with no request (`source_status == "background"`), a
compressor-reported cache miss stays deferred with `model not cached`,
and a failure carrying a path and a token leaves exactly `warm failed`.
- `tests/test_proxy_health.py`: sentinel tests over `/health` and
`/readyz` for both leak paths (`slot.info["detail"]` and `slot.error`).
- `tests/test_kompress_health_detail_policy.py`: covers the policy
directly.
- `tests/test_proxy/test_kompress_background_warmup.py` and
`tests/test_proxy_eager_preload_bind.py`: expectations updated for the
slot promotion.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Type checking: `mypy headroom --ignore-missing-imports` (mypy 1.20.2)
reports one error, in `headroom/fileperms.py` (`os.fchmod` does not
exist on Windows). That file is not touched by this PR, and there are no
errors in `server.py` or the new module.
### Test Output
```text
$ pytest tests/test_kompress_preload_deferral.py tests/test_proxy_health.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_health_detail_policy.py tests/test_proxy_eager_preload_bind.py tests/test_proxy_warmup.py -q
73 passed in 22.36s
$ ruff check headroom tests
All checks passed!
$ ruff format --check headroom tests
1519 files already formatted
$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:71: error: Module has no attribute "fchmod"; maybe "chmod"? [attr-defined]
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, headroom installed from this
branch, Kompress model `chopratejas/kompress-v2-base` in the local HF
cache, `HF_HUB_OFFLINE=1`, branch rebased on `upstream/main` 540f4da1
- Exact command / steps: built the real app with
`create_app(ProxyConfig(optimize=True, cache_enabled=False,
rate_limit_enabled=False, cost_tracking_enabled=False))`, started it
through the FastAPI lifespan with
`HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`, sent no compression request,
and polled `GET /health` every 250 ms until Kompress reported ready,
then read `/readyz` and the warmup slot
- Observed result: at t=0.25 s `/health` returned `{"ready": false,
"status": "degraded", "backend": null, "detail": "warming"}`. The log
showed `Kompress: warmed in the background in 7618 ms (backend onnx)`,
and at t=6.44 s `/health` returned `{"ready": true, "status": "healthy",
"backend": "onnx", "detail": "loaded"}`. `/readyz` matched, and the slot
was `{"status": "loaded", "info": {"source_status": "background",
"backend": "onnx"}}`. No request was needed.
- Not tested: macOS arm64 (the reporter's platform), a real cold-cache
download through the warm thread, and glibc < 2.28 hosts, where #3619
skips the warm-up and `/health` keeps reporting `deferred`.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. Health reporting for the existing
background warm-up from #3619.
- Minimum rollout channel: stable
- Stable/default behavior changed: Yes, additively. The kompress health
check gains a `detail` field, and a successful background warm marks the
slot `loaded` straight away instead of at the next `/health` call.
`enabled`, `ready`, `status` and `backend` are unchanged.
- Kill switch / disable path: `HEADROOM_KOMPRESS_WARMUP=0` disables the
background warm-up (from #3619). The `detail` field has no separate
switch; it only reads state.
- Unsafe override required: No
- Qualification impact: None expected. Health payloads gain one field,
and every value comes from a fixed allowlist.
- Rollback path: Revert the PR. There is no stored state or migration.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`. It is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A. No user-facing UI change.
## Additional Notes
- No documentation change: the `detail` field is diagnostic.
`HEADROOM_KOMPRESS_WARMUP` (from #3619) is not in the env-var reference
yet either; happy to add it here if maintainers prefer.
- An earlier revision of this PR added its own
`HEADROOM_KOMPRESS_BACKGROUND_WARM` switch. It was removed after the
rebase because it did the same job as `HEADROOM_KOMPRESS_WARMUP` and was
never released.
- Follow-up, out of scope: the #3619 warm thread is not told about
`shutdown()`. It is a daemon thread and production shutdown ends the
process, but an in-process restart could still run one load after stop.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
289 lines
9.9 KiB
Python
289 lines
9.9 KiB
Python
"""Startup must bind its port even when eager preload hangs (#790).
|
|
|
|
``HeadroomProxy.startup()`` runs inside the ASGI lifespan, which completes
|
|
*before* uvicorn binds the socket. The eager compressor/parser preload used to
|
|
run synchronously there, so a hang or an uncatchable native stall during a model
|
|
load (observed on Windows) left the proxy "never opening its port". The preload
|
|
now runs off the event loop under ``asyncio.wait_for`` with
|
|
``EAGER_PRELOAD_TIMEOUT_SECONDS``; on timeout startup logs and continues so the
|
|
bind still happens and transforms fall back to lazy loading.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import threading
|
|
import time
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
import headroom.proxy.server as server_mod
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
def _make_proxy(
|
|
*,
|
|
optimize: bool,
|
|
disable_kompress: bool = False,
|
|
disable_kompress_anthropic: bool | None = None,
|
|
disable_kompress_openai: bool | None = None,
|
|
compressors: set[str] | None = None,
|
|
):
|
|
config = ProxyConfig(
|
|
optimize=optimize,
|
|
disable_kompress=disable_kompress,
|
|
disable_kompress_anthropic=disable_kompress_anthropic,
|
|
disable_kompress_openai=disable_kompress_openai,
|
|
compressors=compressors,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
subscription_tracking_enabled=False,
|
|
)
|
|
proxy = create_app(config).state.proxy
|
|
# Every test here substitutes fake pipelines to control exactly what the
|
|
# preload walks. The proxy also eagerly builds the default /v1/compress
|
|
# pipeline (a derived ContentRouter, warmed alongside the request
|
|
# pipelines), which would inject real transform statuses into those
|
|
# assertions — drop it so the fakes remain the only input.
|
|
proxy._compress_pipeline_cache = {}
|
|
return proxy
|
|
|
|
|
|
class _FastTransform:
|
|
def __init__(self, status):
|
|
self._status = status
|
|
|
|
def eager_load_compressors(self):
|
|
return self._status
|
|
|
|
|
|
class _RaisingTransform:
|
|
def eager_load_compressors(self):
|
|
raise RuntimeError("boom")
|
|
|
|
|
|
class _NonDictTransform:
|
|
def eager_load_compressors(self):
|
|
return "not-a-dict"
|
|
|
|
|
|
class _HangingTransform:
|
|
"""Simulates a model load that hangs forever (released via the event)."""
|
|
|
|
def __init__(self, release: threading.Event):
|
|
self._release = release
|
|
|
|
def eager_load_compressors(self):
|
|
# Safety cap so a misbehaving test can never wedge the suite.
|
|
self._release.wait(timeout=30)
|
|
return {"hang": "done"}
|
|
|
|
|
|
class _FakePipeline:
|
|
def __init__(self, transforms):
|
|
self.transforms = transforms
|
|
|
|
|
|
def test_eager_preload_dedupes_and_swallows_failures():
|
|
proxy = _make_proxy(optimize=False)
|
|
shared = _FastTransform({"shared": "enabled"})
|
|
proxy.anthropic_pipeline = _FakePipeline([shared, _FastTransform({"kompress": "enabled"})])
|
|
# ``shared`` appears in both pipelines and must load exactly once; the
|
|
# raising and non-dict transforms must be skipped without aborting.
|
|
proxy.openai_pipeline = _FakePipeline([shared, _RaisingTransform(), _NonDictTransform()])
|
|
|
|
eager_status, statuses = proxy._eager_preload_transforms()
|
|
|
|
# Keys the preload contributes itself rather than collecting from a
|
|
# transform, so this assertion stays about dedupe/swallowing.
|
|
non_transform_keys = {"litellm"}
|
|
assert {k: v for k, v in eager_status.items() if k not in non_transform_keys} == {
|
|
"shared": "enabled",
|
|
"kompress": "enabled",
|
|
}
|
|
assert statuses == [{"shared": "enabled"}, {"kompress": "enabled"}]
|
|
assert eager_status["litellm"] in {"ready", "not installed", "skipped"}
|
|
|
|
|
|
async def test_startup_binds_despite_hung_preload(monkeypatch):
|
|
monkeypatch.setattr(server_mod, "EAGER_PRELOAD_TIMEOUT_SECONDS", 0.3)
|
|
proxy = _make_proxy(optimize=True)
|
|
release = threading.Event()
|
|
proxy.anthropic_pipeline = _FakePipeline([_HangingTransform(release)])
|
|
proxy.openai_pipeline = _FakePipeline([])
|
|
|
|
try:
|
|
start = time.monotonic()
|
|
await proxy.startup() # must NOT wait on the hung load
|
|
elapsed = time.monotonic() - start
|
|
# Returns shortly after the 0.3s preload timeout, far below the 30s hang.
|
|
assert elapsed < 10
|
|
finally:
|
|
release.set()
|
|
await proxy.shutdown()
|
|
|
|
|
|
async def test_startup_merges_warmup_for_normal_transforms(monkeypatch):
|
|
proxy = _make_proxy(optimize=True)
|
|
captured: list[dict] = []
|
|
monkeypatch.setattr(proxy.warmup, "merge_transform_status", captured.append)
|
|
proxy.anthropic_pipeline = _FakePipeline([_FastTransform({"kompress": "enabled"})])
|
|
proxy.openai_pipeline = _FakePipeline([])
|
|
|
|
try:
|
|
await proxy.startup()
|
|
assert {"kompress": "enabled"} in captured
|
|
assert proxy._kompress_status == "enabled"
|
|
finally:
|
|
await proxy.shutdown()
|
|
|
|
|
|
async def test_startup_reports_deferred_kompress(caplog):
|
|
proxy = _make_proxy(optimize=True)
|
|
proxy.anthropic_pipeline = _FakePipeline([_FastTransform({"kompress": "deferred"})])
|
|
proxy.openai_pipeline = _FakePipeline([])
|
|
|
|
try:
|
|
# Attach caplog directly so unrelated propagation mutations cannot
|
|
# affect this assertion.
|
|
server_mod.logger.addHandler(caplog.handler)
|
|
try:
|
|
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
|
await proxy.startup()
|
|
finally:
|
|
server_mod.logger.removeHandler(caplog.handler)
|
|
|
|
assert proxy._kompress_status == "deferred"
|
|
# The wording after the prefix is conditional (a background warm-up
|
|
# thread started or not), so pin the prefix and the real invariant:
|
|
# deferred must never be reported as "not installed".
|
|
assert any(message.startswith("Kompress: DEFERRED") for message in caplog.messages)
|
|
assert not any("Kompress: not installed" in message for message in caplog.messages)
|
|
finally:
|
|
await proxy.shutdown()
|
|
|
|
|
|
@pytest.mark.parametrize("optimize", [True, False])
|
|
async def test_startup_reports_disabled_kompress(caplog, monkeypatch, optimize):
|
|
proxy = _make_proxy(optimize=optimize, disable_kompress=True)
|
|
monkeypatch.setattr(proxy, "_eager_preload_transforms", lambda: ({}, []))
|
|
|
|
try:
|
|
server_mod.logger.addHandler(caplog.handler)
|
|
try:
|
|
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
|
await proxy.startup()
|
|
finally:
|
|
server_mod.logger.removeHandler(caplog.handler)
|
|
|
|
assert "Kompress: DISABLED" in caplog.messages
|
|
assert (
|
|
"Kompress: not installed (pip install headroom-ai[ml] for ML compression)"
|
|
not in caplog.messages
|
|
)
|
|
finally:
|
|
await proxy.shutdown()
|
|
|
|
|
|
async def test_startup_keeps_not_installed_message_when_kompress_is_missing(caplog, monkeypatch):
|
|
proxy = _make_proxy(optimize=True)
|
|
monkeypatch.setattr(proxy, "_eager_preload_transforms", lambda: ({}, []))
|
|
|
|
try:
|
|
server_mod.logger.addHandler(caplog.handler)
|
|
try:
|
|
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
|
await proxy.startup()
|
|
finally:
|
|
server_mod.logger.removeHandler(caplog.handler)
|
|
|
|
assert (
|
|
"Kompress: not installed (pip install headroom-ai[ml] for ML compression)"
|
|
in caplog.messages
|
|
)
|
|
finally:
|
|
await proxy.shutdown()
|
|
|
|
|
|
async def test_startup_keeps_missing_message_when_provider_override_enables_kompress(
|
|
caplog, monkeypatch
|
|
):
|
|
proxy = _make_proxy(
|
|
optimize=True,
|
|
disable_kompress=True,
|
|
disable_kompress_anthropic=False,
|
|
)
|
|
monkeypatch.setattr(proxy, "_eager_preload_transforms", lambda: ({}, []))
|
|
|
|
try:
|
|
server_mod.logger.addHandler(caplog.handler)
|
|
try:
|
|
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
|
await proxy.startup()
|
|
finally:
|
|
server_mod.logger.removeHandler(caplog.handler)
|
|
|
|
assert "Kompress: DISABLED" not in caplog.messages
|
|
assert (
|
|
"Kompress: not installed (pip install headroom-ai[ml] for ML compression)"
|
|
in caplog.messages
|
|
)
|
|
finally:
|
|
await proxy.shutdown()
|
|
|
|
|
|
async def test_startup_reports_disabled_when_both_providers_disable_kompress(caplog, monkeypatch):
|
|
proxy = _make_proxy(
|
|
optimize=True,
|
|
disable_kompress_anthropic=True,
|
|
disable_kompress_openai=True,
|
|
)
|
|
monkeypatch.setattr(proxy, "_eager_preload_transforms", lambda: ({}, []))
|
|
|
|
try:
|
|
server_mod.logger.addHandler(caplog.handler)
|
|
try:
|
|
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
|
await proxy.startup()
|
|
finally:
|
|
server_mod.logger.removeHandler(caplog.handler)
|
|
|
|
assert "Kompress: DISABLED" in caplog.messages
|
|
assert (
|
|
"Kompress: not installed (pip install headroom-ai[ml] for ML compression)"
|
|
not in caplog.messages
|
|
)
|
|
finally:
|
|
await proxy.shutdown()
|
|
|
|
|
|
async def test_startup_reports_disabled_when_compressor_selection_omits_kompress(
|
|
caplog, monkeypatch
|
|
):
|
|
proxy = _make_proxy(optimize=True, compressors={"smart_crusher"})
|
|
monkeypatch.setattr(proxy, "_eager_preload_transforms", lambda: ({}, []))
|
|
|
|
try:
|
|
server_mod.logger.addHandler(caplog.handler)
|
|
try:
|
|
with caplog.at_level(logging.INFO, logger=server_mod.logger.name):
|
|
await proxy.startup()
|
|
finally:
|
|
server_mod.logger.removeHandler(caplog.handler)
|
|
|
|
assert "Kompress: DISABLED" in caplog.messages
|
|
assert (
|
|
"Kompress: not installed (pip install headroom-ai[ml] for ML compression)"
|
|
not in caplog.messages
|
|
)
|
|
finally:
|
|
await proxy.shutdown()
|