mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(kompress): time each model-load stage and stop false not-ready warnings during warm-up (#4195)
## Description After every proxy start, the post-startup Kompress warm-up (`HeadroomProxy._start_kompress_background_warmup`) takes several seconds; 22.8 s was seen on one start. Requests that arrive during that time caused three problems: 1. **A false connectivity warning.** `ContentRouter._try_ml_compressor` logged `Kompress model not ready; requests will not be compressed. Check HuggingFace connectivity ...` at WARNING, although the model was only still loading. 2. **A second loader thread.** The warm-up thread was not in `kompress_compressor`'s per-model loader registry (`_download_threads`). `ensure_background_download` therefore started a second thread, which logged `downloading model ... in the background`, waited on `_kompress_lock` until the warm-up finished, and then returned the warm-up's cache entry. 3. **Misleading or missing load logs.** `_load_kompress_onnx` logged `Downloading Kompress ONNX model` before any cache lookup, even when every file came from disk. The `Kompress ONNX loaded` line gave no breakdown, so there was no way to tell why a load took 22.8 s. This PR has one commit for each problem, plus a fourth that fixes an issue Devin Review found in the second: - `feat(kompress)`: the `Kompress ONNX loaded` line now gives wall-clock and loading-thread CPU milliseconds for each load stage, and the start message says "Loading". - `fix(kompress)`: the warm-up thread is registered as the model's loader before it starts. - `fix(router)`: while a loader thread is running, the router logs at INFO, once per router, that the model is still loading. The WARNING remains for the case where nothing is loading the model. - `fix(kompress)`: a warm-up that replaces a live prefetch in the registry waits for it, so the prefetch stays covered. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/kompress_compressor.py`: - `_timed_stage` / `StageTimings`: record wall-clock and `time.thread_time()` milliseconds for each stage. - `_create_onnx_session(stage_ms=...)` times `artifact_resolve`, `onnxruntime_import`, `session_build` and `smoke_run`. When a candidate fails its smoke run and the next one is tried, both attempts are added to the totals. - `_load_kompress_onnx` also times `transformers_import` and `tokenizer_load`, and logs the stages on the `Kompress ONNX loaded` line. Its start message now says "Loading" instead of "Downloading". - `start_kompress_warmup_thread(model_id, warm)` (new): starts the `kompress-warmup` thread and registers it in `_download_threads` under `_download_threads_lock` before it runs. This covers the warm-up's initial delay too. If a prefetch or download thread is already registered, the warm-up takes its slot but waits for that thread to finish before running `warm()`. A live prefetch therefore can't drop out of the registry and be duplicated after a failed warm-up, and the two don't fetch the same files at once (Devin Review). After the warm-up ends, even if it failed, the next `ensure_background_download` replaces the entry as before. - `headroom/proxy/server.py`: `_start_kompress_background_warmup` starts its thread with `start_kompress_warmup_thread` instead of `threading.Thread(...)`. - `is_kompress_load_in_progress(model_id)` and `KompressCompressor.is_loading()` (new): true while a registered prefetch, download or warm-up thread is alive. - `headroom/transforms/content_router.py`: in the not-ready branch, the router logs at INFO once while `is_loading()` is true, and keeps the original WARNING once when no loader is running (for example, a failed download in its retry backoff; `_background_download` has already logged the actual error). Compressors without `is_loading` keep the WARNING. - Tests: - `tests/test_kompress_failsafe.py`: stage totals include a candidate rejected by its smoke run, and the load log names every stage. - `tests/test_proxy/test_kompress_background_warmup.py`: a request during the warm-up starts no second loader, and one after a failed warm-up still does. - `tests/test_kompress_download_backoff.py`: a warm-up that replaces a live prefetch waits for it. No download starts while the prefetch runs, and one can start once both have ended without a model. - `tests/test_content_router_kompress_not_ready_log.py` (new): INFO while loading, WARNING with no loader, INFO and then WARNING after a failed load, WARNING for a compressor without `is_loading`, and a registered warm-up thread counting as loading. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_kompress_not_ready_log.py tests/test_kompress_failsafe.py tests/test_proxy/test_kompress_background_warmup.py tests/test_kompress_download_backoff.py tests/test_kompress_request_nonblocking.py tests/test_kompress_preload_deferral.py tests/test_router_registry_smartcrusher.py tests/test_content_router_ml_stage_deadline.py tests/test_content_router_concurrency.py tests/test_content_router_exclude_tools.py tests/test_transforms/test_content_router.py -q ======================= 184 passed, 3 warnings in 21.25s ======================= $ ruff check headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py headroom/proxy/server.py tests/test_kompress_failsafe.py tests/test_proxy/test_kompress_background_warmup.py tests/test_content_router_kompress_not_ready_log.py tests/test_kompress_download_backoff.py All checks passed! $ ruff format --check (same 7 files) 7 files already formatted $ mypy headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py headroom/proxy/server.py Found 9 errors in 2 files (checked 3 source files) # All 9 are in headroom/relevance/embedding.py and headroom/providers/model_metadata.py, # which mypy reaches through imports. origin/main (f1bc2609) gives the same 9; # the three changed files report no errors. ``` Regression check: with each fix reverted, its new tests fail. - Without the `server.py` change, `test_request_during_warm_up_does_not_start_a_second_loader` fails with `assert ['test/warmup-model'] == []`. - Without the `content_router.py` change, `test_still_loading_is_info_not_a_connectivity_warning` and `test_warns_once_the_load_ends_without_a_model` fail. - Without the wait in `start_kompress_warmup_thread`, `test_warm_up_waits_for_a_live_loader_it_replaces` fails with `warm-up ran while the prefetch was still live`. Each commit was also tested on its own; at each one the only failure was `TestLazyImports::test_unloadable_native_dep_degrades_instead_of_raising`, which fails the same way on `origin/main` in the same run. ## Real Behavior Proof - Environment: macOS 27 on Apple M5, Python 3.11.x (uv tool install of headroom-ai with onnxruntime 1.31.0), real `chopratejas/kompress-v2-base` ONNX model already in the HF cache. `origin/main` (f1bc2609) and the PR head (4d881f47, rerun on 6b9393e9 with the same output) were run in separate worktrees with the same interpreter. - Exact command / steps: a script calls the real `HeadroomProxy._start_kompress_background_warmup` with a real `ContentRouter` (`HEADROOM_KOMPRESS_WARMUP=1`, `HEADROOM_KOMPRESS_WARMUP_DELAY_SECONDS=0`). After 0.5 s it sends one block through `router._try_ml_compressor` (a request arriving mid-warm-up), waits for the load, and prints the Kompress log lines plus any loader threads still alive. Run as `PYTHONPATH=<worktree> python warmup_proof.py` in each worktree. - Observed result: on `origin/main`, the request logs the connectivity WARNING and starts a second "background download" loader, and the load says "Downloading". On the PR head there is no WARNING and no second loader, the load says "Loading", and the loaded line gives each stage. Output below. - Not tested: a full `headroom proxy` restart with live client traffic, the CoreML and PyTorch backends (the stage timing covers the ONNX path only), and a cold HF cache. ```text ===== before (origin/mainf1bc2609) INFO headroom.transforms.kompress_compressor - Downloading Kompress ONNX model from chopratejas/kompress-v2-base ... INFO headroom.transforms.kompress_compressor - Kompress: downloading model chopratejas/kompress-v2-base in the background ... WARNING headroom.transforms.content_router - Kompress model not ready; requests will not be compressed. Check HuggingFace connectivity or pre-download: headroom-ai[ml] + first-run warmup. INFO headroom.transforms.kompress_compressor - Kompress ONNX loaded: chopratejas/kompress-v2-base backend=onnx INFO headroom.transforms.kompress_compressor - Kompress: background model download complete for chopratejas/kompress-v2-base INFO headroom.proxy - Kompress: warmed in the background in 5241 ms (backend onnx) ready=True loader threads still alive: ['kompress-canary'] ===== after (PR head 4d881f47) INFO headroom.transforms.kompress_compressor - Loading Kompress ONNX model from chopratejas/kompress-v2-base ... INFO headroom.transforms.content_router - Kompress model still loading in the background; requests pass through uncompressed until it is ready. INFO headroom.transforms.kompress_compressor - Kompress ONNX loaded: chopratejas/kompress-v2-base backend=onnx total_ms=3212/2212 stages_ms(wall/cpu): artifact_resolve=2/1 onnxruntime_import=0/0 session_build=424/344 smoke_run=27/27 transformers_import=2687/1770 tokenizer_load=71/68 INFO headroom.proxy - Kompress: warmed in the background in 3212 ms (backend onnx) ready=True loader threads still alive: ['kompress-canary'] ``` ## Runtime Rollout Safety - Rollout-managed feature(s): None. This changes log lines and loader-thread bookkeeping inside the existing Kompress warm-up; there is no new flag. - Minimum rollout channel: N/A (nothing rollout-managed). - Stable/default behavior changed: Log output only, plus no duplicate loader thread during warm-up. One WARNING becomes INFO while a load is in progress. When, how and what Kompress compresses is unchanged. - Kill switch / disable path: Existing `HEADROOM_KOMPRESS_WARMUP=0` still disables the warm-up; no new switch is needed. - Unsafe override required: No. - Qualification impact: None. Compression results and request handling are unchanged. - Rollback path: Revert the PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A (log output is shown above). ## Additional Notes - Documentation: no docs describe these log lines, so there is nothing to update. - Type checking: unchecked because `mypy headroom` was not run in full; on the changed files, the only errors are the 9 that `origin/main` also has (see Test Output). - What the timings showed: in a process doing nothing else, `from transformers import AutoTokenizer` (which also imports torch) takes most of the ~3 s load. On the proxy start that took 22.8 s, the load overlapped with request processing in the same process. The new wall/cpu figures should show whether that time went to contention or to work; if it was contention, importing `AutoTokenizer` earlier could help, subject to the #1908 constraint on native init at startup. --------- Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Kenneth Wong
JerrettDavis
parent
e01ad350dc
commit
89b229ce5f
@@ -2038,7 +2038,11 @@ class HeadroomProxy(
|
||||
backend,
|
||||
)
|
||||
|
||||
threading.Thread(target=_warm, name="kompress-warmup", daemon=True).start()
|
||||
from headroom.transforms.kompress_compressor import start_kompress_warmup_thread
|
||||
|
||||
# Registered as the model's loader, so a request during the warm-up
|
||||
# doesn't start a second load thread.
|
||||
start_kompress_warmup_thread(compressor.config.model_id, _warm)
|
||||
return True
|
||||
|
||||
async def startup(self):
|
||||
|
||||
@@ -2742,6 +2742,21 @@ class ContentRouter(Transform):
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _kompress_load_in_progress(compressor: Any) -> bool:
|
||||
"""Whether a background thread is still fetching or loading ``compressor``'s model.
|
||||
|
||||
Compressors without ``is_loading`` (the remote one is always ready,
|
||||
test doubles) count as not loading.
|
||||
"""
|
||||
is_loading = getattr(compressor, "is_loading", None)
|
||||
if is_loading is None:
|
||||
return False
|
||||
try:
|
||||
return bool(is_loading())
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _record_to_toin(
|
||||
self,
|
||||
strategy: CompressionStrategy,
|
||||
@@ -4725,10 +4740,22 @@ class ContentRouter(Transform):
|
||||
if compressor:
|
||||
if not compressor.is_ready():
|
||||
compressor.ensure_background_load()
|
||||
# Surface: warn once per ContentRouter instance so operators
|
||||
# know compression is degraded — model not cached, or
|
||||
# HuggingFace unreachable (corporate firewall, SSL, etc.).
|
||||
if not getattr(self, "_kompress_warned", False):
|
||||
# Surface once per ContentRouter instance that requests pass
|
||||
# through uncompressed. While a thread is still fetching or
|
||||
# loading the model (the post-startup warm-up, or the
|
||||
# download just started above) that is expected, so say it
|
||||
# at INFO. With no loader running, e.g. a failed download
|
||||
# in its retry backoff, compression is degraded: warn and
|
||||
# name the usual causes (_background_download has already
|
||||
# logged the actual error).
|
||||
if self._kompress_load_in_progress(compressor):
|
||||
if not getattr(self, "_kompress_loading_noted", False):
|
||||
logger.info(
|
||||
"Kompress model still loading in the background; "
|
||||
"requests pass through uncompressed until it is ready."
|
||||
)
|
||||
self._kompress_loading_noted = True
|
||||
elif not getattr(self, "_kompress_warned", False) and not compressor.is_ready():
|
||||
logger.warning(
|
||||
"Kompress model not ready; requests will not be "
|
||||
"compressed. Check HuggingFace connectivity or "
|
||||
|
||||
@@ -23,6 +23,7 @@ import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from typing import Any, Literal
|
||||
@@ -797,8 +798,46 @@ def _smoke_run(session: Any) -> None:
|
||||
)
|
||||
|
||||
|
||||
#: Per-stage load cost: stage name -> (wall-clock ms, loading-thread CPU ms).
|
||||
StageTimings = dict[str, tuple[float, float]]
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _timed_stage(stage_ms: StageTimings | None, stage: str) -> Iterator[None]:
|
||||
"""Add the block's wall-clock and thread-CPU milliseconds to ``stage_ms[stage]``.
|
||||
|
||||
Accumulates rather than overwrites, so a stage that runs once per ONNX
|
||||
candidate reports the cost of every attempt, failed ones included. Records
|
||||
on exceptions too. A no-op when ``stage_ms`` is ``None``.
|
||||
|
||||
CPU is this thread's only (``time.thread_time``): work on onnxruntime's own
|
||||
worker threads is not counted, and wall far above CPU means the thread was
|
||||
waiting (disk, the GIL, a lock) rather than working.
|
||||
"""
|
||||
if stage_ms is None:
|
||||
yield
|
||||
return
|
||||
wall_started = time.perf_counter()
|
||||
cpu_started = time.thread_time()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
wall_ms = (time.perf_counter() - wall_started) * 1000.0
|
||||
cpu_ms = (time.thread_time() - cpu_started) * 1000.0
|
||||
prev_wall, prev_cpu = stage_ms.get(stage, (0.0, 0.0))
|
||||
stage_ms[stage] = (prev_wall + wall_ms, prev_cpu + cpu_ms)
|
||||
|
||||
|
||||
def _format_stage_ms(stage_ms: StageTimings) -> str:
|
||||
return " ".join(f"{stage}={wall:.0f}/{cpu:.0f}" for stage, (wall, cpu) in stage_ms.items())
|
||||
|
||||
|
||||
def _create_onnx_session(
|
||||
model_id: str, providers: list[Any], *, allow_download: bool = True
|
||||
model_id: str,
|
||||
providers: list[Any],
|
||||
*,
|
||||
allow_download: bool = True,
|
||||
stage_ms: StageTimings | None = None,
|
||||
) -> Any:
|
||||
"""Resolve and load the model's ONNX artifact, trying candidates in order.
|
||||
|
||||
@@ -812,6 +851,10 @@ def _create_onnx_session(
|
||||
cache only; if none is cached, :class:`KompressModelNotCached` is raised
|
||||
instead of hitting the network. ``onnxruntime`` is imported only after a
|
||||
candidate resolves, so a cache-only miss never requires it.
|
||||
|
||||
``stage_ms``, when given, accumulates the cost of ``artifact_resolve``,
|
||||
``onnxruntime_import``, ``session_build`` and ``smoke_run`` across every
|
||||
candidate tried. See :func:`_timed_stage`.
|
||||
"""
|
||||
last_err: Exception | None = None
|
||||
cache_miss = False
|
||||
@@ -819,9 +862,10 @@ def _create_onnx_session(
|
||||
ort: Any = None
|
||||
for filename in _onnx_filename_candidates():
|
||||
try:
|
||||
onnx_path = _hf_artifact(
|
||||
model_id, filename, allow_network=allow_download and refused is None
|
||||
)
|
||||
with _timed_stage(stage_ms, "artifact_resolve"):
|
||||
onnx_path = _hf_artifact(
|
||||
model_id, filename, allow_network=allow_download and refused is None
|
||||
)
|
||||
except KompressModelNotCached as exc:
|
||||
# Only _hf_artifact's air-gap translation raises this here. It says
|
||||
# THIS candidate is not cached; a later one may be. Keep looking,
|
||||
@@ -834,16 +878,19 @@ def _create_onnx_session(
|
||||
logger.debug("ONNX artifact %r unavailable for %s: %s", filename, model_id, exc)
|
||||
continue
|
||||
if ort is None:
|
||||
import onnxruntime
|
||||
with _timed_stage(stage_ms, "onnxruntime_import"):
|
||||
import onnxruntime
|
||||
|
||||
ort = onnxruntime
|
||||
try:
|
||||
session = ort.InferenceSession(
|
||||
onnx_path,
|
||||
_onnx_session_options(ort),
|
||||
providers=providers,
|
||||
)
|
||||
_smoke_run(session)
|
||||
with _timed_stage(stage_ms, "session_build"):
|
||||
session = ort.InferenceSession(
|
||||
onnx_path,
|
||||
_onnx_session_options(ort),
|
||||
providers=providers,
|
||||
)
|
||||
with _timed_stage(stage_ms, "smoke_run"):
|
||||
_smoke_run(session)
|
||||
return session
|
||||
except Exception as exc:
|
||||
last_err = exc
|
||||
@@ -878,7 +925,12 @@ def _load_kompress_onnx(
|
||||
if model_id in _kompress_cache:
|
||||
return _kompress_cache[model_id]
|
||||
|
||||
logger.info("Downloading Kompress ONNX model from %s ...", model_id)
|
||||
# Logged before any cache lookup: the artifacts usually come from disk
|
||||
# (startup prefetch), and _hf_artifact goes to the Hub only on a miss.
|
||||
logger.info("Loading Kompress ONNX model from %s ...", model_id)
|
||||
load_started = time.perf_counter()
|
||||
load_cpu_started = time.thread_time()
|
||||
stage_ms: StageTimings = {}
|
||||
|
||||
backend = "onnx_coreml" if use_coreml else "onnx"
|
||||
providers: list[Any]
|
||||
@@ -907,15 +959,26 @@ def _load_kompress_onnx(
|
||||
else:
|
||||
providers = ["CPUExecutionProvider"]
|
||||
|
||||
session = _create_onnx_session(model_id, providers, allow_download=allow_download)
|
||||
session = _create_onnx_session(
|
||||
model_id, providers, allow_download=allow_download, stage_ms=stage_ms
|
||||
)
|
||||
model = _OnnxModel(session)
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
with _timed_stage(stage_ms, "transformers_import"):
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = _load_modernbert_tokenizer(AutoTokenizer, allow_download=allow_download)
|
||||
with _timed_stage(stage_ms, "tokenizer_load"):
|
||||
tokenizer = _load_modernbert_tokenizer(AutoTokenizer, allow_download=allow_download)
|
||||
|
||||
_kompress_cache[model_id] = (model, tokenizer, backend)
|
||||
logger.info("Kompress ONNX loaded: %s backend=%s", model_id, backend)
|
||||
logger.info(
|
||||
"Kompress ONNX loaded: %s backend=%s total_ms=%.0f/%.0f stages_ms(wall/cpu): %s",
|
||||
model_id,
|
||||
backend,
|
||||
(time.perf_counter() - load_started) * 1000.0,
|
||||
(time.thread_time() - load_cpu_started) * 1000.0,
|
||||
_format_stage_ms(stage_ms),
|
||||
)
|
||||
return model, tokenizer, backend
|
||||
|
||||
|
||||
@@ -1341,6 +1404,48 @@ def ensure_background_download(model_id: str = HF_MODEL_ID, device: str = "auto"
|
||||
thread.start()
|
||||
|
||||
|
||||
def is_kompress_load_in_progress(model_id: str = HF_MODEL_ID) -> bool:
|
||||
"""True while a registered thread (prefetch, download or warm-up) is working on ``model_id``."""
|
||||
with _download_threads_lock:
|
||||
thread = _download_threads.get(model_id)
|
||||
return thread is not None and thread.is_alive()
|
||||
|
||||
|
||||
def start_kompress_warmup_thread(model_id: str, warm: Callable[[], None]) -> threading.Thread:
|
||||
"""Run ``warm`` on a daemon ``kompress-warmup`` thread registered as ``model_id``'s loader.
|
||||
|
||||
The proxy's post-startup warm-up loads the model on its own thread. Unless
|
||||
that thread is in the per-model registry, a request arriving mid-warm-up
|
||||
finds the model not ready, sees no live loader, and has
|
||||
:func:`ensure_background_download` start a second thread that only queues
|
||||
on ``_kompress_lock`` and then returns the warm-up's cache entry. Registered
|
||||
before it starts, so the warm-up's initial delay is covered too.
|
||||
|
||||
If a prefetch or download thread is already registered, the warm-up takes
|
||||
its place in the registry and waits for it to finish before running
|
||||
``warm``. Without the wait, a warm-up that failed while a startup prefetch
|
||||
was still downloading would leave the registry pointing at a dead thread,
|
||||
and the next request would start another download next to the hidden
|
||||
prefetch. Waiting also keeps the warm-up from fetching the same files at
|
||||
the same time. Once the warm-up finishes, a failed one included, the next
|
||||
:func:`ensure_background_download` call replaces the dead entry as usual.
|
||||
"""
|
||||
with _download_threads_lock:
|
||||
previous = _download_threads.get(model_id)
|
||||
|
||||
def _run() -> None:
|
||||
# Registry entries are always started threads, so join() is safe;
|
||||
# it returns at once for one that has already finished.
|
||||
if previous is not None:
|
||||
previous.join()
|
||||
warm()
|
||||
|
||||
thread = threading.Thread(target=_run, name="kompress-warmup", daemon=True)
|
||||
_download_threads[model_id] = thread
|
||||
thread.start()
|
||||
return thread
|
||||
|
||||
|
||||
def prefetch_kompress_artifacts(model_id: str = HF_MODEL_ID) -> bool:
|
||||
"""Download the model's ONNX artifact to the local cache. No native init.
|
||||
|
||||
@@ -1751,6 +1856,10 @@ class KompressCompressor(Transform):
|
||||
"""
|
||||
ensure_background_download(self.config.model_id, self.config.device)
|
||||
|
||||
def is_loading(self) -> bool:
|
||||
"""True while a background thread is fetching or loading this model."""
|
||||
return is_kompress_load_in_progress(self.config.model_id)
|
||||
|
||||
def compress(
|
||||
self,
|
||||
content: str,
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""A Kompress model that is still loading is expected, not a connectivity problem.
|
||||
|
||||
The router used to log "Kompress model not ready ... Check HuggingFace
|
||||
connectivity" at WARNING on the first request after every proxy start, because
|
||||
the post-startup warm-up takes ~20 s and requests arrive meanwhile. It now says
|
||||
"still loading" at INFO while a loader thread runs, and keeps the WARNING for
|
||||
when nothing is loading the model (e.g. a failed download in its retry backoff).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.transforms import kompress_compressor as kc
|
||||
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
|
||||
|
||||
_LOGGER = "headroom.transforms.content_router"
|
||||
_CONTENT = "some plain text that the ML model would compress. " * 4
|
||||
|
||||
|
||||
class _NotReadyKompress:
|
||||
def __init__(self, *, loading: bool) -> None:
|
||||
self.loading = loading
|
||||
self.background_loads = 0
|
||||
|
||||
def is_ready(self) -> bool:
|
||||
return False
|
||||
|
||||
def ensure_background_load(self) -> None:
|
||||
self.background_loads += 1
|
||||
|
||||
def is_loading(self) -> bool:
|
||||
return self.loading
|
||||
|
||||
|
||||
def _router(monkeypatch: pytest.MonkeyPatch, compressor: object) -> ContentRouter:
|
||||
router = ContentRouter(
|
||||
ContentRouterConfig(
|
||||
relevance_split=False,
|
||||
lossless_then_lossy=False,
|
||||
ccr_inject_marker=False,
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr(router, "_get_kompress", lambda: compressor)
|
||||
return router
|
||||
|
||||
|
||||
def _logged(caplog: pytest.LogCaptureFixture, level: int, text: str) -> int:
|
||||
return sum(
|
||||
1
|
||||
for r in caplog.records
|
||||
if r.name == _LOGGER and r.levelno == level and text in r.getMessage()
|
||||
)
|
||||
|
||||
|
||||
def test_still_loading_is_info_not_a_connectivity_warning(monkeypatch, caplog):
|
||||
compressor = _NotReadyKompress(loading=True)
|
||||
router = _router(monkeypatch, compressor)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=_LOGGER):
|
||||
for _ in range(3):
|
||||
assert router._try_ml_compressor(_CONTENT, "ctx")[0] == _CONTENT
|
||||
|
||||
assert _logged(caplog, logging.INFO, "still loading in the background") == 1
|
||||
assert _logged(caplog, logging.WARNING, "Check HuggingFace connectivity") == 0
|
||||
assert compressor.background_loads == 3
|
||||
|
||||
|
||||
def test_no_loader_running_still_warns_once(monkeypatch, caplog):
|
||||
router = _router(monkeypatch, _NotReadyKompress(loading=False))
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=_LOGGER):
|
||||
for _ in range(3):
|
||||
router._try_ml_compressor(_CONTENT, "ctx")
|
||||
|
||||
assert _logged(caplog, logging.WARNING, "Check HuggingFace connectivity") == 1
|
||||
assert _logged(caplog, logging.INFO, "still loading in the background") == 0
|
||||
|
||||
|
||||
def test_warns_once_the_load_ends_without_a_model(monkeypatch, caplog):
|
||||
"""A warm-up that fails leaves no loader running: then it is degraded."""
|
||||
compressor = _NotReadyKompress(loading=True)
|
||||
router = _router(monkeypatch, compressor)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=_LOGGER):
|
||||
router._try_ml_compressor(_CONTENT, "ctx")
|
||||
compressor.loading = False
|
||||
router._try_ml_compressor(_CONTENT, "ctx")
|
||||
|
||||
assert _logged(caplog, logging.INFO, "still loading in the background") == 1
|
||||
assert _logged(caplog, logging.WARNING, "Check HuggingFace connectivity") == 1
|
||||
|
||||
|
||||
def test_compressor_without_is_loading_keeps_the_warning(monkeypatch, caplog):
|
||||
class _Legacy:
|
||||
def is_ready(self) -> bool:
|
||||
return False
|
||||
|
||||
def ensure_background_load(self) -> None:
|
||||
pass
|
||||
|
||||
router = _router(monkeypatch, _Legacy())
|
||||
|
||||
with caplog.at_level(logging.INFO, logger=_LOGGER):
|
||||
router._try_ml_compressor(_CONTENT, "ctx")
|
||||
|
||||
assert _logged(caplog, logging.WARNING, "Check HuggingFace connectivity") == 1
|
||||
|
||||
|
||||
def test_registered_warm_up_thread_counts_as_loading():
|
||||
"""The real signal: the warm-up thread is the model's registered loader."""
|
||||
model_id = "test/not-ready-log-model"
|
||||
release = threading.Event()
|
||||
try:
|
||||
thread = kc.start_kompress_warmup_thread(model_id, lambda: release.wait(5))
|
||||
assert kc.is_kompress_load_in_progress(model_id)
|
||||
release.set()
|
||||
thread.join(timeout=5)
|
||||
assert not kc.is_kompress_load_in_progress(model_id)
|
||||
finally:
|
||||
release.set()
|
||||
with kc._download_threads_lock:
|
||||
kc._download_threads.pop(model_id, None)
|
||||
@@ -154,3 +154,30 @@ def test_a_live_thread_is_never_duplicated(monkeypatch):
|
||||
finally:
|
||||
gate.set()
|
||||
_drain()
|
||||
|
||||
|
||||
def test_warm_up_waits_for_a_live_loader_it_replaces(monkeypatch):
|
||||
"""A startup prefetch still downloading stays covered, even if the warm-up fails."""
|
||||
started = _spawned(monkeypatch, fails=False)
|
||||
prefetch_done = threading.Event()
|
||||
prefetch = threading.Thread(target=prefetch_done.wait, args=(10,), daemon=True)
|
||||
with kc._download_threads_lock:
|
||||
prefetch.start()
|
||||
kc._download_threads["some/model"] = prefetch
|
||||
|
||||
# A warm-up that returns without caching the model stands in for a failed one.
|
||||
warmed = threading.Event()
|
||||
warm_up = kc.start_kompress_warmup_thread("some/model", warmed.set)
|
||||
try:
|
||||
kc.ensure_background_download("some/model")
|
||||
assert not warmed.wait(0.2), "warm-up ran while the prefetch was still live"
|
||||
assert started == [], "a request started a download beside the live prefetch"
|
||||
finally:
|
||||
prefetch_done.set()
|
||||
warm_up.join(timeout=10)
|
||||
assert warmed.is_set()
|
||||
|
||||
# Both are done and the model is still not cached: a request may load it now.
|
||||
kc.ensure_background_download("some/model")
|
||||
_drain()
|
||||
assert started == ["some/model"]
|
||||
|
||||
@@ -609,6 +609,71 @@ def test_all_artifacts_failing_at_run_raises_rather_than_returning_a_dead_sessio
|
||||
kc._create_onnx_session("org/model", ["CPUExecutionProvider"])
|
||||
|
||||
|
||||
# ── Load-stage timings: say where a slow warm-up spent its time ────────────────
|
||||
|
||||
|
||||
def _install_ticking_clock(monkeypatch):
|
||||
"""Each clock reading is 1s after the last, so every timed stage costs 1000ms."""
|
||||
|
||||
def ticking_clock():
|
||||
now = [0.0]
|
||||
|
||||
def tick() -> float:
|
||||
now[0] += 1.0
|
||||
return now[0]
|
||||
|
||||
return staticmethod(tick)
|
||||
|
||||
clock = type("_Clock", (), {"perf_counter": ticking_clock(), "thread_time": ticking_clock()})
|
||||
monkeypatch.setattr(kc, "time", clock)
|
||||
|
||||
|
||||
def test_stage_timings_count_every_candidate_attempt(monkeypatch):
|
||||
"""A candidate rejected at smoke-run still costs time, so it shows in the totals."""
|
||||
_install_fake_ort(monkeypatch, run_fails_for={"int8-wo"})
|
||||
_install_ticking_clock(monkeypatch)
|
||||
stage_ms: kc.StageTimings = {}
|
||||
|
||||
kc._create_onnx_session("org/model", ["CPUExecutionProvider"], stage_ms=stage_ms)
|
||||
|
||||
# int8-wo resolved, built and smoke-run (rejected); fp32 the same (accepted).
|
||||
# onnxruntime is imported once, on the first resolved candidate.
|
||||
assert stage_ms == {
|
||||
"artifact_resolve": (2000.0, 2000.0),
|
||||
"onnxruntime_import": (1000.0, 1000.0),
|
||||
"session_build": (2000.0, 2000.0),
|
||||
"smoke_run": (2000.0, 2000.0),
|
||||
}
|
||||
|
||||
|
||||
def test_onnx_load_logs_every_stage(monkeypatch, caplog):
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
|
||||
_install_fake_ort(monkeypatch, run_fails_for=set())
|
||||
monkeypatch.setitem(sys.modules, "transformers", SimpleNamespace(AutoTokenizer=object()))
|
||||
monkeypatch.setattr(kc, "_kompress_cache", {})
|
||||
monkeypatch.setattr(
|
||||
kc, "_load_modernbert_tokenizer", lambda _auto, *, allow_download: "tokenizer"
|
||||
)
|
||||
|
||||
with caplog.at_level("INFO", logger=kc.logger.name):
|
||||
kc._load_kompress_onnx("org/model")
|
||||
|
||||
loaded = [r.getMessage() for r in caplog.records if "Kompress ONNX loaded" in r.getMessage()]
|
||||
assert len(loaded) == 1
|
||||
for stage in (
|
||||
"artifact_resolve=",
|
||||
"onnxruntime_import=",
|
||||
"session_build=",
|
||||
"smoke_run=",
|
||||
"transformers_import=",
|
||||
"tokenizer_load=",
|
||||
):
|
||||
assert stage in loaded[0]
|
||||
assert "Downloading" not in caplog.text
|
||||
|
||||
|
||||
# ── Failure latch: a broken model stops costing us every request ───────────────
|
||||
|
||||
|
||||
|
||||
@@ -14,10 +14,14 @@ from types import SimpleNamespace
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.server import HeadroomProxy
|
||||
from headroom.transforms import kompress_compressor as kc
|
||||
|
||||
_MODEL_ID = "test/warmup-model"
|
||||
|
||||
|
||||
class _Kompress:
|
||||
def __init__(self) -> None:
|
||||
self.config = SimpleNamespace(model_id=_MODEL_ID)
|
||||
self.preloaded = threading.Event()
|
||||
|
||||
def preload(self, *, allow_download: bool = True) -> str:
|
||||
@@ -37,6 +41,16 @@ def _no_delay(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("HEADROOM_KOMPRESS_WARMUP", raising=False)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_registry():
|
||||
yield
|
||||
with kc._download_threads_lock:
|
||||
thread = kc._download_threads.pop(_MODEL_ID, None)
|
||||
kc._download_failures.pop(_MODEL_ID, None)
|
||||
if thread is not None:
|
||||
thread.join(timeout=5)
|
||||
|
||||
|
||||
def test_warms_on_a_background_thread(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))
|
||||
monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False)
|
||||
@@ -74,3 +88,41 @@ def test_test_processes_do_not_warm_unless_forced(monkeypatch: pytest.MonkeyPatc
|
||||
monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))
|
||||
monkeypatch.setenv("PYTEST_CURRENT_TEST", "x")
|
||||
assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(_Kompress())) is False
|
||||
|
||||
|
||||
def test_request_during_warm_up_does_not_start_a_second_loader(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The warm-up is the model's registered loader until it finishes."""
|
||||
monkeypatch.setattr("platform.libc_ver", lambda: ("glibc", "2.35"))
|
||||
monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False)
|
||||
monkeypatch.setattr(kc, "_kompress_cache", {})
|
||||
loads: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
kc,
|
||||
"_load_kompress",
|
||||
lambda model_id, device, allow_download=True: (
|
||||
loads.append(model_id) or (None, None, "onnx")
|
||||
),
|
||||
)
|
||||
release = threading.Event()
|
||||
|
||||
class _SlowKompress(_Kompress):
|
||||
def preload(self, *, allow_download: bool = True) -> str:
|
||||
self.preloaded.set()
|
||||
release.wait(5)
|
||||
raise OSError("warm-up failed")
|
||||
|
||||
kompress = _SlowKompress()
|
||||
assert HeadroomProxy._start_kompress_background_warmup(_proxy_with(kompress)) is True
|
||||
assert kompress.preloaded.wait(5)
|
||||
|
||||
kc.ensure_background_download(_MODEL_ID)
|
||||
assert loads == []
|
||||
|
||||
# Once the warm-up is over (here, failed), a request still starts its own load.
|
||||
release.set()
|
||||
kc._download_threads[_MODEL_ID].join(timeout=5)
|
||||
kc.ensure_background_download(_MODEL_ID)
|
||||
kc._download_threads[_MODEL_ID].join(timeout=5)
|
||||
assert loads == [_MODEL_ID]
|
||||
|
||||
Reference in New Issue
Block a user