mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description The supply-chain pin on the default relevance embedder never took effect. `EmbeddingScorer` passed `revision="52398278…"` to fastembed's `TextEmbedding`, which accepts it and then drops it before `snapshot_download` (fastembed 0.8.0 and 0.9.0: `OnnxTextEmbedding.__init__` calls `download_model(...)` without `**kwargs`). So the model floated on `Qdrant/bge-small-en-v1.5-onnx-Q@main`, cached under the system temp dir. Today the floating `main` (`aa8f8b0`) differs from the pin only in `README.md`, so no unreviewed weights have shipped. The pin just wasn't protecting anything. This PR makes the Python default model load the pinned snapshot and adds tests that would have caught the dropped revision. The Rust `EmbeddingScorer::try_new` change that was in this PR is now its own PR, stacked on this one: #4190. No linked issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/onnx_runtime.py`: pin `Qdrant/bge-small-en-v1.5-onnx-Q` at `52398278842ec682c6f32300af41344b1c0b0bb2` in `_PINNED_REVISIONS`, so `HEADROOM_HF_PIN=off` applies to it like every other pinned repo. - `headroom/relevance/embedding.py`: - The default model resolves its 5 files through `hf_hub_download_local_first`, which is pinned, cache-first and guarded on a miss. - fastembed then gets only the snapshot directory as `specific_model_path`. That construction runs with the Hub forced offline, so a future fastembed that ignored the path would fail instead of downloading an unpinned copy. - Other catalog models keep the existing `_load_text_embedding` path. - The air-gapped error for the default model names the exact snapshot and the command that seeds it, and says a fastembed cache is no longer read (details under Release note). - The private `_DEFAULT_MODEL_PINNED_REVISION` / `_pinned_revision` are removed. - `_load_text_embedding` now types its kwargs `dict[str, Any]`, fixing 8 mypy errors that only appear with fastembed installed. - `pyproject.toml` / `uv.lock`: `fastembed>=0.6.0`. `specific_model_path` first appears in fastembed v0.6.0; it isn't in v0.4.x or v0.5.x. The lock still resolves 0.8.0; only the `requires-dist` specifier changes. - `.github/workflows/ci.yml` (`test-extras`): - The prefetch warms the pinned snapshot through the loader's own resolver, and the cache entry's key carries the SHA. The old prefetch warmed fastembed's own cache, which the offline relevance tests would now miss: `conftest.py:199` turns that into silent skips. - The job runs the new test file, and the unused `FASTEMBED_CACHE_PATH` is gone. - Tests: - New `tests/test_fastembed_pinned_snapshot.py`: - the pinned revision reaches every file download; - fastembed gets only the snapshot directory, built with the Hub forced offline; - `HEADROOM_HF_PIN=off` floats; - files resolved from two different snapshots are refused; - a cold cache while offline degrades without a download, and the error names `repo@sha` and `--revision <sha>`; - real fastembed skips the Hub when given `specific_model_path`. - `tests/test_stateless_writers.py`: the three `_pinned_revision` tests are removed with the helper; `tests/test_hf_revision_pinning.py` covers `_resolve_revision`. - `tests/test_offline_egress_chokepoint.py`: `relevance/embedding.py`'s cache-only count goes from 1 to 2 for the `_load_default_model` construction. ## Release note > **relevance:** the default embedder (`BAAI/bge-small-en-v1.5`) now loads from the pinned snapshot `Qdrant/bge-small-en-v1.5-onnx-Q@52398278842ec682c6f32300af41344b1c0b0bb2` in the HuggingFace hub cache (`HF_HUB_CACHE` / `HF_HOME/hub` / `~/.cache/huggingface/hub`), not from fastembed's cache. Existing installs download ~67 MB once. Air-gapped hosts that pre-seeded the fastembed cache must re-seed, e.g. `hf download Qdrant/bge-small-en-v1.5-onnx-Q model_optimized.onnx tokenizer.json tokenizer_config.json special_tokens_map.json config.json --revision 52398278842ec682c6f32300af41344b1c0b0bb2`. Without that, relevance scoring falls back to BM25, and the error message says how to re-seed. `CHANGELOG.md` is generated from the PR title, so this text is here for whoever writes the release notes. ## Dependency Change (`fastembed>=0.4.0` → `>=0.6.0`) This raises the minimum version of an existing dependency; no package is added. `uv.lock` still resolves `fastembed 0.8.0`. The only lock change is the `requires-dist` specifier, so the locked install set is byte-for-byte the same. - **Why this package:** it's already the `relevance` extra's embedder (since Stage 3c.1). This PR adds no package and swaps none. - **Who maintains it:** Qdrant (`qdrant/fastembed`), Apache-2.0, actively released (0.6.0 on 2025-02-26 through 0.9.0 on 2026-10-07). The GitHub Advisory Database has no advisories for `fastembed` (pip). - **Install surface:** unchanged for the locked install (0.8.0). Installs that resolve their own versions lose access to 0.4.x/0.5.x. Runtime network shrinks: given `specific_model_path`, fastembed makes no Hub call, and the only download is Headroom's own pinned, offline-guarded `hf_hub_download_local_first`. - **Why this version:** `specific_model_path` on `TextEmbedding` first appears in v0.6.0, and the pinned-snapshot load depends on it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HOME=$(mktemp -d) pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py \ tests/test_relevance.py tests/test_relevance_extra.py tests/test_stateless_writers.py tests/test_hf_revision_pinning.py 7 failed, 209 passed, 3 skipped # The 7 failures are TestOtlpExporterOffline / TestLangfuseExporterOffline / TestStartupRefusal / # TestHeadroomCloudCompressionOffline: the OTel and Cloud extras aren't in this venv. The same 7 fail on # origin/main (f1bc2609) here. $ mypy headroom # with fastembed 0.8.0 installed Success: no issues found in 569 source files $ ruff check . # All checks passed! $ ruff format --check . # 1790 files already formatted ``` ## Real Behavior Proof - Exact command / steps: additional maintainer verification on Windows/Python 3.12.13: python -m pytest tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_stateless_writers.py -q. - Observed result: the additional maintainer run passed all 160 selected tests onb3eb7f349c, using the available development native extension for startup; no fresh Rust implementation qualification is claimed. - Environment: macOS arm64, repo `.venv` (fastembed 0.8.0 = `uv.lock`), rebased on `origin/main` `f1bc2609`. - Default load: `EmbeddingScorer().score("authentication failed for user admin", "login error")` loads from `~/.cache/huggingface/hub/models--Qdrant--bge-small-en-v1.5-onnx-Q/snapshots/52398278842ec682c6f32300af41344b1c0b0bb2` and scores 0.7389. `origin/main` loads `<tmp>/fastembed_cache/…/snapshots/aa8f8b06…` (the floating `main`, under `$TMPDIR`). - Air-gapped re-seed, from an empty `HF_HOME`: 1. `HEADROOM_OFFLINE=1` gives `Embedding model 'BAAI/bge-small-en-v1.5' is unavailable: HEADROOM_OFFLINE is set: … Pre-seed the HuggingFace hub cache with Qdrant/bge-small-en-v1.5-onnx-Q@52398278… (`hf download … --revision 52398278…`), or use the BM25-only scorer. A fastembed cache seeded for an older Headroom is not read for this model.` 2. I ran the `hf download` command from that message as printed. 3. `HEADROOM_OFFLINE=1 HF_HUB_OFFLINE=1` then loads and scores 0.7389. 4. With `HEADROOM_HF_PIN=off`, the message names `@main` and `--revision main` instead. - Not tested: Windows, and Linux outside CI. fastembed 0.6.x/0.7.x at runtime (the 0.6.0 floor comes from reading the v0.6.0 tag's source). ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. This is the default code path, with no flag. - Stable/default behavior changed: yes. The default relevance model loads the pinned snapshot from the standard HF hub cache, not fastembed's temp-dir cache, so existing installs download ~67 MB once. The model files are byte-identical to what `main` served; only `README.md` differs. - Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`. Air-gapped hosts re-seed as described in the Release note. - Unsafe override required: no. - Qualification impact: none expected. Same weights and tokenizer as the current floating `main`. - Rollback path: revert this PR. Nothing persistent changes beyond an extra snapshot in the HF cache. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes - `_load_text_embedding` was typed `kwargs: dict[str, str]`, which made mypy report 8 `arg-type` errors on `TextEmbedding(**kwargs)` whenever fastembed is installed in the type-checking environment. CI's mypy run doesn't install fastembed, so it never saw them. - Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches are still unpinned (there's no Rust `HEADROOM_HF_PIN`). Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
98 lines
3.1 KiB
Python
98 lines
3.1 KiB
Python
"""Tests for the complete stateless write guarantee.
|
|
|
|
Covers the process-wide stateless flag and the opt-in serving writers gated by
|
|
it: the output-savings recorder and persistent memory. (Savings tracker and
|
|
TOIN are covered in their own test modules.)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from headroom import paths
|
|
from headroom.proxy.output_savings import SavingsRecorder
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_stateless_globals():
|
|
"""The stateless flag and TOIN singleton are process-global — never leak."""
|
|
yield
|
|
paths.set_process_stateless(False)
|
|
try:
|
|
from headroom.telemetry.toin import reset_toin
|
|
|
|
reset_toin()
|
|
except Exception:
|
|
pass
|
|
|
|
|
|
# ---- process-wide stateless flag ------------------------------------------
|
|
|
|
|
|
def test_process_stateless_flag_set_and_clear(monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
|
paths.set_process_stateless(False)
|
|
assert paths.process_is_stateless() is False
|
|
paths.set_process_stateless(True)
|
|
assert paths.process_is_stateless() is True
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"value,expected", [("1", True), ("true", True), ("on", True), ("off", False), ("", False)]
|
|
)
|
|
def test_process_stateless_env(monkeypatch, value, expected):
|
|
paths.set_process_stateless(False)
|
|
monkeypatch.setenv("HEADROOM_STATELESS", value)
|
|
assert paths.process_is_stateless() is expected
|
|
|
|
|
|
def test_install_id_stays_in_memory_when_stateless(tmp_path, monkeypatch):
|
|
from headroom.telemetry import session
|
|
|
|
workspace = tmp_path / "state"
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(workspace))
|
|
monkeypatch.setattr(session, "_install_id", None)
|
|
paths.set_process_stateless(True)
|
|
|
|
assert len(session.install_id()) == 32
|
|
assert not workspace.exists()
|
|
|
|
|
|
# ---- output-savings recorder ----------------------------------------------
|
|
|
|
|
|
def test_output_savings_flush_writes_nothing_when_stateless(tmp_path, monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
|
path = tmp_path / "output_savings.json"
|
|
paths.set_process_stateless(True)
|
|
rec = SavingsRecorder(path)
|
|
rec.flush()
|
|
assert not path.exists()
|
|
|
|
|
|
def test_output_savings_flush_persists_when_not_stateless(tmp_path, monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
|
path = tmp_path / "output_savings.json"
|
|
paths.set_process_stateless(False)
|
|
rec = SavingsRecorder(path)
|
|
rec.flush()
|
|
assert path.exists()
|
|
|
|
|
|
# ---- persistent memory ----------------------------------------------------
|
|
|
|
|
|
def test_memory_disabled_under_stateless(tmp_path, monkeypatch):
|
|
"""A stateless proxy with --memory must not initialize memory or write a DB."""
|
|
workspace = tmp_path / "state"
|
|
monkeypatch.chdir(tmp_path)
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(workspace))
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
app = create_app(ProxyConfig(memory_enabled=True, stateless=True))
|
|
proxy = app.state.proxy
|
|
assert proxy.memory_handler is None
|
|
assert not workspace.exists()
|