fix(relevance): load the default embedder from one pinned snapshot (#4152)

## Description

The supply-chain pin on the default relevance embedder never took
effect. `EmbeddingScorer` passed `revision="52398278…"` to fastembed's
`TextEmbedding`, which accepts it and then drops it before
`snapshot_download` (fastembed 0.8.0 and 0.9.0:
`OnnxTextEmbedding.__init__` calls `download_model(...)` without
`**kwargs`). So the model floated on
`Qdrant/bge-small-en-v1.5-onnx-Q@main`, cached under the system temp
dir.

Today the floating `main` (`aa8f8b0`) differs from the pin only in
`README.md`, so no unreviewed weights have shipped. The pin just wasn't
protecting anything.

This PR makes the Python default model load the pinned snapshot and adds
tests that would have caught the dropped revision. The Rust
`EmbeddingScorer::try_new` change that was in this PR is now its own PR,
stacked on this one: #4190.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/onnx_runtime.py`: pin `Qdrant/bge-small-en-v1.5-onnx-Q` at
`52398278842ec682c6f32300af41344b1c0b0bb2` in `_PINNED_REVISIONS`, so
`HEADROOM_HF_PIN=off` applies to it like every other pinned repo.
- `headroom/relevance/embedding.py`:
- The default model resolves its 5 files through
`hf_hub_download_local_first`, which is pinned, cache-first and guarded
on a miss.
- fastembed then gets only the snapshot directory as
`specific_model_path`. That construction runs with the Hub forced
offline, so a future fastembed that ignored the path would fail instead
of downloading an unpinned copy.
  - Other catalog models keep the existing `_load_text_embedding` path.
- The air-gapped error for the default model names the exact snapshot
and the command that seeds it, and says a fastembed cache is no longer
read (details under Release note).
- The private `_DEFAULT_MODEL_PINNED_REVISION` / `_pinned_revision` are
removed.
- `_load_text_embedding` now types its kwargs `dict[str, Any]`, fixing 8
mypy errors that only appear with fastembed installed.
- `pyproject.toml` / `uv.lock`: `fastembed>=0.6.0`.
`specific_model_path` first appears in fastembed v0.6.0; it isn't in
v0.4.x or v0.5.x. The lock still resolves 0.8.0; only the
`requires-dist` specifier changes.
- `.github/workflows/ci.yml` (`test-extras`):
- The prefetch warms the pinned snapshot through the loader's own
resolver, and the cache entry's key carries the SHA. The old prefetch
warmed fastembed's own cache, which the offline relevance tests would
now miss: `conftest.py:199` turns that into silent skips.
- The job runs the new test file, and the unused `FASTEMBED_CACHE_PATH`
is gone.
- Tests:
  - New `tests/test_fastembed_pinned_snapshot.py`:
    - the pinned revision reaches every file download;
- fastembed gets only the snapshot directory, built with the Hub forced
offline;
    - `HEADROOM_HF_PIN=off` floats;
    - files resolved from two different snapshots are refused;
- a cold cache while offline degrades without a download, and the error
names `repo@sha` and `--revision <sha>`;
    - real fastembed skips the Hub when given `specific_model_path`.
- `tests/test_stateless_writers.py`: the three `_pinned_revision` tests
are removed with the helper; `tests/test_hf_revision_pinning.py` covers
`_resolve_revision`.
- `tests/test_offline_egress_chokepoint.py`: `relevance/embedding.py`'s
cache-only count goes from 1 to 2 for the `_load_default_model`
construction.

## Release note

> **relevance:** the default embedder (`BAAI/bge-small-en-v1.5`) now
loads from the pinned snapshot
`Qdrant/bge-small-en-v1.5-onnx-Q@52398278842ec682c6f32300af41344b1c0b0bb2`
in the HuggingFace hub cache (`HF_HUB_CACHE` / `HF_HOME/hub` /
`~/.cache/huggingface/hub`), not from fastembed's cache. Existing
installs download ~67 MB once. Air-gapped hosts that pre-seeded the
fastembed cache must re-seed, e.g. `hf download
Qdrant/bge-small-en-v1.5-onnx-Q model_optimized.onnx tokenizer.json
tokenizer_config.json special_tokens_map.json config.json --revision
52398278842ec682c6f32300af41344b1c0b0bb2`. Without that, relevance
scoring falls back to BM25, and the error message says how to re-seed.

`CHANGELOG.md` is generated from the PR title, so this text is here for
whoever writes the release notes.

## Dependency Change (`fastembed>=0.4.0` → `>=0.6.0`)

This raises the minimum version of an existing dependency; no package is
added. `uv.lock` still resolves `fastembed 0.8.0`. The only lock change
is the `requires-dist` specifier, so the locked install set is
byte-for-byte the same.

- **Why this package:** it's already the `relevance` extra's embedder
(since Stage 3c.1). This PR adds no package and swaps none.
- **Who maintains it:** Qdrant (`qdrant/fastembed`), Apache-2.0,
actively released (0.6.0 on 2025-02-26 through 0.9.0 on 2026-10-07). The
GitHub Advisory Database has no advisories for `fastembed` (pip).
- **Install surface:** unchanged for the locked install (0.8.0).
Installs that resolve their own versions lose access to 0.4.x/0.5.x.
Runtime network shrinks: given `specific_model_path`, fastembed makes no
Hub call, and the only download is Headroom's own pinned,
offline-guarded `hf_hub_download_local_first`.
- **Why this version:** `specific_model_path` on `TextEmbedding` first
appears in v0.6.0, and the pinned-snapshot load depends on it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py \
    tests/test_relevance.py tests/test_relevance_extra.py tests/test_stateless_writers.py tests/test_hf_revision_pinning.py
7 failed, 209 passed, 3 skipped
# The 7 failures are TestOtlpExporterOffline / TestLangfuseExporterOffline / TestStartupRefusal /
# TestHeadroomCloudCompressionOffline: the OTel and Cloud extras aren't in this venv. The same 7 fail on
# origin/main (f1bc2609) here.

$ mypy headroom            # with fastembed 0.8.0 installed
Success: no issues found in 569 source files
$ ruff check .             # All checks passed!
$ ruff format --check .    # 1790 files already formatted
```

## Real Behavior Proof
- Exact command / steps: additional maintainer verification on
Windows/Python 3.12.13: python -m pytest
tests/test_fastembed_pinned_snapshot.py
tests/test_offline_egress_chokepoint.py tests/test_stateless_writers.py
-q.
- Observed result: the additional maintainer run passed all 160 selected
tests on b3eb7f349c, using the available
development native extension for startup; no fresh Rust implementation
qualification is claimed.

- Environment: macOS arm64, repo `.venv` (fastembed 0.8.0 = `uv.lock`),
rebased on `origin/main` `f1bc2609`.
- Default load: `EmbeddingScorer().score("authentication failed for user
admin", "login error")` loads from
`~/.cache/huggingface/hub/models--Qdrant--bge-small-en-v1.5-onnx-Q/snapshots/52398278842ec682c6f32300af41344b1c0b0bb2`
and scores 0.7389. `origin/main` loads
`<tmp>/fastembed_cache/…/snapshots/aa8f8b06…` (the floating `main`,
under `$TMPDIR`).
- Air-gapped re-seed, from an empty `HF_HOME`:
1. `HEADROOM_OFFLINE=1` gives `Embedding model 'BAAI/bge-small-en-v1.5'
is unavailable: HEADROOM_OFFLINE is set: … Pre-seed the HuggingFace hub
cache with Qdrant/bge-small-en-v1.5-onnx-Q@52398278… (`hf download …
--revision 52398278…`), or use the BM25-only scorer. A fastembed cache
seeded for an older Headroom is not read for this model.`
  2. I ran the `hf download` command from that message as printed.
  3. `HEADROOM_OFFLINE=1 HF_HUB_OFFLINE=1` then loads and scores 0.7389.
4. With `HEADROOM_HF_PIN=off`, the message names `@main` and `--revision
main` instead.
- Not tested: Windows, and Linux outside CI. fastembed 0.6.x/0.7.x at
runtime (the 0.6.0 floor comes from reading the v0.6.0 tag's source).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A. This is the default code path, with no
flag.
- Stable/default behavior changed: yes. The default relevance model
loads the pinned snapshot from the standard HF hub cache, not
fastembed's temp-dir cache, so existing installs download ~67 MB once.
The model files are byte-identical to what `main` served; only
`README.md` differs.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`.
Air-gapped hosts re-seed as described in the Release note.
- Unsafe override required: no.
- Qualification impact: none expected. Same weights and tokenizer as the
current floating `main`.
- Rollback path: revert this PR. Nothing persistent changes beyond an
extra snapshot in the HF cache.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- `_load_text_embedding` was typed `kwargs: dict[str, str]`, which made
mypy report 8 `arg-type` errors on `TextEmbedding(**kwargs)` whenever
fastembed is installed in the type-checking environment. CI's mypy run
doesn't install fastembed, so it never saw them.
- Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches
are still unpinned (there's no Rust `HEADROOM_HF_PIN`).

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
This commit is contained in:
Kenneth Wong
2026-10-20 19:45:26 -05:00
committed by GitHub
co-authored by Kenneth Wong
parent 62b3bedaba
commit f380692b46
8 changed files with 312 additions and 85 deletions
+7 -6
View File
@@ -443,8 +443,6 @@ jobs:
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
env:
FASTEMBED_CACHE_PATH: ${{ github.workspace }}/.fastembed-cache
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
@@ -459,8 +457,11 @@ jobs:
- name: Cache fastembed model
uses: actions/cache@v6
with:
path: ${{ github.workspace }}/.fastembed-cache
key: ${{ runner.os }}-fastembed-bge-small-v1
# The default relevance model loads from its pinned snapshot in the
# HuggingFace hub cache (headroom/onnx_runtime.py _PINNED_REVISIONS);
# the key carries the pinned SHA so a bump re-fetches.
path: ~/.cache/huggingface/hub/models--Qdrant--bge-small-en-v1.5-onnx-Q
key: ${{ runner.os }}-hf-bge-small-en-v1.5-onnx-Q-52398278842e
- name: Download prebuilt wheel
uses: actions/download-artifact@v8
with:
@@ -481,7 +482,7 @@ jobs:
HF_HUB_DISABLE_TELEMETRY: "1"
run: |
for i in 1 2 3 4 5; do
if python -c "from fastembed import TextEmbedding; TextEmbedding('BAAI/bge-small-en-v1.5')"; then exit 0; fi
if python -c "from headroom.relevance.embedding import _resolve_default_model_snapshot as r; print(r())"; then exit 0; fi
echo "::warning::fastembed fetch attempt $i failed; backing off"; sleep $((i * 20))
done
echo "::error::could not fetch fastembed model from HuggingFace"; exit 1
@@ -491,7 +492,7 @@ jobs:
env:
HF_HUB_OFFLINE: "1"
TRANSFORMERS_OFFLINE: "1"
run: pytest tests/test_relevance.py -v
run: pytest tests/test_relevance.py tests/test_fastembed_pinned_snapshot.py -v
- name: Run HTML extraction and tool reversibility tests
run: |
+3
View File
@@ -81,6 +81,9 @@ _PINNED_REVISIONS: dict[str, str] = {
"chopratejas/siglip-image-encoder-onnx": "d0a9fbd66d4bd8c761bff592d44831f7c2ae184e",
# Third-party repo — pinning matters most here.
"Qdrant/all-MiniLM-L6-v2-onnx": "5f1b8cd78bc4fb444dd171e59b18f3a3af89a079",
# Default relevance embedder (BAAI/bge-small-en-v1.5), third-party too.
# relevance/embedding.py downloads each of its files at this SHA.
"Qdrant/bge-small-en-v1.5-onnx-Q": "52398278842ec682c6f32300af41344b1c0b0bb2",
}
+118 -47
View File
@@ -1,10 +1,9 @@
"""Embedding-based relevance scorer for Headroom SDK.
This module provides semantic relevance scoring using `fastembed`
(BAAI/bge-small-en-v1.5 by default — 33M params, 384 dims, ~30 MB
int8-quantized ONNX). Same library + same model used by the Rust
SmartCrusher (fastembed-rs crate) so embeddings agree byte-for-byte
across the language boundary.
(BAAI/bge-small-en-v1.5 by default — 33M params, 384 dims, ~67 MB
quantized ONNX), loaded from a pinned Qdrant/bge-small-en-v1.5-onnx-Q
snapshot in the HuggingFace hub cache.
Key features:
- Semantic understanding ("errors" matches "failed", "issues")
@@ -28,9 +27,13 @@ from __future__ import annotations
import logging
import os
from typing import TYPE_CHECKING
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
from typing import TYPE_CHECKING, Any
from headroom.offline import OfflineEgressBlocked, guard_egress
from headroom.onnx_runtime import _resolve_revision, hf_hub_download_local_first
from .base import RelevanceScore, RelevanceScorer
@@ -60,7 +63,31 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
def _load_text_embedding(kwargs: dict[str, str]) -> TextEmbedding:
@contextmanager
def _hf_hub_forced_offline() -> Iterator[None]:
"""Force the HuggingFace stack offline for the duration of the block.
huggingface_hub reads HF_HUB_OFFLINE once, at import, into
``constants.HF_HUB_OFFLINE``; if it was imported before this call the env
var alone changes nothing. Force both, and restore both on exit.
"""
from huggingface_hub import constants as hf_constants
previous = os.environ.get("HF_HUB_OFFLINE")
previous_constant = hf_constants.HF_HUB_OFFLINE
os.environ["HF_HUB_OFFLINE"] = "1"
hf_constants.HF_HUB_OFFLINE = True
try:
yield
finally:
hf_constants.HF_HUB_OFFLINE = previous_constant
if previous is None:
os.environ.pop("HF_HUB_OFFLINE", None)
else:
os.environ["HF_HUB_OFFLINE"] = previous
def _load_text_embedding(kwargs: dict[str, Any]) -> TextEmbedding:
"""Build a fastembed ``TextEmbedding``, cache first and network second.
``onnx_runtime.hf_hub_download_local_first`` can hang the air-gap guard on
@@ -83,25 +110,12 @@ def _load_text_embedding(kwargs: dict[str, str]) -> TextEmbedding:
never sent somewhere it would not otherwise go.
"""
from fastembed import TextEmbedding
from huggingface_hub import constants as hf_constants
# huggingface_hub reads HF_HUB_OFFLINE once, at import, into
# ``constants.HF_HUB_OFFLINE``; if it was imported before this call the
# env var alone changes nothing. Force both.
previous = os.environ.get("HF_HUB_OFFLINE")
previous_constant = hf_constants.HF_HUB_OFFLINE
os.environ["HF_HUB_OFFLINE"] = "1"
hf_constants.HF_HUB_OFFLINE = True
try:
return TextEmbedding(**kwargs)
with _hf_hub_forced_offline():
return TextEmbedding(**kwargs)
except Exception as cache_miss: # noqa: BLE001 - any local-lookup failure
logger.debug("fastembed cache lookup failed, falling back to network: %s", cache_miss)
finally:
hf_constants.HF_HUB_OFFLINE = previous_constant
if previous is None:
os.environ.pop("HF_HUB_OFFLINE", None)
else:
os.environ["HF_HUB_OFFLINE"] = previous
guard_egress(
f"fastembed weight download for {kwargs['model_name']}",
@@ -113,25 +127,77 @@ def _load_text_embedding(kwargs: dict[str, str]) -> TextEmbedding:
# Default model name. Same string used by the Rust embedding scorer.
DEFAULT_MODEL_NAME = "BAAI/bge-small-en-v1.5"
# Pinned revision of fastembed's underlying HF repo for the default model
# (fastembed resolves "BAAI/bge-small-en-v1.5" -> "qdrant/bge-small-en-v1.5-onnx-q").
# fastembed's TextEmbedding signature omits ``revision`` but forwards **kwargs to
# huggingface_hub.snapshot_download, so passing it pins the download for
# supply-chain integrity. Only the default model is pinned; custom models float.
# Set HEADROOM_HF_PIN=off to bypass (mirrors headroom.onnx_runtime pinning).
_DEFAULT_MODEL_PINNED_REVISION = "52398278842ec682c6f32300af41344b1c0b0bb2"
# The HF repo fastembed resolves DEFAULT_MODEL_NAME to, and every file a
# fastembed load of it reads. Its revision is pinned in
# ``onnx_runtime._PINNED_REVISIONS`` (HEADROOM_HF_PIN=off floats it).
DEFAULT_MODEL_REPO = "Qdrant/bge-small-en-v1.5-onnx-Q"
DEFAULT_MODEL_FILES = (
"model_optimized.onnx",
"tokenizer.json",
"tokenizer_config.json",
"special_tokens_map.json",
"config.json",
)
def _pinned_revision(model_name: str) -> str | None:
"""Return the pinned HF revision for ``model_name`` (default model only).
def _resolve_default_model_snapshot() -> str:
"""Return the local directory holding the pinned default-model snapshot.
Returns ``None`` for custom models or when ``HEADROOM_HF_PIN=off``.
fastembed cannot pin this itself: ``TextEmbedding`` accepts ``revision``
but drops it before ``snapshot_download`` (fastembed 0.9.0), so passing it
left the download floating on the repo's ``main``. Each file is resolved
here through ``hf_hub_download_local_first`` instead, which applies the
pinned SHA, tries the cache first and guards the network fallback.
The files land in the standard HuggingFace hub cache rather than
fastembed's default under the system temp dir, which macOS purges after a
few idle days.
Raises:
OfflineEgressBlocked: ``HEADROOM_OFFLINE`` is set and a file is not cached.
RuntimeError: the files resolved to more than one snapshot, which only
happens when ``main`` moved mid-download with pinning turned off.
"""
if os.environ.get("HEADROOM_HF_PIN", "").strip().lower() in ("off", "0", "false", "no"):
return None
if model_name == DEFAULT_MODEL_NAME:
return _DEFAULT_MODEL_PINNED_REVISION
return None
paths = [
Path(hf_hub_download_local_first(DEFAULT_MODEL_REPO, name)) for name in DEFAULT_MODEL_FILES
]
snapshot_dirs = {path.parent for path in paths}
if len(snapshot_dirs) != 1:
raise RuntimeError(
f"{DEFAULT_MODEL_REPO} files resolved to more than one snapshot: "
f"{sorted(str(d) for d in snapshot_dirs)}"
)
return str(snapshot_dirs.pop())
def _default_model_seed_hint() -> str:
"""How to pre-seed DEFAULT_MODEL_NAME on a host that cannot download it.
Names the exact snapshot the load looks up, because a fastembed cache
seeded for an older Headroom no longer satisfies it.
"""
revision = _resolve_revision(DEFAULT_MODEL_REPO, None) or "main"
return (
f"Pre-seed the HuggingFace hub cache with {DEFAULT_MODEL_REPO}@{revision} "
f"(`hf download {DEFAULT_MODEL_REPO} {' '.join(DEFAULT_MODEL_FILES)} "
f"--revision {revision}`), or use the BM25-only scorer. A fastembed cache "
"seeded for an older Headroom is not read for this model."
)
def _load_default_model() -> TextEmbedding:
"""Load DEFAULT_MODEL_NAME from its pinned snapshot, with no fastembed lookup.
Given ``specific_model_path`` fastembed loads that directory without
touching the Hub; the construction still runs with the Hub forced offline
so that a fastembed that stopped honouring it fails here instead of
downloading an unpinned copy.
"""
from fastembed import TextEmbedding
snapshot = _resolve_default_model_snapshot()
with _hf_hub_forced_offline():
return TextEmbedding(model_name=DEFAULT_MODEL_NAME, specific_model_path=snapshot)
def _cosine_similarity(a, b) -> float:
@@ -160,8 +226,8 @@ class EmbeddingScorer(RelevanceScorer):
"""Semantic relevance scorer using fastembed (ONNX-backed).
Default model: BAAI/bge-small-en-v1.5 (33M params, 384 dims).
Auto-downloads from HuggingFace Hub on first use (~30 MB
int8-quantized ONNX).
Auto-downloads its pinned snapshot from HuggingFace Hub on first
use (~67 MB quantized ONNX).
Example:
scorer = EmbeddingScorer()
@@ -223,23 +289,28 @@ class EmbeddingScorer(RelevanceScorer):
)
if self._model is None:
revision = _pinned_revision(self.model_name)
kwargs = {"model_name": self.model_name}
if revision is not None:
# fastembed forwards **kwargs to snapshot_download(revision=...).
kwargs["revision"] = revision
try:
self._model = _load_text_embedding(kwargs)
if self.model_name == DEFAULT_MODEL_NAME:
self._model = _load_default_model()
else:
# Other catalog models are not pinned: fastembed resolves
# them on the repo's main, cache first.
self._model = _load_text_embedding({"model_name": self.model_name})
except OfflineEgressBlocked as blocked:
# Translate at the boundary that owns the degradation, the way
# the Kompress and ONNX loaders do. Public model weights are
# not data leaving the box, so what the caller needs to hear
# is "this model is not available here and why", not an
# air-gap type it has never seen.
if self.model_name == DEFAULT_MODEL_NAME:
seed_hint = _default_model_seed_hint()
else:
seed_hint = (
"Pre-seed the fastembed cache on this host, or use the BM25-only scorer."
)
raise RuntimeError(
f"Embedding model {self.model_name!r} is unavailable: {blocked}. "
"Pre-seed the fastembed/HuggingFace cache on this host, or "
"use the BM25-only scorer."
f"Embedding model {self.model_name!r} is unavailable: "
f"{str(blocked).rstrip('.')}. {seed_hint}"
) from None
return self._model
+4 -4
View File
@@ -172,13 +172,13 @@ pytorch-mps = [
]
# Semantic relevance scoring with embeddings.
# Uses `fastembed` (BAAI/bge-small-en-v1.5 by default — 33M params,
# 384 dims, ~30 MB int8-quantized ONNX). Same library + model used by
# the Rust SmartCrusher (`fastembed` crate), giving byte-equal embeddings
# across the language boundary. Replaced sentence-transformers in
# 384 dims, ~67 MB quantized ONNX), loaded from a pinned
# Qdrant/bge-small-en-v1.5-onnx-Q snapshot. Replaced sentence-transformers in
# Stage 3c.1 — fastembed is faster (~2-3x), smaller (no torch
# dependency), and outranks all-MiniLM-L6-v2 on MTEB by ~6 points.
# >=0.6.0 for `specific_model_path`, which the pinned-snapshot load needs.
relevance = [
"fastembed>=0.4.0; python_version<'3.14' or sys_platform != 'darwin' or platform_machine != 'x86_64'",
"fastembed>=0.6.0; python_version<'3.14' or sys_platform != 'darwin' or platform_machine != 'x86_64'",
"numpy>=1.24.0",
]
# Image compression (ML-based routing + OCR)
+173
View File
@@ -0,0 +1,173 @@
"""The default relevance embedder loads one pinned snapshot.
fastembed's ``TextEmbedding`` accepts ``revision`` but drops it before
``snapshot_download``, so the old pin never reached the Hub and the model
floated on ``main``. These tests hold the replacement in place: the pinned
SHA reaches every file download and fastembed only gets the resolved
directory.
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
from types import SimpleNamespace
import pytest
from headroom.offline import OFFLINE_ENV
from headroom.onnx_runtime import _PINNED_REVISIONS
from headroom.relevance import embedding
from headroom.relevance.embedding import (
DEFAULT_MODEL_FILES,
DEFAULT_MODEL_NAME,
DEFAULT_MODEL_REPO,
EmbeddingScorer,
)
PIN = _PINNED_REVISIONS[DEFAULT_MODEL_REPO]
@pytest.fixture
def fake_hub(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> list[dict[str, object]]:
"""Stand in for ``huggingface_hub.hf_hub_download``: record each call and
return the path the real cache layout would give for that revision."""
import huggingface_hub
calls: list[dict[str, object]] = []
def _download(repo_id: str, filename: str, **kwargs: object) -> str:
calls.append({"repo_id": repo_id, "filename": filename, **kwargs})
commit = kwargs.get("revision") or "main-commit"
return str(tmp_path / "snapshots" / str(commit) / filename)
monkeypatch.setattr(huggingface_hub, "hf_hub_download", _download)
monkeypatch.delenv("HEADROOM_HF_PIN", raising=False)
return calls
@pytest.fixture
def fake_fastembed(monkeypatch: pytest.MonkeyPatch) -> list[dict[str, object]]:
"""Install a stub fastembed whose constructor records its kwargs."""
constructed: list[dict[str, object]] = []
class _TextEmbedding:
def __init__(self, **kwargs: object) -> None:
constructed.append(kwargs)
monkeypatch.setitem(sys.modules, "fastembed", SimpleNamespace(TextEmbedding=_TextEmbedding))
return constructed
def test_every_file_is_downloaded_at_the_pinned_revision(fake_hub, fake_fastembed, tmp_path):
EmbeddingScorer()._get_model()
assert {c["filename"] for c in fake_hub} == set(DEFAULT_MODEL_FILES)
assert {c["repo_id"] for c in fake_hub} == {DEFAULT_MODEL_REPO}
assert {c["revision"] for c in fake_hub} == {PIN}
def test_fastembed_gets_the_snapshot_directory_not_a_revision(fake_hub, fake_fastembed, tmp_path):
EmbeddingScorer()._get_model()
assert fake_fastembed == [
{
"model_name": DEFAULT_MODEL_NAME,
"specific_model_path": str(tmp_path / "snapshots" / PIN),
}
]
def test_fastembed_is_constructed_with_the_hub_forced_offline(fake_hub, monkeypatch):
"""If fastembed ever stopped honouring ``specific_model_path`` it would
try the Hub; forcing it offline makes that a load failure, not an
unpinned download."""
from huggingface_hub import constants as hf_constants
seen: list[tuple[str | None, bool]] = []
class _TextEmbedding:
def __init__(self, **kwargs: object) -> None:
seen.append((os.environ.get("HF_HUB_OFFLINE"), hf_constants.HF_HUB_OFFLINE))
monkeypatch.setitem(sys.modules, "fastembed", SimpleNamespace(TextEmbedding=_TextEmbedding))
monkeypatch.setenv("HF_HUB_OFFLINE", "0")
monkeypatch.setattr(hf_constants, "HF_HUB_OFFLINE", False)
EmbeddingScorer()._get_model()
assert seen == [("1", True)]
assert os.environ["HF_HUB_OFFLINE"] == "0"
assert hf_constants.HF_HUB_OFFLINE is False
def test_pin_off_floats_on_main(fake_hub, fake_fastembed, monkeypatch):
monkeypatch.setenv("HEADROOM_HF_PIN", "off")
EmbeddingScorer()._get_model()
assert {c["revision"] for c in fake_hub} == {None}
def test_files_from_two_snapshots_are_refused(monkeypatch, fake_fastembed, tmp_path):
import huggingface_hub
def _download(repo_id: str, filename: str, **kwargs: object) -> str:
commit = "a" if filename == DEFAULT_MODEL_FILES[0] else "b"
return str(tmp_path / commit / filename)
monkeypatch.setattr(huggingface_hub, "hf_hub_download", _download)
with pytest.raises(RuntimeError, match="more than one snapshot"):
EmbeddingScorer()._get_model()
assert fake_fastembed == []
def test_a_cold_cache_offline_degrades_without_a_download(monkeypatch, fake_fastembed):
import huggingface_hub
from huggingface_hub.errors import LocalEntryNotFoundError
def _download(repo_id: str, filename: str, **kwargs: object) -> str:
if not kwargs.get("local_files_only"):
raise AssertionError("the network fallback ran while offline")
raise LocalEntryNotFoundError("not cached")
monkeypatch.setattr(huggingface_hub, "hf_hub_download", _download)
monkeypatch.setenv(OFFLINE_ENV, "1")
with pytest.raises(RuntimeError, match="is unavailable") as excinfo:
EmbeddingScorer()._get_model()
assert fake_fastembed == []
# The hint names the exact snapshot to seed, not the fastembed cache an
# older Headroom read.
assert f"{DEFAULT_MODEL_REPO}@{PIN}" in str(excinfo.value)
assert f"--revision {PIN}" in str(excinfo.value)
def test_other_models_keep_fastembed_resolution(monkeypatch, fake_hub):
seen: list[dict[str, str]] = []
monkeypatch.setattr(embedding, "_load_text_embedding", lambda kwargs: seen.append(kwargs))
monkeypatch.setitem(sys.modules, "fastembed", SimpleNamespace())
EmbeddingScorer("intfloat/e5-small-v2")._get_model()
assert seen == [{"model_name": "intfloat/e5-small-v2"}]
assert fake_hub == []
def test_fastembed_loads_specific_model_path_without_a_download(monkeypatch, tmp_path):
"""The contract the fix rests on, against the real fastembed: given
``specific_model_path`` it uses that directory and never reaches the Hub."""
pytest.importorskip("fastembed")
import fastembed.common.model_management as model_management
from fastembed import TextEmbedding
def _no_hub(*args: object, **kwargs: object) -> None:
raise AssertionError("fastembed reached the Hub despite specific_model_path")
monkeypatch.setattr(model_management, "snapshot_download", _no_hub)
model = TextEmbedding(
model_name=DEFAULT_MODEL_NAME,
specific_model_path=str(tmp_path),
cache_dir=str(tmp_path / "unused-cache"),
lazy_load=True,
)
assert Path(model.model._model_dir) == tmp_path
+6 -2
View File
@@ -2351,13 +2351,17 @@ _EGRESS_ALLOWLIST: dict[str, tuple[int, str]] = {
"calls guard_egress first.",
),
"relevance/embedding.py": (
1,
2,
"cache-only: the first of the two fastembed loads in "
"_load_text_embedding runs with HF_HUB_OFFLINE forced to 1, so "
"huggingface_hub resolves from the local cache or raises without "
"opening a socket. The network retry directly below it calls "
"guard_egress, which is why a pre-seeded air-gapped host still loads "
"the model and a cold one refuses instead of dialling.",
"the model and a cold one refuses instead of dialling. The load in "
"_load_default_model runs under the same forced HF_HUB_OFFLINE and "
"passes specific_model_path, which fastembed loads without any Hub "
"lookup; its files come through hf_hub_download_local_first, guarded "
"on the miss.",
),
}
-25
View File
@@ -13,11 +13,6 @@ pytest.importorskip("fastapi")
from headroom import paths
from headroom.proxy.output_savings import SavingsRecorder
from headroom.relevance.embedding import (
_DEFAULT_MODEL_PINNED_REVISION,
DEFAULT_MODEL_NAME,
_pinned_revision,
)
@pytest.fixture(autouse=True)
@@ -100,23 +95,3 @@ def test_memory_disabled_under_stateless(tmp_path, monkeypatch):
proxy = app.state.proxy
assert proxy.memory_handler is None
assert not workspace.exists()
# ---- fastembed model pinning ----------------------------------------------
def test_fastembed_default_model_is_pinned_to_sha(monkeypatch):
monkeypatch.delenv("HEADROOM_HF_PIN", raising=False)
rev = _pinned_revision(DEFAULT_MODEL_NAME)
assert rev == _DEFAULT_MODEL_PINNED_REVISION
assert len(rev) == 40 and all(c in "0123456789abcdef" for c in rev)
def test_fastembed_custom_model_not_pinned(monkeypatch):
monkeypatch.delenv("HEADROOM_HF_PIN", raising=False)
assert _pinned_revision("intfloat/e5-small-v2") is None
def test_fastembed_pin_can_be_disabled(monkeypatch):
monkeypatch.setenv("HEADROOM_HF_PIN", "off")
assert _pinned_revision(DEFAULT_MODEL_NAME) is None
Generated
+1 -1
View File
@@ -1813,7 +1813,7 @@ requires-dist = [
{ name = "datasets", marker = "extra == 'voice-train'", specifier = ">=5.0.1" },
{ name = "fastapi", marker = "extra == 'dev'", specifier = ">=0.100.0" },
{ name = "fastapi", marker = "extra == 'proxy'", specifier = ">=0.100.0" },
{ name = "fastembed", marker = "(python_full_version < '3.14' and extra == 'relevance') or (platform_machine != 'x86_64' and extra == 'relevance') or (sys_platform != 'darwin' and extra == 'relevance')", specifier = ">=0.4.0" },
{ name = "fastembed", marker = "(python_full_version < '3.14' and extra == 'relevance') or (platform_machine != 'x86_64' and extra == 'relevance') or (sys_platform != 'darwin' and extra == 'relevance')", specifier = ">=0.6.0" },
{ name = "gunicorn", marker = "sys_platform != 'win32' and extra == 'proxy-prod'", specifier = ">=21.0.0" },
{ name = "headroom-ai", extras = ["proxy"], marker = "extra == 'proxy-prod'" },
{ name = "headroom-ai", extras = ["proxy", "code", "ml", "memory", "relevance", "image", "reports", "otel", "evals", "voice", "html", "mcp", "spreadsheet"], marker = "extra == 'all'" },