fix(relevance): load the Rust default embedder from Python's pinned snapshot (#4190)

## Description

Parent status: #4152 has merged. The pinned-loader and batch-cap merge
resolution was reviewed at e3a495cbc5. The
original split/stack history is retained below.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now four
linear commits on top of main, with no merge commits:

- `bfae4ade` fix(relevance): load the Rust default embedder from
Python's pinned snapshot
- `6fbae380` fix(relevance): look up and download the Rust embedder in
one HF cache dir
- `70afece8` fix(relevance): expand ~ and $VAR in the Rust HF cache path
like huggingface_hub
- `62b792f0` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`. Its files matched main; the only difference was
`uv.lock`, where main's later `filelock>=4.2.1` entry was kept.
- The six main-sync merges (`185dc9b7`, `052977b9`, `029c72ac`,
`4405b12c`, `8b0a10e8`, and the reviewed `e3a495cb`) are linearized
away, so those SHAs are no longer on the branch.
- `embedding.rs` conflicted again with main's batch-size cap (#4507). I
resolved it to the same content as the reviewed merge: every file this
PR touches is byte-identical between `e3a495cb` and the new head
`62b792f0` (`git diff e3a495cb 62b792f0 -- <files changed vs main>` is
empty). The review and qualification below still apply to the content;
only the history and SHAs changed.

Rust `EmbeddingScorer::try_new` used `EmbeddingModel::BGESmallENV15`.
fastembed-rs maps that to a different repo and file from what Python
runs: `Xenova/bge-small-en-v1.5` `onnx/model.onnx` (fp32), always on
`main`. So the two languages never ran the same ONNX file, and the
"byte-equal across languages" claim in both modules was wrong.

This PR makes `try_new` load the same pinned
`Qdrant/bge-small-en-v1.5-onnx-Q` snapshot that #4152 makes Python load.
It adds a cross-language parity fixture checked from both sides, and
replaces the parity claim with measured numbers.

Nothing in production calls `EmbeddingScorer::try_new` today:
`create_scorer` and `HybridScorer` use `default()`, which never loads a
model. This keeps the Rust scorer correct and pinned for when it is
wired in.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

(This PR's own commits only.)

- `crates/headroom-core/src/relevance/embedding.rs`:
- `try_new` loads the snapshot through
`TextEmbedding::try_new_from_user_defined`, with CLS pooling and static
quantization, which is what fastembed-rs uses for `BGESmallENV15Q`.
- It uses exactly one hub cache directory, resolved like Python's
`huggingface_hub.constants.HF_HUB_CACHE`: `HF_HUB_CACHE` →
`HUGGINGFACE_HUB_CACHE` → `HF_HOME/hub` →
`XDG_CACHE_HOME/huggingface/hub` → `~/.cache/huggingface/hub`, with a
leading `~` expanded. It looks up and downloads only there, so a stale
snapshot in a lower-priority cache is never used. The home directory
comes from `std::env::home_dir`, which falls back to the user database
when `HOME` is unset, as Python's `expanduser` does.
- It finds the snapshot as `snapshots/<sha>` in that directory.
`hf_hub::Cache` resolves through `refs/<rev>`, which Python's
`huggingface_hub` doesn't write for a commit SHA.
- On a miss it downloads with `Repo::with_revision`. `guard_egress` sits
in front of the Hub client and runs only on a cache miss, so a
pre-seeded air-gapped host now loads; previously Rust refused even with
a warm cache.
- `HEADROOM_HF_PIN=off` floats it on `main`, with the same values as
Python's `_resolve_revision`.
  - `try_new_with_model` is unchanged.
- Parity fixture: `tests/parity/record_embedding.py` and
`tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json` (6 texts
× 384 dims, recorded on ORT 1.31.0). The fixture is written one
embedding row per line, so it's 22 lines. Python checks it in CI
(`test-extras` sets `RUN_FASTEMBED_TESTS=1`); Rust checks it under
`RUN_FASTEMBED_TESTS=1`.
- `tests/test_fastembed_pinned_snapshot.py`: the Rust constants name the
same repo, SHA and files as Python; the fixture was recorded at the pin;
plus the gated Python parity check.
- 10 new Rust unit tests in `embedding.rs`: the lookup ignores other
commits and needs every file; the download URL requests the pinned
revision; the offline guard applies on a miss only;
`HEADROOM_HF_PIN=off` floats; the cache directory follows
`huggingface_hub`'s order variable by variable (including an empty
variable and `~`); with no `HOME` it still resolves through the user
database; and the review case: an empty explicit `HF_HUB_CACHE` next to
a complete pinned snapshot and an older `refs/main` under `HOME` finds
neither, for the pinned revision or `main`, while both are found once
`HF_HUB_CACHE` is unset.
- `tests/test_offline_egress_chokepoint.py`: the Rust sweep's regex now
also matches `ApiBuilder::from_cache` / `from_env`. It missed the new
download client, which is guarded; it flags nothing else in `crates/`.
The user-defined model load, which only reads bytes already on disk, is
allowlisted as `cache-only`.
- Docs: the "byte-equal" wording in both embedder modules,
`pyproject.toml` and `onnx_runtime.py` is replaced with measured numbers
and the cross-language keep-equal notes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `62b792f0` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 991 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_hf_revision_pinning.py
235 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

The 7 offline-egress failures noted below no longer reproduce here.

Before the rebase:

```text
$ cargo test -p headroom-core --lib
test result: ok. 985 passed; 0 failed; 1 ignored      # see the note on #4151 below

$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean

$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.31.0 dylib> cargo test -p headroom-core --lib relevance::embedding
test relevance::embedding::tests::fastembed_matches_parity_fixture ... ok
test result: ok. 24 passed; 0 failed

$ RUN_FASTEMBED_TESTS=1 pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_relevance.py \
    tests/test_stateless_writers.py tests/test_hf_revision_pinning.py
71 passed

$ pytest -q tests/test_offline_egress_chokepoint.py
7 failed, 132 passed, 3 skipped
# The 7 are the OTel / Langfuse / Headroom Cloud offline tests, whose extras aren't in this venv;
# the same 7 fail on origin/main here. The Rust sweep tests pass.
```

**Note on #4151:** with `ORT_DYLIB_PATH` set on macOS, a full `cargo
test -p headroom-core --lib` fails
`tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error`
most of the time (5/5 on `afdb60d3`, before the cache change too).
`a_cached_snapshot_is_not_refused_offline` holds `HEADROOM_OFFLINE=1`
under the env lock while ONNX Runtime loads, which is slow there, and
the tokenizer test reads the variable without the lock. That's the race
#4151 fixes; with its commit applied, 6/6 full runs pass. Linux CI
passes either way.

The cache-precedence test was checked against the bug: with the lookup
changed back to also scanning `~/.cache`, it fails on the first case
(pinned revision, empty `HF_HUB_CACHE`).

## Real Behavior Proof
- Exact command / steps: additional maintainer verification on the
main-synced head: cargo test -p headroom-core --locked hf_hub_cache_dir;
cargo test -p headroom-core --locked
expand_vars_matches_python_posixpath_and_ntpath; python -m pytest
tests/test_fastembed_pinned_snapshot.py -q; repeat its fixture cases
with huggingface-hub==2.2.0.
- Observed result: three Rust tests passed across both POSIX and Windows
expansion fixtures and cache-directory selection; Python passed 46 tests
with one gated skip using Hugging Face Hub 1.16.1, and all 36 shared
fixture cases passed separately with locked Hub 2.2.0.

- Environment: macOS arm64; Rust fastembed 5.17.3 with hf-hub 0.5.0;
repo `.venv` (fastembed 0.8.0, onnxruntime 1.23.2) and a second
interpreter on onnxruntime 1.31.0.
- A cold-cache Rust load with `HF_HUB_CACHE=<empty dir>` wrote
`snapshots/52398278…` plus `refs/52398278…` and passed the parity check.
Per-component max |Δ| against the fixture:
  - Rust vs Python on the same ORT (1.31.0): ≤ 1e-6.
  - Python on ORT 1.23.2 vs 1.31.0: ≤ 4.1e-4.
  - The fixture's tolerance is 1e-3.
- For reference, Xenova's fp32 export (what Rust used to load) is ≤
4.7e-4 from the fixture. The old cross-language gap was real but
numerically small, and this fixture bounds drift; it cannot tell the two
exports apart. Model identity is enforced by the pinned SHA and the unit
tests.
- The fixture's new one-row-per-line layout was re-emitted from the
recorded data, not re-recorded: the parsed JSON is identical.
- Not tested: Windows, and Linux outside CI. Rust's gated tests in CI
(they need the model and an ORT dylib, so they stay manual behind
`RUN_FASTEMBED_TESTS=1`). Rust `HybridScorer` thresholds against the
Qdrant export.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no production path.
`EmbeddingScorer::try_new` loads the Qdrant export instead of Xenova's,
and now loads offline from a warm cache instead of refusing, but nothing
in production calls it today.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches
are still unpinned (there's no Rust `HEADROOM_HF_PIN`), and Rust's gated
embedding tests could get a CI job once one ships an ORT dylib.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4190"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer qualification

The environment-reference mismatch from the previous review is
addressed. The shared corpus covers the reported explicit-cache case,
platform-specific variable syntax, and the two-stage home/hub expansion
ordering. The Rust test runs both POSIX and Windows expansion rules;
Python validates the same corpus against posixpath, ntpath, and fresh
huggingface_hub.constants imports.

A clean current-main sync was published as
052977b9e88ae3bbda96797615b64ee350a2f695; it does not change the
contributor's resolver or fixture files. Qualification used Windows,
Hugging Face Hub 1.16.1 and an isolated locked 2.2.0 target. No model
download or numerical embedding-parity result is claimed by these path
tests; the pre-existing gated numerical-parity test remained skipped.

Hosted checks on the pushed head must finish before approval. No
contributor CI action requested or PR merged.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Kenneth Wong
2026-11-03 21:31:59 -06:00
committed by GitHub
co-authored by Kenneth Wong JD Davis
parent 23a5bb1c81
commit 9bb4b130ff
10 changed files with 1419 additions and 72 deletions
+2
View File
@@ -489,9 +489,11 @@ jobs:
- name: Run relevance tests
# Offline so fastembed reads the cache the prefetch step just warmed,
# without an unauthenticated cache-validation HEAD that could 429.
# RUN_FASTEMBED_TESTS runs the Python half of the embedding parity check.
env:
HF_HUB_OFFLINE: "1"
TRANSFORMERS_OFFLINE: "1"
RUN_FASTEMBED_TESTS: "1"
run: pytest tests/test_relevance.py tests/test_fastembed_pinned_snapshot.py -v
- name: Run HTML extraction and tool reversibility tests
File diff suppressed because it is too large Load Diff
+4 -1
View File
@@ -82,7 +82,10 @@ _PINNED_REVISIONS: dict[str, str] = {
# Third-party repo — pinning matters most here.
"Qdrant/all-MiniLM-L6-v2-onnx": "5f1b8cd78bc4fb444dd171e59b18f3a3af89a079",
# Default relevance embedder (BAAI/bge-small-en-v1.5), third-party too.
# relevance/embedding.py downloads each of its files at this SHA.
# relevance/embedding.py downloads each of its files at this SHA. The Rust
# scorer loads the same snapshot: keep this SHA equal to
# DEFAULT_MODEL_REVISION in crates/headroom-core/src/relevance/embedding.rs
# (tests/test_fastembed_pinned_snapshot.py enforces it).
"Qdrant/bge-small-en-v1.5-onnx-Q": "52398278842ec682c6f32300af41344b1c0b0bb2",
}
+15 -9
View File
@@ -2,8 +2,12 @@
This module provides semantic relevance scoring using `fastembed`
(BAAI/bge-small-en-v1.5 by default — 33M params, 384 dims, ~67 MB
quantized ONNX), loaded from a pinned Qdrant/bge-small-en-v1.5-onnx-Q
snapshot in the HuggingFace hub cache.
quantized ONNX). The Rust scorer (fastembed-rs crate) loads the same
pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, so both languages run
the same ONNX file and tokenizer. On the same ONNX Runtime build the two
agree within 1e-6 per component; across ORT versions they drift by up to
~4e-4, so they are close, not byte-equal. The gated parity fixture in
tests/parity/fixtures/embedding/ bounds that drift.
Key features:
- Semantic understanding ("errors" matches "failed", "issues")
@@ -16,8 +20,8 @@ Install with: pip install "headroom-ai[relevance]"
History: this module previously wrapped `sentence-transformers`
(PyTorch). Switched to fastembed in Stage 3c.1 of the Rust port to:
1. Match the Rust embedding scorer byte-for-byte (both call into
ONNX Runtime over the identical ONNX file).
1. Run the same ONNX model as the Rust embedding scorer (both call
into ONNX Runtime over the identical pinned ONNX file).
2. Remove the torch dependency from the relevance/ path
(Phase 6: "drop torch from Python").
3. Get a better default model (bge-small-en-v1.5 vs MiniLM-L6-v2).
@@ -157,12 +161,14 @@ def _load_text_embedding(kwargs: dict[str, Any]) -> TextEmbedding:
return TextEmbedding(**kwargs)
# Default model name. Same string used by the Rust embedding scorer.
# Default model name. Same model the Rust embedding scorer loads.
DEFAULT_MODEL_NAME = "BAAI/bge-small-en-v1.5"
# The HF repo fastembed resolves DEFAULT_MODEL_NAME to, and every file a
# fastembed load of it reads. Its revision is pinned in
# ``onnx_runtime._PINNED_REVISIONS`` (HEADROOM_HF_PIN=off floats it).
# ``onnx_runtime._PINNED_REVISIONS`` (HEADROOM_HF_PIN=off floats it). The Rust
# scorer loads the same snapshot: keep both equal to DEFAULT_MODEL_REPO and
# DEFAULT_MODEL_FILES in crates/headroom-core/src/relevance/embedding.rs.
DEFAULT_MODEL_REPO = "Qdrant/bge-small-en-v1.5-onnx-Q"
DEFAULT_MODEL_FILES = (
"model_optimized.onnx",
@@ -182,9 +188,9 @@ def _resolve_default_model_snapshot() -> str:
here through ``hf_hub_download_local_first`` instead, which applies the
pinned SHA, tries the cache first and guards the network fallback.
The files land in the standard HuggingFace hub cache rather than
fastembed's default under the system temp dir, which macOS purges after a
few idle days.
The files land in the standard HuggingFace hub cache, which the Rust
scorer searches too, rather than fastembed's default under the system temp
dir, which macOS purges after a few idle days.
Raises:
OfflineEgressBlocked: ``HEADROOM_OFFLINE`` is set and a file is not cached.
+4 -2
View File
@@ -177,8 +177,10 @@ pytorch-mps = [
]
# Semantic relevance scoring with embeddings.
# Uses `fastembed` (BAAI/bge-small-en-v1.5 by default — 33M params,
# 384 dims, ~67 MB quantized ONNX), loaded from a pinned
# Qdrant/bge-small-en-v1.5-onnx-Q snapshot. Replaced sentence-transformers in
# 384 dims, ~67 MB quantized ONNX). The Rust scorer (`fastembed` crate)
# loads the same pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, so both
# languages run the same ONNX file; tests/parity/fixtures/embedding/
# bounds the remaining ONNX Runtime drift. Replaced sentence-transformers in
# Stage 3c.1 — fastembed is faster (~2-3x), smaller (no torch
# dependency), and outranks all-MiniLM-L6-v2 on MTEB by ~6 points.
# >=0.6.0 for `specific_model_path`, which the pinned-snapshot load needs.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,279 @@
{
"description": "Cases both languages are checked against so Rust's hf_hub_cache_dir() names the directory huggingface_hub would. tests/test_fastembed_pinned_snapshot.py checks them against Python's posixpath/ntpath and huggingface_hub.constants; crates/headroom-core/src/relevance/embedding.rs checks its port against the same file.",
"expandvars": {
"comment": "posix = posixpath.expandvars, nt = ntpath.expandvars, with env set and unset absent (identical on CPython 3.10, 3.11, 3.13).",
"env": {
"HEADROOM_EV_A": "x",
"HEADROOM_EV_B_C": "y$HEADROOM_EV_A",
"HEADROOM_EV_D-E": "z"
},
"unset": "HEADROOM_EV_UNSET",
"cases": [
{
"input": "plain/path",
"posix": "plain/path",
"nt": "plain/path"
},
{
"input": "$HEADROOM_EV_A",
"posix": "x",
"nt": "x"
},
{
"input": "${HEADROOM_EV_A}",
"posix": "x",
"nt": "x"
},
{
"input": "$HEADROOM_EV_A/$HEADROOM_EV_B_C",
"posix": "x/y$HEADROOM_EV_A",
"nt": "x/y$HEADROOM_EV_A"
},
{
"input": "$$HEADROOM_EV_A",
"posix": "$x",
"nt": "$HEADROOM_EV_A"
},
{
"input": "$",
"posix": "$",
"nt": "$"
},
{
"input": "a$",
"posix": "a$",
"nt": "a$"
},
{
"input": "$/x",
"posix": "$/x",
"nt": "$/x"
},
{
"input": "${HEADROOM_EV_A",
"posix": "${HEADROOM_EV_A",
"nt": "${HEADROOM_EV_A"
},
{
"input": "${}",
"posix": "${}",
"nt": "${}"
},
{
"input": "$HEADROOM_EV_UNSET/x",
"posix": "$HEADROOM_EV_UNSET/x",
"nt": "$HEADROOM_EV_UNSET/x"
},
{
"input": "${HEADROOM_EV_UNSET}/x",
"posix": "${HEADROOM_EV_UNSET}/x",
"nt": "${HEADROOM_EV_UNSET}/x"
},
{
"input": "%HEADROOM_EV_A%/x",
"posix": "%HEADROOM_EV_A%/x",
"nt": "x/x"
},
{
"input": "%%HEADROOM_EV_A%%",
"posix": "%%HEADROOM_EV_A%%",
"nt": "%HEADROOM_EV_A%"
},
{
"input": "%HEADROOM_EV_A",
"posix": "%HEADROOM_EV_A",
"nt": "%HEADROOM_EV_A"
},
{
"input": "%HEADROOM_EV_UNSET%/x",
"posix": "%HEADROOM_EV_UNSET%/x",
"nt": "%HEADROOM_EV_UNSET%/x"
},
{
"input": "'$HEADROOM_EV_A'/$HEADROOM_EV_A",
"posix": "'x'/x",
"nt": "'$HEADROOM_EV_A'/x"
},
{
"input": "'$HEADROOM_EV_A",
"posix": "'x",
"nt": "'$HEADROOM_EV_A"
},
{
"input": "$HEADROOM_EV_D-E",
"posix": "$HEADROOM_EV_D-E",
"nt": "z"
},
{
"input": "${HEADROOM_EV_D-E}",
"posix": "z",
"nt": "z"
},
{
"input": "$HEADROOM_EV_A-x",
"posix": "x-x",
"nt": "$HEADROOM_EV_A-x"
},
{
"input": "~$HEADROOM_EV_A",
"posix": "~x",
"nt": "~x"
},
{
"input": "é/$HEADROOM_EV_A/é",
"posix": "é/x/é",
"nt": "é/x/é"
}
]
},
"cache_dir": {
"comment": "{root} and {home} are temporary directories; HOME and USERPROFILE are set to {home}, every other cache variable is cleared first. expected may be {posix, nt}.",
"probe_vars": [
"HEADROOM_HF_PROBE_ROOT",
"HEADROOM_HF_PROBE_UNSET"
],
"cases": [
{
"name": "HF_HUB_CACHE through ${VAR} (the #4190 review case)",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HF_HUB_CACHE": "${HEADROOM_HF_PROBE_ROOT}/hub"
},
"expected": "{root}/hub"
},
{
"name": "HF_HUB_CACHE through $VAR",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HF_HUB_CACHE": "$HEADROOM_HF_PROBE_ROOT/hub"
},
"expected": "{root}/hub"
},
{
"name": "legacy HUGGINGFACE_HUB_CACHE through ${VAR}",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HUGGINGFACE_HUB_CACHE": "${HEADROOM_HF_PROBE_ROOT}/legacy"
},
"expected": "{root}/legacy"
},
{
"name": "HF_HOME through ${VAR}",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HF_HOME": "${HEADROOM_HF_PROBE_ROOT}/hf-home"
},
"expected": "{root}/hf-home/hub"
},
{
"name": "XDG_CACHE_HOME through ${VAR}",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"XDG_CACHE_HOME": "${HEADROOM_HF_PROBE_ROOT}/xdg"
},
"expected": "{root}/xdg/huggingface/hub"
},
{
"name": "HF_HUB_CACHE still wins over an expanded HF_HOME",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HF_HOME": "${HEADROOM_HF_PROBE_ROOT}/hf-home",
"HF_HUB_CACHE": "{root}/explicit"
},
"expected": "{root}/explicit"
},
{
"name": "an unset variable stays literal",
"env": {
"HF_HUB_CACHE": "{root}/${HEADROOM_HF_PROBE_UNSET}/hub"
},
"expected": "{root}/${HEADROOM_HF_PROBE_UNSET}/hub"
},
{
"name": "leading ~ in HF_HUB_CACHE",
"env": {
"HF_HUB_CACHE": "~/models"
},
"expected": "{home}/models"
},
{
"name": "leading ~ in HF_HOME",
"env": {
"HF_HOME": "~/hf"
},
"expected": "{home}/hf/hub"
},
{
"name": "a ~ that a variable produces in HF_HUB_CACHE stays literal (expanduser runs first)",
"env": {
"HEADROOM_HF_PROBE_ROOT": "~/r",
"HF_HUB_CACHE": "${HEADROOM_HF_PROBE_ROOT}/hub"
},
"expected": "~/r/hub"
},
{
"name": "a ~ that a variable produces in HF_HOME expands on the second pass",
"env": {
"HEADROOM_HF_PROBE_ROOT": "~/r",
"HF_HOME": "${HEADROOM_HF_PROBE_ROOT}/hf"
},
"expected": "{home}/r/hf/hub"
},
{
"name": "%VAR% is a variable on Windows only",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HF_HUB_CACHE": "%HEADROOM_HF_PROBE_ROOT%/hub"
},
"expected": {
"posix": "%HEADROOM_HF_PROBE_ROOT%/hub",
"nt": "{root}/hub"
}
},
{
"name": "an empty HF_HUB_CACHE is the working directory, not HF_HOME (the #4438 review case)",
"env": {
"HF_HUB_CACHE": "",
"HF_HOME": "{root}/fallback-home"
},
"expected": ""
},
{
"name": "an empty legacy HUGGINGFACE_HUB_CACHE is the working directory",
"env": {
"HUGGINGFACE_HUB_CACHE": "",
"HF_HOME": "{root}/fallback-home"
},
"expected": ""
},
{
"name": "an empty HF_HUB_CACHE still wins over a set legacy HUGGINGFACE_HUB_CACHE",
"env": {
"HF_HUB_CACHE": "",
"HUGGINGFACE_HUB_CACHE": "{root}/legacy"
},
"expected": ""
},
{
"name": "an empty HF_HOME is hub under the working directory, not XDG_CACHE_HOME",
"env": {
"HF_HOME": "",
"XDG_CACHE_HOME": "{root}/xdg"
},
"expected": "hub"
},
{
"name": "an empty XDG_CACHE_HOME is huggingface/hub under the working directory, not the home default",
"env": {
"XDG_CACHE_HOME": ""
},
"expected": "huggingface/hub"
},
{
"name": "nothing set: the default under the home directory",
"env": {},
"expected": "{home}/.cache/huggingface/hub"
}
]
}
}
+85
View File
@@ -0,0 +1,85 @@
"""Record the embedding parity fixture for the default relevance model.
Embeds a fixed set of texts with the Python ``EmbeddingScorer``, which loads
the pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, and writes them to
``fixtures/embedding/bge_small_en_v15_pinned.json``. Two gated tests compare
against it: ``tests/test_fastembed_pinned_snapshot.py`` (Python) and
``fastembed_matches_parity_fixture`` in
``crates/headroom-core/src/relevance/embedding.rs`` (Rust). Both reading one
fixture bounds the drift between the languages by ``MAX_ABS_DIFF``.
The languages run the same ONNX file, so the gap between them comes from the
ONNX Runtime build, not the pipeline. Measured when this fixture was recorded:
Rust and Python on the same ORT (1.31.0) agree within 1e-6 per component;
Python on ORT 1.23.2 vs 1.31.0 differs by up to 4.1e-4. ``MAX_ABS_DIFF``
covers the ORT range the project supports with ~2x headroom.
This bounds numeric drift; it does not identify the model. Xenova's fp32
export of the same model sits within 4.7e-4 of this fixture, inside the
tolerance. Which files load is guaranteed by the pinned SHA and the unit tests
on both sides, not by this fixture.
Re-record after bumping the pinned revision (needs network on a cold cache),
with an interpreter whose onnxruntime is >= 1.24, the runtime the Rust side
needs:
python tests/parity/record_embedding.py
"""
from __future__ import annotations
import json
import os
from headroom.onnx_runtime import _PINNED_REVISIONS
from headroom.relevance.embedding import DEFAULT_MODEL_NAME, DEFAULT_MODEL_REPO, EmbeddingScorer
FIXTURE_PATH = os.path.join(
os.path.dirname(__file__), "fixtures", "embedding", "bge_small_en_v15_pinned.json"
)
# Largest allowed |embedding component - fixture|, on either side. See the
# module docstring for how it was chosen.
MAX_ABS_DIFF = 1e-3
TEXTS = [
"login error",
"authentication failed for user admin: invalid credentials",
'{"status": "failed", "error": "connection refused", "retries": 3}',
"def retry(fn, attempts=3):\n for i in range(attempts):\n return fn()",
"Ünïcödé text, emoji 🚀 and CJK 中文 mixed in one line.",
# Longer than the 512-token window, so truncation is covered too.
" ".join(f"log line {i}: request handled in {i % 97} ms" for i in range(400)),
]
def write_fixture(fixture: dict[str, object], path: str = FIXTURE_PATH) -> None:
"""Write ``fixture`` with each list item, so each embedding row, on one line."""
lines = []
for key, value in fixture.items():
if isinstance(value, list):
items = ",\n".join(f" {json.dumps(item, ensure_ascii=False)}" for item in value)
lines.append(f" {json.dumps(key)}: [\n{items}\n ]")
else:
lines.append(f" {json.dumps(key)}: {json.dumps(value, ensure_ascii=False)}")
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8", newline="\n") as f:
f.write("{\n" + ",\n".join(lines) + "\n}\n")
def main() -> None:
scorer = EmbeddingScorer(DEFAULT_MODEL_NAME)
embeddings = scorer._encode(TEXTS)
fixture = {
"model": DEFAULT_MODEL_NAME,
"repo": DEFAULT_MODEL_REPO,
"revision": _PINNED_REVISIONS[DEFAULT_MODEL_REPO],
"max_abs_diff": MAX_ABS_DIFF,
"texts": TEXTS,
"embeddings": [[round(float(v), 8) for v in row] for row in embeddings],
}
write_fixture(fixture)
print(f"wrote {FIXTURE_PATH} ({len(TEXTS)} texts x {len(embeddings[0])} dims)")
if __name__ == "__main__":
main()
+120 -3
View File
@@ -1,15 +1,17 @@
"""The default relevance embedder loads one pinned snapshot.
"""The default relevance embedder loads one pinned snapshot, in both languages.
fastembed's ``TextEmbedding`` accepts ``revision`` but drops it before
``snapshot_download``, so the old pin never reached the Hub and the model
floated on ``main``. These tests hold the replacement in place: the pinned
SHA reaches every file download and fastembed only gets the resolved
directory.
SHA reaches every file download, fastembed only gets the resolved directory,
and the Rust scorer names the same repo, SHA and files.
"""
from __future__ import annotations
import json
import os
import re
import sys
from pathlib import Path
from types import SimpleNamespace
@@ -26,6 +28,12 @@ from headroom.relevance.embedding import (
EmbeddingScorer,
)
REPO_ROOT = Path(__file__).resolve().parents[1]
RUST_EMBEDDING = REPO_ROOT / "crates/headroom-core/src/relevance/embedding.rs"
PARITY_FIXTURE = REPO_ROOT / "tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json"
HF_CACHE_FIXTURE = json.loads(
(REPO_ROOT / "tests/parity/fixtures/hf_hub_cache/cases.json").read_text(encoding="utf-8")
)
PIN = _PINNED_REVISIONS[DEFAULT_MODEL_REPO]
@@ -171,3 +179,112 @@ def test_fastembed_loads_specific_model_path_without_a_download(monkeypatch, tmp
lazy_load=True,
)
assert Path(model.model._model_dir) == tmp_path
def _rust_const(name: str) -> str:
match = re.search(rf'const {name}: &str = "([^"]+)";', RUST_EMBEDDING.read_text())
assert match, f"{name} not found in {RUST_EMBEDDING}"
return match.group(1)
def test_rust_loads_the_same_snapshot():
assert _rust_const("DEFAULT_MODEL_REPO") == DEFAULT_MODEL_REPO
assert _rust_const("DEFAULT_MODEL_REVISION") == PIN
assert _rust_const("DEFAULT_MODEL_FILE") == DEFAULT_MODEL_FILES[0]
files = re.search(
r"const DEFAULT_MODEL_FILES: \[&str; \d+\] = \[(.*?)\];",
RUST_EMBEDDING.read_text(),
re.S,
)
assert files, "DEFAULT_MODEL_FILES not found"
entries = [part.strip() for part in files.group(1).split(",") if part.strip()]
assert entries == ["DEFAULT_MODEL_FILE", *(f'"{name}"' for name in DEFAULT_MODEL_FILES[1:])]
def test_parity_fixture_was_recorded_at_the_pin():
fixture = json.loads(PARITY_FIXTURE.read_text(encoding="utf-8"))
assert fixture["repo"] == DEFAULT_MODEL_REPO
assert fixture["revision"] == PIN
assert len(fixture["embeddings"]) == len(fixture["texts"])
@pytest.mark.skipif(
not os.environ.get("RUN_FASTEMBED_TESTS"),
reason="loads the real model (~67 MB download on a cold cache); set RUN_FASTEMBED_TESTS=1",
)
def test_python_matches_parity_fixture():
"""Python half of the cross-language check; the Rust half is
``fastembed_matches_parity_fixture`` in the Rust scorer."""
np = pytest.importorskip("numpy")
pytest.importorskip("fastembed")
fixture = json.loads(PARITY_FIXTURE.read_text(encoding="utf-8"))
got = EmbeddingScorer()._encode(fixture["texts"])
for text, row, expected in zip(fixture["texts"], got, fixture["embeddings"]):
worst = float(np.abs(np.asarray(row, dtype=np.float64) - np.asarray(expected)).max())
assert worst <= fixture["max_abs_diff"], f"{text!r}: max |python - fixture| = {worst:e}"
# ── the shared HuggingFace cache directory ──────────────────────────────────
#
# Rust's ``hf_hub_cache_dir`` re-implements how ``huggingface_hub`` picks the
# hub cache, so both languages look up and download into one directory. These
# check the shared fixture against Python itself; the Rust tests
# ``expand_vars_matches_python_posixpath_and_ntpath`` and
# ``hf_hub_cache_dir_matches_the_shared_fixture`` check the port against the
# same file.
_CACHE_VARS = ("HF_HUB_CACHE", "HUGGINGFACE_HUB_CACHE", "HF_HOME", "XDG_CACHE_HOME")
@pytest.mark.parametrize(
"case", HF_CACHE_FIXTURE["expandvars"]["cases"], ids=lambda case: case["input"]
)
def test_expandvars_fixture_matches_posixpath_and_ntpath(
case: dict[str, str], monkeypatch: pytest.MonkeyPatch
) -> None:
import ntpath
import posixpath
for name, value in HF_CACHE_FIXTURE["expandvars"]["env"].items():
monkeypatch.setenv(name, value)
monkeypatch.delenv(HF_CACHE_FIXTURE["expandvars"]["unset"], raising=False)
assert posixpath.expandvars(case["input"]) == case["posix"]
assert ntpath.expandvars(case["input"]) == case["nt"]
@pytest.mark.parametrize(
"case", HF_CACHE_FIXTURE["cache_dir"]["cases"], ids=lambda case: case["name"]
)
def test_cache_dir_fixture_matches_huggingface_hub(case: dict, tmp_path: Path) -> None:
"""``huggingface_hub`` reads these variables once, at import, so each case
runs in a fresh interpreter."""
import subprocess
pytest.importorskip("huggingface_hub")
root, home = tmp_path / "root", tmp_path / "home"
def fill(text: str) -> str:
return text.replace("{root}", str(root)).replace("{home}", str(home))
probe_vars = HF_CACHE_FIXTURE["cache_dir"]["probe_vars"]
env = {k: v for k, v in os.environ.items() if k not in (*_CACHE_VARS, *probe_vars)}
env["HOME"] = env["USERPROFILE"] = str(home)
env.update({name: fill(value) for name, value in case["env"].items()})
result = subprocess.run(
[
sys.executable,
"-c",
"import sys, huggingface_hub.constants as c; sys.stdout.write(c.HF_HUB_CACHE)",
],
capture_output=True,
text=True,
env=env,
timeout=60,
check=True,
)
expected = case["expected"]
if isinstance(expected, dict):
expected = expected["nt" if os.name == "nt" else "posix"]
assert Path(result.stdout) == Path(fill(expected))
+17 -3
View File
@@ -3205,8 +3205,8 @@ class TestDocsMatchTheGuarantee:
_RUST_EGRESS_PATTERNS = re.compile(
r"""
hf_hub::api::(?:sync|tokio)::Api(?:Builder)?::new\(
| (?<![\w:])Api(?:Builder)?::new\(
hf_hub::api::(?:sync|tokio)::Api(?:Builder)?::(?:new|from_env|from_cache)\(
| (?<![\w:])Api(?:Builder)?::(?:new|from_env|from_cache)\(
| (?<![\w:])TextEmbedding::try_new\w*\(
| (?<![\w:])reqwest::(?:Client::(?:new|builder)|get|post)\(
| (?<![\w:])ureq::(?:agent|builder|get|post|put|delete|request)\(
@@ -3220,6 +3220,14 @@ _RUST_FN = re.compile(
)
_RUST_ALLOWLIST: dict[str, tuple[int, str]] = {
"headroom-core/src/relevance/embedding.rs": (
1,
"cache-only: TextEmbedding::try_new_from_user_defined in load_snapshot "
"builds the ONNX session from bytes already read off disk and has no "
"Hub client at all. The files reach the disk through "
"default_model_repo, whose ApiBuilder is guarded, and only on a cache "
"miss, so a pre-seeded air-gapped host still loads.",
),
"headroom-proxy/src/proxy.rs": (
1,
"user-traffic: the reqwest client the Rust proxy forwards the caller's "
@@ -3326,7 +3334,13 @@ class TestRustEgressChokepointCoverage:
"headroom-core/src/relevance/embedding.rs",
):
assert relpath in sites, f"{relpath} has no detected egress site at all"
assert all(site.guarded for site in sites[relpath]), (
# Every site is guarded, except the cache-only loads the allowlist
# counts for the file (embedding.rs: the user-defined model load).
unguarded = [site for site in sites[relpath] if not site.guarded]
assert len(unguarded) < len(sites[relpath]), (
f"{relpath} has no egress site the Rust sweep sees as guarded"
)
assert len(unguarded) == _RUST_ALLOWLIST.get(relpath, (0, ""))[0], (
f"{relpath} has an egress site the Rust sweep does not see as guarded"
)