mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(relevance): load the Rust default embedder from Python's pinned snapshot (#4190)
## Description Parent status: #4152 has merged. The pinned-loader and batch-cap merge resolution was reviewed ate3a495cbc5. The original split/stack history is retained below. **Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now four linear commits on top of main, with no merge commits: - `bfae4ade` fix(relevance): load the Rust default embedder from Python's pinned snapshot - `6fbae380` fix(relevance): look up and download the Rust embedder in one HF cache dir - `70afece8` fix(relevance): expand ~ and $VAR in the Rust HF cache path like huggingface_hub - `62b792f0` fix(relevance): keep an empty HF cache variable, as huggingface_hub does What the rebase changed: - The #4152 commit (`cb1c3a33`) is gone from the branch: main already has it as `f380692b`. Its files matched main; the only difference was `uv.lock`, where main's later `filelock>=4.2.1` entry was kept. - The six main-sync merges (`185dc9b7`, `052977b9`, `029c72ac`, `4405b12c`, `8b0a10e8`, and the reviewed `e3a495cb`) are linearized away, so those SHAs are no longer on the branch. - `embedding.rs` conflicted again with main's batch-size cap (#4507). I resolved it to the same content as the reviewed merge: every file this PR touches is byte-identical between `e3a495cb` and the new head `62b792f0` (`git diffe3a495cb62b792f0-- <files changed vs main>` is empty). The review and qualification below still apply to the content; only the history and SHAs changed. Rust `EmbeddingScorer::try_new` used `EmbeddingModel::BGESmallENV15`. fastembed-rs maps that to a different repo and file from what Python runs: `Xenova/bge-small-en-v1.5` `onnx/model.onnx` (fp32), always on `main`. So the two languages never ran the same ONNX file, and the "byte-equal across languages" claim in both modules was wrong. This PR makes `try_new` load the same pinned `Qdrant/bge-small-en-v1.5-onnx-Q` snapshot that #4152 makes Python load. It adds a cross-language parity fixture checked from both sides, and replaces the parity claim with measured numbers. Nothing in production calls `EmbeddingScorer::try_new` today: `create_scorer` and `HybridScorer` use `default()`, which never loads a model. This keeps the Rust scorer correct and pinned for when it is wired in. No linked issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made (This PR's own commits only.) - `crates/headroom-core/src/relevance/embedding.rs`: - `try_new` loads the snapshot through `TextEmbedding::try_new_from_user_defined`, with CLS pooling and static quantization, which is what fastembed-rs uses for `BGESmallENV15Q`. - It uses exactly one hub cache directory, resolved like Python's `huggingface_hub.constants.HF_HUB_CACHE`: `HF_HUB_CACHE` → `HUGGINGFACE_HUB_CACHE` → `HF_HOME/hub` → `XDG_CACHE_HOME/huggingface/hub` → `~/.cache/huggingface/hub`, with a leading `~` expanded. It looks up and downloads only there, so a stale snapshot in a lower-priority cache is never used. The home directory comes from `std::env::home_dir`, which falls back to the user database when `HOME` is unset, as Python's `expanduser` does. - It finds the snapshot as `snapshots/<sha>` in that directory. `hf_hub::Cache` resolves through `refs/<rev>`, which Python's `huggingface_hub` doesn't write for a commit SHA. - On a miss it downloads with `Repo::with_revision`. `guard_egress` sits in front of the Hub client and runs only on a cache miss, so a pre-seeded air-gapped host now loads; previously Rust refused even with a warm cache. - `HEADROOM_HF_PIN=off` floats it on `main`, with the same values as Python's `_resolve_revision`. - `try_new_with_model` is unchanged. - Parity fixture: `tests/parity/record_embedding.py` and `tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json` (6 texts × 384 dims, recorded on ORT 1.31.0). The fixture is written one embedding row per line, so it's 22 lines. Python checks it in CI (`test-extras` sets `RUN_FASTEMBED_TESTS=1`); Rust checks it under `RUN_FASTEMBED_TESTS=1`. - `tests/test_fastembed_pinned_snapshot.py`: the Rust constants name the same repo, SHA and files as Python; the fixture was recorded at the pin; plus the gated Python parity check. - 10 new Rust unit tests in `embedding.rs`: the lookup ignores other commits and needs every file; the download URL requests the pinned revision; the offline guard applies on a miss only; `HEADROOM_HF_PIN=off` floats; the cache directory follows `huggingface_hub`'s order variable by variable (including an empty variable and `~`); with no `HOME` it still resolves through the user database; and the review case: an empty explicit `HF_HUB_CACHE` next to a complete pinned snapshot and an older `refs/main` under `HOME` finds neither, for the pinned revision or `main`, while both are found once `HF_HUB_CACHE` is unset. - `tests/test_offline_egress_chokepoint.py`: the Rust sweep's regex now also matches `ApiBuilder::from_cache` / `from_env`. It missed the new download client, which is guarded; it flags nothing else in `crates/`. The user-defined model load, which only reads bytes already on disk, is allowlisted as `cache-only`. - Docs: the "byte-equal" wording in both embedder modules, `pyproject.toml` and `onnx_runtime.py` is replaced with measured numbers and the cross-language keep-equal notes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output After the rebase, on `62b792f0` (macOS arm64, no `ORT_DYLIB_PATH`, so the gated model tests skip): ```text $ cargo test -p headroom-core --lib test result: ok. 991 passed; 0 failed; 1 ignored $ cargo clippy -p headroom-core --all-targets -- -D warnings # clean $ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings # clean $ cargo fmt --all -- --check # clean $ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_hf_revision_pinning.py 235 passed, 3 skipped $ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2) # clean ``` The 7 offline-egress failures noted below no longer reproduce here. Before the rebase: ```text $ cargo test -p headroom-core --lib test result: ok. 985 passed; 0 failed; 1 ignored # see the note on #4151 below $ cargo clippy -p headroom-core --all-targets -- -D warnings # clean $ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings # clean $ cargo fmt --all -- --check # clean $ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.31.0 dylib> cargo test -p headroom-core --lib relevance::embedding test relevance::embedding::tests::fastembed_matches_parity_fixture ... ok test result: ok. 24 passed; 0 failed $ RUN_FASTEMBED_TESTS=1 pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_relevance.py \ tests/test_stateless_writers.py tests/test_hf_revision_pinning.py 71 passed $ pytest -q tests/test_offline_egress_chokepoint.py 7 failed, 132 passed, 3 skipped # The 7 are the OTel / Langfuse / Headroom Cloud offline tests, whose extras aren't in this venv; # the same 7 fail on origin/main here. The Rust sweep tests pass. ``` **Note on #4151:** with `ORT_DYLIB_PATH` set on macOS, a full `cargo test -p headroom-core --lib` fails `tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error` most of the time (5/5 on `afdb60d3`, before the cache change too). `a_cached_snapshot_is_not_refused_offline` holds `HEADROOM_OFFLINE=1` under the env lock while ONNX Runtime loads, which is slow there, and the tokenizer test reads the variable without the lock. That's the race #4151 fixes; with its commit applied, 6/6 full runs pass. Linux CI passes either way. The cache-precedence test was checked against the bug: with the lookup changed back to also scanning `~/.cache`, it fails on the first case (pinned revision, empty `HF_HUB_CACHE`). ## Real Behavior Proof - Exact command / steps: additional maintainer verification on the main-synced head: cargo test -p headroom-core --locked hf_hub_cache_dir; cargo test -p headroom-core --locked expand_vars_matches_python_posixpath_and_ntpath; python -m pytest tests/test_fastembed_pinned_snapshot.py -q; repeat its fixture cases with huggingface-hub==2.2.0. - Observed result: three Rust tests passed across both POSIX and Windows expansion fixtures and cache-directory selection; Python passed 46 tests with one gated skip using Hugging Face Hub 1.16.1, and all 36 shared fixture cases passed separately with locked Hub 2.2.0. - Environment: macOS arm64; Rust fastembed 5.17.3 with hf-hub 0.5.0; repo `.venv` (fastembed 0.8.0, onnxruntime 1.23.2) and a second interpreter on onnxruntime 1.31.0. - A cold-cache Rust load with `HF_HUB_CACHE=<empty dir>` wrote `snapshots/52398278…` plus `refs/52398278…` and passed the parity check. Per-component max |Δ| against the fixture: - Rust vs Python on the same ORT (1.31.0): ≤ 1e-6. - Python on ORT 1.23.2 vs 1.31.0: ≤ 4.1e-4. - The fixture's tolerance is 1e-3. - For reference, Xenova's fp32 export (what Rust used to load) is ≤ 4.7e-4 from the fixture. The old cross-language gap was real but numerically small, and this fixture bounds drift; it cannot tell the two exports apart. Model identity is enforced by the pinned SHA and the unit tests. - The fixture's new one-row-per-line layout was re-emitted from the recorded data, not re-recorded: the parsed JSON is identical. - Not tested: Windows, and Linux outside CI. Rust's gated tests in CI (they need the model and an ORT dylib, so they stay manual behind `RUN_FASTEMBED_TESTS=1`). Rust `HybridScorer` thresholds against the Qdrant export. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: no production path. `EmbeddingScorer::try_new` loads the Qdrant export instead of Xenova's, and now loads offline from a warm cache instead of refusing, but nothing in production calls it today. - Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes - Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches are still unpinned (there's no Rust `HEADROOM_HF_PIN`), and Rust's gated embedding tests could get a CI job once one ships an ORT dylib. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4190" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Maintainer qualification The environment-reference mismatch from the previous review is addressed. The shared corpus covers the reported explicit-cache case, platform-specific variable syntax, and the two-stage home/hub expansion ordering. The Rust test runs both POSIX and Windows expansion rules; Python validates the same corpus against posixpath, ntpath, and fresh huggingface_hub.constants imports. A clean current-main sync was published as 052977b9e88ae3bbda96797615b64ee350a2f695; it does not change the contributor's resolver or fixture files. Qualification used Windows, Hugging Face Hub 1.16.1 and an isolated locked 2.2.0 target. No model download or numerical embedding-parity result is claimed by these path tests; the pre-existing gated numerical-parity test remained skipped. Hosted checks on the pushed head must finish before approval. No contributor CI action requested or PR merged. --------- Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Kenneth Wong
JD Davis
parent
23a5bb1c81
commit
9bb4b130ff
@@ -489,9 +489,11 @@ jobs:
|
||||
- name: Run relevance tests
|
||||
# Offline so fastembed reads the cache the prefetch step just warmed,
|
||||
# without an unauthenticated cache-validation HEAD that could 429.
|
||||
# RUN_FASTEMBED_TESTS runs the Python half of the embedding parity check.
|
||||
env:
|
||||
HF_HUB_OFFLINE: "1"
|
||||
TRANSFORMERS_OFFLINE: "1"
|
||||
RUN_FASTEMBED_TESTS: "1"
|
||||
run: pytest tests/test_relevance.py tests/test_fastembed_pinned_snapshot.py -v
|
||||
|
||||
- name: Run HTML extraction and tool reversibility tests
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -82,7 +82,10 @@ _PINNED_REVISIONS: dict[str, str] = {
|
||||
# Third-party repo — pinning matters most here.
|
||||
"Qdrant/all-MiniLM-L6-v2-onnx": "5f1b8cd78bc4fb444dd171e59b18f3a3af89a079",
|
||||
# Default relevance embedder (BAAI/bge-small-en-v1.5), third-party too.
|
||||
# relevance/embedding.py downloads each of its files at this SHA.
|
||||
# relevance/embedding.py downloads each of its files at this SHA. The Rust
|
||||
# scorer loads the same snapshot: keep this SHA equal to
|
||||
# DEFAULT_MODEL_REVISION in crates/headroom-core/src/relevance/embedding.rs
|
||||
# (tests/test_fastembed_pinned_snapshot.py enforces it).
|
||||
"Qdrant/bge-small-en-v1.5-onnx-Q": "52398278842ec682c6f32300af41344b1c0b0bb2",
|
||||
}
|
||||
|
||||
|
||||
@@ -2,8 +2,12 @@
|
||||
|
||||
This module provides semantic relevance scoring using `fastembed`
|
||||
(BAAI/bge-small-en-v1.5 by default — 33M params, 384 dims, ~67 MB
|
||||
quantized ONNX), loaded from a pinned Qdrant/bge-small-en-v1.5-onnx-Q
|
||||
snapshot in the HuggingFace hub cache.
|
||||
quantized ONNX). The Rust scorer (fastembed-rs crate) loads the same
|
||||
pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, so both languages run
|
||||
the same ONNX file and tokenizer. On the same ONNX Runtime build the two
|
||||
agree within 1e-6 per component; across ORT versions they drift by up to
|
||||
~4e-4, so they are close, not byte-equal. The gated parity fixture in
|
||||
tests/parity/fixtures/embedding/ bounds that drift.
|
||||
|
||||
Key features:
|
||||
- Semantic understanding ("errors" matches "failed", "issues")
|
||||
@@ -16,8 +20,8 @@ Install with: pip install "headroom-ai[relevance]"
|
||||
|
||||
History: this module previously wrapped `sentence-transformers`
|
||||
(PyTorch). Switched to fastembed in Stage 3c.1 of the Rust port to:
|
||||
1. Match the Rust embedding scorer byte-for-byte (both call into
|
||||
ONNX Runtime over the identical ONNX file).
|
||||
1. Run the same ONNX model as the Rust embedding scorer (both call
|
||||
into ONNX Runtime over the identical pinned ONNX file).
|
||||
2. Remove the torch dependency from the relevance/ path
|
||||
(Phase 6: "drop torch from Python").
|
||||
3. Get a better default model (bge-small-en-v1.5 vs MiniLM-L6-v2).
|
||||
@@ -157,12 +161,14 @@ def _load_text_embedding(kwargs: dict[str, Any]) -> TextEmbedding:
|
||||
return TextEmbedding(**kwargs)
|
||||
|
||||
|
||||
# Default model name. Same string used by the Rust embedding scorer.
|
||||
# Default model name. Same model the Rust embedding scorer loads.
|
||||
DEFAULT_MODEL_NAME = "BAAI/bge-small-en-v1.5"
|
||||
|
||||
# The HF repo fastembed resolves DEFAULT_MODEL_NAME to, and every file a
|
||||
# fastembed load of it reads. Its revision is pinned in
|
||||
# ``onnx_runtime._PINNED_REVISIONS`` (HEADROOM_HF_PIN=off floats it).
|
||||
# ``onnx_runtime._PINNED_REVISIONS`` (HEADROOM_HF_PIN=off floats it). The Rust
|
||||
# scorer loads the same snapshot: keep both equal to DEFAULT_MODEL_REPO and
|
||||
# DEFAULT_MODEL_FILES in crates/headroom-core/src/relevance/embedding.rs.
|
||||
DEFAULT_MODEL_REPO = "Qdrant/bge-small-en-v1.5-onnx-Q"
|
||||
DEFAULT_MODEL_FILES = (
|
||||
"model_optimized.onnx",
|
||||
@@ -182,9 +188,9 @@ def _resolve_default_model_snapshot() -> str:
|
||||
here through ``hf_hub_download_local_first`` instead, which applies the
|
||||
pinned SHA, tries the cache first and guards the network fallback.
|
||||
|
||||
The files land in the standard HuggingFace hub cache rather than
|
||||
fastembed's default under the system temp dir, which macOS purges after a
|
||||
few idle days.
|
||||
The files land in the standard HuggingFace hub cache, which the Rust
|
||||
scorer searches too, rather than fastembed's default under the system temp
|
||||
dir, which macOS purges after a few idle days.
|
||||
|
||||
Raises:
|
||||
OfflineEgressBlocked: ``HEADROOM_OFFLINE`` is set and a file is not cached.
|
||||
|
||||
+4
-2
@@ -177,8 +177,10 @@ pytorch-mps = [
|
||||
]
|
||||
# Semantic relevance scoring with embeddings.
|
||||
# Uses `fastembed` (BAAI/bge-small-en-v1.5 by default — 33M params,
|
||||
# 384 dims, ~67 MB quantized ONNX), loaded from a pinned
|
||||
# Qdrant/bge-small-en-v1.5-onnx-Q snapshot. Replaced sentence-transformers in
|
||||
# 384 dims, ~67 MB quantized ONNX). The Rust scorer (`fastembed` crate)
|
||||
# loads the same pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, so both
|
||||
# languages run the same ONNX file; tests/parity/fixtures/embedding/
|
||||
# bounds the remaining ONNX Runtime drift. Replaced sentence-transformers in
|
||||
# Stage 3c.1 — fastembed is faster (~2-3x), smaller (no torch
|
||||
# dependency), and outranks all-MiniLM-L6-v2 on MTEB by ~6 points.
|
||||
# >=0.6.0 for `specific_model_path`, which the pinned-snapshot load needs.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,279 @@
|
||||
{
|
||||
"description": "Cases both languages are checked against so Rust's hf_hub_cache_dir() names the directory huggingface_hub would. tests/test_fastembed_pinned_snapshot.py checks them against Python's posixpath/ntpath and huggingface_hub.constants; crates/headroom-core/src/relevance/embedding.rs checks its port against the same file.",
|
||||
"expandvars": {
|
||||
"comment": "posix = posixpath.expandvars, nt = ntpath.expandvars, with env set and unset absent (identical on CPython 3.10, 3.11, 3.13).",
|
||||
"env": {
|
||||
"HEADROOM_EV_A": "x",
|
||||
"HEADROOM_EV_B_C": "y$HEADROOM_EV_A",
|
||||
"HEADROOM_EV_D-E": "z"
|
||||
},
|
||||
"unset": "HEADROOM_EV_UNSET",
|
||||
"cases": [
|
||||
{
|
||||
"input": "plain/path",
|
||||
"posix": "plain/path",
|
||||
"nt": "plain/path"
|
||||
},
|
||||
{
|
||||
"input": "$HEADROOM_EV_A",
|
||||
"posix": "x",
|
||||
"nt": "x"
|
||||
},
|
||||
{
|
||||
"input": "${HEADROOM_EV_A}",
|
||||
"posix": "x",
|
||||
"nt": "x"
|
||||
},
|
||||
{
|
||||
"input": "$HEADROOM_EV_A/$HEADROOM_EV_B_C",
|
||||
"posix": "x/y$HEADROOM_EV_A",
|
||||
"nt": "x/y$HEADROOM_EV_A"
|
||||
},
|
||||
{
|
||||
"input": "$$HEADROOM_EV_A",
|
||||
"posix": "$x",
|
||||
"nt": "$HEADROOM_EV_A"
|
||||
},
|
||||
{
|
||||
"input": "$",
|
||||
"posix": "$",
|
||||
"nt": "$"
|
||||
},
|
||||
{
|
||||
"input": "a$",
|
||||
"posix": "a$",
|
||||
"nt": "a$"
|
||||
},
|
||||
{
|
||||
"input": "$/x",
|
||||
"posix": "$/x",
|
||||
"nt": "$/x"
|
||||
},
|
||||
{
|
||||
"input": "${HEADROOM_EV_A",
|
||||
"posix": "${HEADROOM_EV_A",
|
||||
"nt": "${HEADROOM_EV_A"
|
||||
},
|
||||
{
|
||||
"input": "${}",
|
||||
"posix": "${}",
|
||||
"nt": "${}"
|
||||
},
|
||||
{
|
||||
"input": "$HEADROOM_EV_UNSET/x",
|
||||
"posix": "$HEADROOM_EV_UNSET/x",
|
||||
"nt": "$HEADROOM_EV_UNSET/x"
|
||||
},
|
||||
{
|
||||
"input": "${HEADROOM_EV_UNSET}/x",
|
||||
"posix": "${HEADROOM_EV_UNSET}/x",
|
||||
"nt": "${HEADROOM_EV_UNSET}/x"
|
||||
},
|
||||
{
|
||||
"input": "%HEADROOM_EV_A%/x",
|
||||
"posix": "%HEADROOM_EV_A%/x",
|
||||
"nt": "x/x"
|
||||
},
|
||||
{
|
||||
"input": "%%HEADROOM_EV_A%%",
|
||||
"posix": "%%HEADROOM_EV_A%%",
|
||||
"nt": "%HEADROOM_EV_A%"
|
||||
},
|
||||
{
|
||||
"input": "%HEADROOM_EV_A",
|
||||
"posix": "%HEADROOM_EV_A",
|
||||
"nt": "%HEADROOM_EV_A"
|
||||
},
|
||||
{
|
||||
"input": "%HEADROOM_EV_UNSET%/x",
|
||||
"posix": "%HEADROOM_EV_UNSET%/x",
|
||||
"nt": "%HEADROOM_EV_UNSET%/x"
|
||||
},
|
||||
{
|
||||
"input": "'$HEADROOM_EV_A'/$HEADROOM_EV_A",
|
||||
"posix": "'x'/x",
|
||||
"nt": "'$HEADROOM_EV_A'/x"
|
||||
},
|
||||
{
|
||||
"input": "'$HEADROOM_EV_A",
|
||||
"posix": "'x",
|
||||
"nt": "'$HEADROOM_EV_A"
|
||||
},
|
||||
{
|
||||
"input": "$HEADROOM_EV_D-E",
|
||||
"posix": "$HEADROOM_EV_D-E",
|
||||
"nt": "z"
|
||||
},
|
||||
{
|
||||
"input": "${HEADROOM_EV_D-E}",
|
||||
"posix": "z",
|
||||
"nt": "z"
|
||||
},
|
||||
{
|
||||
"input": "$HEADROOM_EV_A-x",
|
||||
"posix": "x-x",
|
||||
"nt": "$HEADROOM_EV_A-x"
|
||||
},
|
||||
{
|
||||
"input": "~$HEADROOM_EV_A",
|
||||
"posix": "~x",
|
||||
"nt": "~x"
|
||||
},
|
||||
{
|
||||
"input": "é/$HEADROOM_EV_A/é",
|
||||
"posix": "é/x/é",
|
||||
"nt": "é/x/é"
|
||||
}
|
||||
]
|
||||
},
|
||||
"cache_dir": {
|
||||
"comment": "{root} and {home} are temporary directories; HOME and USERPROFILE are set to {home}, every other cache variable is cleared first. expected may be {posix, nt}.",
|
||||
"probe_vars": [
|
||||
"HEADROOM_HF_PROBE_ROOT",
|
||||
"HEADROOM_HF_PROBE_UNSET"
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"name": "HF_HUB_CACHE through ${VAR} (the #4190 review case)",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"HF_HUB_CACHE": "${HEADROOM_HF_PROBE_ROOT}/hub"
|
||||
},
|
||||
"expected": "{root}/hub"
|
||||
},
|
||||
{
|
||||
"name": "HF_HUB_CACHE through $VAR",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"HF_HUB_CACHE": "$HEADROOM_HF_PROBE_ROOT/hub"
|
||||
},
|
||||
"expected": "{root}/hub"
|
||||
},
|
||||
{
|
||||
"name": "legacy HUGGINGFACE_HUB_CACHE through ${VAR}",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"HUGGINGFACE_HUB_CACHE": "${HEADROOM_HF_PROBE_ROOT}/legacy"
|
||||
},
|
||||
"expected": "{root}/legacy"
|
||||
},
|
||||
{
|
||||
"name": "HF_HOME through ${VAR}",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"HF_HOME": "${HEADROOM_HF_PROBE_ROOT}/hf-home"
|
||||
},
|
||||
"expected": "{root}/hf-home/hub"
|
||||
},
|
||||
{
|
||||
"name": "XDG_CACHE_HOME through ${VAR}",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"XDG_CACHE_HOME": "${HEADROOM_HF_PROBE_ROOT}/xdg"
|
||||
},
|
||||
"expected": "{root}/xdg/huggingface/hub"
|
||||
},
|
||||
{
|
||||
"name": "HF_HUB_CACHE still wins over an expanded HF_HOME",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"HF_HOME": "${HEADROOM_HF_PROBE_ROOT}/hf-home",
|
||||
"HF_HUB_CACHE": "{root}/explicit"
|
||||
},
|
||||
"expected": "{root}/explicit"
|
||||
},
|
||||
{
|
||||
"name": "an unset variable stays literal",
|
||||
"env": {
|
||||
"HF_HUB_CACHE": "{root}/${HEADROOM_HF_PROBE_UNSET}/hub"
|
||||
},
|
||||
"expected": "{root}/${HEADROOM_HF_PROBE_UNSET}/hub"
|
||||
},
|
||||
{
|
||||
"name": "leading ~ in HF_HUB_CACHE",
|
||||
"env": {
|
||||
"HF_HUB_CACHE": "~/models"
|
||||
},
|
||||
"expected": "{home}/models"
|
||||
},
|
||||
{
|
||||
"name": "leading ~ in HF_HOME",
|
||||
"env": {
|
||||
"HF_HOME": "~/hf"
|
||||
},
|
||||
"expected": "{home}/hf/hub"
|
||||
},
|
||||
{
|
||||
"name": "a ~ that a variable produces in HF_HUB_CACHE stays literal (expanduser runs first)",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "~/r",
|
||||
"HF_HUB_CACHE": "${HEADROOM_HF_PROBE_ROOT}/hub"
|
||||
},
|
||||
"expected": "~/r/hub"
|
||||
},
|
||||
{
|
||||
"name": "a ~ that a variable produces in HF_HOME expands on the second pass",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "~/r",
|
||||
"HF_HOME": "${HEADROOM_HF_PROBE_ROOT}/hf"
|
||||
},
|
||||
"expected": "{home}/r/hf/hub"
|
||||
},
|
||||
{
|
||||
"name": "%VAR% is a variable on Windows only",
|
||||
"env": {
|
||||
"HEADROOM_HF_PROBE_ROOT": "{root}",
|
||||
"HF_HUB_CACHE": "%HEADROOM_HF_PROBE_ROOT%/hub"
|
||||
},
|
||||
"expected": {
|
||||
"posix": "%HEADROOM_HF_PROBE_ROOT%/hub",
|
||||
"nt": "{root}/hub"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "an empty HF_HUB_CACHE is the working directory, not HF_HOME (the #4438 review case)",
|
||||
"env": {
|
||||
"HF_HUB_CACHE": "",
|
||||
"HF_HOME": "{root}/fallback-home"
|
||||
},
|
||||
"expected": ""
|
||||
},
|
||||
{
|
||||
"name": "an empty legacy HUGGINGFACE_HUB_CACHE is the working directory",
|
||||
"env": {
|
||||
"HUGGINGFACE_HUB_CACHE": "",
|
||||
"HF_HOME": "{root}/fallback-home"
|
||||
},
|
||||
"expected": ""
|
||||
},
|
||||
{
|
||||
"name": "an empty HF_HUB_CACHE still wins over a set legacy HUGGINGFACE_HUB_CACHE",
|
||||
"env": {
|
||||
"HF_HUB_CACHE": "",
|
||||
"HUGGINGFACE_HUB_CACHE": "{root}/legacy"
|
||||
},
|
||||
"expected": ""
|
||||
},
|
||||
{
|
||||
"name": "an empty HF_HOME is hub under the working directory, not XDG_CACHE_HOME",
|
||||
"env": {
|
||||
"HF_HOME": "",
|
||||
"XDG_CACHE_HOME": "{root}/xdg"
|
||||
},
|
||||
"expected": "hub"
|
||||
},
|
||||
{
|
||||
"name": "an empty XDG_CACHE_HOME is huggingface/hub under the working directory, not the home default",
|
||||
"env": {
|
||||
"XDG_CACHE_HOME": ""
|
||||
},
|
||||
"expected": "huggingface/hub"
|
||||
},
|
||||
{
|
||||
"name": "nothing set: the default under the home directory",
|
||||
"env": {},
|
||||
"expected": "{home}/.cache/huggingface/hub"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Record the embedding parity fixture for the default relevance model.
|
||||
|
||||
Embeds a fixed set of texts with the Python ``EmbeddingScorer``, which loads
|
||||
the pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, and writes them to
|
||||
``fixtures/embedding/bge_small_en_v15_pinned.json``. Two gated tests compare
|
||||
against it: ``tests/test_fastembed_pinned_snapshot.py`` (Python) and
|
||||
``fastembed_matches_parity_fixture`` in
|
||||
``crates/headroom-core/src/relevance/embedding.rs`` (Rust). Both reading one
|
||||
fixture bounds the drift between the languages by ``MAX_ABS_DIFF``.
|
||||
|
||||
The languages run the same ONNX file, so the gap between them comes from the
|
||||
ONNX Runtime build, not the pipeline. Measured when this fixture was recorded:
|
||||
Rust and Python on the same ORT (1.31.0) agree within 1e-6 per component;
|
||||
Python on ORT 1.23.2 vs 1.31.0 differs by up to 4.1e-4. ``MAX_ABS_DIFF``
|
||||
covers the ORT range the project supports with ~2x headroom.
|
||||
|
||||
This bounds numeric drift; it does not identify the model. Xenova's fp32
|
||||
export of the same model sits within 4.7e-4 of this fixture, inside the
|
||||
tolerance. Which files load is guaranteed by the pinned SHA and the unit tests
|
||||
on both sides, not by this fixture.
|
||||
|
||||
Re-record after bumping the pinned revision (needs network on a cold cache),
|
||||
with an interpreter whose onnxruntime is >= 1.24, the runtime the Rust side
|
||||
needs:
|
||||
python tests/parity/record_embedding.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
from headroom.onnx_runtime import _PINNED_REVISIONS
|
||||
from headroom.relevance.embedding import DEFAULT_MODEL_NAME, DEFAULT_MODEL_REPO, EmbeddingScorer
|
||||
|
||||
FIXTURE_PATH = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "embedding", "bge_small_en_v15_pinned.json"
|
||||
)
|
||||
|
||||
# Largest allowed |embedding component - fixture|, on either side. See the
|
||||
# module docstring for how it was chosen.
|
||||
MAX_ABS_DIFF = 1e-3
|
||||
|
||||
TEXTS = [
|
||||
"login error",
|
||||
"authentication failed for user admin: invalid credentials",
|
||||
'{"status": "failed", "error": "connection refused", "retries": 3}',
|
||||
"def retry(fn, attempts=3):\n for i in range(attempts):\n return fn()",
|
||||
"Ünïcödé text, emoji 🚀 and CJK 中文 mixed in one line.",
|
||||
# Longer than the 512-token window, so truncation is covered too.
|
||||
" ".join(f"log line {i}: request handled in {i % 97} ms" for i in range(400)),
|
||||
]
|
||||
|
||||
|
||||
def write_fixture(fixture: dict[str, object], path: str = FIXTURE_PATH) -> None:
|
||||
"""Write ``fixture`` with each list item, so each embedding row, on one line."""
|
||||
lines = []
|
||||
for key, value in fixture.items():
|
||||
if isinstance(value, list):
|
||||
items = ",\n".join(f" {json.dumps(item, ensure_ascii=False)}" for item in value)
|
||||
lines.append(f" {json.dumps(key)}: [\n{items}\n ]")
|
||||
else:
|
||||
lines.append(f" {json.dumps(key)}: {json.dumps(value, ensure_ascii=False)}")
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8", newline="\n") as f:
|
||||
f.write("{\n" + ",\n".join(lines) + "\n}\n")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
scorer = EmbeddingScorer(DEFAULT_MODEL_NAME)
|
||||
embeddings = scorer._encode(TEXTS)
|
||||
fixture = {
|
||||
"model": DEFAULT_MODEL_NAME,
|
||||
"repo": DEFAULT_MODEL_REPO,
|
||||
"revision": _PINNED_REVISIONS[DEFAULT_MODEL_REPO],
|
||||
"max_abs_diff": MAX_ABS_DIFF,
|
||||
"texts": TEXTS,
|
||||
"embeddings": [[round(float(v), 8) for v in row] for row in embeddings],
|
||||
}
|
||||
write_fixture(fixture)
|
||||
print(f"wrote {FIXTURE_PATH} ({len(TEXTS)} texts x {len(embeddings[0])} dims)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,15 +1,17 @@
|
||||
"""The default relevance embedder loads one pinned snapshot.
|
||||
"""The default relevance embedder loads one pinned snapshot, in both languages.
|
||||
|
||||
fastembed's ``TextEmbedding`` accepts ``revision`` but drops it before
|
||||
``snapshot_download``, so the old pin never reached the Hub and the model
|
||||
floated on ``main``. These tests hold the replacement in place: the pinned
|
||||
SHA reaches every file download and fastembed only gets the resolved
|
||||
directory.
|
||||
SHA reaches every file download, fastembed only gets the resolved directory,
|
||||
and the Rust scorer names the same repo, SHA and files.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
@@ -26,6 +28,12 @@ from headroom.relevance.embedding import (
|
||||
EmbeddingScorer,
|
||||
)
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
RUST_EMBEDDING = REPO_ROOT / "crates/headroom-core/src/relevance/embedding.rs"
|
||||
PARITY_FIXTURE = REPO_ROOT / "tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json"
|
||||
HF_CACHE_FIXTURE = json.loads(
|
||||
(REPO_ROOT / "tests/parity/fixtures/hf_hub_cache/cases.json").read_text(encoding="utf-8")
|
||||
)
|
||||
PIN = _PINNED_REVISIONS[DEFAULT_MODEL_REPO]
|
||||
|
||||
|
||||
@@ -171,3 +179,112 @@ def test_fastembed_loads_specific_model_path_without_a_download(monkeypatch, tmp
|
||||
lazy_load=True,
|
||||
)
|
||||
assert Path(model.model._model_dir) == tmp_path
|
||||
|
||||
|
||||
def _rust_const(name: str) -> str:
|
||||
match = re.search(rf'const {name}: &str = "([^"]+)";', RUST_EMBEDDING.read_text())
|
||||
assert match, f"{name} not found in {RUST_EMBEDDING}"
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def test_rust_loads_the_same_snapshot():
|
||||
assert _rust_const("DEFAULT_MODEL_REPO") == DEFAULT_MODEL_REPO
|
||||
assert _rust_const("DEFAULT_MODEL_REVISION") == PIN
|
||||
assert _rust_const("DEFAULT_MODEL_FILE") == DEFAULT_MODEL_FILES[0]
|
||||
files = re.search(
|
||||
r"const DEFAULT_MODEL_FILES: \[&str; \d+\] = \[(.*?)\];",
|
||||
RUST_EMBEDDING.read_text(),
|
||||
re.S,
|
||||
)
|
||||
assert files, "DEFAULT_MODEL_FILES not found"
|
||||
entries = [part.strip() for part in files.group(1).split(",") if part.strip()]
|
||||
assert entries == ["DEFAULT_MODEL_FILE", *(f'"{name}"' for name in DEFAULT_MODEL_FILES[1:])]
|
||||
|
||||
|
||||
def test_parity_fixture_was_recorded_at_the_pin():
|
||||
fixture = json.loads(PARITY_FIXTURE.read_text(encoding="utf-8"))
|
||||
assert fixture["repo"] == DEFAULT_MODEL_REPO
|
||||
assert fixture["revision"] == PIN
|
||||
assert len(fixture["embeddings"]) == len(fixture["texts"])
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not os.environ.get("RUN_FASTEMBED_TESTS"),
|
||||
reason="loads the real model (~67 MB download on a cold cache); set RUN_FASTEMBED_TESTS=1",
|
||||
)
|
||||
def test_python_matches_parity_fixture():
|
||||
"""Python half of the cross-language check; the Rust half is
|
||||
``fastembed_matches_parity_fixture`` in the Rust scorer."""
|
||||
np = pytest.importorskip("numpy")
|
||||
pytest.importorskip("fastembed")
|
||||
fixture = json.loads(PARITY_FIXTURE.read_text(encoding="utf-8"))
|
||||
|
||||
got = EmbeddingScorer()._encode(fixture["texts"])
|
||||
|
||||
for text, row, expected in zip(fixture["texts"], got, fixture["embeddings"]):
|
||||
worst = float(np.abs(np.asarray(row, dtype=np.float64) - np.asarray(expected)).max())
|
||||
assert worst <= fixture["max_abs_diff"], f"{text!r}: max |python - fixture| = {worst:e}"
|
||||
|
||||
|
||||
# ── the shared HuggingFace cache directory ──────────────────────────────────
|
||||
#
|
||||
# Rust's ``hf_hub_cache_dir`` re-implements how ``huggingface_hub`` picks the
|
||||
# hub cache, so both languages look up and download into one directory. These
|
||||
# check the shared fixture against Python itself; the Rust tests
|
||||
# ``expand_vars_matches_python_posixpath_and_ntpath`` and
|
||||
# ``hf_hub_cache_dir_matches_the_shared_fixture`` check the port against the
|
||||
# same file.
|
||||
|
||||
_CACHE_VARS = ("HF_HUB_CACHE", "HUGGINGFACE_HUB_CACHE", "HF_HOME", "XDG_CACHE_HOME")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"case", HF_CACHE_FIXTURE["expandvars"]["cases"], ids=lambda case: case["input"]
|
||||
)
|
||||
def test_expandvars_fixture_matches_posixpath_and_ntpath(
|
||||
case: dict[str, str], monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
import ntpath
|
||||
import posixpath
|
||||
|
||||
for name, value in HF_CACHE_FIXTURE["expandvars"]["env"].items():
|
||||
monkeypatch.setenv(name, value)
|
||||
monkeypatch.delenv(HF_CACHE_FIXTURE["expandvars"]["unset"], raising=False)
|
||||
assert posixpath.expandvars(case["input"]) == case["posix"]
|
||||
assert ntpath.expandvars(case["input"]) == case["nt"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"case", HF_CACHE_FIXTURE["cache_dir"]["cases"], ids=lambda case: case["name"]
|
||||
)
|
||||
def test_cache_dir_fixture_matches_huggingface_hub(case: dict, tmp_path: Path) -> None:
|
||||
"""``huggingface_hub`` reads these variables once, at import, so each case
|
||||
runs in a fresh interpreter."""
|
||||
import subprocess
|
||||
|
||||
pytest.importorskip("huggingface_hub")
|
||||
root, home = tmp_path / "root", tmp_path / "home"
|
||||
|
||||
def fill(text: str) -> str:
|
||||
return text.replace("{root}", str(root)).replace("{home}", str(home))
|
||||
|
||||
probe_vars = HF_CACHE_FIXTURE["cache_dir"]["probe_vars"]
|
||||
env = {k: v for k, v in os.environ.items() if k not in (*_CACHE_VARS, *probe_vars)}
|
||||
env["HOME"] = env["USERPROFILE"] = str(home)
|
||||
env.update({name: fill(value) for name, value in case["env"].items()})
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"import sys, huggingface_hub.constants as c; sys.stdout.write(c.HF_HUB_CACHE)",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
timeout=60,
|
||||
check=True,
|
||||
)
|
||||
expected = case["expected"]
|
||||
if isinstance(expected, dict):
|
||||
expected = expected["nt" if os.name == "nt" else "posix"]
|
||||
assert Path(result.stdout) == Path(fill(expected))
|
||||
|
||||
@@ -3205,8 +3205,8 @@ class TestDocsMatchTheGuarantee:
|
||||
|
||||
_RUST_EGRESS_PATTERNS = re.compile(
|
||||
r"""
|
||||
hf_hub::api::(?:sync|tokio)::Api(?:Builder)?::new\(
|
||||
| (?<![\w:])Api(?:Builder)?::new\(
|
||||
hf_hub::api::(?:sync|tokio)::Api(?:Builder)?::(?:new|from_env|from_cache)\(
|
||||
| (?<![\w:])Api(?:Builder)?::(?:new|from_env|from_cache)\(
|
||||
| (?<![\w:])TextEmbedding::try_new\w*\(
|
||||
| (?<![\w:])reqwest::(?:Client::(?:new|builder)|get|post)\(
|
||||
| (?<![\w:])ureq::(?:agent|builder|get|post|put|delete|request)\(
|
||||
@@ -3220,6 +3220,14 @@ _RUST_FN = re.compile(
|
||||
)
|
||||
|
||||
_RUST_ALLOWLIST: dict[str, tuple[int, str]] = {
|
||||
"headroom-core/src/relevance/embedding.rs": (
|
||||
1,
|
||||
"cache-only: TextEmbedding::try_new_from_user_defined in load_snapshot "
|
||||
"builds the ONNX session from bytes already read off disk and has no "
|
||||
"Hub client at all. The files reach the disk through "
|
||||
"default_model_repo, whose ApiBuilder is guarded, and only on a cache "
|
||||
"miss, so a pre-seeded air-gapped host still loads.",
|
||||
),
|
||||
"headroom-proxy/src/proxy.rs": (
|
||||
1,
|
||||
"user-traffic: the reqwest client the Rust proxy forwards the caller's "
|
||||
@@ -3326,7 +3334,13 @@ class TestRustEgressChokepointCoverage:
|
||||
"headroom-core/src/relevance/embedding.rs",
|
||||
):
|
||||
assert relpath in sites, f"{relpath} has no detected egress site at all"
|
||||
assert all(site.guarded for site in sites[relpath]), (
|
||||
# Every site is guarded, except the cache-only loads the allowlist
|
||||
# counts for the file (embedding.rs: the user-defined model load).
|
||||
unguarded = [site for site in sites[relpath] if not site.guarded]
|
||||
assert len(unguarded) < len(sites[relpath]), (
|
||||
f"{relpath} has no egress site the Rust sweep sees as guarded"
|
||||
)
|
||||
assert len(unguarded) == _RUST_ALLOWLIST.get(relpath, (0, ""))[0], (
|
||||
f"{relpath} has an egress site the Rust sweep does not see as guarded"
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user