Files
headroom/tests/parity/record_embedding.py
9bb4b130ff fix(relevance): load the Rust default embedder from Python's pinned snapshot (#4190)
## Description

Parent status: #4152 has merged. The pinned-loader and batch-cap merge
resolution was reviewed at e3a495cbc5. The
original split/stack history is retained below.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now four
linear commits on top of main, with no merge commits:

- `bfae4ade` fix(relevance): load the Rust default embedder from
Python's pinned snapshot
- `6fbae380` fix(relevance): look up and download the Rust embedder in
one HF cache dir
- `70afece8` fix(relevance): expand ~ and $VAR in the Rust HF cache path
like huggingface_hub
- `62b792f0` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`. Its files matched main; the only difference was
`uv.lock`, where main's later `filelock>=4.2.1` entry was kept.
- The six main-sync merges (`185dc9b7`, `052977b9`, `029c72ac`,
`4405b12c`, `8b0a10e8`, and the reviewed `e3a495cb`) are linearized
away, so those SHAs are no longer on the branch.
- `embedding.rs` conflicted again with main's batch-size cap (#4507). I
resolved it to the same content as the reviewed merge: every file this
PR touches is byte-identical between `e3a495cb` and the new head
`62b792f0` (`git diff e3a495cb 62b792f0 -- <files changed vs main>` is
empty). The review and qualification below still apply to the content;
only the history and SHAs changed.

Rust `EmbeddingScorer::try_new` used `EmbeddingModel::BGESmallENV15`.
fastembed-rs maps that to a different repo and file from what Python
runs: `Xenova/bge-small-en-v1.5` `onnx/model.onnx` (fp32), always on
`main`. So the two languages never ran the same ONNX file, and the
"byte-equal across languages" claim in both modules was wrong.

This PR makes `try_new` load the same pinned
`Qdrant/bge-small-en-v1.5-onnx-Q` snapshot that #4152 makes Python load.
It adds a cross-language parity fixture checked from both sides, and
replaces the parity claim with measured numbers.

Nothing in production calls `EmbeddingScorer::try_new` today:
`create_scorer` and `HybridScorer` use `default()`, which never loads a
model. This keeps the Rust scorer correct and pinned for when it is
wired in.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

(This PR's own commits only.)

- `crates/headroom-core/src/relevance/embedding.rs`:
- `try_new` loads the snapshot through
`TextEmbedding::try_new_from_user_defined`, with CLS pooling and static
quantization, which is what fastembed-rs uses for `BGESmallENV15Q`.
- It uses exactly one hub cache directory, resolved like Python's
`huggingface_hub.constants.HF_HUB_CACHE`: `HF_HUB_CACHE` →
`HUGGINGFACE_HUB_CACHE` → `HF_HOME/hub` →
`XDG_CACHE_HOME/huggingface/hub` → `~/.cache/huggingface/hub`, with a
leading `~` expanded. It looks up and downloads only there, so a stale
snapshot in a lower-priority cache is never used. The home directory
comes from `std::env::home_dir`, which falls back to the user database
when `HOME` is unset, as Python's `expanduser` does.
- It finds the snapshot as `snapshots/<sha>` in that directory.
`hf_hub::Cache` resolves through `refs/<rev>`, which Python's
`huggingface_hub` doesn't write for a commit SHA.
- On a miss it downloads with `Repo::with_revision`. `guard_egress` sits
in front of the Hub client and runs only on a cache miss, so a
pre-seeded air-gapped host now loads; previously Rust refused even with
a warm cache.
- `HEADROOM_HF_PIN=off` floats it on `main`, with the same values as
Python's `_resolve_revision`.
  - `try_new_with_model` is unchanged.
- Parity fixture: `tests/parity/record_embedding.py` and
`tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json` (6 texts
× 384 dims, recorded on ORT 1.31.0). The fixture is written one
embedding row per line, so it's 22 lines. Python checks it in CI
(`test-extras` sets `RUN_FASTEMBED_TESTS=1`); Rust checks it under
`RUN_FASTEMBED_TESTS=1`.
- `tests/test_fastembed_pinned_snapshot.py`: the Rust constants name the
same repo, SHA and files as Python; the fixture was recorded at the pin;
plus the gated Python parity check.
- 10 new Rust unit tests in `embedding.rs`: the lookup ignores other
commits and needs every file; the download URL requests the pinned
revision; the offline guard applies on a miss only;
`HEADROOM_HF_PIN=off` floats; the cache directory follows
`huggingface_hub`'s order variable by variable (including an empty
variable and `~`); with no `HOME` it still resolves through the user
database; and the review case: an empty explicit `HF_HUB_CACHE` next to
a complete pinned snapshot and an older `refs/main` under `HOME` finds
neither, for the pinned revision or `main`, while both are found once
`HF_HUB_CACHE` is unset.
- `tests/test_offline_egress_chokepoint.py`: the Rust sweep's regex now
also matches `ApiBuilder::from_cache` / `from_env`. It missed the new
download client, which is guarded; it flags nothing else in `crates/`.
The user-defined model load, which only reads bytes already on disk, is
allowlisted as `cache-only`.
- Docs: the "byte-equal" wording in both embedder modules,
`pyproject.toml` and `onnx_runtime.py` is replaced with measured numbers
and the cross-language keep-equal notes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `62b792f0` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 991 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_hf_revision_pinning.py
235 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

The 7 offline-egress failures noted below no longer reproduce here.

Before the rebase:

```text
$ cargo test -p headroom-core --lib
test result: ok. 985 passed; 0 failed; 1 ignored      # see the note on #4151 below

$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean

$ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.31.0 dylib> cargo test -p headroom-core --lib relevance::embedding
test relevance::embedding::tests::fastembed_matches_parity_fixture ... ok
test result: ok. 24 passed; 0 failed

$ RUN_FASTEMBED_TESTS=1 pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_relevance.py \
    tests/test_stateless_writers.py tests/test_hf_revision_pinning.py
71 passed

$ pytest -q tests/test_offline_egress_chokepoint.py
7 failed, 132 passed, 3 skipped
# The 7 are the OTel / Langfuse / Headroom Cloud offline tests, whose extras aren't in this venv;
# the same 7 fail on origin/main here. The Rust sweep tests pass.
```

**Note on #4151:** with `ORT_DYLIB_PATH` set on macOS, a full `cargo
test -p headroom-core --lib` fails
`tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error`
most of the time (5/5 on `afdb60d3`, before the cache change too).
`a_cached_snapshot_is_not_refused_offline` holds `HEADROOM_OFFLINE=1`
under the env lock while ONNX Runtime loads, which is slow there, and
the tokenizer test reads the variable without the lock. That's the race
#4151 fixes; with its commit applied, 6/6 full runs pass. Linux CI
passes either way.

The cache-precedence test was checked against the bug: with the lookup
changed back to also scanning `~/.cache`, it fails on the first case
(pinned revision, empty `HF_HUB_CACHE`).

## Real Behavior Proof
- Exact command / steps: additional maintainer verification on the
main-synced head: cargo test -p headroom-core --locked hf_hub_cache_dir;
cargo test -p headroom-core --locked
expand_vars_matches_python_posixpath_and_ntpath; python -m pytest
tests/test_fastembed_pinned_snapshot.py -q; repeat its fixture cases
with huggingface-hub==2.2.0.
- Observed result: three Rust tests passed across both POSIX and Windows
expansion fixtures and cache-directory selection; Python passed 46 tests
with one gated skip using Hugging Face Hub 1.16.1, and all 36 shared
fixture cases passed separately with locked Hub 2.2.0.

- Environment: macOS arm64; Rust fastembed 5.17.3 with hf-hub 0.5.0;
repo `.venv` (fastembed 0.8.0, onnxruntime 1.23.2) and a second
interpreter on onnxruntime 1.31.0.
- A cold-cache Rust load with `HF_HUB_CACHE=<empty dir>` wrote
`snapshots/52398278…` plus `refs/52398278…` and passed the parity check.
Per-component max |Δ| against the fixture:
  - Rust vs Python on the same ORT (1.31.0): ≤ 1e-6.
  - Python on ORT 1.23.2 vs 1.31.0: ≤ 4.1e-4.
  - The fixture's tolerance is 1e-3.
- For reference, Xenova's fp32 export (what Rust used to load) is ≤
4.7e-4 from the fixture. The old cross-language gap was real but
numerically small, and this fixture bounds drift; it cannot tell the two
exports apart. Model identity is enforced by the pinned SHA and the unit
tests.
- The fixture's new one-row-per-line layout was re-emitted from the
recorded data, not re-recorded: the parsed JSON is identical.
- Not tested: Windows, and Linux outside CI. Rust's gated tests in CI
(they need the model and an ORT dylib, so they stay manual behind
`RUN_FASTEMBED_TESTS=1`). Rust `HybridScorer` thresholds against the
Qdrant export.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no production path.
`EmbeddingScorer::try_new` loads the Qdrant export instead of Xenova's,
and now loads offline from a warm cache instead of refusing, but nothing
in production calls it today.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches
are still unpinned (there's no Rust `HEADROOM_HF_PIN`), and Rust's gated
embedding tests could get a CI job once one ships an ORT dylib.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4190"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

## Maintainer qualification

The environment-reference mismatch from the previous review is
addressed. The shared corpus covers the reported explicit-cache case,
platform-specific variable syntax, and the two-stage home/hub expansion
ordering. The Rust test runs both POSIX and Windows expansion rules;
Python validates the same corpus against posixpath, ntpath, and fresh
huggingface_hub.constants imports.

A clean current-main sync was published as
052977b9e88ae3bbda96797615b64ee350a2f695; it does not change the
contributor's resolver or fixture files. Qualification used Windows,
Hugging Face Hub 1.16.1 and an isolated locked 2.2.0 target. No model
download or numerical embedding-parity result is claimed by these path
tests; the pre-existing gated numerical-parity test remained skipped.

Hosted checks on the pushed head must finish before approval. No
contributor CI action requested or PR merged.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 21:31:59 -06:00

86 lines
3.5 KiB
Python

"""Record the embedding parity fixture for the default relevance model.
Embeds a fixed set of texts with the Python ``EmbeddingScorer``, which loads
the pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, and writes them to
``fixtures/embedding/bge_small_en_v15_pinned.json``. Two gated tests compare
against it: ``tests/test_fastembed_pinned_snapshot.py`` (Python) and
``fastembed_matches_parity_fixture`` in
``crates/headroom-core/src/relevance/embedding.rs`` (Rust). Both reading one
fixture bounds the drift between the languages by ``MAX_ABS_DIFF``.
The languages run the same ONNX file, so the gap between them comes from the
ONNX Runtime build, not the pipeline. Measured when this fixture was recorded:
Rust and Python on the same ORT (1.31.0) agree within 1e-6 per component;
Python on ORT 1.23.2 vs 1.31.0 differs by up to 4.1e-4. ``MAX_ABS_DIFF``
covers the ORT range the project supports with ~2x headroom.
This bounds numeric drift; it does not identify the model. Xenova's fp32
export of the same model sits within 4.7e-4 of this fixture, inside the
tolerance. Which files load is guaranteed by the pinned SHA and the unit tests
on both sides, not by this fixture.
Re-record after bumping the pinned revision (needs network on a cold cache),
with an interpreter whose onnxruntime is >= 1.24, the runtime the Rust side
needs:
python tests/parity/record_embedding.py
"""
from __future__ import annotations
import json
import os
from headroom.onnx_runtime import _PINNED_REVISIONS
from headroom.relevance.embedding import DEFAULT_MODEL_NAME, DEFAULT_MODEL_REPO, EmbeddingScorer
FIXTURE_PATH = os.path.join(
os.path.dirname(__file__), "fixtures", "embedding", "bge_small_en_v15_pinned.json"
)
# Largest allowed |embedding component - fixture|, on either side. See the
# module docstring for how it was chosen.
MAX_ABS_DIFF = 1e-3
TEXTS = [
"login error",
"authentication failed for user admin: invalid credentials",
'{"status": "failed", "error": "connection refused", "retries": 3}',
"def retry(fn, attempts=3):\n for i in range(attempts):\n return fn()",
"Ünïcödé text, emoji 🚀 and CJK 中文 mixed in one line.",
# Longer than the 512-token window, so truncation is covered too.
" ".join(f"log line {i}: request handled in {i % 97} ms" for i in range(400)),
]
def write_fixture(fixture: dict[str, object], path: str = FIXTURE_PATH) -> None:
"""Write ``fixture`` with each list item, so each embedding row, on one line."""
lines = []
for key, value in fixture.items():
if isinstance(value, list):
items = ",\n".join(f" {json.dumps(item, ensure_ascii=False)}" for item in value)
lines.append(f" {json.dumps(key)}: [\n{items}\n ]")
else:
lines.append(f" {json.dumps(key)}: {json.dumps(value, ensure_ascii=False)}")
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8", newline="\n") as f:
f.write("{\n" + ",\n".join(lines) + "\n}\n")
def main() -> None:
scorer = EmbeddingScorer(DEFAULT_MODEL_NAME)
embeddings = scorer._encode(TEXTS)
fixture = {
"model": DEFAULT_MODEL_NAME,
"repo": DEFAULT_MODEL_REPO,
"revision": _PINNED_REVISIONS[DEFAULT_MODEL_REPO],
"max_abs_diff": MAX_ABS_DIFF,
"texts": TEXTS,
"embeddings": [[round(float(v), 8) for v in row] for row in embeddings],
}
write_fixture(fixture)
print(f"wrote {FIXTURE_PATH} ({len(TEXTS)} texts x {len(embeddings[0])} dims)")
if __name__ == "__main__":
main()