mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Parent status: #4152 has merged. The pinned-loader and batch-cap merge resolution was reviewed ate3a495cbc5. The original split/stack history is retained below. **Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now four linear commits on top of main, with no merge commits: - `bfae4ade` fix(relevance): load the Rust default embedder from Python's pinned snapshot - `6fbae380` fix(relevance): look up and download the Rust embedder in one HF cache dir - `70afece8` fix(relevance): expand ~ and $VAR in the Rust HF cache path like huggingface_hub - `62b792f0` fix(relevance): keep an empty HF cache variable, as huggingface_hub does What the rebase changed: - The #4152 commit (`cb1c3a33`) is gone from the branch: main already has it as `f380692b`. Its files matched main; the only difference was `uv.lock`, where main's later `filelock>=4.2.1` entry was kept. - The six main-sync merges (`185dc9b7`, `052977b9`, `029c72ac`, `4405b12c`, `8b0a10e8`, and the reviewed `e3a495cb`) are linearized away, so those SHAs are no longer on the branch. - `embedding.rs` conflicted again with main's batch-size cap (#4507). I resolved it to the same content as the reviewed merge: every file this PR touches is byte-identical between `e3a495cb` and the new head `62b792f0` (`git diffe3a495cb62b792f0-- <files changed vs main>` is empty). The review and qualification below still apply to the content; only the history and SHAs changed. Rust `EmbeddingScorer::try_new` used `EmbeddingModel::BGESmallENV15`. fastembed-rs maps that to a different repo and file from what Python runs: `Xenova/bge-small-en-v1.5` `onnx/model.onnx` (fp32), always on `main`. So the two languages never ran the same ONNX file, and the "byte-equal across languages" claim in both modules was wrong. This PR makes `try_new` load the same pinned `Qdrant/bge-small-en-v1.5-onnx-Q` snapshot that #4152 makes Python load. It adds a cross-language parity fixture checked from both sides, and replaces the parity claim with measured numbers. Nothing in production calls `EmbeddingScorer::try_new` today: `create_scorer` and `HybridScorer` use `default()`, which never loads a model. This keeps the Rust scorer correct and pinned for when it is wired in. No linked issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made (This PR's own commits only.) - `crates/headroom-core/src/relevance/embedding.rs`: - `try_new` loads the snapshot through `TextEmbedding::try_new_from_user_defined`, with CLS pooling and static quantization, which is what fastembed-rs uses for `BGESmallENV15Q`. - It uses exactly one hub cache directory, resolved like Python's `huggingface_hub.constants.HF_HUB_CACHE`: `HF_HUB_CACHE` → `HUGGINGFACE_HUB_CACHE` → `HF_HOME/hub` → `XDG_CACHE_HOME/huggingface/hub` → `~/.cache/huggingface/hub`, with a leading `~` expanded. It looks up and downloads only there, so a stale snapshot in a lower-priority cache is never used. The home directory comes from `std::env::home_dir`, which falls back to the user database when `HOME` is unset, as Python's `expanduser` does. - It finds the snapshot as `snapshots/<sha>` in that directory. `hf_hub::Cache` resolves through `refs/<rev>`, which Python's `huggingface_hub` doesn't write for a commit SHA. - On a miss it downloads with `Repo::with_revision`. `guard_egress` sits in front of the Hub client and runs only on a cache miss, so a pre-seeded air-gapped host now loads; previously Rust refused even with a warm cache. - `HEADROOM_HF_PIN=off` floats it on `main`, with the same values as Python's `_resolve_revision`. - `try_new_with_model` is unchanged. - Parity fixture: `tests/parity/record_embedding.py` and `tests/parity/fixtures/embedding/bge_small_en_v15_pinned.json` (6 texts × 384 dims, recorded on ORT 1.31.0). The fixture is written one embedding row per line, so it's 22 lines. Python checks it in CI (`test-extras` sets `RUN_FASTEMBED_TESTS=1`); Rust checks it under `RUN_FASTEMBED_TESTS=1`. - `tests/test_fastembed_pinned_snapshot.py`: the Rust constants name the same repo, SHA and files as Python; the fixture was recorded at the pin; plus the gated Python parity check. - 10 new Rust unit tests in `embedding.rs`: the lookup ignores other commits and needs every file; the download URL requests the pinned revision; the offline guard applies on a miss only; `HEADROOM_HF_PIN=off` floats; the cache directory follows `huggingface_hub`'s order variable by variable (including an empty variable and `~`); with no `HOME` it still resolves through the user database; and the review case: an empty explicit `HF_HUB_CACHE` next to a complete pinned snapshot and an older `refs/main` under `HOME` finds neither, for the pinned revision or `main`, while both are found once `HF_HUB_CACHE` is unset. - `tests/test_offline_egress_chokepoint.py`: the Rust sweep's regex now also matches `ApiBuilder::from_cache` / `from_env`. It missed the new download client, which is guarded; it flags nothing else in `crates/`. The user-defined model load, which only reads bytes already on disk, is allowlisted as `cache-only`. - Docs: the "byte-equal" wording in both embedder modules, `pyproject.toml` and `onnx_runtime.py` is replaced with measured numbers and the cross-language keep-equal notes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output After the rebase, on `62b792f0` (macOS arm64, no `ORT_DYLIB_PATH`, so the gated model tests skip): ```text $ cargo test -p headroom-core --lib test result: ok. 991 passed; 0 failed; 1 ignored $ cargo clippy -p headroom-core --all-targets -- -D warnings # clean $ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings # clean $ cargo fmt --all -- --check # clean $ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py tests/test_hf_revision_pinning.py 235 passed, 3 skipped $ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2) # clean ``` The 7 offline-egress failures noted below no longer reproduce here. Before the rebase: ```text $ cargo test -p headroom-core --lib test result: ok. 985 passed; 0 failed; 1 ignored # see the note on #4151 below $ cargo clippy -p headroom-core --all-targets -- -D warnings # clean $ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings # clean $ cargo fmt --all -- --check # clean $ RUN_FASTEMBED_TESTS=1 ORT_DYLIB_PATH=<onnxruntime 1.31.0 dylib> cargo test -p headroom-core --lib relevance::embedding test relevance::embedding::tests::fastembed_matches_parity_fixture ... ok test result: ok. 24 passed; 0 failed $ RUN_FASTEMBED_TESTS=1 pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_relevance.py \ tests/test_stateless_writers.py tests/test_hf_revision_pinning.py 71 passed $ pytest -q tests/test_offline_egress_chokepoint.py 7 failed, 132 passed, 3 skipped # The 7 are the OTel / Langfuse / Headroom Cloud offline tests, whose extras aren't in this venv; # the same 7 fail on origin/main here. The Rust sweep tests pass. ``` **Note on #4151:** with `ORT_DYLIB_PATH` set on macOS, a full `cargo test -p headroom-core --lib` fails `tokenizer::hf_impl::tests::from_pretrained_invalid_repo_returns_hub_error` most of the time (5/5 on `afdb60d3`, before the cache change too). `a_cached_snapshot_is_not_refused_offline` holds `HEADROOM_OFFLINE=1` under the env lock while ONNX Runtime loads, which is slow there, and the tokenizer test reads the variable without the lock. That's the race #4151 fixes; with its commit applied, 6/6 full runs pass. Linux CI passes either way. The cache-precedence test was checked against the bug: with the lookup changed back to also scanning `~/.cache`, it fails on the first case (pinned revision, empty `HF_HUB_CACHE`). ## Real Behavior Proof - Exact command / steps: additional maintainer verification on the main-synced head: cargo test -p headroom-core --locked hf_hub_cache_dir; cargo test -p headroom-core --locked expand_vars_matches_python_posixpath_and_ntpath; python -m pytest tests/test_fastembed_pinned_snapshot.py -q; repeat its fixture cases with huggingface-hub==2.2.0. - Observed result: three Rust tests passed across both POSIX and Windows expansion fixtures and cache-directory selection; Python passed 46 tests with one gated skip using Hugging Face Hub 1.16.1, and all 36 shared fixture cases passed separately with locked Hub 2.2.0. - Environment: macOS arm64; Rust fastembed 5.17.3 with hf-hub 0.5.0; repo `.venv` (fastembed 0.8.0, onnxruntime 1.23.2) and a second interpreter on onnxruntime 1.31.0. - A cold-cache Rust load with `HF_HUB_CACHE=<empty dir>` wrote `snapshots/52398278…` plus `refs/52398278…` and passed the parity check. Per-component max |Δ| against the fixture: - Rust vs Python on the same ORT (1.31.0): ≤ 1e-6. - Python on ORT 1.23.2 vs 1.31.0: ≤ 4.1e-4. - The fixture's tolerance is 1e-3. - For reference, Xenova's fp32 export (what Rust used to load) is ≤ 4.7e-4 from the fixture. The old cross-language gap was real but numerically small, and this fixture bounds drift; it cannot tell the two exports apart. Model identity is enforced by the pinned SHA and the unit tests. - The fixture's new one-row-per-line layout was re-emitted from the recorded data, not re-recorded: the parsed JSON is identical. - Not tested: Windows, and Linux outside CI. Rust's gated tests in CI (they need the model and an ORT dylib, so they stay manual behind `RUN_FASTEMBED_TESTS=1`). Rust `HybridScorer` thresholds against the Qdrant export. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: no production path. `EmbeddingScorer::try_new` loads the Qdrant export instead of Xenova's, and now loads offline from a warm cache instead of refusing, but nothing in production calls it today. - Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes - Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches are still unpinned (there's no Rust `HEADROOM_HF_PIN`), and Rust's gated embedding tests could get a CI job once one ships an ORT dylib. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4190" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Maintainer qualification The environment-reference mismatch from the previous review is addressed. The shared corpus covers the reported explicit-cache case, platform-specific variable syntax, and the two-stage home/hub expansion ordering. The Rust test runs both POSIX and Windows expansion rules; Python validates the same corpus against posixpath, ntpath, and fresh huggingface_hub.constants imports. A clean current-main sync was published as 052977b9e88ae3bbda96797615b64ee350a2f695; it does not change the contributor's resolver or fixture files. Qualification used Windows, Hugging Face Hub 1.16.1 and an isolated locked 2.2.0 target. No model download or numerical embedding-parity result is claimed by these path tests; the pre-existing gated numerical-parity test remained skipped. Hosted checks on the pushed head must finish before approval. No contributor CI action requested or PR merged. --------- Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
86 lines
3.5 KiB
Python
86 lines
3.5 KiB
Python
"""Record the embedding parity fixture for the default relevance model.
|
|
|
|
Embeds a fixed set of texts with the Python ``EmbeddingScorer``, which loads
|
|
the pinned Qdrant/bge-small-en-v1.5-onnx-Q snapshot, and writes them to
|
|
``fixtures/embedding/bge_small_en_v15_pinned.json``. Two gated tests compare
|
|
against it: ``tests/test_fastembed_pinned_snapshot.py`` (Python) and
|
|
``fastembed_matches_parity_fixture`` in
|
|
``crates/headroom-core/src/relevance/embedding.rs`` (Rust). Both reading one
|
|
fixture bounds the drift between the languages by ``MAX_ABS_DIFF``.
|
|
|
|
The languages run the same ONNX file, so the gap between them comes from the
|
|
ONNX Runtime build, not the pipeline. Measured when this fixture was recorded:
|
|
Rust and Python on the same ORT (1.31.0) agree within 1e-6 per component;
|
|
Python on ORT 1.23.2 vs 1.31.0 differs by up to 4.1e-4. ``MAX_ABS_DIFF``
|
|
covers the ORT range the project supports with ~2x headroom.
|
|
|
|
This bounds numeric drift; it does not identify the model. Xenova's fp32
|
|
export of the same model sits within 4.7e-4 of this fixture, inside the
|
|
tolerance. Which files load is guaranteed by the pinned SHA and the unit tests
|
|
on both sides, not by this fixture.
|
|
|
|
Re-record after bumping the pinned revision (needs network on a cold cache),
|
|
with an interpreter whose onnxruntime is >= 1.24, the runtime the Rust side
|
|
needs:
|
|
python tests/parity/record_embedding.py
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
|
|
from headroom.onnx_runtime import _PINNED_REVISIONS
|
|
from headroom.relevance.embedding import DEFAULT_MODEL_NAME, DEFAULT_MODEL_REPO, EmbeddingScorer
|
|
|
|
FIXTURE_PATH = os.path.join(
|
|
os.path.dirname(__file__), "fixtures", "embedding", "bge_small_en_v15_pinned.json"
|
|
)
|
|
|
|
# Largest allowed |embedding component - fixture|, on either side. See the
|
|
# module docstring for how it was chosen.
|
|
MAX_ABS_DIFF = 1e-3
|
|
|
|
TEXTS = [
|
|
"login error",
|
|
"authentication failed for user admin: invalid credentials",
|
|
'{"status": "failed", "error": "connection refused", "retries": 3}',
|
|
"def retry(fn, attempts=3):\n for i in range(attempts):\n return fn()",
|
|
"Ünïcödé text, emoji 🚀 and CJK 中文 mixed in one line.",
|
|
# Longer than the 512-token window, so truncation is covered too.
|
|
" ".join(f"log line {i}: request handled in {i % 97} ms" for i in range(400)),
|
|
]
|
|
|
|
|
|
def write_fixture(fixture: dict[str, object], path: str = FIXTURE_PATH) -> None:
|
|
"""Write ``fixture`` with each list item, so each embedding row, on one line."""
|
|
lines = []
|
|
for key, value in fixture.items():
|
|
if isinstance(value, list):
|
|
items = ",\n".join(f" {json.dumps(item, ensure_ascii=False)}" for item in value)
|
|
lines.append(f" {json.dumps(key)}: [\n{items}\n ]")
|
|
else:
|
|
lines.append(f" {json.dumps(key)}: {json.dumps(value, ensure_ascii=False)}")
|
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
|
with open(path, "w", encoding="utf-8", newline="\n") as f:
|
|
f.write("{\n" + ",\n".join(lines) + "\n}\n")
|
|
|
|
|
|
def main() -> None:
|
|
scorer = EmbeddingScorer(DEFAULT_MODEL_NAME)
|
|
embeddings = scorer._encode(TEXTS)
|
|
fixture = {
|
|
"model": DEFAULT_MODEL_NAME,
|
|
"repo": DEFAULT_MODEL_REPO,
|
|
"revision": _PINNED_REVISIONS[DEFAULT_MODEL_REPO],
|
|
"max_abs_diff": MAX_ABS_DIFF,
|
|
"texts": TEXTS,
|
|
"embeddings": [[round(float(v), 8) for v in row] for row in embeddings],
|
|
}
|
|
write_fixture(fixture)
|
|
print(f"wrote {FIXTURE_PATH} ({len(TEXTS)} texts x {len(embeddings[0])} dims)")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|