Files
headroom/tests/test_stateless_writers.py
Kenneth WongandKenneth Wong f380692b46 fix(relevance): load the default embedder from one pinned snapshot (#4152)
## Description

The supply-chain pin on the default relevance embedder never took
effect. `EmbeddingScorer` passed `revision="52398278…"` to fastembed's
`TextEmbedding`, which accepts it and then drops it before
`snapshot_download` (fastembed 0.8.0 and 0.9.0:
`OnnxTextEmbedding.__init__` calls `download_model(...)` without
`**kwargs`). So the model floated on
`Qdrant/bge-small-en-v1.5-onnx-Q@main`, cached under the system temp
dir.

Today the floating `main` (`aa8f8b0`) differs from the pin only in
`README.md`, so no unreviewed weights have shipped. The pin just wasn't
protecting anything.

This PR makes the Python default model load the pinned snapshot and adds
tests that would have caught the dropped revision. The Rust
`EmbeddingScorer::try_new` change that was in this PR is now its own PR,
stacked on this one: #4190.

No linked issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/onnx_runtime.py`: pin `Qdrant/bge-small-en-v1.5-onnx-Q` at
`52398278842ec682c6f32300af41344b1c0b0bb2` in `_PINNED_REVISIONS`, so
`HEADROOM_HF_PIN=off` applies to it like every other pinned repo.
- `headroom/relevance/embedding.py`:
- The default model resolves its 5 files through
`hf_hub_download_local_first`, which is pinned, cache-first and guarded
on a miss.
- fastembed then gets only the snapshot directory as
`specific_model_path`. That construction runs with the Hub forced
offline, so a future fastembed that ignored the path would fail instead
of downloading an unpinned copy.
  - Other catalog models keep the existing `_load_text_embedding` path.
- The air-gapped error for the default model names the exact snapshot
and the command that seeds it, and says a fastembed cache is no longer
read (details under Release note).
- The private `_DEFAULT_MODEL_PINNED_REVISION` / `_pinned_revision` are
removed.
- `_load_text_embedding` now types its kwargs `dict[str, Any]`, fixing 8
mypy errors that only appear with fastembed installed.
- `pyproject.toml` / `uv.lock`: `fastembed>=0.6.0`.
`specific_model_path` first appears in fastembed v0.6.0; it isn't in
v0.4.x or v0.5.x. The lock still resolves 0.8.0; only the
`requires-dist` specifier changes.
- `.github/workflows/ci.yml` (`test-extras`):
- The prefetch warms the pinned snapshot through the loader's own
resolver, and the cache entry's key carries the SHA. The old prefetch
warmed fastembed's own cache, which the offline relevance tests would
now miss: `conftest.py:199` turns that into silent skips.
- The job runs the new test file, and the unused `FASTEMBED_CACHE_PATH`
is gone.
- Tests:
  - New `tests/test_fastembed_pinned_snapshot.py`:
    - the pinned revision reaches every file download;
- fastembed gets only the snapshot directory, built with the Hub forced
offline;
    - `HEADROOM_HF_PIN=off` floats;
    - files resolved from two different snapshots are refused;
- a cold cache while offline degrades without a download, and the error
names `repo@sha` and `--revision <sha>`;
    - real fastembed skips the Hub when given `specific_model_path`.
- `tests/test_stateless_writers.py`: the three `_pinned_revision` tests
are removed with the helper; `tests/test_hf_revision_pinning.py` covers
`_resolve_revision`.
- `tests/test_offline_egress_chokepoint.py`: `relevance/embedding.py`'s
cache-only count goes from 1 to 2 for the `_load_default_model`
construction.

## Release note

> **relevance:** the default embedder (`BAAI/bge-small-en-v1.5`) now
loads from the pinned snapshot
`Qdrant/bge-small-en-v1.5-onnx-Q@52398278842ec682c6f32300af41344b1c0b0bb2`
in the HuggingFace hub cache (`HF_HUB_CACHE` / `HF_HOME/hub` /
`~/.cache/huggingface/hub`), not from fastembed's cache. Existing
installs download ~67 MB once. Air-gapped hosts that pre-seeded the
fastembed cache must re-seed, e.g. `hf download
Qdrant/bge-small-en-v1.5-onnx-Q model_optimized.onnx tokenizer.json
tokenizer_config.json special_tokens_map.json config.json --revision
52398278842ec682c6f32300af41344b1c0b0bb2`. Without that, relevance
scoring falls back to BM25, and the error message says how to re-seed.

`CHANGELOG.md` is generated from the PR title, so this text is here for
whoever writes the release notes.

## Dependency Change (`fastembed>=0.4.0` → `>=0.6.0`)

This raises the minimum version of an existing dependency; no package is
added. `uv.lock` still resolves `fastembed 0.8.0`. The only lock change
is the `requires-dist` specifier, so the locked install set is
byte-for-byte the same.

- **Why this package:** it's already the `relevance` extra's embedder
(since Stage 3c.1). This PR adds no package and swaps none.
- **Who maintains it:** Qdrant (`qdrant/fastembed`), Apache-2.0,
actively released (0.6.0 on 2025-02-26 through 0.9.0 on 2026-10-07). The
GitHub Advisory Database has no advisories for `fastembed` (pip).
- **Install surface:** unchanged for the locked install (0.8.0).
Installs that resolve their own versions lose access to 0.4.x/0.5.x.
Runtime network shrinks: given `specific_model_path`, fastembed makes no
Hub call, and the only download is Headroom's own pinned,
offline-guarded `hf_hub_download_local_first`.
- **Why this version:** `specific_model_path` on `TextEmbedding` first
appears in v0.6.0, and the pinned-snapshot load depends on it.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HOME=$(mktemp -d) pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py \
    tests/test_relevance.py tests/test_relevance_extra.py tests/test_stateless_writers.py tests/test_hf_revision_pinning.py
7 failed, 209 passed, 3 skipped
# The 7 failures are TestOtlpExporterOffline / TestLangfuseExporterOffline / TestStartupRefusal /
# TestHeadroomCloudCompressionOffline: the OTel and Cloud extras aren't in this venv. The same 7 fail on
# origin/main (f1bc2609) here.

$ mypy headroom            # with fastembed 0.8.0 installed
Success: no issues found in 569 source files
$ ruff check .             # All checks passed!
$ ruff format --check .    # 1790 files already formatted
```

## Real Behavior Proof
- Exact command / steps: additional maintainer verification on
Windows/Python 3.12.13: python -m pytest
tests/test_fastembed_pinned_snapshot.py
tests/test_offline_egress_chokepoint.py tests/test_stateless_writers.py
-q.
- Observed result: the additional maintainer run passed all 160 selected
tests on b3eb7f349c, using the available
development native extension for startup; no fresh Rust implementation
qualification is claimed.

- Environment: macOS arm64, repo `.venv` (fastembed 0.8.0 = `uv.lock`),
rebased on `origin/main` `f1bc2609`.
- Default load: `EmbeddingScorer().score("authentication failed for user
admin", "login error")` loads from
`~/.cache/huggingface/hub/models--Qdrant--bge-small-en-v1.5-onnx-Q/snapshots/52398278842ec682c6f32300af41344b1c0b0bb2`
and scores 0.7389. `origin/main` loads
`<tmp>/fastembed_cache/…/snapshots/aa8f8b06…` (the floating `main`,
under `$TMPDIR`).
- Air-gapped re-seed, from an empty `HF_HOME`:
1. `HEADROOM_OFFLINE=1` gives `Embedding model 'BAAI/bge-small-en-v1.5'
is unavailable: HEADROOM_OFFLINE is set: … Pre-seed the HuggingFace hub
cache with Qdrant/bge-small-en-v1.5-onnx-Q@52398278… (`hf download …
--revision 52398278…`), or use the BM25-only scorer. A fastembed cache
seeded for an older Headroom is not read for this model.`
  2. I ran the `hf download` command from that message as printed.
  3. `HEADROOM_OFFLINE=1 HF_HUB_OFFLINE=1` then loads and scores 0.7389.
4. With `HEADROOM_HF_PIN=off`, the message names `@main` and `--revision
main` instead.
- Not tested: Windows, and Linux outside CI. fastembed 0.6.x/0.7.x at
runtime (the 0.6.0 floor comes from reading the v0.6.0 tag's source).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A. This is the default code path, with no
flag.
- Stable/default behavior changed: yes. The default relevance model
loads the pinned snapshot from the standard HF hub cache, not
fastembed's temp-dir cache, so existing installs download ~67 MB once.
The model files are byte-identical to what `main` served; only
`README.md` differs.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats it on `main`.
Air-gapped hosts re-seed as described in the Release note.
- Unsafe override required: no.
- Qualification impact: none expected. Same weights and tokenizer as the
current floating `main`.
- Rollback path: revert this PR. Nothing persistent changes beyond an
extra snapshot in the HF cache.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

- `_load_text_embedding` was typed `kwargs: dict[str, str]`, which made
mypy report 8 `arg-type` errors on `TextEmbedding(**kwargs)` whenever
fastembed is installed in the type-checking environment. CI's mypy run
doesn't install fastembed, so it never saw them.
- Follow-ups, not in this PR: Rust's Kompress and tokenizer Hub fetches
are still unpinned (there's no Rust `HEADROOM_HF_PIN`).

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
2026-10-20 19:45:26 -05:00

98 lines
3.1 KiB
Python

"""Tests for the complete stateless write guarantee.
Covers the process-wide stateless flag and the opt-in serving writers gated by
it: the output-savings recorder and persistent memory. (Savings tracker and
TOIN are covered in their own test modules.)
"""
from __future__ import annotations
import pytest
pytest.importorskip("fastapi")
from headroom import paths
from headroom.proxy.output_savings import SavingsRecorder
@pytest.fixture(autouse=True)
def _reset_stateless_globals():
"""The stateless flag and TOIN singleton are process-global — never leak."""
yield
paths.set_process_stateless(False)
try:
from headroom.telemetry.toin import reset_toin
reset_toin()
except Exception:
pass
# ---- process-wide stateless flag ------------------------------------------
def test_process_stateless_flag_set_and_clear(monkeypatch):
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
paths.set_process_stateless(False)
assert paths.process_is_stateless() is False
paths.set_process_stateless(True)
assert paths.process_is_stateless() is True
@pytest.mark.parametrize(
"value,expected", [("1", True), ("true", True), ("on", True), ("off", False), ("", False)]
)
def test_process_stateless_env(monkeypatch, value, expected):
paths.set_process_stateless(False)
monkeypatch.setenv("HEADROOM_STATELESS", value)
assert paths.process_is_stateless() is expected
def test_install_id_stays_in_memory_when_stateless(tmp_path, monkeypatch):
from headroom.telemetry import session
workspace = tmp_path / "state"
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(workspace))
monkeypatch.setattr(session, "_install_id", None)
paths.set_process_stateless(True)
assert len(session.install_id()) == 32
assert not workspace.exists()
# ---- output-savings recorder ----------------------------------------------
def test_output_savings_flush_writes_nothing_when_stateless(tmp_path, monkeypatch):
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
path = tmp_path / "output_savings.json"
paths.set_process_stateless(True)
rec = SavingsRecorder(path)
rec.flush()
assert not path.exists()
def test_output_savings_flush_persists_when_not_stateless(tmp_path, monkeypatch):
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
path = tmp_path / "output_savings.json"
paths.set_process_stateless(False)
rec = SavingsRecorder(path)
rec.flush()
assert path.exists()
# ---- persistent memory ----------------------------------------------------
def test_memory_disabled_under_stateless(tmp_path, monkeypatch):
"""A stateless proxy with --memory must not initialize memory or write a DB."""
workspace = tmp_path / "state"
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(workspace))
from headroom.proxy.server import ProxyConfig, create_app
app = create_app(ProxyConfig(memory_enabled=True, stateless=True))
proxy = app.state.proxy
assert proxy.memory_handler is None
assert not workspace.exists()