fix(kompress): read the pinned snapshot from Python's one HF cache in the native proxy (#4438)

## Description

**Stacked on #4190.** This branch contains #4190's commits (it reuses
#4190's Rust HF-cache resolver). Please review only the five commits
listed under Changes Made. Once #4190 merges, I'll rebase onto main and
the diff will shrink to those commits.

**Rebased onto main (`7e0094d6`), 2026-11-03.** The branch is now
linear, with no merge commits:

- #4190's first three commits: `e173e100`, `344cf365`, `41918cf0`.
- This PR's five commits (SHAs updated under Changes Made).
- `d9f16d39` fix(relevance): keep an empty HF cache variable, as
huggingface_hub does. This is #4190's last commit, applied to
`crate::hf_cache` since this PR moves the resolver there. It replaces
the earlier merge `a2e89118`.

What the rebase changed:

- The #4152 commit (`cb1c3a33`) is gone from the branch: main already
has it as `f380692b`.
- The main-sync merges (`7e17a2d3`, `6904fd19`, `8be25204`, `79e3a5cb`,
`19b1adc5`, and #4190's merges) are linearized away.
- The Rust embedder conflicts with main's batch-size cap (#4507) were
re-resolved to the same content as before: every file this PR touches is
byte-identical between the previous head `19b1adc5` and the new head
`d9f16d39` (`git diff 19b1adc5 d9f16d39 -- <files changed vs main>` is
empty). Only the history and SHAs changed.

The native proxy's live-zone Kompress is cache-only
(`Kompress::from_cache`): it reads the files Python downloaded. Its
cache lookup (`hf_hub_roots` + `hf_cache_file` on main) disagrees with
`huggingface_hub` in several ways. When it misses, `kompress_ready`
stays false and live-zone passes text through uncompressed. When it
loads, it can load the wrong files. This is the follow-up planned in
https://github.com/headroomlabs-ai/headroom/pull/4190#issuecomment-6209746394
(Devin's note on `hf_hub_roots`). The Python proxy resolves Kompress
through `huggingface_hub` and is unaffected.

| setup | Python | native Kompress on main |
|---|---|---|
| only `HUGGINGFACE_HUB_CACHE` | found | miss |
| only `XDG_CACHE_HOME` | found | miss |
| `HF_HUB_CACHE=${ROOT}/hub` | `<ROOT>/hub` | miss |
| `HF_HUB_CACHE` set + a copy under `$HOME` | looks only in
`HF_HUB_CACHE` | loads the `$HOME` copy |
| pinned snapshot plus older/newer ones | the `_PINNED_REVISIONS` commit
| first snapshot in `read_dir` order; tokenizer and ONNX looked up
independently |

The last row matters most: every Kompress pin bump leaves the old
snapshot in the cache.

Closes # (no issue; follow-up to #4190)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

Five commits on top of #4190:

- `af191f19` (was `ddec6b88`) fix(kompress): read the pinned snapshot
from Python's one HF cache in the native proxy.
- #4190's resolver moves to a shared `crate::hf_cache` module:
`hub_cache_dir`, the `expandvars` port, `pinned_or_main`, and a
pin-aware `snapshot_path`.
- Kompress reads one snapshot of each repo, at `DEFAULT_MODEL_REVISION`
/ `DEFAULT_TOKENIZER_REVISION`. A new `tests/test_kompress_rust_pins.py`
keeps these equal to `_PINNED_REVISIONS`.
- `HEADROOM_HF_PIN=off` and unpinned repos follow `refs/main`, as in
Python.
  - `kompress_cache_miss` logs the one cache and revision it looked in.
- New `kompress::cached_artifacts` lets the parity harnesses load the
files production would load.
- `7cdc3d94` (was `5e71eeea`) fix(tokenizer):
`HfTokenizer::from_pretrained` downloads into Python's hub cache with
Python's `HF_TOKEN` / `HF_ENDPOINT`. `hf-hub`'s defaults read neither,
and with `Api::new` not even `HF_HOME`.
- `c43c5eb8` (was `5fd3a87d`) fix(hf-cache): changes to the shared Hub
client:
- The tokenizer and embedder share one Hub client, `hf_cache::hub_api`.
- Trailing slashes are stripped from `HF_ENDPOINT`, as Python's
`constants.ENDPOINT` does.
- The token is withheld from a plain-`http://` non-loopback endpoint,
with a warning.
- `91128294` (was `70e61b02`) fix(hf-cache): `hub_api(cache, purpose,
repo)` runs `guard_egress` itself, before it builds the client. A new
caller can't skip the air-gap check, and the offline-egress sweep in
`tests/test_offline_egress_chokepoint.py` sees the site as guarded. The
tokenizer still returns `HfTokenizerError::Offline`, and the embedder
keeps its messages.
- `bbee29cd` (was `14dbac29`) review follow-ups:
  - `HEADROOM_OFFLINE` wins over a missing cache directory.
  - The `hub_api` docs are scoped to the tokenizer and embedder.
  - The order test checks inside `hub_api`'s body.
  - Stale comments are fixed.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

After the rebase, on `d9f16d39` (macOS arm64, no `ORT_DYLIB_PATH`, so
the gated model tests skip):

```text
$ cargo test -p headroom-core --lib
test result: ok. 1002 passed; 0 failed; 1 ignored
$ cargo clippy -p headroom-core --all-targets -- -D warnings                  # clean
$ cargo clippy -p headroom-core --lib --no-default-features -- -D warnings    # clean
$ cargo fmt --all -- --check                                                  # clean
$ pytest -q tests/test_fastembed_pinned_snapshot.py tests/test_offline_egress_chokepoint.py \
    tests/test_hf_revision_pinning.py tests/test_kompress_rust_pins.py
247 passed, 3 skipped
$ ruff check / ruff format --check on the PR's Python files (ruff 0.17.2)     # clean
```

Before the rebase:

```text
$ cargo test -p headroom-core -- --skip kompress_matches_python_fixtures_byte_for_byte --skip short_input_passes_through
  (HF_HUB_CACHE=$(mktemp -d), cold like CI; own CARGO_TARGET_DIR)
20 test binaries, all "test result: ok", 0 failed
$ cargo clippy --workspace --all-targets -- -D warnings      clean
$ cargo clippy -p headroom-core --no-default-features --lib -- -D warnings   clean
$ cargo fmt --all -- --check                                  clean
$ pytest tests/test_offline_egress_chokepoint.py tests/test_kompress_rust_pins.py tests/test_fastembed_pinned_snapshot.py
195 passed, 4 skipped, 2 failed
  the 2 failures are TestHeadroomCloudCompressionOffline (litellm fetching its
  pricing table from raw.githubusercontent.com); they fail identically on
  origin/main e995eb9f with the locked LiteLLM 1.101.0, and #4242 fixes them
$ ruff check / ruff format --check (0.16.8)                   clean
```

mypy is not checked: this PR changes no `headroom/` Python.

## Real Behavior Proof

- Environment: macOS 27 arm64, debug build of `headroom-core` (default
`ml` feature), `ORT_DYLIB_PATH` = onnxruntime 1.31.0, the real cached
`chopratejas/kompress-v2-base` and `answerdotai/ModernBERT-base`
snapshots at their pins, symlinked into throwaway hub caches;
huggingface_hub 1.16.1 for the Python column.
- Exact command / steps: a throwaway example (not committed) calls
`Kompress::from_cache(KompressConfig::default())` and prints the
outcome. It runs once per cache layout under `env -i` with an empty
`HOME`, built from `origin/main` (e995eb9f) and from this branch. The
Python column prints `huggingface_hub.constants.HF_HUB_CACHE` under the
same environment. In the pin-bump layout, the older (`0000…`) and newer
(`ffff…`, `refs/main`) snapshots hold garbage `tokenizer.json` / ONNX
files.
- Observed result: the branch now agrees with Python in every layout. On
main, four layouts miss where Python finds the cache, one loads a copy
Python would never read, and the pin-bump layout loads a non-pinned
snapshot. Each side was built in its own target dir.

```text
== origin/main (e995eb9f)
HF_HUB_CACHE=<T>/hub (control)               loaded              | Python HF_HUB_CACHE=<T>/hub
only HUGGINGFACE_HUB_CACHE                   cache miss (None)   | Python HF_HUB_CACHE=<T>/hub
only XDG_CACHE_HOME                          cache miss (None)   | Python HF_HUB_CACHE=<T>/xdg/huggingface/hub
HF_HUB_CACHE=${PROBE_ROOT}/hub               cache miss (None)   | Python HF_HUB_CACHE=<T>/root/hub
HF_HUB_CACHE=<empty>, copy under $HOME       loaded              | Python HF_HUB_CACHE=<T>/empty
pin + older/newer snapshots, main=newer      error: failed to load tokenizer for `answerdotai/ModernBERT-base`: expected value at line 1 column 1 | Python HF_HUB_CACHE=<T>/pinbump
== this branch (14dbac29)
HF_HUB_CACHE=<T>/hub (control)               loaded              | Python HF_HUB_CACHE=<T>/hub
only HUGGINGFACE_HUB_CACHE                   loaded              | Python HF_HUB_CACHE=<T>/hub
only XDG_CACHE_HOME                          loaded              | Python HF_HUB_CACHE=<T>/xdg/huggingface/hub
HF_HUB_CACHE=${PROBE_ROOT}/hub               loaded              | Python HF_HUB_CACHE=<T>/root/hub
HF_HUB_CACHE=<empty>, copy under $HOME       cache miss (None)   | Python HF_HUB_CACHE=<T>/empty
pin + older/newer snapshots, main=newer      loaded              | Python HF_HUB_CACHE=<T>/pinbump
```

On main, the pin-bump row read a garbage non-pinned snapshot (here it
errors; with two real model versions it would silently load the wrong
one). The branch reads the pinned snapshot.
- Not tested: a Windows build (the `ntpath` expandvars rules are covered
by the shared fixture on macOS/Linux); a live Hub download through
`hub_api` with `HF_ENDPOINT` pointing at a real mirror; the native proxy
end to end (the probe calls the same `Kompress::from_cache` the proxy's
live-zone warm-up calls).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this changes where the native
proxy's cache-only Kompress and the Rust tokenizer/embedder look for Hub
files.
- Minimum rollout channel: stable (no rollout gate).
- Stable/default behavior changed: yes. The native proxy now loads
Kompress from the cache Python uses, at the pinned snapshot. It no
longer loads from other roots or from whichever snapshot `read_dir`
returns first. Rust tokenizer downloads land in Python's hub cache.
- Kill switch / disable path: `HEADROOM_HF_PIN=off` floats the pinned
repos on `refs/main`, as in Python; Kompress itself is disabled as
before.
- Unsafe override required: no.
- Qualification impact: native live-zone Kompress becomes ready on hosts
whose Python cache is configured via `HUGGINGFACE_HUB_CACHE`,
`XDG_CACHE_HOME` or `$VAR` paths; parity harnesses read the pinned
artifacts.
- Rollback path: revert this PR; no persisted state or migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no user-facing docs describe the native Kompress cache
lookup.
- Follow-up, not in this PR: `Kompress::from_pretrained` still builds
its own `hf_hub::api::sync::Api::new()` client behind its own guard, so
it ignores `HF_ENDPOINT` / `HF_TOKEN` and downloads into `hf-hub`'s
default cache. It has no production caller (the proxy uses
`from_cache`); moving it onto `hf_cache::hub_api` is a separate change.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4438"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Kenneth Wong
2026-11-04 12:29:32 -06:00
committed by GitHub
co-authored by Kenneth Wong JD Davis
parent d8ff0fe9d2
commit eef44f60cc
11 changed files with 1325 additions and 578 deletions
+712
View File
@@ -0,0 +1,712 @@
//! The HuggingFace hub cache, located and read the way Python's
//! `huggingface_hub` does, so the Rust loaders (the relevance embedder,
//! Kompress and the HuggingFace tokenizer) find the files the Python side
//! downloaded and pinned, and authenticate with the same token.
//!
//! All network-free: [`hub_cache_dir`] picks the one cache directory Python
//! would use, [`snapshot_path`] resolves a repo at a revision inside it,
//! [`hf_token`] finds the access token and [`hub_api`] builds a client from
//! them. A path variable that is set but empty counts as set, as in Python
//! (see [`hub_cache_dir`]). The pinned revisions themselves live
//! next to each loader and are kept equal to `_PINNED_REVISIONS` in
//! `headroom/onnx_runtime.py` by tests on the Python side.
//!
//! Not behind the `ml` feature: the tokenizer, which is not either, uses the
//! cache and token lookups.
use std::path::{Path, PathBuf};
use thiserror::Error;
/// True when `HEADROOM_HF_PIN` turns revision pinning off, so pinned repos
/// float on `main` (same values as Python's `onnx_runtime._resolve_revision`).
#[cfg_attr(not(feature = "ml"), allow(dead_code))] // the pinned loaders are `ml`-only
pub(crate) fn pin_disabled() -> bool {
let pin = std::env::var("HEADROOM_HF_PIN").unwrap_or_default();
matches!(
pin.trim().to_ascii_lowercase().as_str(),
"off" | "0" | "false" | "no"
)
}
/// The revision to load for a repo pinned at `pinned`: the pin, or `main`
/// when `HEADROOM_HF_PIN` turns pinning off.
#[cfg_attr(not(feature = "ml"), allow(dead_code))]
pub(crate) fn pinned_or_main(pinned: &str) -> &str {
if pin_disabled() {
"main"
} else {
pinned
}
}
/// The snapshot directory of `repo` (`"owner/name"`) at `revision` in the hub
/// cache `cache`, whether or not it exists. Never touches the network.
///
/// A branch name such as `main` resolves through `refs/<revision>`; a commit
/// SHA is used as the directory name directly. `hf_hub::Cache` resolves every
/// revision through `refs/`, which Python's `huggingface_hub` does not write
/// for a commit SHA, so a snapshot Python downloaded at a pin would be missed.
#[cfg_attr(not(feature = "ml"), allow(dead_code))]
pub(crate) fn snapshot_path(cache: &Path, repo: &str, revision: &str) -> PathBuf {
let repo_dir = cache.join(format!("models--{}", repo.replace('/', "--")));
let commit = std::fs::read_to_string(repo_dir.join("refs").join(revision))
.map(|c| c.trim().to_string())
.unwrap_or_else(|_| revision.to_string());
repo_dir.join("snapshots").join(commit)
}
/// The HuggingFace hub cache directory, resolved the way Python's
/// `huggingface_hub` resolves `constants.HF_HUB_CACHE`, so both languages look
/// up and download into the same place: `HF_HUB_CACHE`, then
/// `HUGGINGFACE_HUB_CACHE` (its legacy name), then `HF_HOME/hub`, then
/// `XDG_CACHE_HOME/huggingface/hub`, then `~/.cache/huggingface/hub`.
///
/// It is exactly one directory. A snapshot in a lower-priority cache is not
/// used when a higher one is configured, which is also what Python does: with
/// `HF_HUB_CACHE` set, a model under `~/.cache` is neither found nor reused.
///
/// Both expansions happen as in `huggingface_hub.constants`:
/// `expandvars(expanduser(...))`, first on `HF_HOME` (or its
/// `XDG_CACHE_HOME` default) and then on the hub path itself. So
/// `HF_HUB_CACHE=${ROOT}/hub` names the same directory in both languages, and
/// a `~` that only appears once a variable is expanded stays literal, as it
/// does in Python. The home directory comes from `std::env::home_dir`, which
/// falls back to the user database when `HOME` is unset, as `expanduser`
/// does. A variable that is set but empty still counts, as with `os.getenv`:
/// `HF_HUB_CACHE=''` is the working directory itself, `HF_HOME=''` is `hub`
/// and `XDG_CACHE_HOME=''` is `huggingface/hub` under it, and none of them
/// falls through to a lower-priority cache. `tests/parity/fixtures/hf_hub_cache/`
/// holds the cases both languages are checked against.
pub(crate) fn hub_cache_dir() -> Option<PathBuf> {
let hub = match env_path("HF_HUB_CACHE").or_else(|| env_path("HUGGINGFACE_HUB_CACHE")) {
Some(dir) => dir,
None => hf_home_dir()?.join("hub"),
};
Some(expand(hub))
}
/// `huggingface_hub.constants.HF_HOME`: `HF_HOME`, else
/// `XDG_CACHE_HOME/huggingface`, else `~/.cache/huggingface`, expanded as
/// Python does. The hub cache defaults to its `hub` folder and the login
/// token to its `token` file.
fn hf_home_dir() -> Option<PathBuf> {
let dir = match env_path("HF_HOME") {
Some(dir) => dir,
None => env_path("XDG_CACHE_HOME")
.or_else(|| std::env::home_dir().map(|home| home.join(".cache")))?
.join("huggingface"),
};
Some(expand(dir))
}
/// The HuggingFace access token, found the way `huggingface_hub.get_token()`
/// finds it: `HF_TOKEN`, then its legacy name `HUGGING_FACE_HUB_TOKEN`, then
/// the file at `HF_TOKEN_PATH` (default `HF_HOME/token`, where
/// `hf auth login` writes it). Each value is cleaned like Python's
/// `_clean_token`: line breaks removed, outer whitespace trimmed, and an
/// empty result counts as no token.
///
/// `hf-hub`'s own lookup reads only a `token` file beside the hub cache, so
/// `HF_TOKEN`, an explicit `HF_TOKEN_PATH`, or a hub cache moved away from
/// `HF_HOME` would leave a gated download unauthenticated.
pub(crate) fn hf_token() -> Option<String> {
let clean = |token: String| {
let token = token.replace(['\r', '\n'], "");
let token = token.trim();
(!token.is_empty()).then(|| token.to_string())
};
// `HF_TOKEN or HUGGING_FACE_HUB_TOKEN`, then clean: a whitespace-only
// `HF_TOKEN` is still the one picked, and cleans to no token, so the
// legacy name is not consulted (the login file is).
["HF_TOKEN", "HUGGING_FACE_HUB_TOKEN"]
.into_iter()
.find_map(|name| std::env::var(name).ok().filter(|v| !v.is_empty()))
.and_then(clean)
.or_else(|| {
let path = match env_path("HF_TOKEN_PATH") {
Some(path) => expand(path),
None => hf_home_dir()?.join("token"),
};
std::fs::read_to_string(path).ok().and_then(clean)
})
}
/// The Hub endpoint: `HF_ENDPOINT` without trailing slashes (Python's
/// `constants.ENDPOINT` strips them, and `hf-hub` appends `/{repo}/…`, so a
/// kept slash would request `//repo`), else `https://huggingface.co`.
pub(crate) fn hf_endpoint() -> String {
match std::env::var("HF_ENDPOINT") {
Ok(endpoint) => endpoint.trim_end_matches('/').to_string(),
Err(_) => "https://huggingface.co".to_string(),
}
}
/// Why [`hub_api`] returned no client.
#[derive(Debug, Error)]
pub(crate) enum HubApiError {
/// `HEADROOM_OFFLINE` is set; refused before the client existed.
#[error(transparent)]
Offline(#[from] crate::offline::OfflineEgressBlocked),
/// `hf-hub` could not build the client.
#[error(transparent)]
Build(#[from] hf_hub::api::sync::ApiError),
}
/// An `hf-hub` client that caches in `cache` and talks to [`hf_endpoint`]
/// with [`hf_token`], as Python's `huggingface_hub` would, for fetching
/// `destination` (a repo id) for `purpose`.
///
/// This is the air-gap chokepoint for the tokenizer and embedder downloads:
/// under `HEADROOM_OFFLINE` it returns [`HubApiError::Offline`] before
/// building the client, because building it resolves the endpoint and the
/// `ureq` agent, and `hf-hub` downloads on a cache miss even with
/// `HF_HUB_OFFLINE` set. Guarding here rather than in each caller means no
/// caller of this function can skip it. (`Kompress::from_pretrained` still
/// builds its own `Api::new()` client behind its own guard; production loads
/// Kompress cache-only through `Kompress::from_cache`.)
///
/// The token is not sent over plain `http://` to a host other than loopback:
/// it would cross the network in clear text on every request, so the client
/// goes without it and says why. Python would send it; a gated repo behind
/// such a mirror then answers 401 rather than leak the credential.
pub(crate) fn hub_api(
cache: &Path,
purpose: &str,
destination: &str,
) -> Result<hf_hub::api::sync::Api, HubApiError> {
crate::offline::guard_egress(purpose, destination)?;
let endpoint = hf_endpoint();
let token = hf_token().filter(|_| {
let exposed = sends_in_clear_text(&endpoint);
if exposed {
tracing::warn!(
event = "hf_token_withheld",
endpoint = %endpoint,
"HF_ENDPOINT is plain http to a non-loopback host; not sending the \
HuggingFace token to it. Use an https endpoint for gated repos."
);
}
!exposed
});
hf_hub::api::sync::ApiBuilder::from_cache(hf_hub::Cache::new(cache.to_path_buf()))
.with_progress(false)
.with_endpoint(endpoint)
.with_token(token)
.build()
.map_err(HubApiError::Build)
}
/// Whether a request to `endpoint` would carry its headers unencrypted off
/// this machine: `http://` to anything but a loopback host.
fn sends_in_clear_text(endpoint: &str) -> bool {
let Some(rest) = endpoint
.get(..7)
.filter(|scheme| scheme.eq_ignore_ascii_case("http://"))
.map(|_| &endpoint[7..])
else {
return false;
};
let authority = rest.split(['/', '?', '#']).next().unwrap_or("");
let host_port = authority.rsplit('@').next().unwrap_or("");
let host = match host_port.strip_prefix('[') {
Some(v6) => v6.split(']').next().unwrap_or(""),
None => host_port.split(':').next().unwrap_or(""),
};
let loopback = host.eq_ignore_ascii_case("localhost")
|| host
.parse::<std::net::IpAddr>()
.is_ok_and(|ip| ip.is_loopback());
!loopback
}
/// A path-valued variable. An empty one is still set, as `os.getenv` reads
/// it, so it never falls through to the next variable.
fn env_path(name: &str) -> Option<PathBuf> {
std::env::var_os(name).map(PathBuf::from)
}
/// `expandvars(expanduser(path))`, the order `huggingface_hub.constants` uses.
fn expand(path: PathBuf) -> PathBuf {
expand_vars(&expand_user(path, std::env::home_dir().as_deref()))
}
/// Python's `os.path.expanduser` for a leading bare `~` component. `~user`
/// stays as written, which is also what `expanduser` does when it cannot
/// resolve the user.
fn expand_user(path: PathBuf, home: Option<&Path>) -> PathBuf {
match (path.strip_prefix("~"), home) {
(Ok(rest), Some(home)) => home.join(rest),
_ => path,
}
}
/// Python's `os.path.expandvars`: `posixpath` rules off Windows, `ntpath`
/// rules on it. A path that is not valid Unicode is returned unchanged.
fn expand_vars(path: &Path) -> PathBuf {
let Some(text) = path.to_str() else {
return path.to_path_buf();
};
#[cfg(windows)]
let expanded = expand_vars_nt(text, env_lookup);
#[cfg(not(windows))]
let expanded = expand_vars_posix(text, env_lookup);
PathBuf::from(expanded)
}
/// An environment value for `expand_vars`. Names the OS cannot hold (empty,
/// or containing `=` or NUL) are unset, which is how Python's `os.environ`
/// lookup fails for them too.
fn env_lookup(name: &str) -> Option<std::ffi::OsString> {
if name.is_empty() || name.contains(['=', '\0']) {
return None;
}
std::env::var_os(name)
}
/// `posixpath.expandvars`: `$name` (ASCII letters, digits, `_`) or
/// `${anything but a brace}`. An unset name, and a `$` that starts neither
/// form, stay as written. Expanded values are not scanned again.
#[cfg_attr(windows, allow(dead_code))] // tests run both rule sets everywhere
fn expand_vars_posix(
text: &str,
lookup: impl Fn(&str) -> Option<std::ffi::OsString>,
) -> std::ffi::OsString {
let mut out = std::ffi::OsString::new();
let mut rest = text;
while let Some(at) = rest.find('$') {
out.push(&rest[..at]);
let after = &rest[at + 1..];
let word = after
.find(|c: char| !(c.is_ascii_alphanumeric() || c == '_'))
.unwrap_or(after.len());
// (name, length of the reference after the `$`)
let reference = if word > 0 {
Some((&after[..word], word))
} else {
after
.strip_prefix('{')
.and_then(|braced| braced.find('}').map(|end| (&braced[..end], end + 2)))
};
match reference {
Some((name, len)) => {
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[at..at + 1 + len]),
}
rest = &after[len..];
}
None => {
out.push("$");
rest = after;
}
}
}
out.push(rest);
out
}
/// `ntpath.expandvars`: `%name%`, `${name}` and `$name` (ASCII letters,
/// digits, `_`, `-`); `%%` and `$$` are a literal `%` and `$`; nothing inside
/// single quotes expands. An unset or unterminated reference stays as
/// written.
#[cfg_attr(not(windows), allow(dead_code))] // tests run both rule sets everywhere
fn expand_vars_nt(
text: &str,
lookup: impl Fn(&str) -> Option<std::ffi::OsString>,
) -> std::ffi::OsString {
let mut out = std::ffi::OsString::new();
let mut rest = text;
while let Some(c) = rest.chars().next() {
let after = &rest[c.len_utf8()..];
match c {
'\'' => match after.find('\'') {
Some(end) => {
out.push(&rest[..end + 2]);
rest = &after[end + 1..];
}
None => {
out.push(rest);
rest = "";
}
},
'%' if after.starts_with('%') => {
out.push("%");
rest = &after[1..];
}
'%' => match after.find('%') {
Some(end) => {
let name = &after[..end];
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[..end + 2]),
}
rest = &after[end + 1..];
}
None => {
out.push(rest);
rest = "";
}
},
'$' if after.starts_with('$') => {
out.push("$");
rest = &after[1..];
}
'$' if after.starts_with('{') => match after[1..].find('}') {
Some(end) => {
let name = &after[1..1 + end];
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[..end + 3]),
}
rest = &after[end + 2..];
}
None => {
out.push(rest);
rest = "";
}
},
'$' => {
let end = after
.find(|c: char| !(c.is_ascii_alphanumeric() || c == '_' || c == '-'))
.unwrap_or(after.len());
let name = &after[..end];
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[..end + 1]),
}
rest = &after[end..];
}
_ => {
out.push(&rest[..c.len_utf8()]);
rest = after;
}
}
}
out
}
#[cfg(test)]
pub(crate) mod tests {
use super::*;
/// Every variable `hub_cache_dir` reads.
const CACHE_ENV: [&str; 6] = [
"HF_HUB_CACHE",
"HUGGINGFACE_HUB_CACHE",
"HF_HOME",
"XDG_CACHE_HOME",
"HOME",
"USERPROFILE",
];
/// Saves [`CACHE_ENV`], clears it, and puts it back on drop, so a failing
/// assertion cannot leak a fake cache into later tests. Take
/// `env_lock()` first.
pub(crate) struct CacheEnv(Vec<(String, Option<std::ffi::OsString>)>);
impl CacheEnv {
pub(crate) fn cleared() -> Self {
Self::cleared_with(std::iter::empty())
}
/// [`Self::cleared`], plus `extra` variables (the ones a test's
/// paths reference).
pub(crate) fn cleared_with(extra: impl IntoIterator<Item = String>) -> Self {
let names: Vec<String> = CACHE_ENV
.iter()
.map(|name| name.to_string())
.chain(extra)
.collect();
let saved = names
.into_iter()
.map(|name| {
let value = std::env::var_os(&name);
std::env::remove_var(&name);
(name, value)
})
.collect();
CacheEnv(saved)
}
}
impl Drop for CacheEnv {
fn drop(&mut self) {
for (name, value) in &self.0 {
match value {
Some(value) => std::env::set_var(name, value),
None => std::env::remove_var(name),
}
}
}
}
/// Same order as `huggingface_hub.constants.HF_HUB_CACHE`: each variable
/// wins over every one after it, and only one directory comes back.
#[test]
fn hub_cache_dir_follows_huggingface_hub_precedence() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared();
let t = tempfile::tempdir().unwrap();
let p = |name: &str| t.path().join(name);
std::env::set_var("HOME", p("home"));
std::env::set_var("USERPROFILE", p("home"));
std::env::set_var("XDG_CACHE_HOME", p("xdg"));
std::env::set_var("HF_HOME", p("hf-home"));
std::env::set_var("HUGGINGFACE_HUB_CACHE", p("legacy"));
std::env::set_var("HF_HUB_CACHE", p("explicit"));
assert_eq!(hub_cache_dir(), Some(p("explicit")));
std::env::remove_var("HF_HUB_CACHE");
assert_eq!(hub_cache_dir(), Some(p("legacy")));
std::env::remove_var("HUGGINGFACE_HUB_CACHE");
assert_eq!(hub_cache_dir(), Some(p("hf-home").join("hub")));
std::env::remove_var("HF_HOME");
assert_eq!(
hub_cache_dir(),
Some(p("xdg").join("huggingface").join("hub"))
);
std::env::remove_var("XDG_CACHE_HOME");
assert_eq!(
hub_cache_dir(),
Some(p("home").join(".cache").join("huggingface").join("hub"))
);
// A leading `~` expands.
std::env::set_var("HF_HOME", "~/models");
assert_eq!(hub_cache_dir(), Some(p("home").join("models").join("hub")));
// A set but empty variable is the working directory, as in Python,
// and still wins over every variable after it.
std::env::set_var("HF_HUB_CACHE", "");
assert_eq!(hub_cache_dir(), Some(PathBuf::new()));
std::env::remove_var("HF_HUB_CACHE");
std::env::set_var("HF_HOME", "");
std::env::set_var("XDG_CACHE_HOME", p("xdg"));
assert_eq!(hub_cache_dir(), Some(PathBuf::from("hub")));
}
/// An empty `HF_TOKEN_PATH` names no readable file, so there is no token;
/// it must not fall back to `HF_HOME/token`. Python's `get_token()` raises
/// `IsADirectoryError` here (it reads `.`), so this case is Rust-only
/// rather than in the shared fixture.
#[test]
fn an_empty_hf_token_path_does_not_fall_back_to_hf_home() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with(
["HF_TOKEN", "HUGGING_FACE_HUB_TOKEN", "HF_TOKEN_PATH"].map(String::from),
);
let hf_home = tempfile::tempdir().unwrap();
std::fs::write(hf_home.path().join("token"), "hf_from_home\n").unwrap();
std::env::set_var("HF_HOME", hf_home.path());
std::env::remove_var("HF_TOKEN_PATH");
assert_eq!(hf_token().as_deref(), Some("hf_from_home"));
std::env::set_var("HF_TOKEN_PATH", "");
assert_eq!(hf_token(), None);
}
/// Cases checked against Python itself (`posixpath`, `ntpath` and
/// `huggingface_hub.constants`) by `tests/test_fastembed_pinned_snapshot.py`.
const HF_CACHE_FIXTURE: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/../../tests/parity/fixtures/hf_hub_cache/cases.json"
));
/// Both rule sets on every platform, so the Windows port is exercised by
/// the macOS and Linux runs too.
#[test]
fn expand_vars_matches_python_posixpath_and_ntpath() {
let fixture: serde_json::Value = serde_json::from_str(HF_CACHE_FIXTURE).unwrap();
let section = &fixture["expandvars"];
let env: std::collections::HashMap<String, String> =
serde_json::from_value(section["env"].clone()).unwrap();
let lookup = |name: &str| env.get(name).map(std::ffi::OsString::from);
for case in section["cases"].as_array().unwrap() {
let input = case["input"].as_str().unwrap();
let want = |key: &str| std::ffi::OsString::from(case[key].as_str().unwrap());
assert_eq!(
expand_vars_posix(input, lookup),
want("posix"),
"posix {input:?}"
);
assert_eq!(expand_vars_nt(input, lookup), want("nt"), "nt {input:?}");
}
}
/// The review case among them: `HF_HUB_CACHE=${ROOT}/hub` is
/// `<ROOT>/hub` in Python, so Rust must not look up or download into a
/// literal `${ROOT}` directory instead.
#[test]
fn hub_cache_dir_matches_the_shared_fixture() {
let fixture: serde_json::Value = serde_json::from_str(HF_CACHE_FIXTURE).unwrap();
let section = &fixture["cache_dir"];
let probe_vars: Vec<String> =
serde_json::from_value(section["probe_vars"].clone()).unwrap();
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with(probe_vars.clone());
let t = tempfile::tempdir().unwrap();
let (root, home) = (t.path().join("root"), t.path().join("home"));
let fill = |text: &str| {
text.replace("{root}", root.to_str().unwrap())
.replace("{home}", home.to_str().unwrap())
};
for case in section["cases"].as_array().unwrap() {
for name in CACHE_ENV
.iter()
.copied()
.chain(probe_vars.iter().map(String::as_str))
{
std::env::remove_var(name);
}
std::env::set_var("HOME", &home);
std::env::set_var("USERPROFILE", &home);
for (name, value) in case["env"].as_object().unwrap() {
std::env::set_var(name, fill(value.as_str().unwrap()));
}
let expected = match &case["expected"] {
serde_json::Value::Object(by_os) => {
&by_os[if cfg!(windows) { "nt" } else { "posix" }]
}
expected => expected,
};
assert_eq!(
hub_cache_dir(),
Some(PathBuf::from(fill(expected.as_str().unwrap()))),
"{}",
case["name"]
);
}
}
/// With no `HOME` (a service started with a bare environment) the home
/// directory still comes from the user database, as Python's
/// `expanduser` does, instead of leaving the scorer with no cache.
#[cfg(all(feature = "ml", unix))]
#[test]
fn hub_cache_dir_without_home_uses_the_user_database() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared();
let dir = hub_cache_dir().expect("a cache directory without HOME");
assert!(dir.is_absolute(), "{}", dir.display());
assert!(
dir.ends_with(Path::new(".cache").join("huggingface").join("hub")),
"{}",
dir.display()
);
}
/// The fixture's `token` cases, which the Python half checks against
/// `huggingface_hub.get_token()`.
#[test]
fn hf_token_matches_the_shared_fixture() {
let fixture: serde_json::Value = serde_json::from_str(HF_CACHE_FIXTURE).unwrap();
let probe_vars: Vec<String> =
serde_json::from_value(fixture["cache_dir"]["probe_vars"].clone()).unwrap();
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with(probe_vars.clone());
for case in fixture["token"]["cases"].as_array().unwrap() {
let t = tempfile::tempdir().unwrap();
let (root, home) = (t.path().join("root"), t.path().join("home"));
let fill = |text: &str| {
text.replace("{root}", root.to_str().unwrap())
.replace("{home}", home.to_str().unwrap())
};
for name in CACHE_ENV
.iter()
.copied()
.chain(probe_vars.iter().map(String::as_str))
{
std::env::remove_var(name);
}
std::env::set_var("HOME", &home);
std::env::set_var("USERPROFILE", &home);
for (name, value) in case["env"].as_object().unwrap() {
std::env::set_var(name, fill(value.as_str().unwrap()));
}
for (path, content) in case["files"].as_object().unwrap() {
let path = PathBuf::from(fill(path));
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
std::fs::write(path, content.as_str().unwrap()).unwrap();
}
assert_eq!(
hf_token().as_deref(),
case["expected"].as_str(),
"{}",
case["name"]
);
}
}
#[test]
fn hf_endpoint_drops_trailing_slashes_like_python() {
let _guard = crate::test_support::env_lock();
let saved = std::env::var_os("HF_ENDPOINT");
std::env::remove_var("HF_ENDPOINT");
assert_eq!(hf_endpoint(), "https://huggingface.co");
std::env::set_var("HF_ENDPOINT", "https://mirror.example.com//");
assert_eq!(hf_endpoint(), "https://mirror.example.com");
match saved {
Some(v) => std::env::set_var("HF_ENDPOINT", v),
None => std::env::remove_var("HF_ENDPOINT"),
}
}
/// The token is withheld only where it would cross the network unencrypted.
#[test]
fn plain_http_off_loopback_is_the_only_clear_text_endpoint() {
for endpoint in [
"http://mirror.example.com",
"HTTP://mirror.example.com:8080/hub",
"http://10.0.0.5",
"http://user@hub.internal",
"http://localhost.example.com",
] {
assert!(sends_in_clear_text(endpoint), "{endpoint}");
}
for endpoint in [
"https://huggingface.co",
"https://mirror.example.com",
"http://localhost:8000",
"http://127.0.0.1:9",
"http://127.8.0.1",
"http://[::1]:8080/",
] {
assert!(!sends_in_clear_text(endpoint), "{endpoint}");
}
}
/// A commit SHA names the snapshot directly; a branch goes through its
/// ref, as in Python's cache layout.
#[test]
fn snapshot_path_resolves_commits_directly_and_branches_through_refs() {
let cache = tempfile::tempdir().unwrap();
let repo = cache.path().join("models--owner--name");
std::fs::create_dir_all(repo.join("refs")).unwrap();
std::fs::write(repo.join("refs").join("main"), "abc123\n").unwrap();
let snapshots = repo.join("snapshots");
assert_eq!(
snapshot_path(cache.path(), "owner/name", "def456"),
snapshots.join("def456")
);
assert_eq!(
snapshot_path(cache.path(), "owner/name", "main"),
snapshots.join("abc123")
);
}
#[test]
fn pin_can_be_turned_off() {
let _guard = crate::test_support::env_lock();
let saved = std::env::var_os("HEADROOM_HF_PIN");
for (value, disabled) in [("", false), ("on", false), ("off", true), (" No ", true)] {
std::env::set_var("HEADROOM_HF_PIN", value);
assert_eq!(pin_disabled(), disabled, "{value:?}");
assert_eq!(pinned_or_main("sha"), if disabled { "main" } else { "sha" });
}
match saved {
Some(v) => std::env::set_var("HEADROOM_HF_PIN", v),
None => std::env::remove_var("HEADROOM_HF_PIN"),
}
}
}
+1
View File
@@ -8,6 +8,7 @@ pub mod auth_mode;
pub mod cache_control;
pub mod ccr;
pub mod compression_policy;
mod hf_cache;
pub mod offline;
#[cfg(feature = "ml")]
mod onnx_cpu;
+24 -435
View File
@@ -15,7 +15,7 @@
//! construction may need network access to fetch the model. The files
//! live in the HuggingFace hub cache directory Python's `huggingface_hub`
//! would pick (`HF_HUB_CACHE`, `HF_HOME/hub`, `~/.cache/huggingface/hub`,
//! …; see `hf_hub_cache_dir`), shared with Python.
//! …; see `crate::hf_cache::hub_cache_dir`), shared with Python.
//!
//! When constructed, `is_available()` returns `true` and `HybridScorer`
//! switches off the BM25-fallback path automatically. If construction
@@ -95,230 +95,16 @@ const DEFAULT_MODEL_FILES: [&str; 5] = [
/// pinning off (same values as Python's `onnx_runtime._resolve_revision`).
#[cfg(feature = "ml")]
fn default_model_revision() -> &'static str {
let pin = std::env::var("HEADROOM_HF_PIN").unwrap_or_default();
match pin.trim().to_ascii_lowercase().as_str() {
"off" | "0" | "false" | "no" => "main",
_ => DEFAULT_MODEL_REVISION,
}
}
/// The HuggingFace hub cache directory, resolved the way Python's
/// `huggingface_hub` resolves `constants.HF_HUB_CACHE`, so both languages look
/// up and download into the same place: `HF_HUB_CACHE`, then
/// `HUGGINGFACE_HUB_CACHE` (its legacy name), then `HF_HOME/hub`, then
/// `XDG_CACHE_HOME/huggingface/hub`, then `~/.cache/huggingface/hub`.
///
/// It is exactly one directory. A snapshot in a lower-priority cache is not
/// used when a higher one is configured, which is also what Python does: with
/// `HF_HUB_CACHE` set, a model under `~/.cache` is neither found nor reused.
///
/// Both expansions happen as in `huggingface_hub.constants`:
/// `expandvars(expanduser(...))`, first on `HF_HOME` (or its
/// `XDG_CACHE_HOME` default) and then on the hub path itself. So
/// `HF_HUB_CACHE=${ROOT}/hub` names the same directory in both languages, and
/// a `~` that only appears once a variable is expanded stays literal, as it
/// does in Python. The home directory comes from `std::env::home_dir`, which
/// falls back to the user database when `HOME` is unset, as `expanduser`
/// does. A variable that is set but empty still counts, as with `os.getenv`:
/// `HF_HUB_CACHE=''` is the working directory itself, `HF_HOME=''` is `hub`
/// and `XDG_CACHE_HOME=''` is `huggingface/hub` under it, and none of them
/// falls through to a lower-priority cache. `tests/parity/fixtures/hf_hub_cache/`
/// holds the cases both languages are checked against.
#[cfg(feature = "ml")]
fn hf_hub_cache_dir() -> Option<PathBuf> {
let var = |name: &str| std::env::var_os(name).map(PathBuf::from);
let home = std::env::home_dir();
let expand = |path: PathBuf| expand_vars(&expand_user(path, home.as_deref()));
let hub = match var("HF_HUB_CACHE").or_else(|| var("HUGGINGFACE_HUB_CACHE")) {
Some(dir) => dir,
None => {
let hf_home = match var("HF_HOME") {
Some(dir) => dir,
None => var("XDG_CACHE_HOME")
.or_else(|| home.as_ref().map(|home| home.join(".cache")))?
.join("huggingface"),
};
expand(hf_home).join("hub")
}
};
Some(expand(hub))
}
/// Python's `os.path.expanduser` for a leading bare `~` component. `~user`
/// stays as written, which is also what `expanduser` does when it cannot
/// resolve the user.
#[cfg(feature = "ml")]
fn expand_user(path: PathBuf, home: Option<&Path>) -> PathBuf {
match (path.strip_prefix("~"), home) {
(Ok(rest), Some(home)) => home.join(rest),
_ => path,
}
}
/// Python's `os.path.expandvars`: `posixpath` rules off Windows, `ntpath`
/// rules on it. A path that is not valid Unicode is returned unchanged.
#[cfg(feature = "ml")]
fn expand_vars(path: &Path) -> PathBuf {
let Some(text) = path.to_str() else {
return path.to_path_buf();
};
#[cfg(windows)]
let expanded = expand_vars_nt(text, env_lookup);
#[cfg(not(windows))]
let expanded = expand_vars_posix(text, env_lookup);
PathBuf::from(expanded)
}
/// An environment value for `expand_vars`. Names the OS cannot hold (empty,
/// or containing `=` or NUL) are unset, which is how Python's `os.environ`
/// lookup fails for them too.
#[cfg(feature = "ml")]
fn env_lookup(name: &str) -> Option<std::ffi::OsString> {
if name.is_empty() || name.contains(['=', '\0']) {
return None;
}
std::env::var_os(name)
}
/// `posixpath.expandvars`: `$name` (ASCII letters, digits, `_`) or
/// `${anything but a brace}`. An unset name, and a `$` that starts neither
/// form, stay as written. Expanded values are not scanned again.
#[cfg(feature = "ml")]
#[cfg_attr(windows, allow(dead_code))] // tests run both rule sets everywhere
fn expand_vars_posix(
text: &str,
lookup: impl Fn(&str) -> Option<std::ffi::OsString>,
) -> std::ffi::OsString {
let mut out = std::ffi::OsString::new();
let mut rest = text;
while let Some(at) = rest.find('$') {
out.push(&rest[..at]);
let after = &rest[at + 1..];
let word = after
.find(|c: char| !(c.is_ascii_alphanumeric() || c == '_'))
.unwrap_or(after.len());
// (name, length of the reference after the `$`)
let reference = if word > 0 {
Some((&after[..word], word))
} else {
after
.strip_prefix('{')
.and_then(|braced| braced.find('}').map(|end| (&braced[..end], end + 2)))
};
match reference {
Some((name, len)) => {
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[at..at + 1 + len]),
}
rest = &after[len..];
}
None => {
out.push("$");
rest = after;
}
}
}
out.push(rest);
out
}
/// `ntpath.expandvars`: `%name%`, `${name}` and `$name` (ASCII letters,
/// digits, `_`, `-`); `%%` and `$$` are a literal `%` and `$`; nothing inside
/// single quotes expands. An unset or unterminated reference stays as
/// written.
#[cfg(feature = "ml")]
#[cfg_attr(not(windows), allow(dead_code))] // tests run both rule sets everywhere
fn expand_vars_nt(
text: &str,
lookup: impl Fn(&str) -> Option<std::ffi::OsString>,
) -> std::ffi::OsString {
let mut out = std::ffi::OsString::new();
let mut rest = text;
while let Some(c) = rest.chars().next() {
let after = &rest[c.len_utf8()..];
match c {
'\'' => match after.find('\'') {
Some(end) => {
out.push(&rest[..end + 2]);
rest = &after[end + 1..];
}
None => {
out.push(rest);
rest = "";
}
},
'%' if after.starts_with('%') => {
out.push("%");
rest = &after[1..];
}
'%' => match after.find('%') {
Some(end) => {
let name = &after[..end];
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[..end + 2]),
}
rest = &after[end + 1..];
}
None => {
out.push(rest);
rest = "";
}
},
'$' if after.starts_with('$') => {
out.push("$");
rest = &after[1..];
}
'$' if after.starts_with('{') => match after[1..].find('}') {
Some(end) => {
let name = &after[1..1 + end];
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[..end + 3]),
}
rest = &after[end + 2..];
}
None => {
out.push(rest);
rest = "";
}
},
'$' => {
let end = after
.find(|c: char| !(c.is_ascii_alphanumeric() || c == '_' || c == '-'))
.unwrap_or(after.len());
let name = &after[..end];
match lookup(name) {
Some(value) => out.push(value),
None => out.push(&rest[..end + 1]),
}
rest = &after[end..];
}
_ => {
out.push(&rest[..c.len_utf8()]);
rest = after;
}
}
}
out
crate::hf_cache::pinned_or_main(DEFAULT_MODEL_REVISION)
}
/// Find a snapshot of [`DEFAULT_MODEL_REPO`] at `revision` in the hub cache
/// `cache` that holds every file in [`DEFAULT_MODEL_FILES`]. Never touches the
/// network.
///
/// Looks the snapshot up by directory rather than through `hf_hub::Cache`:
/// that resolves every revision through `refs/<revision>`, which Python's
/// `huggingface_hub` does not write for a commit SHA, so a snapshot Python
/// downloaded would be missed. A branch name still goes through its ref.
/// network; see [`crate::hf_cache::snapshot_path`] for how `revision`
/// resolves.
#[cfg(feature = "ml")]
fn cached_snapshot(cache: &Path, revision: &str) -> Option<PathBuf> {
let repo = cache.join(format!("models--{}", DEFAULT_MODEL_REPO.replace('/', "--")));
let commit = std::fs::read_to_string(repo.join("refs").join(revision))
.map(|c| c.trim().to_string())
.unwrap_or_else(|_| revision.to_string());
let snapshot = repo.join("snapshots").join(commit);
let snapshot = crate::hf_cache::snapshot_path(cache, DEFAULT_MODEL_REPO, revision);
// `is_file` follows the snapshot's symlinks, so blobs deleted from under
// it count as a miss rather than a load failure.
DEFAULT_MODEL_FILES
@@ -328,32 +114,30 @@ fn cached_snapshot(cache: &Path, revision: &str) -> Option<PathBuf> {
}
/// The Hub handle for [`DEFAULT_MODEL_REPO`] at `revision`, caching under
/// `cache_root`. Honors `HF_ENDPOINT` like the Python side does.
/// `cache_root`, with Python's endpoint and token ([`crate::hf_cache::hub_api`]).
#[cfg(feature = "ml")]
fn default_model_repo(
cache_root: &Path,
revision: &str,
) -> Result<hf_hub::api::sync::ApiRepo, String> {
// Air-gap chokepoint, ahead of the client: building it resolves the Hub
// endpoint and the `ureq` agent, and `hf-hub` downloads on a cache miss
// even with `HF_HUB_OFFLINE` set. Callers reach this only on a cache
// `hub_api` is the air-gap chokepoint: under `HEADROOM_OFFLINE` it
// refuses before building the client. Callers reach this only on a cache
// miss, so a pre-seeded air-gapped host still loads.
//
// The refusal is soft by design: every caller degrades to the BM25
// scorer on `Err`, so an air-gapped box loses embedding relevance instead
// of failing a request. The message names the switch so the operator can
// tell the causes apart in the log.
crate::offline::guard_egress("fastembed embedding model download", DEFAULT_MODEL_REPO)
.map_err(|e| format!("EmbeddingScorer model load refused: {e}"))?;
let mut builder =
hf_hub::api::sync::ApiBuilder::from_cache(hf_hub::Cache::new(cache_root.to_path_buf()))
.with_progress(false);
if let Ok(endpoint) = std::env::var("HF_ENDPOINT") {
builder = builder.with_endpoint(endpoint);
}
let api = builder
.build()
.map_err(|e| format!("EmbeddingScorer: HuggingFace client init failed: {e}"))?;
use crate::hf_cache::HubApiError;
let api = crate::hf_cache::hub_api(
cache_root,
"fastembed embedding model download",
DEFAULT_MODEL_REPO,
)
.map_err(|e| match e {
HubApiError::Offline(e) => format!("EmbeddingScorer model load refused: {e}"),
HubApiError::Build(e) => format!("EmbeddingScorer: HuggingFace client init failed: {e}"),
})?;
Ok(api.repo(hf_hub::Repo::with_revision(
DEFAULT_MODEL_REPO.to_string(),
hf_hub::RepoType::Model,
@@ -493,7 +277,7 @@ impl EmbeddingScorer {
/// offline, network failure during download, missing ONNX runtime
/// binaries, etc.).
pub fn try_new() -> Result<Self, String> {
let cache = hf_hub_cache_dir().ok_or_else(|| {
let cache = crate::hf_cache::hub_cache_dir().ok_or_else(|| {
"EmbeddingScorer: no HuggingFace cache directory (set HF_HUB_CACHE or HF_HOME)"
.to_string()
})?;
@@ -729,6 +513,8 @@ fn cosine_similarity(a: &[f32], b: &[f32]) -> f64 {
#[cfg(test)]
mod tests {
use super::*;
#[cfg(feature = "ml")]
use crate::hf_cache::tests::CacheEnv;
// The real-model tests are gated behind RUN_FASTEMBED_TESTS=1
// since they require network access on first run (~67 MB model
@@ -843,209 +629,12 @@ mod tests {
assert_eq!(cached_snapshot(cache.path(), DEFAULT_MODEL_REVISION), None);
}
/// Every variable `hf_hub_cache_dir` reads.
#[cfg(feature = "ml")]
const CACHE_ENV: [&str; 6] = [
"HF_HUB_CACHE",
"HUGGINGFACE_HUB_CACHE",
"HF_HOME",
"XDG_CACHE_HOME",
"HOME",
"USERPROFILE",
];
/// Saves [`CACHE_ENV`], clears it, and puts it back on drop, so a failing
/// assertion cannot leak a fake cache into later tests. Take
/// `env_lock()` first.
#[cfg(feature = "ml")]
struct CacheEnv(Vec<(String, Option<std::ffi::OsString>)>);
#[cfg(feature = "ml")]
impl CacheEnv {
fn cleared() -> Self {
Self::cleared_with(std::iter::empty())
}
/// [`Self::cleared`], plus `extra` variables (the ones a test's
/// paths reference).
fn cleared_with(extra: impl IntoIterator<Item = String>) -> Self {
let names: Vec<String> = CACHE_ENV
.iter()
.map(|name| name.to_string())
.chain(extra)
.collect();
let saved = names
.into_iter()
.map(|name| {
let value = std::env::var_os(&name);
std::env::remove_var(&name);
(name, value)
})
.collect();
CacheEnv(saved)
}
}
#[cfg(feature = "ml")]
impl Drop for CacheEnv {
fn drop(&mut self) {
for (name, value) in &self.0 {
match value {
Some(value) => std::env::set_var(name, value),
None => std::env::remove_var(name),
}
}
}
}
/// Same order as `huggingface_hub.constants.HF_HUB_CACHE`: each variable
/// wins over every one after it, and only one directory comes back.
#[cfg(feature = "ml")]
#[test]
fn hf_hub_cache_dir_follows_huggingface_hub_precedence() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared();
let t = tempfile::tempdir().unwrap();
let p = |name: &str| t.path().join(name);
std::env::set_var("HOME", p("home"));
std::env::set_var("USERPROFILE", p("home"));
std::env::set_var("XDG_CACHE_HOME", p("xdg"));
std::env::set_var("HF_HOME", p("hf-home"));
std::env::set_var("HUGGINGFACE_HUB_CACHE", p("legacy"));
std::env::set_var("HF_HUB_CACHE", p("explicit"));
assert_eq!(hf_hub_cache_dir(), Some(p("explicit")));
std::env::remove_var("HF_HUB_CACHE");
assert_eq!(hf_hub_cache_dir(), Some(p("legacy")));
std::env::remove_var("HUGGINGFACE_HUB_CACHE");
assert_eq!(hf_hub_cache_dir(), Some(p("hf-home").join("hub")));
std::env::remove_var("HF_HOME");
assert_eq!(
hf_hub_cache_dir(),
Some(p("xdg").join("huggingface").join("hub"))
);
std::env::remove_var("XDG_CACHE_HOME");
assert_eq!(
hf_hub_cache_dir(),
Some(p("home").join(".cache").join("huggingface").join("hub"))
);
// A leading `~` expands.
std::env::set_var("HF_HOME", "~/models");
assert_eq!(
hf_hub_cache_dir(),
Some(p("home").join("models").join("hub"))
);
// A set but empty variable is the working directory, as in Python,
// and still wins over every variable after it.
std::env::set_var("HF_HUB_CACHE", "");
assert_eq!(hf_hub_cache_dir(), Some(PathBuf::new()));
std::env::remove_var("HF_HUB_CACHE");
std::env::set_var("HF_HOME", "");
std::env::set_var("XDG_CACHE_HOME", p("xdg"));
assert_eq!(hf_hub_cache_dir(), Some(PathBuf::from("hub")));
}
/// Cases checked against Python itself (`posixpath`, `ntpath` and
/// `huggingface_hub.constants`) by `tests/test_fastembed_pinned_snapshot.py`.
#[cfg(feature = "ml")]
const HF_CACHE_FIXTURE: &str = include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/../../tests/parity/fixtures/hf_hub_cache/cases.json"
));
/// Both rule sets on every platform, so the Windows port is exercised by
/// the macOS and Linux runs too.
#[cfg(feature = "ml")]
#[test]
fn expand_vars_matches_python_posixpath_and_ntpath() {
let fixture: serde_json::Value = serde_json::from_str(HF_CACHE_FIXTURE).unwrap();
let section = &fixture["expandvars"];
let env: std::collections::HashMap<String, String> =
serde_json::from_value(section["env"].clone()).unwrap();
let lookup = |name: &str| env.get(name).map(std::ffi::OsString::from);
for case in section["cases"].as_array().unwrap() {
let input = case["input"].as_str().unwrap();
let want = |key: &str| std::ffi::OsString::from(case[key].as_str().unwrap());
assert_eq!(
expand_vars_posix(input, lookup),
want("posix"),
"posix {input:?}"
);
assert_eq!(expand_vars_nt(input, lookup), want("nt"), "nt {input:?}");
}
}
/// The review case among them: `HF_HUB_CACHE=${ROOT}/hub` is
/// `<ROOT>/hub` in Python, so Rust must not look up or download into a
/// literal `${ROOT}` directory instead.
#[cfg(feature = "ml")]
#[test]
fn hf_hub_cache_dir_matches_the_shared_fixture() {
let fixture: serde_json::Value = serde_json::from_str(HF_CACHE_FIXTURE).unwrap();
let section = &fixture["cache_dir"];
let probe_vars: Vec<String> =
serde_json::from_value(section["probe_vars"].clone()).unwrap();
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with(probe_vars.clone());
let t = tempfile::tempdir().unwrap();
let (root, home) = (t.path().join("root"), t.path().join("home"));
let fill = |text: &str| {
text.replace("{root}", root.to_str().unwrap())
.replace("{home}", home.to_str().unwrap())
};
for case in section["cases"].as_array().unwrap() {
for name in CACHE_ENV
.iter()
.copied()
.chain(probe_vars.iter().map(String::as_str))
{
std::env::remove_var(name);
}
std::env::set_var("HOME", &home);
std::env::set_var("USERPROFILE", &home);
for (name, value) in case["env"].as_object().unwrap() {
std::env::set_var(name, fill(value.as_str().unwrap()));
}
let expected = match &case["expected"] {
serde_json::Value::Object(by_os) => {
&by_os[if cfg!(windows) { "nt" } else { "posix" }]
}
expected => expected,
};
assert_eq!(
hf_hub_cache_dir(),
Some(PathBuf::from(fill(expected.as_str().unwrap()))),
"{}",
case["name"]
);
}
}
/// With no `HOME` (a service started with a bare environment) the home
/// directory still comes from the user database, as Python's
/// `expanduser` does, instead of leaving the scorer with no cache.
#[cfg(all(feature = "ml", unix))]
#[test]
fn hf_hub_cache_dir_without_home_uses_the_user_database() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared();
let dir = hf_hub_cache_dir().expect("a cache directory without HOME");
assert!(dir.is_absolute(), "{}", dir.display());
assert!(
dir.ends_with(Path::new(".cache").join("huggingface").join("hub")),
"{}",
dir.display()
);
}
/// The case from review: an explicit, empty `HF_HUB_CACHE` next to a
/// complete snapshot in the default cache under `HOME`. Python looks only
/// in the explicit cache and downloads there, so Rust must not find the
/// `HOME` copy, for the pinned revision or for floating `main`
/// (`HEADROOM_HF_PIN=off`). `try_new` is `try_new_pinned` on
/// `hf_hub_cache_dir()`, which looks up and downloads only in the
/// `hub_cache_dir()`, which looks up and downloads only in the
/// directory it is given (see the two offline tests above), so checking
/// the resolver and the lookup here covers it without flipping
/// `HEADROOM_OFFLINE` for a long stretch of the parallel run.
@@ -1068,14 +657,14 @@ mod tests {
std::env::set_var("USERPROFILE", home.path());
std::env::set_var("HF_HUB_CACHE", explicit.path());
let cache = hf_hub_cache_dir().expect("cache directory");
let cache = crate::hf_cache::hub_cache_dir().expect("cache directory");
assert_eq!(cache, explicit.path());
assert_eq!(cached_snapshot(&cache, DEFAULT_MODEL_REVISION), None);
assert_eq!(cached_snapshot(&cache, "main"), None);
// Without the explicit cache, the same HOME snapshots are the cache.
std::env::remove_var("HF_HUB_CACHE");
let cache = hf_hub_cache_dir().expect("cache directory");
let cache = crate::hf_cache::hub_cache_dir().expect("cache directory");
assert_eq!(cache, default_cache);
assert_eq!(
cached_snapshot(&cache, DEFAULT_MODEL_REVISION),
+156 -17
View File
@@ -12,11 +12,12 @@
//! - [`HfTokenizer::from_bytes`] — for tokenizers embedded via `include_bytes!`.
//! - [`HfTokenizer::from_file`] — for a local `tokenizer.json`.
//! - [`HfTokenizer::from_pretrained`] — pulls `tokenizer.json` from the
//! HuggingFace Hub via `hf-hub`. First call downloads to
//! `~/.cache/huggingface/hub`, subsequent calls hit the cache. Blocking;
//! HuggingFace Hub via `hf-hub`. First call downloads into the hub cache
//! Python's `huggingface_hub` uses (`HF_HUB_CACHE`, `HF_HOME/hub`,
//! `~/.cache/huggingface/hub`, …), subsequent calls hit it. Blocking;
//! call from `main()` or `tokio::task::spawn_blocking`. Gated repos
//! (Llama, Mistral) require an `HF_TOKEN` env var or a token in
//! `~/.cache/huggingface/token`.
//! (Llama, Mistral) need a token: `HF_TOKEN`, or the file
//! `hf auth login` writes (`HF_HOME/token`, or `HF_TOKEN_PATH`).
//!
//! # What's NOT here
//! - **No tokenizer.json bundled in the binary.** Bundling Llama / Cohere
@@ -112,9 +113,13 @@ impl HfTokenizer {
/// e.g. `"CohereForAI/c4ai-command-r-v01"` or `"meta-llama/Meta-Llama-3-8B"`.
///
/// Uses the `main` revision. Blocking — calls into `ureq` synchronously.
/// First successful call writes the file to `~/.cache/huggingface/hub`
/// (or `$HF_HOME` if set); subsequent calls in the same or later
/// processes hit the on-disk cache.
/// The file lands in the hub cache directory Python's `huggingface_hub`
/// would use ([`crate::hf_cache::hub_cache_dir`]: `HF_HUB_CACHE`, then
/// `HUGGINGFACE_HUB_CACHE`, `HF_HOME/hub`, `XDG_CACHE_HOME/huggingface/hub`,
/// `~/.cache/huggingface/hub`), so later calls, and the Python side, hit
/// the same copy. The token and `HF_ENDPOINT` are read as Python reads
/// them too ([`crate::hf_cache::hf_token`]); `hf-hub`'s own defaults
/// ignore every variable but `HF_HOME`, and with `Api::new` even that.
///
/// Errors:
/// - [`HfTokenizerError::Offline`] when `HEADROOM_OFFLINE` is set. Refused
@@ -129,11 +134,11 @@ impl HfTokenizer {
// HEADROOM_OFFLINE=1 gets the same refusal whichever runtime reaches
// for the Hub.
//
// It has to sit here rather than around the `get` below: `Api::new`
// builds the `ureq` agent and resolves the endpoint, and hf-hub's own
// `crate::hf_cache::hub_api` runs the guard before it builds the
// client, rather than around the `get` below: building it creates the
// `ureq` agent and resolves the endpoint, and hf-hub's own
// HF_HUB_OFFLINE handling only covers the cache lookup — a cache miss
// still dials out. Guarding first means the air-gapped case never
// constructs a client at all.
// still dials out. So the air-gapped case never constructs a client.
//
// The refusal is hard on purpose. Silently returning a cached or
// estimating tokenizer would leave token counts subtly wrong with no
@@ -145,11 +150,7 @@ impl HfTokenizer {
// client exists" and "serve from the cache" cannot both hold here. An
// air-gapped deployment with pre-seeded artifacts should point at them
// with `from_file`, which is network-free by construction.
crate::offline::guard_egress("HuggingFace tokenizer download", repo)?;
let api = hf_hub::api::sync::Api::new().map_err(|e| HfTokenizerError::Hub {
repo: repo.to_string(),
source: Box::new(e),
})?;
let api = hub_api(repo)?;
let path = api
.model(repo.to_string())
.get("tokenizer.json")
@@ -169,6 +170,31 @@ impl HfTokenizer {
}
}
/// The Hub client for [`HfTokenizer::from_pretrained`] fetching `repo`:
/// caching in Python's hub cache directory, with Python's token and endpoint.
/// Refuses with [`HfTokenizerError::Offline`] under `HEADROOM_OFFLINE`.
fn hub_api(repo: &str) -> Result<hf_hub::api::sync::Api, HfTokenizerError> {
use crate::hf_cache::HubApiError;
const PURPOSE: &str = "HuggingFace tokenizer download";
let hub_error = |source: Box<dyn std::error::Error + Send + Sync>| HfTokenizerError::Hub {
repo: repo.to_string(),
source,
};
let Some(cache) = crate::hf_cache::hub_cache_dir() else {
// Offline still wins: the caller treats `Offline` as final, and a
// missing cache directory is no reason to report a Hub failure.
crate::offline::guard_egress(PURPOSE, repo)?;
return Err(hub_error(
"no HuggingFace cache directory (set HF_HUB_CACHE or HF_HOME)".into(),
));
};
crate::hf_cache::hub_api(&cache, PURPOSE, repo).map_err(|e| match e {
HubApiError::Offline(blocked) => HfTokenizerError::Offline(blocked),
HubApiError::Build(source) => hub_error(Box::new(source)),
})
}
impl Tokenizer for HfTokenizer {
fn count_text(&self, text: &str) -> usize {
if text.is_empty() {
@@ -297,7 +323,7 @@ mod tests {
/// this a real assertion rather than a tautology: without the guard the call
/// reaches `hf-hub`, dials `huggingface.co`, gets a 401/404 and returns
/// `HfTokenizerError::Hub`. Only the guard can produce `Offline`, and only
/// by returning before `Api::new` — so `Offline` *is* the "no socket was
/// by returning before `hf_cache::hub_api` builds the client — so `Offline` *is* the "no socket was
/// opened" assertion for this path. (A packet-level assertion is not
/// available to a `cargo test` unit test the way `socket.socket` patching is
/// in Python; this typed-error distinction is the equivalent signal.)
@@ -349,6 +375,119 @@ mod tests {
let _ = std::fs::remove_dir_all(&dir);
}
/// Lay out `tokenizer.json` for `repo` in hub cache `cache`, the way
/// Python's `huggingface_hub` leaves a `main` download: a snapshot plus
/// `refs/main` naming it.
fn cache_main_tokenizer(cache: &std::path::Path, repo: &str) {
let repo_dir = cache.join(format!("models--{}", repo.replace('/', "--")));
let snapshot = repo_dir.join("snapshots").join("abc123");
std::fs::create_dir_all(&snapshot).unwrap();
std::fs::write(snapshot.join("tokenizer.json"), TINY_TOKENIZER_JSON).unwrap();
std::fs::create_dir_all(repo_dir.join("refs")).unwrap();
std::fs::write(repo_dir.join("refs").join("main"), "abc123").unwrap();
}
/// Variables `from_pretrained` reads, saved and restored around a test.
fn hub_env() -> crate::hf_cache::tests::CacheEnv {
crate::hf_cache::tests::CacheEnv::cleared_with(
[
"HF_TOKEN",
"HUGGING_FACE_HUB_TOKEN",
"HF_TOKEN_PATH",
"HF_ENDPOINT",
"HF_TEST_ROOT",
]
.map(String::from),
)
}
/// The tokenizer is read from (and would be downloaded into) the hub cache
/// Python uses, here an `HF_HUB_CACHE` given through `${VAR}`. Before, it
/// was always `~/.cache/huggingface/hub`, so this was a cache miss and a
/// download. `HF_ENDPOINT` points at a closed loopback port, so a miss
/// fails fast instead of reaching the internet.
#[test]
fn from_pretrained_reads_the_hub_cache_python_uses() {
let _guard = crate::test_support::env_lock();
let _env = hub_env();
let t = tempfile::tempdir().unwrap();
std::env::set_var("HOME", t.path().join("home"));
std::env::set_var("USERPROFILE", t.path().join("home"));
std::env::set_var("HF_TEST_ROOT", t.path());
std::env::set_var("HF_HUB_CACHE", "${HF_TEST_ROOT}/hub");
std::env::set_var("HF_ENDPOINT", "http://127.0.0.1:9");
cache_main_tokenizer(&t.path().join("hub"), "acme/tiny");
let tokenizer = HfTokenizer::from_pretrained("acme/tiny").expect("served from the cache");
assert_eq!(tokenizer.count_text("hello world"), 2);
}
/// A cache miss asks the Hub at `HF_ENDPOINT` with the token from
/// `HF_TOKEN`. `hf-hub`'s `Api::new` read neither, so a gated repo
/// (Llama, Mistral) failed with 401 even with `HF_TOKEN` set. A loopback
/// stand-in for the Hub records the request and answers 404.
#[test]
fn from_pretrained_sends_hf_token_to_hf_endpoint() {
use std::io::{Read, Write};
let _guard = crate::test_support::env_lock();
let _env = hub_env();
let t = tempfile::tempdir().unwrap();
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = listener.local_addr().unwrap().port();
// Waits at most 10 s for the request, so a regression that never
// reaches the endpoint fails instead of hanging the suite.
listener.set_nonblocking(true).unwrap();
let hub = std::thread::spawn(move || {
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
let mut conn = loop {
match listener.accept() {
Ok((conn, _)) => break conn,
Err(_) if std::time::Instant::now() < deadline => {
std::thread::sleep(std::time::Duration::from_millis(20))
}
Err(_) => return String::new(),
}
};
conn.set_nonblocking(false).unwrap();
let mut request = Vec::new();
let mut buf = [0u8; 1024];
while !request.windows(4).any(|w| w == b"\r\n\r\n") {
let n = conn.read(&mut buf).unwrap();
if n == 0 {
break;
}
request.extend_from_slice(&buf[..n]);
}
conn.write_all(
b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n",
)
.unwrap();
String::from_utf8_lossy(&request).to_ascii_lowercase()
});
std::env::set_var("HOME", t.path().join("home"));
std::env::set_var("USERPROFILE", t.path().join("home"));
std::env::set_var("HF_HUB_CACHE", t.path().join("hub"));
// Trailing slash on purpose: Python strips it, and a kept one
// would request `//acme/...`.
std::env::set_var("HF_ENDPOINT", format!("http://127.0.0.1:{port}/"));
std::env::set_var("HF_TOKEN", "hf_test_token");
let result = HfTokenizer::from_pretrained("acme/gated");
let request = hub.join().unwrap();
assert!(
matches!(result, Err(HfTokenizerError::Hub { .. })),
"{result:?}"
);
assert!(
request.contains(" /acme/gated/resolve/main/tokenizer.json "),
"{request}"
);
assert!(
request.contains("authorization: bearer hf_test_token"),
"{request}"
);
}
/// Network-dependent: hits HuggingFace Hub. Run with
/// `cargo test -p headroom-core -- --ignored from_pretrained_downloads_real_tokenizer`.
/// `gpt2` is a small public unauthenticated repo (~1.4 MB tokenizer.json).
+228 -56
View File
@@ -74,6 +74,12 @@ use tokenizers::{EncodeInput, InputSequence, Tokenizer};
pub const DEFAULT_MODEL_ID: &str = "chopratejas/kompress-v2-base";
/// HuggingFace repo holding the tokenizer Kompress reuses.
pub const DEFAULT_TOKENIZER_REPO: &str = "answerdotai/ModernBERT-base";
/// Pinned commits of [`DEFAULT_MODEL_ID`] and [`DEFAULT_TOKENIZER_REPO`], the
/// snapshots the cache-only loader reads. Keep each equal to its entry in
/// `_PINNED_REVISIONS` (`headroom/onnx_runtime.py`), which is what the Python
/// side downloads; `tests/test_kompress_rust_pins.py` enforces it.
pub const DEFAULT_MODEL_REVISION: &str = "b1563631b35bfdcee37587ad530147497d820d4c";
pub const DEFAULT_TOKENIZER_REVISION: &str = "8949b909ec900327062f0ebf497f51aef5e6f0c8";
/// Words per inference chunk. Coupled to the model's training window;
/// kompress-v2-base was trained for 350.
pub const DEFAULT_CHUNK_WORDS: usize = 350;
@@ -261,8 +267,8 @@ impl Kompress {
/// downloading on miss). Blocking — call off the hot path. Tries the
/// [`ONNX_CANDIDATES`] in order.
pub fn from_pretrained(config: KompressConfig) -> Result<Self, KompressError> {
// Air-gap chokepoint — the same one `hf_impl.rs` uses for the
// tokenizer, for the same reason. `HF_HUB_OFFLINE` only shortcuts the
// Air-gap chokepoint — the same guard `hf_cache::hub_api` runs for
// the tokenizer and embedder, for the same reason. `HF_HUB_OFFLINE` only shortcuts the
// cache *lookup*; on a cache miss `hf-hub` still dials huggingface.co,
// so the Kompress model download is a live egress path under
// `HEADROOM_OFFLINE=1` and was not covered.
@@ -333,20 +339,23 @@ impl Kompress {
pub fn from_cache(config: KompressConfig) -> Result<Option<Self>, KompressError> {
let Some(tok_path) = hf_cache_file(&config.tokenizer_repo, &["tokenizer.json"]) else {
// Diagnostic: a `None` here is the #1 cause of a silent
// `kompress_ready=false`. Name the repo + the roots searched so
// operators don't have to guess between "not downloaded" and
// `kompress_ready=false`. Name the repo, the cache and the revision
// looked in so operators don't have to guess between "not
// downloaded" and
// "present but unreadable" (e.g. HF symlinks over `\\wsl$`, which
// native Windows can't follow — `path.exists()` returns false on
// native Windows can't follow — `is_file()` returns false on
// the unresolved symlink). Cache-only, so this is a defer, not an
// error: the caller passes plain text through.
tracing::warn!(
event = "kompress_cache_miss",
stage = "tokenizer",
tokenizer_repo = %config.tokenizer_repo,
searched_roots = ?hf_hub_roots(),
"Kompress deferred: tokenizer.json not found in HF cache \
(not downloaded, or present but unreadable — e.g. HF symlinks \
over \\\\wsl$ which native Windows cannot follow)"
revision = %cache_revision(&config.tokenizer_repo),
hub_cache = ?crate::hf_cache::hub_cache_dir(),
"Kompress deferred: tokenizer.json not found in the HF cache at the \
pinned revision (not downloaded yet — the Python side fetches the \
pin on its next Kompress load — or present but unreadable, e.g. HF \
symlinks over \\\\wsl$ which native Windows cannot follow)"
);
return Ok(None);
};
@@ -384,7 +393,8 @@ impl Kompress {
model_id = %config.model_id,
candidates = ?ONNX_CANDIDATES,
any_onnx_found = found_onnx,
searched_roots = ?hf_hub_roots(),
revision = %cache_revision(&config.model_id),
hub_cache = ?crate::hf_cache::hub_cache_dir(),
"Kompress deferred: no usable ONNX session \
(no candidate file in cache, or every candidate failed to build)"
);
@@ -603,59 +613,221 @@ fn build_session(path: &Path) -> Result<Session, Box<dyn std::error::Error + Sen
Ok(session)
}
/// Resolve `rel` (e.g. `["tokenizer.json"]` or `["onnx", "kompress-int8-wo.onnx"]`)
/// inside the local HuggingFace cache for `repo` (`"owner/name"`), searching
/// every snapshot under every candidate cache root. Returns `None` if not
/// present — never touches the network.
fn hf_cache_file(repo: &str, rel: &[&str]) -> Option<PathBuf> {
let repo_dir = format!("models--{}", repo.replace('/', "--"));
for hub in hf_hub_roots() {
let snapshots = hub.join(&repo_dir).join("snapshots");
let Ok(entries) = std::fs::read_dir(&snapshots) else {
continue;
};
for snap in entries.flatten() {
let mut cand = snap.path();
for part in rel {
cand = cand.join(part);
}
if cand.exists() {
return Some(cand);
}
}
}
None
/// The tokenizer and the ONNX artifact `onnx` (e.g.
/// `"onnx/kompress-int8-wo.onnx"`) that the cache-only loader reads for
/// `config`: the pinned snapshots in Python's hub cache, as
/// [`Kompress::from_cache`] finds them. `None` when either is missing. For
/// parity harnesses that load one specific artifact through
/// [`Kompress::from_files`], so they test the files production would load.
pub fn cached_artifacts(config: &KompressConfig, onnx: &str) -> Option<(PathBuf, PathBuf)> {
let tokenizer = hf_cache_file(&config.tokenizer_repo, &["tokenizer.json"])?;
let rel: Vec<&str> = onnx.split('/').collect();
let model = hf_cache_file(&config.model_id, &rel)?;
Some((tokenizer, model))
}
/// HuggingFace hub cache roots in resolution precedence. Cross-platform so
/// the cache-only loader works on Windows (native `headroom-proxy.exe`) as
/// well as Linux: `HF_HUB_CACHE` (the hub dir directly) → `HF_HOME/hub` →
/// `{HOME|USERPROFILE}/.cache/huggingface/hub`. `HOME` is the unix home; on
/// Windows the process sees `USERPROFILE` (and often no `HOME`), so both are
/// tried. Honoring `HF_HOME` also lets a Windows proxy point at a WSL cache.
fn hf_hub_roots() -> Vec<PathBuf> {
let mut roots = Vec::new();
let push_env = |roots: &mut Vec<PathBuf>, var: &str, suffix: &[&str]| {
if let Ok(v) = std::env::var(var) {
if !v.is_empty() {
let mut p = PathBuf::from(v);
for s in suffix {
p = p.join(s);
}
roots.push(p);
}
}
};
push_env(&mut roots, "HF_HUB_CACHE", &[]);
push_env(&mut roots, "HF_HOME", &["hub"]);
push_env(&mut roots, "HOME", &[".cache", "huggingface", "hub"]);
push_env(&mut roots, "USERPROFILE", &[".cache", "huggingface", "hub"]);
roots
/// The revision of `repo` to load from the cache: its pin when it is one of
/// Kompress's default repos (see [`DEFAULT_MODEL_REVISION`]), otherwise
/// `main`, which is also what Python's `_resolve_revision` gives an unpinned
/// repo. `HEADROOM_HF_PIN=off` floats the pinned ones on `main` too.
fn cache_revision(repo: &str) -> &str {
match repo {
DEFAULT_MODEL_ID => crate::hf_cache::pinned_or_main(DEFAULT_MODEL_REVISION),
DEFAULT_TOKENIZER_REPO => crate::hf_cache::pinned_or_main(DEFAULT_TOKENIZER_REVISION),
_ => "main",
}
}
/// Resolve `rel` (e.g. `["tokenizer.json"]` or `["onnx", "kompress-int8-wo.onnx"]`)
/// in `repo`'s snapshot at [`cache_revision`], inside the one hub cache
/// directory Python's `huggingface_hub` would use
/// ([`crate::hf_cache::hub_cache_dir`]). Returns `None` if not present —
/// never touches the network.
///
/// One snapshot, not the first file found in any: with several revisions in
/// the cache (every pin bump leaves the old one behind), which one a
/// directory scan returned depended on filesystem order, and the tokenizer
/// and the ONNX file could come from different snapshots. And one cache, not
/// every root that exists: Python does not fall back to `~/.cache` when
/// `HF_HUB_CACHE` names another directory, so neither does this. `is_file`
/// follows the cache's symlinks, so a blob missing under a snapshot (or a
/// symlink native Windows cannot follow over `\\wsl$`) is a miss.
fn hf_cache_file(repo: &str, rel: &[&str]) -> Option<PathBuf> {
let cache = crate::hf_cache::hub_cache_dir()?;
let mut path = crate::hf_cache::snapshot_path(&cache, repo, cache_revision(repo));
for part in rel {
path.push(part);
}
path.is_file().then_some(path)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::hf_cache::tests::CacheEnv;
/// Lay out `rel` under `repo`'s snapshot `commit` in the hub cache `cache`.
fn cached_file(cache: &Path, repo: &str, commit: &str, rel: &str) -> PathBuf {
let path = cache
.join(format!("models--{}", repo.replace('/', "--")))
.join("snapshots")
.join(commit)
.join(rel);
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
std::fs::write(&path, commit).unwrap();
path
}
/// The cache after a pin bump: the pinned snapshot sits between an older
/// one and a newer `main`, and both other snapshots hold the same files.
fn cache_with_three_snapshots(cache: &Path) {
for repo in [DEFAULT_MODEL_ID, DEFAULT_TOKENIZER_REPO] {
let pin = cache_revision(repo).to_string();
for commit in [
"0000000000000000000000000000000000000000",
&pin,
"ffffffffffffffffffffffffffffffffffffffff",
] {
cached_file(cache, repo, commit, "tokenizer.json");
cached_file(cache, repo, commit, ONNX_CANDIDATES[1]);
}
let refs = cache
.join(format!("models--{}", repo.replace('/', "--")))
.join("refs");
std::fs::create_dir_all(&refs).unwrap();
std::fs::write(
refs.join("main"),
"ffffffffffffffffffffffffffffffffffffffff",
)
.unwrap();
}
}
/// Python loads the pinned snapshot of both repos; so must the cache-only
/// loader, whatever order the filesystem lists the snapshots in.
#[test]
fn the_cache_lookup_reads_the_pinned_snapshot() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with(["HEADROOM_HF_PIN".to_string()]);
let cache = tempfile::tempdir().unwrap();
std::env::set_var("HF_HUB_CACHE", cache.path());
cache_with_three_snapshots(cache.path());
let tokenizer = hf_cache_file(DEFAULT_TOKENIZER_REPO, &["tokenizer.json"]).unwrap();
assert!(tokenizer.ends_with(Path::new(DEFAULT_TOKENIZER_REVISION).join("tokenizer.json")));
let rel: Vec<&str> = ONNX_CANDIDATES[1].split('/').collect();
let onnx = hf_cache_file(DEFAULT_MODEL_ID, &rel).unwrap();
assert!(onnx.ends_with(Path::new(DEFAULT_MODEL_REVISION).join(ONNX_CANDIDATES[1])));
// A candidate the pinned snapshot lacks is a miss, even when another
// snapshot has it.
cached_file(
cache.path(),
DEFAULT_MODEL_ID,
"ffffffffffffffffffffffffffffffffffffffff",
ONNX_CANDIDATES[2],
);
let rel: Vec<&str> = ONNX_CANDIDATES[2].split('/').collect();
assert_eq!(hf_cache_file(DEFAULT_MODEL_ID, &rel), None);
}
/// `HEADROOM_HF_PIN=off` floats on `main` through `refs/main`, and a repo
/// Kompress does not pin (a domain model) always does, as in Python.
#[test]
fn unpinned_lookups_follow_refs_main() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with(["HEADROOM_HF_PIN".to_string()]);
let cache = tempfile::tempdir().unwrap();
std::env::set_var("HF_HUB_CACHE", cache.path());
cache_with_three_snapshots(cache.path());
let newest = Path::new("ffffffffffffffffffffffffffffffffffffffff").join("tokenizer.json");
std::env::set_var("HEADROOM_HF_PIN", "off");
let tokenizer = hf_cache_file(DEFAULT_TOKENIZER_REPO, &["tokenizer.json"]).unwrap();
assert!(tokenizer.ends_with(&newest));
std::env::remove_var("HEADROOM_HF_PIN");
let domain = "acme/kompress-legal";
cached_file(
cache.path(),
domain,
"ffffffffffffffffffffffffffffffffffffffff",
"tokenizer.json",
);
let refs = cache
.path()
.join("models--acme--kompress-legal")
.join("refs");
std::fs::create_dir_all(&refs).unwrap();
std::fs::write(
refs.join("main"),
"ffffffffffffffffffffffffffffffffffffffff",
)
.unwrap();
assert!(hf_cache_file(domain, &["tokenizer.json"])
.unwrap()
.ends_with(&newest));
}
/// The Kompress lookup now names the directory Python would, including
/// the cases the old root list missed: the legacy `HUGGINGFACE_HUB_CACHE`,
/// `XDG_CACHE_HOME` and a `${VAR}` reference. And it reads only that one
/// directory: a copy under `HOME` is not used when `HF_HUB_CACHE` points
/// elsewhere.
#[test]
fn the_cache_lookup_uses_the_one_directory_python_uses() {
let _guard = crate::test_support::env_lock();
let _env = CacheEnv::cleared_with([
"HEADROOM_HF_PIN".to_string(),
"KOMPRESS_TEST_ROOT".to_string(),
]);
let t = tempfile::tempdir().unwrap();
let pin = cache_revision(DEFAULT_TOKENIZER_REPO).to_string();
let lookup = || hf_cache_file(DEFAULT_TOKENIZER_REPO, &["tokenizer.json"]);
std::env::set_var("HOME", t.path().join("home"));
std::env::set_var("USERPROFILE", t.path().join("home"));
let legacy = t.path().join("legacy");
let want = cached_file(&legacy, DEFAULT_TOKENIZER_REPO, &pin, "tokenizer.json");
std::env::set_var("HUGGINGFACE_HUB_CACHE", &legacy);
assert_eq!(lookup(), Some(want));
std::env::remove_var("HUGGINGFACE_HUB_CACHE");
let xdg = t.path().join("xdg");
let want = cached_file(
&xdg.join("huggingface").join("hub"),
DEFAULT_TOKENIZER_REPO,
&pin,
"tokenizer.json",
);
std::env::set_var("XDG_CACHE_HOME", &xdg);
assert_eq!(lookup(), Some(want));
std::env::remove_var("XDG_CACHE_HOME");
let root = t.path().join("root");
let want = cached_file(
&root.join("hub"),
DEFAULT_TOKENIZER_REPO,
&pin,
"tokenizer.json",
);
std::env::set_var("KOMPRESS_TEST_ROOT", &root);
std::env::set_var("HF_HUB_CACHE", "${KOMPRESS_TEST_ROOT}/hub");
assert_eq!(lookup(), Some(want));
cached_file(
&t.path()
.join("home")
.join(".cache")
.join("huggingface")
.join("hub"),
DEFAULT_TOKENIZER_REPO,
&pin,
"tokenizer.json",
);
std::env::set_var("HF_HUB_CACHE", t.path().join("empty"));
assert_eq!(lookup(), None);
}
/// `from_pretrained` must refuse before `Api::new()` when the operator
/// air-gapped the box.
+9 -31
View File
@@ -14,37 +14,20 @@
use std::fs;
use std::path::{Path, PathBuf};
use headroom_core::transforms::kompress::{Kompress, KompressConfig};
use headroom_core::transforms::kompress::{self, Kompress, KompressConfig};
use serde_json::Value;
fn hf_cache_file(repo_dir: &str, rel: &[&str]) -> Option<PathBuf> {
let home = std::env::var("HOME").ok()?;
let snapshots = Path::new(&home)
.join(".cache/huggingface/hub")
.join(repo_dir)
.join("snapshots");
for snap in fs::read_dir(snapshots).ok()?.filter_map(|e| e.ok()) {
let mut cand = snap.path();
for part in rel {
cand = cand.join(part);
}
if cand.exists() {
return Some(cand);
}
}
None
/// The tokenizer + int8 ONNX artifact production would load: the pinned
/// snapshots in Python's hub cache. The fixtures were recorded against them.
fn cached_artifacts() -> Option<(PathBuf, PathBuf)> {
kompress::cached_artifacts(&KompressConfig::default(), "onnx/kompress-int8-wo.onnx")
}
#[test]
fn kompress_matches_python_fixtures_byte_for_byte() {
let tok = hf_cache_file("models--answerdotai--ModernBERT-base", &["tokenizer.json"]);
let onnx = hf_cache_file(
"models--chopratejas--kompress-v2-base",
&["onnx", "kompress-int8-wo.onnx"],
);
let (tok, onnx) = match (tok, onnx) {
(Some(t), Some(o)) => (t, o),
_ => {
let (tok, onnx) = match cached_artifacts() {
Some(paths) => paths,
None => {
eprintln!(
"SKIP: kompress model/tokenizer not in HF cache; \
run `python scripts/record_kompress_trace.py` first"
@@ -112,12 +95,7 @@ fn short_input_passes_through() {
// Pure-logic check — no model needed. Fewer than MIN_WORDS words must
// pass through unchanged regardless of model availability... but the
// engine needs a model to construct. Guard on cache like the main test.
let tok = hf_cache_file("models--answerdotai--ModernBERT-base", &["tokenizer.json"]);
let onnx = hf_cache_file(
"models--chopratejas--kompress-v2-base",
&["onnx", "kompress-int8-wo.onnx"],
);
let (Some(tok), Some(onnx)) = (tok, onnx) else {
let Some((tok, onnx)) = cached_artifacts() else {
eprintln!("SKIP: model not cached");
return;
};
+7 -28
View File
@@ -648,37 +648,16 @@ impl KompressComparator {
Self::default()
}
fn hf_cache_file(repo_dir: &str, rel: &[&str]) -> Option<PathBuf> {
let home = std::env::var("HOME").ok()?;
let snapshots = Path::new(&home)
.join(".cache/huggingface/hub")
.join(repo_dir)
.join("snapshots");
for snap in fs::read_dir(snapshots).ok()?.filter_map(|e| e.ok()) {
let mut cand = snap.path();
for part in rel {
cand = cand.join(part);
}
if cand.exists() {
return Some(cand);
}
}
None
}
fn model(&self) -> Option<&headroom_core::transforms::kompress::Kompress> {
self.model
.get_or_init(|| {
use headroom_core::transforms::kompress::{Kompress, KompressConfig};
let tok = Self::hf_cache_file(
"models--answerdotai--ModernBERT-base",
&["tokenizer.json"],
)?;
let onnx = Self::hf_cache_file(
"models--chopratejas--kompress-v2-base",
&["onnx", "kompress-int8-wo.onnx"],
)?;
Kompress::from_files(&tok, &onnx, KompressConfig::default()).ok()
use headroom_core::transforms::kompress::{self, Kompress, KompressConfig};
// The pinned snapshots in Python's hub cache, as production
// loads them.
let config = KompressConfig::default();
let (tok, onnx) =
kompress::cached_artifacts(&config, "onnx/kompress-int8-wo.onnx")?;
Kompress::from_files(&tok, &onnx, config).ok()
})
.as_ref()
}
+102 -2
View File
@@ -1,5 +1,5 @@
{
"description": "Cases both languages are checked against so Rust's hf_hub_cache_dir() names the directory huggingface_hub would. tests/test_fastembed_pinned_snapshot.py checks them against Python's posixpath/ntpath and huggingface_hub.constants; crates/headroom-core/src/relevance/embedding.rs checks its port against the same file.",
"description": "Cases both languages are checked against so Rust's hf_hub_cache_dir() names the directory huggingface_hub would. tests/test_fastembed_pinned_snapshot.py checks them against Python's posixpath/ntpath and huggingface_hub.constants; crates/headroom-core/src/hf_cache.rs checks its port against the same file.",
"expandvars": {
"comment": "posix = posixpath.expandvars, nt = ntpath.expandvars, with env set and unset absent (identical on CPython 3.10, 3.11, 3.13).",
"env": {
@@ -130,7 +130,10 @@
"comment": "{root} and {home} are temporary directories; HOME and USERPROFILE are set to {home}, every other cache variable is cleared first. expected may be {posix, nt}.",
"probe_vars": [
"HEADROOM_HF_PROBE_ROOT",
"HEADROOM_HF_PROBE_UNSET"
"HEADROOM_HF_PROBE_UNSET",
"HF_TOKEN",
"HF_TOKEN_PATH",
"HUGGING_FACE_HUB_TOKEN"
],
"cases": [
{
@@ -275,5 +278,102 @@
"expected": "{home}/.cache/huggingface/hub"
}
]
},
"token": {
"comment": "huggingface_hub.get_token(): env, files written first ({root}/{home} as in cache_dir), expected token or null. HF_TOKEN, HUGGING_FACE_HUB_TOKEN and HF_TOKEN_PATH are cleared with the cache variables.",
"cases": [
{
"name": "HF_TOKEN wins over the login file",
"env": {
"HF_TOKEN": "hf_env"
},
"files": {
"{home}/.cache/huggingface/token": "hf_file"
},
"expected": "hf_env"
},
{
"name": "the legacy HUGGING_FACE_HUB_TOKEN",
"env": {
"HUGGING_FACE_HUB_TOKEN": "hf_legacy"
},
"files": {},
"expected": "hf_legacy"
},
{
"name": "an empty HF_TOKEN falls through to the legacy name",
"env": {
"HF_TOKEN": "",
"HUGGING_FACE_HUB_TOKEN": "hf_legacy"
},
"files": {},
"expected": "hf_legacy"
},
{
"name": "a whitespace-only HF_TOKEN is still the one picked, so the legacy name is skipped",
"env": {
"HF_TOKEN": " ",
"HUGGING_FACE_HUB_TOKEN": "hf_legacy"
},
"files": {
"{home}/.cache/huggingface/token": "hf_file"
},
"expected": "hf_file"
},
{
"name": "the login file under the default HF_HOME, cleaned",
"env": {},
"files": {
"{home}/.cache/huggingface/token": " hf_file\r\n"
},
"expected": "hf_file"
},
{
"name": "the login file under HF_HOME",
"env": {
"HF_HOME": "{root}/hf"
},
"files": {
"{root}/hf/token": "hf_home"
},
"expected": "hf_home"
},
{
"name": "HF_TOKEN_PATH through ${VAR}",
"env": {
"HEADROOM_HF_PROBE_ROOT": "{root}",
"HF_TOKEN_PATH": "${HEADROOM_HF_PROBE_ROOT}/tok"
},
"files": {
"{root}/tok": "hf_path"
},
"expected": "hf_path"
},
{
"name": "moving the hub cache does not move the token",
"env": {
"HF_HUB_CACHE": "{root}/elsewhere/hub"
},
"files": {
"{home}/.cache/huggingface/token": "hf_file",
"{root}/elsewhere/token": "hf_wrong"
},
"expected": "hf_file"
},
{
"name": "a blank login file is no token",
"env": {},
"files": {
"{home}/.cache/huggingface/token": " \n"
},
"expected": null
},
{
"name": "nothing set",
"env": {},
"files": {},
"expected": null
}
]
}
}
+38 -2
View File
@@ -228,11 +228,11 @@ def test_python_matches_parity_fixture():
# ── the shared HuggingFace cache directory ──────────────────────────────────
#
# Rust's ``hf_hub_cache_dir`` re-implements how ``huggingface_hub`` picks the
# Rust's ``hf_cache::hub_cache_dir`` re-implements how ``huggingface_hub`` picks the
# hub cache, so both languages look up and download into one directory. These
# check the shared fixture against Python itself; the Rust tests
# ``expand_vars_matches_python_posixpath_and_ntpath`` and
# ``hf_hub_cache_dir_matches_the_shared_fixture`` check the port against the
# ``hub_cache_dir_matches_the_shared_fixture`` (``crates/headroom-core/src/hf_cache.rs``) check the port against the
# same file.
_CACHE_VARS = ("HF_HUB_CACHE", "HUGGINGFACE_HUB_CACHE", "HF_HOME", "XDG_CACHE_HOME")
@@ -288,3 +288,39 @@ def test_cache_dir_fixture_matches_huggingface_hub(case: dict, tmp_path: Path) -
if isinstance(expected, dict):
expected = expected["nt" if os.name == "nt" else "posix"]
assert Path(result.stdout) == Path(fill(expected))
@pytest.mark.parametrize("case", HF_CACHE_FIXTURE["token"]["cases"], ids=lambda case: case["name"])
def test_token_fixture_matches_huggingface_hub(case: dict, tmp_path: Path) -> None:
"""The token Rust's ``hf_cache::hf_token`` must find, from
``huggingface_hub.get_token()`` itself."""
import subprocess
pytest.importorskip("huggingface_hub")
root, home = tmp_path / "root", tmp_path / "home"
def fill(text: str) -> str:
return text.replace("{root}", str(root)).replace("{home}", str(home))
for path, content in case["files"].items():
target = Path(fill(path))
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(content, encoding="utf-8")
probe_vars = HF_CACHE_FIXTURE["cache_dir"]["probe_vars"]
env = {k: v for k, v in os.environ.items() if k not in (*_CACHE_VARS, *probe_vars)}
env["HOME"] = env["USERPROFILE"] = str(home)
env.update({name: fill(value) for name, value in case["env"].items()})
result = subprocess.run(
[
sys.executable,
"-c",
"import json, sys; from huggingface_hub import get_token; "
"sys.stdout.write(json.dumps(get_token()))",
],
capture_output=True,
text=True,
env=env,
timeout=60,
check=True,
)
assert json.loads(result.stdout) == case["expected"]
+37
View File
@@ -0,0 +1,37 @@
"""The Rust Kompress loader reads the snapshots the Python side pinned.
The native proxy's Kompress is cache-only: it never downloads, it reads the
files Python's ``hf_hub_download_local_first`` put in the HuggingFace cache at
``_PINNED_REVISIONS``. ``crates/headroom-core/src/transforms/kompress.rs`` looks
up the same commits, so the two must not drift: a pin bump on one side only
would leave the Rust proxy on a snapshot Python no longer downloads.
"""
from __future__ import annotations
import re
from pathlib import Path
from headroom.onnx_runtime import _PINNED_REVISIONS
RUST_KOMPRESS = (
Path(__file__).resolve().parents[1] / "crates/headroom-core/src/transforms/kompress.rs"
)
def _rust_const(name: str) -> str:
match = re.search(rf'pub const {name}: &str = "([^"]+)";', RUST_KOMPRESS.read_text())
assert match, f"{name} not found in {RUST_KOMPRESS}"
return match.group(1)
def test_rust_kompress_model_pin_matches_python():
repo = _rust_const("DEFAULT_MODEL_ID")
assert repo in _PINNED_REVISIONS, f"{repo} has no Python pin"
assert _rust_const("DEFAULT_MODEL_REVISION") == _PINNED_REVISIONS[repo]
def test_rust_kompress_tokenizer_pin_matches_python():
repo = _rust_const("DEFAULT_TOKENIZER_REPO")
assert repo in _PINNED_REVISIONS, f"{repo} has no Python pin"
assert _rust_const("DEFAULT_TOKENIZER_REVISION") == _PINNED_REVISIONS[repo]
+11 -7
View File
@@ -868,7 +868,8 @@ class TestHuggingFaceDownloadOffline:
# ───────────────── path 3: the Rust HuggingFace fetch (parity) ──────────────
_RUST_OFFLINE = REPO_ROOT / "crates" / "headroom-core" / "src" / "offline.rs"
_RUST_HF = REPO_ROOT / "crates" / "headroom-core" / "src" / "tokenizer" / "hf_impl.rs"
# Every Rust Hub download (tokenizer, embedder) builds its client in hf_cache::hub_api.
_RUST_HF = REPO_ROOT / "crates" / "headroom-core" / "src" / "hf_cache.rs"
class TestRustOfflineParity:
@@ -976,11 +977,14 @@ class TestRustOfflineParity:
def test_hf_fetch_guards_before_it_builds_a_client(self) -> None:
source = _RUST_HF.read_text(encoding="utf-8")
assert "guard_egress(" in source, "hf_impl.rs does not consult the offline guard"
guard_at = source.index("guard_egress(")
api_at = source.index("hf_hub::api::sync::Api::new()")
# Within hub_api itself: a guard elsewhere in the file does not count.
fn_at = source.index("pub(crate) fn hub_api(")
body = source[fn_at : source.index("\n}\n", fn_at)]
assert "guard_egress(" in body, "hf_cache::hub_api does not consult the offline guard"
guard_at = body.index("guard_egress(")
api_at = body.index("hf_hub::api::sync::ApiBuilder::from_cache(")
assert guard_at < api_at, (
"the offline guard runs after Api::new(), which already resolves the "
"the offline guard runs after ApiBuilder::build(), which already resolves the "
"Hub endpoint and builds the ureq agent — guard before the client "
"exists, not before the request"
)
@@ -3225,7 +3229,7 @@ _RUST_ALLOWLIST: dict[str, tuple[int, str]] = {
"cache-only: TextEmbedding::try_new_from_user_defined in load_snapshot "
"builds the ONNX session from bytes already read off disk and has no "
"Hub client at all. The files reach the disk through "
"default_model_repo, whose ApiBuilder is guarded, and only on a cache "
"default_model_repo, whose hf_cache::hub_api is guarded, and only on a cache "
"miss, so a pre-seeded air-gapped host still loads.",
),
"headroom-proxy/src/proxy.rs": (
@@ -3329,7 +3333,7 @@ class TestRustEgressChokepointCoverage:
def test_the_guarded_rust_paths_are_seen_as_guarded(self) -> None:
sites = _rust_egress_sites()
for relpath in (
"headroom-core/src/tokenizer/hf_impl.rs",
"headroom-core/src/hf_cache.rs",
"headroom-core/src/transforms/kompress.rs",
"headroom-core/src/relevance/embedding.rs",
):