mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description `HEADROOM_OFFLINE=1` must not fall through from a cold HuggingFace tokenizer cache to a network download. This change applies the existing offline environment defaults before importing Transformers and returns to estimation after a cache miss when offline mode is active. Closes #3777 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Apply `apply_offline_env()` before the Transformers import when `HEADROOM_OFFLINE` is enabled. - Skip the network fallback and use estimation after an offline cache miss. - Add regression coverage for cold-cache no-egress behavior, explicit offline-flag overrides, and warm-cache loading. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text 22 passed in 1.09s 2 files already formatted All checks passed! Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.13.15, `uv` 0.12.15, Transformers 5.x, empty temporary `HF_HOME`, and a loopback HTTP server used as `HF_ENDPOINT`; no provider or HuggingFace credentials. - Exact command / steps: ran `env -u HF_HUB_OFFLINE -u TRANSFORMERS_OFFLINE PYTHONPATH=. uv run --no-project --with 'transformers>=5.5,<6' --with opentelemetry-api --with pydantic --with tiktoken --with click --with rich --with pyyaml --with tomlkit python -` with a script that started the loopback server, set `HF_HOME` to an empty temporary directory, set `HEADROOM_OFFLINE=1`, called `_load_tokenizer("Qwen/Qwen2.5-7B")`, and recorded requests and cache entries. - Observed result: `_load_tokenizer` returned `None` (estimation fallback), the loopback HuggingFace endpoint received no requests, the empty `HF_HOME` stayed empty, and both `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` were set to `1`: ```text {"cache_entries": [], "hf_hub_offline": "1", "loader_http_requests": [], "result": null, "transformers_offline": "1"} ``` - Not tested: a real external `huggingface.co` endpoint, a full proxy request, or a real warm tokenizer cache; those paths are covered by the existing loader tests and the local proof isolates the changed behavior. ## Runtime Rollout Safety - Rollout-managed feature(s): HuggingFace tokenizer network fallback. - Minimum rollout channel: next patch release. - Stable/default behavior changed: only when `HEADROOM_OFFLINE` is enabled; online behavior remains unchanged. - Kill switch / disable path: unset `HEADROOM_OFFLINE` and restart the process; existing explicit `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` overrides remain respected by `apply_offline_env()`. - Unsafe override required: none. - Qualification impact: no provider, model, or API qualification changes. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title ## Screenshots (if applicable) Not applicable. ## Additional Notes No dependencies or generated files were added. The implementation reuses the existing offline-mode contract rather than introducing a second configuration switch. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3783" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
190 lines
7.8 KiB
Python
190 lines
7.8 KiB
Python
"""HF tokenizer loading must be bounded (GH #1701): AutoTokenizer.from_pretrained
|
|
performs unbounded network downloads/retries; called lazily from the proxy's request
|
|
path it blocked the event loop for ~10 minutes and zombified the server. The fix
|
|
tries the local HF cache first (local_files_only=True), bounds the network attempt
|
|
with HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS on a daemon thread, and fails open to
|
|
estimation — caching the failure so the hub is probed at most once per process.
|
|
|
|
The repo ids below are real shipped ones because _load_tokenizer now refuses
|
|
anything off the tokenizer allowlist before it reaches the loading path at all
|
|
(see test_huggingface_tokenizer_allowlist.py); a placeholder name would short
|
|
out these tests by being rejected rather than exercising the timeout.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import time
|
|
import types
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.tokenizers import huggingface as hf_mod
|
|
from headroom.tokenizers.huggingface import (
|
|
HuggingFaceTokenizer,
|
|
_load_tokenizer,
|
|
get_tokenizer_name,
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _fresh_cache(monkeypatch: pytest.MonkeyPatch):
|
|
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
|
monkeypatch.delenv("HF_HUB_OFFLINE", raising=False)
|
|
monkeypatch.delenv("TRANSFORMERS_OFFLINE", raising=False)
|
|
_load_tokenizer.cache_clear()
|
|
yield
|
|
_load_tokenizer.cache_clear()
|
|
|
|
|
|
def _install_fake_transformers(monkeypatch: pytest.MonkeyPatch, from_pretrained) -> None:
|
|
fake = types.ModuleType("transformers")
|
|
fake.AutoTokenizer = type(
|
|
"AutoTokenizer", (), {"from_pretrained": staticmethod(from_pretrained)}
|
|
)
|
|
monkeypatch.setitem(sys.modules, "transformers", fake)
|
|
|
|
|
|
def test_local_cache_tried_before_network(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
calls: list[dict[str, Any]] = []
|
|
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
calls.append(kwargs)
|
|
if kwargs.get("local_files_only"):
|
|
raise OSError("not in cache")
|
|
return "network-tokenizer"
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
monkeypatch.setenv("HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS", "5")
|
|
|
|
assert _load_tokenizer("Qwen/Qwen2.5-7B") == "network-tokenizer"
|
|
assert calls[0].get("local_files_only") is True, "first attempt must be cache-only"
|
|
assert not calls[1].get("local_files_only")
|
|
|
|
|
|
def test_cache_hit_never_touches_network(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
calls: list[dict[str, Any]] = []
|
|
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
calls.append(kwargs)
|
|
return "cached-tokenizer"
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
|
|
assert _load_tokenizer("Qwen/Qwen2.5-7B") == "cached-tokenizer"
|
|
assert len(calls) == 1
|
|
assert calls[0].get("local_files_only") is True
|
|
|
|
|
|
def test_slow_network_load_times_out_and_fails_open(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
if kwargs.get("local_files_only"):
|
|
raise OSError("not in cache")
|
|
time.sleep(60) # simulates hung huggingface_hub download
|
|
return "never"
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
monkeypatch.setenv("HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS", "0.2")
|
|
|
|
start = time.monotonic()
|
|
assert _load_tokenizer("Qwen/Qwen2-7B") is None
|
|
assert time.monotonic() - start < 5, "load must unblock at the timeout, not the download"
|
|
|
|
# Failure is cached (lru_cache) — the second call must not re-probe the hub.
|
|
start = time.monotonic()
|
|
assert _load_tokenizer("Qwen/Qwen2-7B") is None
|
|
assert time.monotonic() - start < 0.05
|
|
|
|
|
|
def test_timeout_zero_disables_network_loading(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
if kwargs.get("local_files_only"):
|
|
raise OSError("not in cache")
|
|
raise AssertionError("network load attempted despite timeout=0")
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
monkeypatch.setenv("HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS", "0")
|
|
|
|
assert _load_tokenizer("google/gemma-7b") is None
|
|
|
|
|
|
def test_offline_mode_prevents_network_loading(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
calls: list[dict[str, Any]] = []
|
|
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
calls.append(kwargs)
|
|
if kwargs.get("local_files_only"):
|
|
raise OSError("not in cache")
|
|
raise AssertionError("network load attempted in offline mode")
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
monkeypatch.setenv("HF_HUB_OFFLINE", "0")
|
|
monkeypatch.setenv("TRANSFORMERS_OFFLINE", "0")
|
|
monkeypatch.setenv("HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS", "5")
|
|
|
|
assert _load_tokenizer("Qwen/Qwen2.5-7B") is None
|
|
assert len(calls) == 1
|
|
assert calls[0].get("local_files_only") is True
|
|
|
|
|
|
def test_offline_mode_allows_cached_tokenizer(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
calls: list[dict[str, Any]] = []
|
|
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
calls.append(kwargs)
|
|
return "cached-tokenizer"
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
|
|
|
assert _load_tokenizer("Qwen/Qwen2.5-7B") == "cached-tokenizer"
|
|
assert len(calls) == 1
|
|
assert calls[0].get("local_files_only") is True
|
|
|
|
|
|
def test_count_messages_fails_open_to_estimation(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
def fake_from_pretrained(name: str, **kwargs: Any):
|
|
raise OSError("unavailable")
|
|
|
|
_install_fake_transformers(monkeypatch, fake_from_pretrained)
|
|
monkeypatch.setenv("HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS", "0.2")
|
|
|
|
counter = HuggingFaceTokenizer("deepseek-chat")
|
|
tokens = counter.count_messages([{"role": "user", "content": "hello world" * 50}])
|
|
assert tokens > 0 # estimation fallback, no exception, no hang
|
|
|
|
|
|
def test_deepseek_model_aliases_resolve_to_expected_tokenizers() -> None:
|
|
assert get_tokenizer_name("deepseek-v3.2") == "deepseek-ai/DeepSeek-V3.2"
|
|
assert get_tokenizer_name("deepseek-v4-pro") == "deepseek-ai/DeepSeek-V4-Pro"
|
|
assert get_tokenizer_name("deepseek-v4-flash") == "deepseek-ai/DeepSeek-V4.1-Flash"
|
|
assert get_tokenizer_name("deepseek-flash") == "deepseek-ai/DeepSeek-V4.1-Flash"
|
|
assert get_tokenizer_name("deepseek-v4-flash-vision-exp") == "deepseek-ai/DeepSeek-V4.1-Flash"
|
|
assert get_tokenizer_name("deepseek-r1") == "deepseek-ai/DeepSeek-R1"
|
|
assert get_tokenizer_name("deepseek-r1-0528") == "deepseek-ai/DeepSeek-R1-0528"
|
|
|
|
|
|
def test_invalid_timeout_env_falls_back_to_default(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setenv("HEADROOM_HF_TOKENIZER_LOAD_TIMEOUT_SECS", "not-a-number")
|
|
assert hf_mod._load_timeout_secs() == hf_mod._LOAD_TIMEOUT_DEFAULT
|
|
|
|
|
|
def test_get_tokenizer_name_prefers_most_specific_prefix() -> None:
|
|
"""A more-specific family key must win over a shorter one.
|
|
|
|
Prefix matching used to scan MODEL_TO_TOKENIZER in dict-insertion order, so
|
|
the short "qwen" key preceded "qwen2"/"qwen2.5" and shadowed them —
|
|
"qwen2-7b-instruct" resolved to the Qwen1 tokenizer (a different vocabulary,
|
|
hence wrong counts). The resolver now picks the longest matching prefix.
|
|
"""
|
|
# Versioned models not present as literal keys must hit the right family.
|
|
assert get_tokenizer_name("qwen2-7b-instruct") == "Qwen/Qwen2-7B"
|
|
assert get_tokenizer_name("qwen2.5-turbo") == "Qwen/Qwen2.5-7B"
|
|
assert get_tokenizer_name("deepseek-v2.5") == "deepseek-ai/DeepSeek-V2"
|
|
# Direct hits and shorter family fallbacks still resolve through their
|
|
# longest matching tokenizer aliases.
|
|
assert get_tokenizer_name("qwen-14b") == "Qwen/Qwen-14B"
|
|
assert get_tokenizer_name("deepseek-chat") == "deepseek-ai/DeepSeek-V3"
|