mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
34da21e89e |
fix(tokenizers): disable trust_remote_code and allowlist HF tokenizer repos (#3723)
## A-1 (S1, remote code execution): HF tokenizer loader ran caller-named repositories `headroom/tokenizers/huggingface.py` called `AutoTokenizer.from_pretrained(..., trust_remote_code=True)` on an identifier that traces back to a proxied request body. `get_tokenizer_name()` ended in *"assume the model name is the tokenizer name"*, so an unrecognised `model` field was handed straight to the loader, which resolves it against the HuggingFace Hub and — with `trust_remote_code=True` — executes whatever Python that repository ships. Publish a repo, name it in a request body, get code execution in the proxy process. Nothing constrained the identifier. ### The fix 1. **`trust_remote_code=False` on every path**, passed explicitly rather than leaning on a library default upstream can change. 2. **`ALLOWED_TOKENIZER_REPOS`**, derived from the mappings the product actually ships (`MODEL_TO_TOKENIZER.values()`, so adding a family allows it and nothing drifts in). `_load_tokenizer` checks it *before* importing transformers: an unallowlisted name is refused outright and never becomes a Hub lookup. It fails closed to character estimation rather than silently substituting a different vocabulary under the caller's name. 3. **`get_tokenizer_name()` fails closed.** The fallthrough now returns `DEFAULT_TOKENIZER` instead of echoing the caller's string. A full repo id is accepted only when it is on the allowlist, and then the value returned is *our* canonical spelling, never the caller's. Two layers on purpose: the resolver is the normal path, the loader is the chokepoint — `_load_tokenizer` is importable and callable with an arbitrary string. `benchmarks/comprehensive_eval.py` also passed `trust_remote_code=True` to `datasets.load_dataset`. The brief only named the two tokenizer call sites; that third one is fixed here too, and the tree-wide test below now holds the line for all of them. ### Operator escape hatch `HEADROOM_HF_TOKENIZER_ALLOWLIST` (comma-separated repo ids) lets an operator admit private or self-hosted tokenizer repositories. It is deliberately environment-only — server-side configuration, never anything a client can reach through a request body — and repos added there are still loaded with `trust_remote_code=False`. ### Default change a customer could depend on An unrecognised model previously resolved to *itself* as a tokenizer name. In practice a made-up id failed to resolve on the Hub and counting fell back to estimation, so most deployments see no change. Two cases do change: - A caller who relied on passing an **arbitrary HF repo id** not in `MODEL_TO_TOKENIZER` (e.g. a private fine-tune) and getting exact counts now gets the default tokenizer instead. Remedy: add the repo to `HEADROOM_HF_TOKENIZER_ALLOWLIST`. - Unknown models now resolve to `DEFAULT_TOKENIZER` (Llama 3), so a deployment with that vocabulary in its HF cache gets a real BPE count where it previously got estimation. Repo ids that already resolved correctly are unchanged. Alias and prefix matching run first and are untouched, so existing precedence — including `mistralai/Mixtral-8x7B-v0.1` hitting the `mistral` prefix — resolves exactly as before; there is an explicit test pinning that so a future reorder has to acknowledge it. ### Tests - `tests/test_huggingface_tokenizer_allowlist.py` — a crafted `model` never reaches `from_pretrained`; every resolution lands inside the shipped set; the loader refuses an unallowlisted repo without touching the Hub; `trust_remote_code` is `False` on both the cache-only and bounded-network calls; every shipped alias still resolves to its mapping; the operator env hatch admits only what it names; and, since matching is case-insensitive, the string handed to the Hub is the allowlist's own spelling rather than the caller's variant. - `tests/test_no_trust_remote_code.py` — tree-wide scan over every git-tracked file asserting every `trust_remote_code` assignment is `False`. - `tests/test_huggingface_tokenizer_timeout.py` — the existing GH #1701 timeout tests used placeholder repo ids (`some/model`, `slow/model`, `offline/model`) which the allowlist now rejects before the timeout logic runs, short-circuiting what those tests exist to prove. Switched to real shipped repo ids so they still exercise the bounded-load path. ### Fail-before / pass-after evidence Method: commit, `git checkout origin/main -- headroom/tokenizers/huggingface.py benchmarks/comprehensive_eval.py` (source reverted, new tests kept), run, then `git checkout HEAD -- ...` to restore. **Before (source reverted, tests kept) — 10 failed, 4 passed:** ``` tests/test_huggingface_tokenizer_allowlist.py FFFF..F..FFF [ 85%] tests/test_no_trust_remote_code.py FF [100%] =================================== FAILURES =================================== E AssertionError: attacker-named repository was handed to from_pretrained assert 'attacker-controlled/headroom-rce-poc' not in ['attacker-controlled/headroom-rce-poc'] E AssertionError: 'attacker-controlled/headroom-rce-poc' escaped the allowlist assert 'attacker-controlled/headroom-rce-poc' in frozenset({'01-ai/Yi-1.5-6B', ...}) + where 'attacker-controlled/headroom-rce-poc' = get_tokenizer_name('attacker-controlled/headroom-rce-poc') E AttributeError: module 'headroom.tokenizers.huggingface' has no attribute 'DEFAULT_TOKENIZER' E AssertionError: trust_remote_code must be False everywhere — it executes repository-supplied Python in this process: benchmarks/comprehensive_eval.py:44: trust_remote_code=True, headroom/tokenizers/huggingface.py:162: trust_remote_code=True, headroom/tokenizers/huggingface.py:184: trust_remote_code=True, E AssertionError: assert {'True'} == {'False'} =========================== short test summary info ============================ FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_never_reaches_the_loader FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_a_shipped_tokenizer FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_the_declared_default FAILED tests/test_huggingface_tokenizer_allowlist.py::test_loader_fetches_the_allowlists_own_spelling FAILED tests/test_huggingface_tokenizer_allowlist.py::test_remote_code_disabled_on_cache_and_network_paths FAILED tests/test_huggingface_tokenizer_allowlist.py::test_repo_id_matching_is_case_insensitive_but_returns_our_spelling FAILED tests/test_huggingface_tokenizer_allowlist.py::test_operator_env_allowlist_admits_a_private_repo FAILED tests/test_huggingface_tokenizer_allowlist.py::test_empty_operator_env_admits_nothing_extra FAILED tests/test_no_trust_remote_code.py::test_remote_code_is_never_enabled_anywhere_in_the_tree FAILED tests/test_no_trust_remote_code.py::test_the_tokenizer_loader_sets_the_flag_explicitly ========================= 10 failed, 4 passed in 1.47s ========================= ``` **After (source restored):** ``` tests/test_huggingface_tokenizer_allowlist.py ............ [ 54%] tests/test_no_trust_remote_code.py .. [ 63%] tests/test_huggingface_tokenizer_timeout.py ........ [100%] ============================== 22 passed in 2.73s ============================== ``` Wider tokenizer suites (`test_tokenizers.py`, `test_tokenizer.py`, `test_tokenizers/`, `test_tokenizer_encoding_resolution.py`, `test_tokenizer_selection_coverage.py`, `test_provider_tokenizer_one_ruler.py`, `test_tokenizer_count_offload.py`, `test_strands_tokenizer.py`, `test_compress_route_tokenizer_by_model.py`, `test_evals_cjk_tokenization.py`): **166 passed, 14 skipped**. Lint: `ruff 0.16.7 check .` and `format --check .` both clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
bd2d447c26 |
Add quality retention eval and fix linting for Python 3.12
- Add quality_retention_eval.py for needle-in-haystack testing to verify intelligent compression retains critical information (100% retention achieved) - Add intelligent_context_integration_test.py for comprehensive pipeline testing - Add test_progressive_summarizer.py with 36 tests for ProgressiveSummarizer - Add HeadroomConfig parameter to HeadroomClient for direct config injection - Update pipeline.py with IntelligentContextManager wiring and logging - Fix all ruff linting issues and format for Python 3.12 compatibility - Add comprehensive_eval.py benchmark for multi-scenario evaluation - Add real_data_demo.py for production-scale volume testing - Add reasoning agent test examples (groq, debug) |