mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
48f650ad04 |
test: make the trust_remote_code scan see every truthy value and run without git (#4785)
## Description
`tests/test_no_trust_remote_code.py` scans the tracked tree so that
nothing re-enables HuggingFace remote code execution. Its regex captured
the assigned value only when it started with a letter
(`([A-Za-z_][\w.]*)`), but the library tests the flag for truthiness, so
these also enable it and were not seen: `trust_remote_code=1`,
`trust_remote_code=(True)`, `trust_remote_code="yes"`, `setattr(cfg,
"trust_remote_code", True)` and `kwargs.setdefault("trust_remote_code",
True)`. The test also called `pytest.skip` when `git ls-files` failed
(for example in an exported tarball), so the guard silently stopped
running.
This PR widens the value rule (a quoted string, or a name or number,
with an optional leading `(`, treating anything except the exact name
`False` as enabling the flag), adds a second pattern for the `"name",
value` form, matches across a line break, and falls back to scanning
`headroom/**/*.py` when git is unavailable instead of skipping. The scan
of the current tree finds no offender with the wider rules.
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_no_trust_remote_code.py`: `_VALUE`, a wider `ASSIGNMENT`,
a new `NAMED_ARGUMENT` limited to
`setattr`/`getattr`/`setdefault`/`get`/`pop` calls, a `_values()` helper
that reports `(line, value)`, a git-less fallback in `_tracked_files()`,
and 17 new cases: calls that do not set the flag (`log.info("...",
True)`) must be ignored, every truthy shape must be flagged, a value on
the next line must be flagged, and an explicit `False` (including
`(False)`, `kwargs["..."] = False` and `setattr(..., False)`) must be
accepted.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format` (0.17.3) are clean.
### Test Output
The old regex on the shapes the old test never saw (`re.findall` with
the old pattern):
```text
old regex matches=True trust_remote_code=True
old regex matches=False trust_remote_code=1
old regex matches=False trust_remote_code=(True)
old regex matches=False trust_remote_code="yes"
old regex matches=False setattr(cfg, "trust_remote_code", True)
old regex matches=False kwargs.setdefault("trust_remote_code", True)
```
The new tests with the old scan rule (value must start with a letter, no
`"name", value` pattern):
```text
6 failed, 10 passed (the =1, ="yes", mapping =1, setattr, setdefault and setattr(..., False) cases fail)
```
The new tests with this branch's scan:
```text
tests/test_no_trust_remote_code.py 19 passed (the tree scan finds no offender)
same file run from a copy of the tree without .git 19 passed, 0 skipped
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0, `main` at
`3c66c91e`.
- Exact command / steps: `python -m pytest
tests/test_no_trust_remote_code.py`; then the same file from a copy of
the repository without `.git`; then the new tests with the scan logic
swapped for the old rule.
- Observed result: as above. The tree scan passes on the current tree
with the wider rules, so no existing code needs to change.
- Not tested: CI, and a deliberately injected `trust_remote_code=1` in a
real module (the shapes are covered by the unit cases instead).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
scan is stricter and no longer skips without git.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- A value such as `"False"` (a string) is truthy in Python, so the scan
flags it; I think that is right for this guard.
- The git-less fallback scans only `headroom/`: an exported tarball has
no other tracked-file list.
AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4785"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
34da21e89e |
fix(tokenizers): disable trust_remote_code and allowlist HF tokenizer repos (#3723)
## A-1 (S1, remote code execution): HF tokenizer loader ran caller-named repositories `headroom/tokenizers/huggingface.py` called `AutoTokenizer.from_pretrained(..., trust_remote_code=True)` on an identifier that traces back to a proxied request body. `get_tokenizer_name()` ended in *"assume the model name is the tokenizer name"*, so an unrecognised `model` field was handed straight to the loader, which resolves it against the HuggingFace Hub and — with `trust_remote_code=True` — executes whatever Python that repository ships. Publish a repo, name it in a request body, get code execution in the proxy process. Nothing constrained the identifier. ### The fix 1. **`trust_remote_code=False` on every path**, passed explicitly rather than leaning on a library default upstream can change. 2. **`ALLOWED_TOKENIZER_REPOS`**, derived from the mappings the product actually ships (`MODEL_TO_TOKENIZER.values()`, so adding a family allows it and nothing drifts in). `_load_tokenizer` checks it *before* importing transformers: an unallowlisted name is refused outright and never becomes a Hub lookup. It fails closed to character estimation rather than silently substituting a different vocabulary under the caller's name. 3. **`get_tokenizer_name()` fails closed.** The fallthrough now returns `DEFAULT_TOKENIZER` instead of echoing the caller's string. A full repo id is accepted only when it is on the allowlist, and then the value returned is *our* canonical spelling, never the caller's. Two layers on purpose: the resolver is the normal path, the loader is the chokepoint — `_load_tokenizer` is importable and callable with an arbitrary string. `benchmarks/comprehensive_eval.py` also passed `trust_remote_code=True` to `datasets.load_dataset`. The brief only named the two tokenizer call sites; that third one is fixed here too, and the tree-wide test below now holds the line for all of them. ### Operator escape hatch `HEADROOM_HF_TOKENIZER_ALLOWLIST` (comma-separated repo ids) lets an operator admit private or self-hosted tokenizer repositories. It is deliberately environment-only — server-side configuration, never anything a client can reach through a request body — and repos added there are still loaded with `trust_remote_code=False`. ### Default change a customer could depend on An unrecognised model previously resolved to *itself* as a tokenizer name. In practice a made-up id failed to resolve on the Hub and counting fell back to estimation, so most deployments see no change. Two cases do change: - A caller who relied on passing an **arbitrary HF repo id** not in `MODEL_TO_TOKENIZER` (e.g. a private fine-tune) and getting exact counts now gets the default tokenizer instead. Remedy: add the repo to `HEADROOM_HF_TOKENIZER_ALLOWLIST`. - Unknown models now resolve to `DEFAULT_TOKENIZER` (Llama 3), so a deployment with that vocabulary in its HF cache gets a real BPE count where it previously got estimation. Repo ids that already resolved correctly are unchanged. Alias and prefix matching run first and are untouched, so existing precedence — including `mistralai/Mixtral-8x7B-v0.1` hitting the `mistral` prefix — resolves exactly as before; there is an explicit test pinning that so a future reorder has to acknowledge it. ### Tests - `tests/test_huggingface_tokenizer_allowlist.py` — a crafted `model` never reaches `from_pretrained`; every resolution lands inside the shipped set; the loader refuses an unallowlisted repo without touching the Hub; `trust_remote_code` is `False` on both the cache-only and bounded-network calls; every shipped alias still resolves to its mapping; the operator env hatch admits only what it names; and, since matching is case-insensitive, the string handed to the Hub is the allowlist's own spelling rather than the caller's variant. - `tests/test_no_trust_remote_code.py` — tree-wide scan over every git-tracked file asserting every `trust_remote_code` assignment is `False`. - `tests/test_huggingface_tokenizer_timeout.py` — the existing GH #1701 timeout tests used placeholder repo ids (`some/model`, `slow/model`, `offline/model`) which the allowlist now rejects before the timeout logic runs, short-circuiting what those tests exist to prove. Switched to real shipped repo ids so they still exercise the bounded-load path. ### Fail-before / pass-after evidence Method: commit, `git checkout origin/main -- headroom/tokenizers/huggingface.py benchmarks/comprehensive_eval.py` (source reverted, new tests kept), run, then `git checkout HEAD -- ...` to restore. **Before (source reverted, tests kept) — 10 failed, 4 passed:** ``` tests/test_huggingface_tokenizer_allowlist.py FFFF..F..FFF [ 85%] tests/test_no_trust_remote_code.py FF [100%] =================================== FAILURES =================================== E AssertionError: attacker-named repository was handed to from_pretrained assert 'attacker-controlled/headroom-rce-poc' not in ['attacker-controlled/headroom-rce-poc'] E AssertionError: 'attacker-controlled/headroom-rce-poc' escaped the allowlist assert 'attacker-controlled/headroom-rce-poc' in frozenset({'01-ai/Yi-1.5-6B', ...}) + where 'attacker-controlled/headroom-rce-poc' = get_tokenizer_name('attacker-controlled/headroom-rce-poc') E AttributeError: module 'headroom.tokenizers.huggingface' has no attribute 'DEFAULT_TOKENIZER' E AssertionError: trust_remote_code must be False everywhere — it executes repository-supplied Python in this process: benchmarks/comprehensive_eval.py:44: trust_remote_code=True, headroom/tokenizers/huggingface.py:162: trust_remote_code=True, headroom/tokenizers/huggingface.py:184: trust_remote_code=True, E AssertionError: assert {'True'} == {'False'} =========================== short test summary info ============================ FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_never_reaches_the_loader FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_a_shipped_tokenizer FAILED tests/test_huggingface_tokenizer_allowlist.py::test_crafted_model_resolves_to_the_declared_default FAILED tests/test_huggingface_tokenizer_allowlist.py::test_loader_fetches_the_allowlists_own_spelling FAILED tests/test_huggingface_tokenizer_allowlist.py::test_remote_code_disabled_on_cache_and_network_paths FAILED tests/test_huggingface_tokenizer_allowlist.py::test_repo_id_matching_is_case_insensitive_but_returns_our_spelling FAILED tests/test_huggingface_tokenizer_allowlist.py::test_operator_env_allowlist_admits_a_private_repo FAILED tests/test_huggingface_tokenizer_allowlist.py::test_empty_operator_env_admits_nothing_extra FAILED tests/test_no_trust_remote_code.py::test_remote_code_is_never_enabled_anywhere_in_the_tree FAILED tests/test_no_trust_remote_code.py::test_the_tokenizer_loader_sets_the_flag_explicitly ========================= 10 failed, 4 passed in 1.47s ========================= ``` **After (source restored):** ``` tests/test_huggingface_tokenizer_allowlist.py ............ [ 54%] tests/test_no_trust_remote_code.py .. [ 63%] tests/test_huggingface_tokenizer_timeout.py ........ [100%] ============================== 22 passed in 2.73s ============================== ``` Wider tokenizer suites (`test_tokenizers.py`, `test_tokenizer.py`, `test_tokenizers/`, `test_tokenizer_encoding_resolution.py`, `test_tokenizer_selection_coverage.py`, `test_provider_tokenizer_one_ruler.py`, `test_tokenizer_count_offload.py`, `test_strands_tokenizer.py`, `test_compress_route_tokenizer_by_model.py`, `test_evals_cjk_tokenization.py`): **166 passed, 14 skipped**. Lint: `ruff 0.16.7 check .` and `format --check .` both clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |