mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description
`tests/test_no_trust_remote_code.py` scans the tracked tree so that
nothing re-enables HuggingFace remote code execution. Its regex captured
the assigned value only when it started with a letter
(`([A-Za-z_][\w.]*)`), but the library tests the flag for truthiness, so
these also enable it and were not seen: `trust_remote_code=1`,
`trust_remote_code=(True)`, `trust_remote_code="yes"`, `setattr(cfg,
"trust_remote_code", True)` and `kwargs.setdefault("trust_remote_code",
True)`. The test also called `pytest.skip` when `git ls-files` failed
(for example in an exported tarball), so the guard silently stopped
running.
This PR widens the value rule (a quoted string, or a name or number,
with an optional leading `(`, treating anything except the exact name
`False` as enabling the flag), adds a second pattern for the `"name",
value` form, matches across a line break, and falls back to scanning
`headroom/**/*.py` when git is unavailable instead of skipping. The scan
of the current tree finds no offender with the wider rules.
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_no_trust_remote_code.py`: `_VALUE`, a wider `ASSIGNMENT`,
a new `NAMED_ARGUMENT` limited to
`setattr`/`getattr`/`setdefault`/`get`/`pop` calls, a `_values()` helper
that reports `(line, value)`, a git-less fallback in `_tracked_files()`,
and 17 new cases: calls that do not set the flag (`log.info("...",
True)`) must be ignored, every truthy shape must be flagged, a value on
the next line must be flagged, and an explicit `False` (including
`(False)`, `kwargs["..."] = False` and `setattr(..., False)`) must be
accepted.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format` (0.17.3) are clean.
### Test Output
The old regex on the shapes the old test never saw (`re.findall` with
the old pattern):
```text
old regex matches=True trust_remote_code=True
old regex matches=False trust_remote_code=1
old regex matches=False trust_remote_code=(True)
old regex matches=False trust_remote_code="yes"
old regex matches=False setattr(cfg, "trust_remote_code", True)
old regex matches=False kwargs.setdefault("trust_remote_code", True)
```
The new tests with the old scan rule (value must start with a letter, no
`"name", value` pattern):
```text
6 failed, 10 passed (the =1, ="yes", mapping =1, setattr, setdefault and setattr(..., False) cases fail)
```
The new tests with this branch's scan:
```text
tests/test_no_trust_remote_code.py 19 passed (the tree scan finds no offender)
same file run from a copy of the tree without .git 19 passed, 0 skipped
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0, `main` at
`3c66c91e`.
- Exact command / steps: `python -m pytest
tests/test_no_trust_remote_code.py`; then the same file from a copy of
the repository without `.git`; then the new tests with the scan logic
swapped for the old rule.
- Observed result: as above. The tree scan passes on the current tree
with the wider rules, so no existing code needs to change.
- Not tested: CI, and a deliberately injected `trust_remote_code=1` in a
real module (the shapes are covered by the unit cases instead).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
scan is stricter and no longer skips without git.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- A value such as `"False"` (a string) is truthy in Python, so the scan
flags it; I think that is right for this guard.
- The git-less fallback scans only `headroom/`: an exported tarball has
no other tracked-file list.
AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4785"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
157 lines
5.7 KiB
Python
157 lines
5.7 KiB
Python
"""Nothing in this tree may enable HuggingFace remote code execution (A-1).
|
|
|
|
``trust_remote_code`` makes a model/dataset repository's own Python run inside
|
|
the process that loads it. Headroom loads tokenizers from identifiers that trace
|
|
back to proxied request bodies, so a single re-introduced ``=True`` anywhere is
|
|
remote code execution in the proxy. This is a tree-wide scan rather than a test
|
|
of one module: the tokenizer loader was not the only caller, and the next one
|
|
will not be either.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
ROOT = Path(__file__).resolve().parents[1]
|
|
|
|
# Assembled at runtime so this file contributes no literal occurrence of the
|
|
# token to its own scan.
|
|
FLAG = "trust_remote" + "_code"
|
|
|
|
# The value: a quoted string, or a name / number (``True``, ``1``, ``-1``, ``cfg.x``).
|
|
# Anything but the exact name ``False`` is treated as enabling the flag.
|
|
_VALUE = r"\"[^\"]*\"|'[^']*'|[-+]?[\w.]+"
|
|
|
|
# Matches the kwarg form (``flag=True``), the mapping form (``"flag": True``)
|
|
# and the subscript form (``kwargs["flag"] = True``), capturing whatever it is
|
|
# set to up to the next delimiter. The optional quote and bracket matter:
|
|
# without them the mapping form slips through, and
|
|
# ``from_pretrained(**{"flag": True})`` is exactly how this would come back.
|
|
# The value is NOT required to start with a letter: the library tests the flag
|
|
# for truthiness, so ``=1``, ``=(True)`` and ``="yes"`` enable it just as
|
|
# ``=True`` does. A value on the next line is still a value, because ``\s*``
|
|
# spans the newline.
|
|
ASSIGNMENT = re.compile(re.escape(FLAG) + r"[\"']?\s*\]?\s*[=:]\s*\(*\s*(" + _VALUE + ")")
|
|
|
|
# ``setattr(cfg, "flag", True)`` and ``kwargs.setdefault("flag", True)``: the
|
|
# name is a string literal followed by a comma, not by ``=`` or ``:``.
|
|
# Only calls that set or default the flag count: ``log.info("flag", True)`` does not.
|
|
NAMED_ARGUMENT = re.compile(
|
|
r"(?:\b(?:setattr|getattr)\s*\([^,()]*,|\b(?:setdefault|get|pop)\s*\()\s*[\"']"
|
|
+ re.escape(FLAG)
|
|
+ r"[\"']\s*,\s*\(*\s*("
|
|
+ _VALUE
|
|
+ ")"
|
|
)
|
|
|
|
|
|
def _values(text: str) -> list[tuple[int, str]]:
|
|
"""Every ``(line number, value)`` the flag is set to in ``text``."""
|
|
found: list[tuple[int, str]] = []
|
|
for pattern in (ASSIGNMENT, NAMED_ARGUMENT):
|
|
for match in pattern.finditer(text):
|
|
found.append((text.count("\n", 0, match.start()) + 1, match.group(1)))
|
|
return found
|
|
|
|
|
|
def _tracked_files() -> list[Path]:
|
|
"""Every git-tracked file, so the scan cannot be dodged by adding a directory.
|
|
|
|
git-tracked rather than a filesystem walk: the repository is commonly checked
|
|
out alongside worktrees and virtualenvs that a naive rglob would sweep in.
|
|
Without git (an exported tarball) the package itself is still scanned, so the
|
|
guard does not silently stop running.
|
|
"""
|
|
try:
|
|
out = subprocess.run(
|
|
["git", "ls-files", "-z"],
|
|
cwd=ROOT,
|
|
capture_output=True,
|
|
check=True,
|
|
).stdout
|
|
except (OSError, subprocess.CalledProcessError):
|
|
return sorted((ROOT / "headroom").rglob("*.py"))
|
|
return [ROOT / name for name in out.decode().split("\0") if name]
|
|
|
|
|
|
def test_remote_code_is_never_enabled_anywhere_in_the_tree() -> None:
|
|
offenders: list[str] = []
|
|
|
|
for path in _tracked_files():
|
|
try:
|
|
text = path.read_text(encoding="utf-8")
|
|
except (OSError, UnicodeDecodeError):
|
|
continue # binary or vanished (submodule, symlink) — nothing to read
|
|
if FLAG not in text:
|
|
continue
|
|
lines = text.splitlines()
|
|
for lineno, value in _values(text):
|
|
if value != "False":
|
|
offenders.append(f"{path.relative_to(ROOT)}:{lineno}: {lines[lineno - 1].strip()}")
|
|
|
|
assert not offenders, (
|
|
f"{FLAG} must be False everywhere — it executes repository-supplied "
|
|
"Python in this process:\n" + "\n".join(offenders)
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"line",
|
|
[
|
|
f"model = load(x, {FLAG}=True)",
|
|
f"model = load(x, {FLAG}=1)",
|
|
f"model = load(x, {FLAG}=(True))",
|
|
f'model = load(x, {FLAG}="yes")',
|
|
f"model = load(x, {FLAG}=enabled)",
|
|
f'kwargs["{FLAG}"] = True',
|
|
f'load(**{{"{FLAG}": 1}})',
|
|
f'setattr(cfg, "{FLAG}", True)',
|
|
f'kwargs.setdefault("{FLAG}", True)',
|
|
],
|
|
)
|
|
def test_the_scan_flags_every_truthy_way_to_enable_the_flag(line: str) -> None:
|
|
values = [value for _, value in _values(line)]
|
|
assert values and all(value != "False" for value in values), line
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"line",
|
|
[
|
|
f'log.info("{FLAG}", True)',
|
|
f'print("{FLAG}", 1)',
|
|
f'logger.debug("{FLAG}", enabled)',
|
|
],
|
|
)
|
|
def test_the_scan_ignores_calls_that_do_not_set_the_flag(line: str) -> None:
|
|
assert _values(line) == []
|
|
|
|
|
|
def test_the_scan_flags_a_value_on_the_next_line() -> None:
|
|
text = f"load(\n x,\n {FLAG}=\n True,\n)\n"
|
|
assert [value for _, value in _values(text)] == ["True"]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"line",
|
|
[
|
|
f"model = load(x, {FLAG}=False)",
|
|
f"model = load(x, {FLAG}=(False))",
|
|
f'kwargs["{FLAG}"] = False',
|
|
f'setattr(cfg, "{FLAG}", False)',
|
|
],
|
|
)
|
|
def test_the_scan_accepts_an_explicit_false(line: str) -> None:
|
|
assert [value for _, value in _values(line)] == ["False"]
|
|
|
|
|
|
def test_the_tokenizer_loader_sets_the_flag_explicitly() -> None:
|
|
"""Explicit beats relying on the library default, which upstream can change."""
|
|
source = (ROOT / "headroom" / "tokenizers" / "huggingface.py").read_text(encoding="utf-8")
|
|
values = [value for _, value in _values(source)]
|
|
assert values, f"tokenizer loader no longer passes {FLAG} explicitly"
|
|
assert set(values) == {"False"}
|