Files
headroom/tests/test_no_trust_remote_code.py
48f650ad04 test: make the trust_remote_code scan see every truthy value and run without git (#4785)
## Description

`tests/test_no_trust_remote_code.py` scans the tracked tree so that
nothing re-enables HuggingFace remote code execution. Its regex captured
the assigned value only when it started with a letter
(`([A-Za-z_][\w.]*)`), but the library tests the flag for truthiness, so
these also enable it and were not seen: `trust_remote_code=1`,
`trust_remote_code=(True)`, `trust_remote_code="yes"`, `setattr(cfg,
"trust_remote_code", True)` and `kwargs.setdefault("trust_remote_code",
True)`. The test also called `pytest.skip` when `git ls-files` failed
(for example in an exported tarball), so the guard silently stopped
running.

This PR widens the value rule (a quoted string, or a name or number,
with an optional leading `(`, treating anything except the exact name
`False` as enabling the flag), adds a second pattern for the `"name",
value` form, matches across a line break, and falls back to scanning
`headroom/**/*.py` when git is unavailable instead of skipping. The scan
of the current tree finds no offender with the wider rules.

Part of #4783

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/test_no_trust_remote_code.py`: `_VALUE`, a wider `ASSIGNMENT`,
a new `NAMED_ARGUMENT` limited to
`setattr`/`getattr`/`setdefault`/`get`/`pop` calls, a `_values()` helper
that reports `(line, value)`, a git-less fallback in `_tracked_files()`,
and 17 new cases: calls that do not set the flag (`log.info("...",
True)`) must be ignored, every truthy shape must be flagged, a value on
the next line must be flagged, and an explicit `False` (including
`(False)`, `kwargs["..."] = False` and `setattr(..., False)`) must be
accepted.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format` (0.17.3) are clean.

### Test Output

The old regex on the shapes the old test never saw (`re.findall` with
the old pattern):

```text
old regex matches=True  trust_remote_code=True
old regex matches=False trust_remote_code=1
old regex matches=False trust_remote_code=(True)
old regex matches=False trust_remote_code="yes"
old regex matches=False setattr(cfg, "trust_remote_code", True)
old regex matches=False kwargs.setdefault("trust_remote_code", True)
```

The new tests with the old scan rule (value must start with a letter, no
`"name", value` pattern):

```text
6 failed, 10 passed   (the =1, ="yes", mapping =1, setattr, setdefault and setattr(..., False) cases fail)
```

The new tests with this branch's scan:

```text
tests/test_no_trust_remote_code.py   19 passed   (the tree scan finds no offender)
same file run from a copy of the tree without .git   19 passed, 0 skipped
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, pytest 9.2.0, `main` at
`3c66c91e`.
- Exact command / steps: `python -m pytest
tests/test_no_trust_remote_code.py`; then the same file from a copy of
the repository without `.git`; then the new tests with the scan logic
swapped for the old rule.
- Observed result: as above. The tree scan passes on the current tree
with the wider rules, so no existing code needs to change.
- Not tested: CI, and a deliberately injected `trust_remote_code=1` in a
real module (the shapes are covered by the unit cases instead).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
scan is stricter and no longer skips without git.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- A value such as `"False"` (a string) is truthy in Python, so the scan
flags it; I think that is right for this guard.
- The git-less fallback scans only `headroom/`: an exported tarball has
no other tracked-file list.

AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4785"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:02:31 -06:00

157 lines
5.7 KiB
Python

"""Nothing in this tree may enable HuggingFace remote code execution (A-1).
``trust_remote_code`` makes a model/dataset repository's own Python run inside
the process that loads it. Headroom loads tokenizers from identifiers that trace
back to proxied request bodies, so a single re-introduced ``=True`` anywhere is
remote code execution in the proxy. This is a tree-wide scan rather than a test
of one module: the tokenizer loader was not the only caller, and the next one
will not be either.
"""
from __future__ import annotations
import re
import subprocess
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1]
# Assembled at runtime so this file contributes no literal occurrence of the
# token to its own scan.
FLAG = "trust_remote" + "_code"
# The value: a quoted string, or a name / number (``True``, ``1``, ``-1``, ``cfg.x``).
# Anything but the exact name ``False`` is treated as enabling the flag.
_VALUE = r"\"[^\"]*\"|'[^']*'|[-+]?[\w.]+"
# Matches the kwarg form (``flag=True``), the mapping form (``"flag": True``)
# and the subscript form (``kwargs["flag"] = True``), capturing whatever it is
# set to up to the next delimiter. The optional quote and bracket matter:
# without them the mapping form slips through, and
# ``from_pretrained(**{"flag": True})`` is exactly how this would come back.
# The value is NOT required to start with a letter: the library tests the flag
# for truthiness, so ``=1``, ``=(True)`` and ``="yes"`` enable it just as
# ``=True`` does. A value on the next line is still a value, because ``\s*``
# spans the newline.
ASSIGNMENT = re.compile(re.escape(FLAG) + r"[\"']?\s*\]?\s*[=:]\s*\(*\s*(" + _VALUE + ")")
# ``setattr(cfg, "flag", True)`` and ``kwargs.setdefault("flag", True)``: the
# name is a string literal followed by a comma, not by ``=`` or ``:``.
# Only calls that set or default the flag count: ``log.info("flag", True)`` does not.
NAMED_ARGUMENT = re.compile(
r"(?:\b(?:setattr|getattr)\s*\([^,()]*,|\b(?:setdefault|get|pop)\s*\()\s*[\"']"
+ re.escape(FLAG)
+ r"[\"']\s*,\s*\(*\s*("
+ _VALUE
+ ")"
)
def _values(text: str) -> list[tuple[int, str]]:
"""Every ``(line number, value)`` the flag is set to in ``text``."""
found: list[tuple[int, str]] = []
for pattern in (ASSIGNMENT, NAMED_ARGUMENT):
for match in pattern.finditer(text):
found.append((text.count("\n", 0, match.start()) + 1, match.group(1)))
return found
def _tracked_files() -> list[Path]:
"""Every git-tracked file, so the scan cannot be dodged by adding a directory.
git-tracked rather than a filesystem walk: the repository is commonly checked
out alongside worktrees and virtualenvs that a naive rglob would sweep in.
Without git (an exported tarball) the package itself is still scanned, so the
guard does not silently stop running.
"""
try:
out = subprocess.run(
["git", "ls-files", "-z"],
cwd=ROOT,
capture_output=True,
check=True,
).stdout
except (OSError, subprocess.CalledProcessError):
return sorted((ROOT / "headroom").rglob("*.py"))
return [ROOT / name for name in out.decode().split("\0") if name]
def test_remote_code_is_never_enabled_anywhere_in_the_tree() -> None:
offenders: list[str] = []
for path in _tracked_files():
try:
text = path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue # binary or vanished (submodule, symlink) — nothing to read
if FLAG not in text:
continue
lines = text.splitlines()
for lineno, value in _values(text):
if value != "False":
offenders.append(f"{path.relative_to(ROOT)}:{lineno}: {lines[lineno - 1].strip()}")
assert not offenders, (
f"{FLAG} must be False everywhere — it executes repository-supplied "
"Python in this process:\n" + "\n".join(offenders)
)
@pytest.mark.parametrize(
"line",
[
f"model = load(x, {FLAG}=True)",
f"model = load(x, {FLAG}=1)",
f"model = load(x, {FLAG}=(True))",
f'model = load(x, {FLAG}="yes")',
f"model = load(x, {FLAG}=enabled)",
f'kwargs["{FLAG}"] = True',
f'load(**{{"{FLAG}": 1}})',
f'setattr(cfg, "{FLAG}", True)',
f'kwargs.setdefault("{FLAG}", True)',
],
)
def test_the_scan_flags_every_truthy_way_to_enable_the_flag(line: str) -> None:
values = [value for _, value in _values(line)]
assert values and all(value != "False" for value in values), line
@pytest.mark.parametrize(
"line",
[
f'log.info("{FLAG}", True)',
f'print("{FLAG}", 1)',
f'logger.debug("{FLAG}", enabled)',
],
)
def test_the_scan_ignores_calls_that_do_not_set_the_flag(line: str) -> None:
assert _values(line) == []
def test_the_scan_flags_a_value_on_the_next_line() -> None:
text = f"load(\n x,\n {FLAG}=\n True,\n)\n"
assert [value for _, value in _values(text)] == ["True"]
@pytest.mark.parametrize(
"line",
[
f"model = load(x, {FLAG}=False)",
f"model = load(x, {FLAG}=(False))",
f'kwargs["{FLAG}"] = False',
f'setattr(cfg, "{FLAG}", False)',
],
)
def test_the_scan_accepts_an_explicit_false(line: str) -> None:
assert [value for _, value in _values(line)] == ["False"]
def test_the_tokenizer_loader_sets_the_flag_explicitly() -> None:
"""Explicit beats relying on the library default, which upstream can change."""
source = (ROOT / "headroom" / "tokenizers" / "huggingface.py").read_text(encoding="utf-8")
values = [value for _, value in _values(source)]
assert values, f"tokenizer loader no longer passes {FLAG} explicitly"
assert set(values) == {"False"}