Files
headroom/tests/test_packaging_extras_and_security_docs.py
09b7e8c1e4 fix(packaging): rename the sandbox extra to lean and correct SECURITY.md claims (#3727)
**Revised 2026-10-05 after review:** tests now patch the deployment root
directly (never HOME) and assert POSIX modes only where enforced, the
supported-versions test follows the latest-release policy, and runner
scripts are recreated rather than truncated so a key is never written at
an old 0755 mode.

## Description

Security-remediation item A-10 (P-21, P-22). Renames the misleading
`sandbox` extra to `lean` (keeping `sandbox` as a deprecated alias for
one release, so `pip install "headroom-ai[sandbox]"` keeps working; not
a breaking change), corrects SECURITY.md's credential claims, and makes
the `headroom install --env` credential-persistence path owner-only and
documented. Details below.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- `pyproject.toml`: `sandbox` extra renamed to `lean`; `sandbox =
["headroom-ai[lean]"]` deprecated alias; `uv.lock` regenerated (`uv lock
--check` clean).
- CI `python-314-wheels`, `run-all-plugins.sh` and the installation
guide now use `[lean]`.
- SECURITY.md: latest-release support policy (from main); discloses the
Copilot OAuth token and the `install --env` persistence path with its
modes.
- `headroom/install/paths.py`: `chmod_owner_only` verifies and reports
the mode it set.
- `headroom/install/state.py`: profile dir 0700 / manifest 0600 on every
save; warns if the directory cannot be narrowed.
- `headroom/install/supervisors.py`: runner scripts 0700 (`.ps1` 0600),
created fresh (old file removed first, so `O_TRUNC` cannot keep a 0755
mode while secrets are written), fail closed on POSIX if the mode cannot
be set.
- Tests: deployment writes are routed through a monkeypatched deploy
root; POSIX-mode assertions are skipped where modes are not enforced
(Windows).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest -q \
    tests/test_packaging_extras_and_security_docs.py tests/test_install/ tests/test_optional_dependencies.py
209 passed, 7 skipped
$ ruff check / ruff format --check (touched files)   -> clean
$ uv lock --check                                    -> Resolved 251 packages
```

Fail-before (2026-10-05 revision):

```text
test_rewriting_an_old_script_never_writes_secrets_at_its_old_mode   (fix reverted) FAILED assert 493 == 448 (0755 != 0700)
test_security_policy_supports_the_latest_release_without_a_version_number
                                                     (pinned 0.39.x table restored) FAILED
test_unrestrictable_profile_dir_warns_and_still_saves with Path.home() forced to ignore $HOME (as on Windows):
  previous version wrote <fake-home>/.headroom/deploy/default/manifest.json; new version writes nothing outside tmp_path
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.6), CPython 3.14 via `uv run --frozen`,
branch head after merging main.
- Exact command / steps: with `HEADROOM_WORKSPACE_DIR` pointed at a temp
dir containing a pre-existing 0755 `deploy/default/` and 0755
`run-headroom.sh` (simulating a profile from an earlier version), ran
the real `build_manifest(..., extra_env={"ANTHROPIC_API_KEY": ...})` ->
`save_manifest` -> `render_runner_scripts`, then `stat` on every file.
- Observed result: `default/` 0700, `manifest.json` 0600,
`run-headroom.sh` 0700, `ensure-headroom.sh` 0700. `uv.lock` records
`headroom-ai[lean]` under marker `extra == 'sandbox'`, so the alias
resolves.
- Not tested: Windows end-to-end (ACL behaviour; mode assertions are
skipped there by design), a real `pip install "headroom-ai[sandbox]"`
from a published wheel, supervisor registration (launchd/systemd).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a (install-time file modes and packaging
metadata).
- Stable/default behavior changed: yes - deployment profile dirs and
runner scripts are owner-only after the next `headroom install apply`
(see release note below). `[sandbox]` keeps installing the same
dependencies.
- Kill switch / disable path: none; re-running an older version restores
old modes.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert the PR; existing files keep their narrowed
modes, which is harmless.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

History of the change and the earlier review rounds follows.

---

Security-remediation item **A-10** (P-21, P-22). Two documentation
defects, each of which leads a reader to a false premise about the
product.

## 1. The `sandbox` extra promises isolation it does not provide

`headroom-ai[sandbox]` is a **dependency profile** — a torch-free subset
of `[all]`. It provides no isolation, no confinement and no privilege
reduction of any kind. The name invites exactly the wrong assumption,
and "we run it in the sandbox extra" is the kind of sentence that ends
up in someone's control narrative.

Renamed to **`lean`**, which describes what it actually is. `sandbox` is
kept as a deprecated alias forwarding to `[lean]` for one release,
because our own `python-314-wheels` CI job installs it
(`.github/workflows/ci.yml:218`) and so may customers. The alias
forwards rather than duplicating the dependency list, so the two cannot
drift.

## 2. SECURITY.md is stale and overclaims

- **Supported-versions table sat at 0.27.x** while the code ships 0.38.0
— four minor releases behind. A lagging table is worse than none: it
tells a reporter their release is unsupported when it is the current
one.
- **"No credential storage: We never store or log API keys"** is broader
than the code supports. `headroom/copilot_auth.py:646` persists the
GitHub Copilot OAuth refresh token to `~/.headroom/copilot_auth.json`.
It does so correctly — `chmod 0600` at `:648` — but it *is* stored, and
a blanket denial is the sort of claim a customer's security reviewer
checks.

The replacement says what is actually true: provider API keys are read
from the environment and forwarded without being written to disk; the
Copilot OAuth token is the one deliberate exception; and
request/response bodies may reach the runtime log when logging is
enabled, so that log is sensitive.

## Tests (original revision)

`tests/test_packaging_extras_and_security_docs.py` — four tests, all
failing before and passing after:

```
FAILED test_sandbox_alias_still_resolves_to_the_lean_profile
FAILED test_lean_profile_stays_torch_free
FAILED test_security_policy_supported_versions_match_the_shipped_version
FAILED test_security_policy_does_not_claim_blanket_no_credential_storage
4 failed  ->  4 passed
```

Method: commit, `git checkout origin/main -- pyproject.toml SECURITY.md`
(keeping the new tests), run, restore.

The version-table test is the one that matters most — main now documents
"latest release on PyPI" as the supported version, and the test (renamed
`test_security_policy_supports_the_latest_release_without_a_version_number`)
pins that and fails if a hard-coded version row returns, so **the table
cannot silently rot again**. That drift, not the wrong number, is the
actual defect.

`test_lean_profile_stays_torch_free` guards the profile's whole reason
for existing: adding a torch-pulling extra to it now fails the build by
name.

## Not done here

The backlog pairs this with a deprecation *warning* on the old extra
name. pip has no mechanism to warn on an extra at install time, so the
alias plus the comment is the available remedy; removal is noted for
after 0.39.x.

Ruff 0.16.7 `check` + `format --check` clean.

---

## Follow-up: the credential-persistence path (review of this PR)

@JerrettDavis blocked the first version because the SECURITY.md
*correction* was
itself inaccurate — it claimed provider API keys are not written to
disk, which
`headroom install --env KEY=VALUE` falsifies. Addressed in 3da8b955 by
disclosing the path accurately **and** narrowing its permissions.

What the trace found: `--env` values are merged into
`DeploymentManifest.base_env`
and persisted verbatim — into `manifest.json` *and*, as cleartext
`export` lines,
into the generated `run-headroom.sh` / `ensure-headroom.sh`. The
manifest was
already 0600, but only incidentally (via `mkstemp`), with nothing
pinning it. The
runner scripts were **0755** and the profile directory **0755** — a live
API key
in a world-readable file.

Persistence is kept: supervisors start from a bare environment, so
`--env` is the
only channel that reaches a supervised proxy. Redacting secret-valued
entries
would leave existing installs starting a proxy with no credential — a
silent 401
after upgrade. Instead: manifest 0600, runner scripts 0700 (0600 for the
`.ps1`),
profile directory 0700, applied on every save so pre-existing profiles
are
narrowed too; scripts created via `os.open(..., mode)` so no
world-readable window
exists between write and chmod.

SECURITY.md now names the files, the modes, that owner-only is not
encryption,
that rotation means re-running `install apply`, and that the Docker
runtime also
places the value in the container command line (`ps`-visible). The
`--env` help
text says the same.

The policy test now drives the real path — `build_manifest` →
`save_manifest` →
`render_runner_scripts` with a provider key against a fake `$HOME` —
asserts the
key really lands in both files, and **parses the documented octal modes
back out
of SECURITY.md** to compare against `stat()`, so the document and the
code cannot
drift apart again.

```
FAILED test_install_env_persists_a_provider_key_exactly_as_the_policy_describes
FAILED test_security_policy_discloses_the_persistent_install_credential_path
FAILED test_save_manifest_writes_owner_only
FAILED test_save_manifest_narrows_a_profile_directory_created_before_this_change
FAILED test_render_runner_scripts_are_owner_only_because_they_export_secrets
       (run-headroom.sh is 0755, exposing an API key)
FAILED test_render_unix_runner_narrows_a_script_written_by_an_earlier_version
6 failed  ->  315 passed
```

### ⚠️ Release note — behaviour change

Existing deployment profiles are **re-permissioned on the next `headroom
install
apply`**: `~/.headroom/deploy/<profile>/` and its runner scripts drop
group and
other bits (0755 -> 0700). Anyone reading or executing `run-headroom.sh`
as a
different, non-root user must now do so as the owner. No supported
supervisor
configuration does this — user scope runs as the installing user, system
scope as
root — but it changes the mode of files that already exist on customer
machines.
`--env` itself is unchanged: no value is dropped, redacted or relocated.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3727"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 14:56:54 -07:00

232 lines
9.8 KiB
Python

"""A-10: the `sandbox` extra promised isolation it never provided, and
SECURITY.md drifted four minor releases behind the code.
Both are documentation defects with a security consequence: a reader who
believes either one makes a deployment decision on a false premise. These tests
pin the rename's compatibility alias and make the stale version table — the part
that silently rots — a build failure instead of a reviewer's job.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
import pytest
try:
import tomllib
except ModuleNotFoundError: # Python 3.10
import tomli as tomllib # type: ignore[no-redef]
_ROOT = Path(__file__).resolve().parents[1]
def _pyproject() -> dict:
with (_ROOT / "pyproject.toml").open("rb") as fh:
return tomllib.load(fh)
def test_sandbox_alias_still_resolves_to_the_lean_profile() -> None:
"""The old name must keep working for one release.
`sandbox` was renamed to `lean`; anything already installing the old extra
(our own python-314-wheels CI job did) must not break on upgrade.
"""
extras = _pyproject()["project"]["optional-dependencies"]
assert "lean" in extras, "the renamed extra is missing"
assert "sandbox" in extras, "the deprecated alias was removed too early"
assert extras["sandbox"] == ["headroom-ai[lean]"], (
"the alias must forward to `lean` rather than duplicate its dependency "
"list, or the two will drift"
)
def test_lean_profile_stays_torch_free() -> None:
"""The profile's whole purpose is avoiding torch; guard it by name.
A torch-pulling extra added here would quietly defeat the reason anyone
selects this profile.
"""
extras = _pyproject()["project"]["optional-dependencies"]
(spec,) = extras["lean"]
selected = set(re.search(r"\[(.*)\]", spec).group(1).split(","))
torch_pulling = {"ml", "memory", "evals", "image", "voice", "pytorch-mps"}
assert not (selected & torch_pulling), (
f"`lean` must stay torch-free; it now selects {sorted(selected & torch_pulling)}"
)
def test_security_policy_supports_the_latest_release_without_a_version_number() -> None:
"""SECURITY.md's table sat at 0.27.x while the code shipped 0.38.0.
A supported-versions table that lags is worse than none: it tells a reporter
their release is unsupported when it is the current one. The policy now
supports "the latest release on PyPI", which cannot go stale; pin that, and
fail if a hard-coded version row (the thing that rotted) comes back.
"""
text = (_ROOT / "SECURITY.md").read_text(encoding="utf-8")
table = text.split("## Supported Versions", 1)[1].split("\n## ", 1)[0]
assert re.search(r"^\| Latest release on PyPI \| :white_check_mark: +\|$", table, re.M), (
"SECURITY.md must list the latest PyPI release as the supported version"
)
pinned = re.findall(r"^\|[^|]*\d+\.\d+(?:\.x)?[^|]*\|", table, re.M)
assert not pinned, f"SECURITY.md pins a version number that will go stale: {pinned}"
def test_security_policy_does_not_claim_blanket_no_credential_storage() -> None:
"""`headroom copilot login` writes an OAuth refresh token to disk.
The policy used to say "We never store or log API keys", which is broader
than the code supports. Keep the honest wording.
"""
text = (_ROOT / "SECURITY.md").read_text(encoding="utf-8")
assert "never store or log API keys" not in text, (
"the blanket no-credential-storage claim is contradicted by "
"headroom/copilot_auth.py, which persists the Copilot OAuth refresh token"
)
assert "copilot_auth.json" in text, "the one stored credential must stay documented"
# --- The persistent-install credential path -------------------------------
#
# The assertions above are negative string checks: they stop the *old* wording
# coming back, but they would happily pass while the replacement wording is
# itself false (review of PR #3727 — the first correction claimed Headroom
# "does not write [API keys] to disk", which `headroom install --env` falsifies).
#
# So the tests below drive the real install path with a provider key, read what
# actually lands in `~/.headroom/deploy/<profile>/`, and compare it against the
# claims parsed out of SECURITY.md. A change to either side that is not matched
# on the other fails here.
_SECRET = "sk-ant-api03-not-a-real-key"
_MANIFEST_MODE_CLAIM = re.compile(
r"`~/\.headroom/deploy/<profile>/manifest\.json` \(mode `(0[0-7]{3})`\)"
)
_SCRIPT_MODE_CLAIM = re.compile(
r"`run-headroom\.sh` / `ensure-headroom\.sh` \(mode `(0[0-7]{3})`\)"
)
_DIR_MODE_CLAIM = re.compile(r"in the same directory \(mode `(0[0-7]{3})`\)")
def _security_policy_prose() -> str:
"""SECURITY.md with its line wrapping flattened, so claims match as written."""
return re.sub(r"\s+", " ", (_ROOT / "SECURITY.md").read_text(encoding="utf-8"))
def _claimed_mode(pattern: re.Pattern[str]) -> int:
match = pattern.search(_security_policy_prose())
assert match is not None, (
f"SECURITY.md no longer states the mode matched by {pattern.pattern!r}. The "
"persistent-install credential path must keep documenting where a key "
"passed to `headroom install --env` is written and how it is protected."
)
return int(match.group(1), 8)
def _install_with_a_provider_key(deploy: Path) -> tuple[Path, Path, Path]:
"""Run the real planner/state/supervisor path with a key in ``--env``.
Returns the profile directory, its manifest and its foreground runner
script. Imports are local so the module-level packaging assertions above
still run if the install package fails to import.
"""
from headroom.install.paths import manifest_path, profile_root, unix_run_script_path
from headroom.install.planner import build_manifest
from headroom.install.state import save_manifest
from headroom.install.supervisors import render_runner_scripts
assert profile_root("default").parent == deploy, "the deploy-root monkeypatch did not take"
manifest = build_manifest(
profile="default",
preset="persistent-service",
runtime_kind="python",
scope="user",
provider_mode="manual",
targets=[],
port=8787,
backend="anthropic",
anyllm_provider=None,
region=None,
proxy_mode="passthrough",
memory_enabled=False,
telemetry_enabled=False,
image="ghcr.io/headroomlabs-ai/headroom:latest",
extra_env={"ANTHROPIC_API_KEY": _SECRET},
)
save_manifest(manifest)
render_runner_scripts(manifest)
return profile_root("default"), manifest_path("default"), unix_run_script_path("default")
@pytest.mark.skipif(
sys.platform.startswith("win"),
reason="POSIX mode bits are advisory on Windows, where access is governed by ACLs",
)
def test_install_env_persists_a_provider_key_exactly_as_the_policy_describes(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""`--env ANTHROPIC_API_KEY=...` really does write the key to disk.
This is deliberate — supervisors start from a bare environment, so `--env`
is the only channel that reaches a supervised proxy — but it means the
policy may not claim keys are never written. Pin both halves: that the key
lands where SECURITY.md says, and that it lands owner-only.
"""
# Patch the deployment root itself, not HOME: Windows resolves the home
# directory without consulting $HOME, so a HOME-only patch writes this
# fixture key into the real user's profile.
deploy = tmp_path / "deploy"
monkeypatch.setattr("headroom.install.paths._paths.deploy_root", lambda: deploy)
profile_dir, manifest_file, run_script = _install_with_a_provider_key(deploy)
# 1. What lands on disk. Both files carry the key in cleartext; the runner
# script `export`s it because launchd/systemd hand the process nothing.
assert manifest_file.exists(), "a persistent install must write a manifest"
assert _SECRET in manifest_file.read_text(encoding="utf-8"), (
"the key passed to --env is stored verbatim in manifest.json; if that "
"stopped being true, SECURITY.md and the --env help must change with it"
)
assert run_script.exists(), "a persistent install must write a runner script"
assert f"export ANTHROPIC_API_KEY={_SECRET}" in run_script.read_text(encoding="utf-8")
# 2. With what protection — read back out of the policy document, so the
# two cannot drift.
assert manifest_file.stat().st_mode & 0o777 == _claimed_mode(_MANIFEST_MODE_CLAIM)
assert run_script.stat().st_mode & 0o777 == _claimed_mode(_SCRIPT_MODE_CLAIM)
assert profile_dir.stat().st_mode & 0o777 == _claimed_mode(_DIR_MODE_CLAIM)
# 3. And, concretely: nothing in that directory is readable by group or
# other, whatever the umask was.
for path in [profile_dir, *profile_dir.iterdir()]:
assert not path.stat().st_mode & 0o077, f"{path.name} is readable beyond its owner"
def test_security_policy_discloses_the_persistent_install_credential_path() -> None:
"""The policy must name the `--env` path, not just the Copilot token."""
prose = _security_policy_prose()
assert "does not write them to disk" not in prose, (
"`headroom install --env ANTHROPIC_API_KEY=...` writes the key to "
"~/.headroom/deploy/<profile>/, so a blanket 'not written to disk' "
"claim is false"
)
assert "headroom install --env KEY=VALUE" in prose, (
"the supported way to hand a supervised proxy a provider key must be "
"documented as a credential-persistence path"
)
assert "~/.headroom/deploy/<profile>/manifest.json" in prose
assert "owner-only, not encrypted" in prose, (
"owner-only file modes are not encryption; the policy must say so "
"rather than let a reader infer the key is protected at rest"
)