Commit Graph
4 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 b324efb1d4 fix(sql): stop granting anon read access to raw proxy telemetry (#4867)
> Replaces #4864: same change as one commit, because a commit message
there failed commitlint and fixing it would need a force-push.

## Description

`sql/create_proxy_telemetry_v2.sql` created an `anon_select` policy and
`GRANT SELECT ... TO anon` on the raw telemetry table. The anon key is
meant to be public, so anyone holding it could read every raw row. The
Supabase beacon was removed in #1526, so the script is now marked legacy
and keeps only the INSERT path that older releases still use. The SELECT
policy and grant are gone. The script now runs `DROP POLICY IF EXISTS
"anon_select"` and `REVOKE SELECT ... FROM anon`, and drops
`anon_insert` before creating it, so re-running the whole script on a
project created with the old version removes the read access.

`run-all-plugins.sh` (dev launcher) silently exported
`HEADROOM_LICENSE_DEV=1`, turning off licence verification, whenever
`~/.headroom/plugins.env` was missing. It now keeps a `HEADROOM_LICENSE`
already in the environment and otherwise exits with a message.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `sql/create_proxy_telemetry_v2.sql`: legacy header; anon INSERT only;
idempotent policy section that drops the old read policy and revokes
SELECT.
- `run-all-plugins.sh`: no `HEADROOM_LICENSE_DEV` fallback.
- `tests/test_sql_anon_access.py`: static checks (there is no SQL
harness in the repo) that no policy or grant gives anon SELECT/ALL on
the raw table, INSERT is still granted, the DROP POLICY / REVOKE run as
live statements, and the dev script does not export
`HEADROOM_LICENSE_DEV`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_sql_anon_access.py
2 passed

# same tests against origin/main files
2 failed
```

## Real Behavior Proof

- Not run against Postgres: a local Postgres container run was attempted
but Docker was unavailable on the test machine. Not run against the live
Supabase project; if it still exists, someone with access needs to
re-run this script (or the DROP POLICY / REVOKE pair) once. This PR does
not change any deployed database.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no runtime change; setup SQL and a
dev script only.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4867"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:31:41 -06:00
09b7e8c1e4 fix(packaging): rename the sandbox extra to lean and correct SECURITY.md claims (#3727)
**Revised 2026-10-05 after review:** tests now patch the deployment root
directly (never HOME) and assert POSIX modes only where enforced, the
supported-versions test follows the latest-release policy, and runner
scripts are recreated rather than truncated so a key is never written at
an old 0755 mode.

## Description

Security-remediation item A-10 (P-21, P-22). Renames the misleading
`sandbox` extra to `lean` (keeping `sandbox` as a deprecated alias for
one release, so `pip install "headroom-ai[sandbox]"` keeps working; not
a breaking change), corrects SECURITY.md's credential claims, and makes
the `headroom install --env` credential-persistence path owner-only and
documented. Details below.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update

## Changes Made

- `pyproject.toml`: `sandbox` extra renamed to `lean`; `sandbox =
["headroom-ai[lean]"]` deprecated alias; `uv.lock` regenerated (`uv lock
--check` clean).
- CI `python-314-wheels`, `run-all-plugins.sh` and the installation
guide now use `[lean]`.
- SECURITY.md: latest-release support policy (from main); discloses the
Copilot OAuth token and the `install --env` persistence path with its
modes.
- `headroom/install/paths.py`: `chmod_owner_only` verifies and reports
the mode it set.
- `headroom/install/state.py`: profile dir 0700 / manifest 0600 on every
save; warns if the directory cannot be narrowed.
- `headroom/install/supervisors.py`: runner scripts 0700 (`.ps1` 0600),
created fresh (old file removed first, so `O_TRUNC` cannot keep a 0755
mode while secrets are written), fail closed on POSIX if the mode cannot
be set.
- Tests: deployment writes are routed through a monkeypatched deploy
root; POSIX-mode assertions are skipped where modes are not enforced
(Windows).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest -q \
    tests/test_packaging_extras_and_security_docs.py tests/test_install/ tests/test_optional_dependencies.py
209 passed, 7 skipped
$ ruff check / ruff format --check (touched files)   -> clean
$ uv lock --check                                    -> Resolved 251 packages
```

Fail-before (2026-10-05 revision):

```text
test_rewriting_an_old_script_never_writes_secrets_at_its_old_mode   (fix reverted) FAILED assert 493 == 448 (0755 != 0700)
test_security_policy_supports_the_latest_release_without_a_version_number
                                                     (pinned 0.39.x table restored) FAILED
test_unrestrictable_profile_dir_warns_and_still_saves with Path.home() forced to ignore $HOME (as on Windows):
  previous version wrote <fake-home>/.headroom/deploy/default/manifest.json; new version writes nothing outside tmp_path
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.6), CPython 3.14 via `uv run --frozen`,
branch head after merging main.
- Exact command / steps: with `HEADROOM_WORKSPACE_DIR` pointed at a temp
dir containing a pre-existing 0755 `deploy/default/` and 0755
`run-headroom.sh` (simulating a profile from an earlier version), ran
the real `build_manifest(..., extra_env={"ANTHROPIC_API_KEY": ...})` ->
`save_manifest` -> `render_runner_scripts`, then `stat` on every file.
- Observed result: `default/` 0700, `manifest.json` 0600,
`run-headroom.sh` 0700, `ensure-headroom.sh` 0700. `uv.lock` records
`headroom-ai[lean]` under marker `extra == 'sandbox'`, so the alias
resolves.
- Not tested: Windows end-to-end (ACL behaviour; mode assertions are
skipped there by design), a real `pip install "headroom-ai[sandbox]"`
from a published wheel, supervisor registration (launchd/systemd).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a (install-time file modes and packaging
metadata).
- Stable/default behavior changed: yes - deployment profile dirs and
runner scripts are owner-only after the next `headroom install apply`
(see release note below). `[sandbox]` keeps installing the same
dependencies.
- Kill switch / disable path: none; re-running an older version restores
old modes.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert the PR; existing files keep their narrowed
modes, which is harmless.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

History of the change and the earlier review rounds follows.

---

Security-remediation item **A-10** (P-21, P-22). Two documentation
defects, each of which leads a reader to a false premise about the
product.

## 1. The `sandbox` extra promises isolation it does not provide

`headroom-ai[sandbox]` is a **dependency profile** — a torch-free subset
of `[all]`. It provides no isolation, no confinement and no privilege
reduction of any kind. The name invites exactly the wrong assumption,
and "we run it in the sandbox extra" is the kind of sentence that ends
up in someone's control narrative.

Renamed to **`lean`**, which describes what it actually is. `sandbox` is
kept as a deprecated alias forwarding to `[lean]` for one release,
because our own `python-314-wheels` CI job installs it
(`.github/workflows/ci.yml:218`) and so may customers. The alias
forwards rather than duplicating the dependency list, so the two cannot
drift.

## 2. SECURITY.md is stale and overclaims

- **Supported-versions table sat at 0.27.x** while the code ships 0.38.0
— four minor releases behind. A lagging table is worse than none: it
tells a reporter their release is unsupported when it is the current
one.
- **"No credential storage: We never store or log API keys"** is broader
than the code supports. `headroom/copilot_auth.py:646` persists the
GitHub Copilot OAuth refresh token to `~/.headroom/copilot_auth.json`.
It does so correctly — `chmod 0600` at `:648` — but it *is* stored, and
a blanket denial is the sort of claim a customer's security reviewer
checks.

The replacement says what is actually true: provider API keys are read
from the environment and forwarded without being written to disk; the
Copilot OAuth token is the one deliberate exception; and
request/response bodies may reach the runtime log when logging is
enabled, so that log is sensitive.

## Tests (original revision)

`tests/test_packaging_extras_and_security_docs.py` — four tests, all
failing before and passing after:

```
FAILED test_sandbox_alias_still_resolves_to_the_lean_profile
FAILED test_lean_profile_stays_torch_free
FAILED test_security_policy_supported_versions_match_the_shipped_version
FAILED test_security_policy_does_not_claim_blanket_no_credential_storage
4 failed  ->  4 passed
```

Method: commit, `git checkout origin/main -- pyproject.toml SECURITY.md`
(keeping the new tests), run, restore.

The version-table test is the one that matters most — main now documents
"latest release on PyPI" as the supported version, and the test (renamed
`test_security_policy_supports_the_latest_release_without_a_version_number`)
pins that and fails if a hard-coded version row returns, so **the table
cannot silently rot again**. That drift, not the wrong number, is the
actual defect.

`test_lean_profile_stays_torch_free` guards the profile's whole reason
for existing: adding a torch-pulling extra to it now fails the build by
name.

## Not done here

The backlog pairs this with a deprecation *warning* on the old extra
name. pip has no mechanism to warn on an extra at install time, so the
alias plus the comment is the available remedy; removal is noted for
after 0.39.x.

Ruff 0.16.7 `check` + `format --check` clean.

---

## Follow-up: the credential-persistence path (review of this PR)

@JerrettDavis blocked the first version because the SECURITY.md
*correction* was
itself inaccurate — it claimed provider API keys are not written to
disk, which
`headroom install --env KEY=VALUE` falsifies. Addressed in 3da8b955 by
disclosing the path accurately **and** narrowing its permissions.

What the trace found: `--env` values are merged into
`DeploymentManifest.base_env`
and persisted verbatim — into `manifest.json` *and*, as cleartext
`export` lines,
into the generated `run-headroom.sh` / `ensure-headroom.sh`. The
manifest was
already 0600, but only incidentally (via `mkstemp`), with nothing
pinning it. The
runner scripts were **0755** and the profile directory **0755** — a live
API key
in a world-readable file.

Persistence is kept: supervisors start from a bare environment, so
`--env` is the
only channel that reaches a supervised proxy. Redacting secret-valued
entries
would leave existing installs starting a proxy with no credential — a
silent 401
after upgrade. Instead: manifest 0600, runner scripts 0700 (0600 for the
`.ps1`),
profile directory 0700, applied on every save so pre-existing profiles
are
narrowed too; scripts created via `os.open(..., mode)` so no
world-readable window
exists between write and chmod.

SECURITY.md now names the files, the modes, that owner-only is not
encryption,
that rotation means re-running `install apply`, and that the Docker
runtime also
places the value in the container command line (`ps`-visible). The
`--env` help
text says the same.

The policy test now drives the real path — `build_manifest` →
`save_manifest` →
`render_runner_scripts` with a provider key against a fake `$HOME` —
asserts the
key really lands in both files, and **parses the documented octal modes
back out
of SECURITY.md** to compare against `stat()`, so the document and the
code cannot
drift apart again.

```
FAILED test_install_env_persists_a_provider_key_exactly_as_the_policy_describes
FAILED test_security_policy_discloses_the_persistent_install_credential_path
FAILED test_save_manifest_writes_owner_only
FAILED test_save_manifest_narrows_a_profile_directory_created_before_this_change
FAILED test_render_runner_scripts_are_owner_only_because_they_export_secrets
       (run-headroom.sh is 0755, exposing an API key)
FAILED test_render_unix_runner_narrows_a_script_written_by_an_earlier_version
6 failed  ->  315 passed
```

### ⚠️ Release note — behaviour change

Existing deployment profiles are **re-permissioned on the next `headroom
install
apply`**: `~/.headroom/deploy/<profile>/` and its runner scripts drop
group and
other bits (0755 -> 0700). Anyone reading or executing `run-headroom.sh`
as a
different, non-root user must now do so as the owner. No supported
supervisor
configuration does this — user scope runs as the installing user, system
scope as
root — but it changes the mode of files that already exist on customer
machines.
`--env` itself is unchanged: no value is dropped, redacted or relocated.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3727"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 14:56:54 -07:00
746038991c fix(proxy)!: read HEADROOM_LICENSE and make usage reporting opt-in (#3857)
## Why

Core read its licence from `HEADROOM_LICENSE_KEY`, while every licensed
extension reads `HEADROOM_LICENSE`. Core's cloud usage reporter also
started whenever a key was present: it posts the licence to the Headroom
cloud for validation and then sends aggregate usage every 5 minutes.
Renaming the variable on its own would have switched the reporter on for
every deployment that already sets `HEADROOM_LICENSE` for its
extensions. So this PR renames the variable and makes reporting opt-in
in the same change.

## What changes

- New `headroom/license_env.py` holds the rules:
  - `HEADROOM_LICENSE` is the variable.
- `HEADROOM_LICENSE_KEY` is still read as a deprecated alias for one
release and logs `event=license_env_deprecated`.
- If both are set to different values, `HEADROOM_LICENSE` wins and
`event=license_env_conflict` is logged.
- `HEADROOM_USAGE_REPORTING=1` (also `true`, `yes`, `on`) is the only
way to turn reporting on.
- `ProxyConfig.usage_reporting` (default `False`) carries the opt-in.
Multi-worker children get it automatically.
- `HeadroomProxy` starts `UsageReporter` only when a licence is set,
`usage_reporting` is true and offline mode is off.
- The startup banner no longer prints the first 8 characters of the
licence. It shows `OSS (no licence)` or `LICENSED (usage reporting off |
ON | suppressed by HEADROOM_OFFLINE)`.
- Docs: two new rows in the proxy environment table. The comment in
`run-all-plugins.sh` is corrected.

Compression gating is unchanged: with no reporter running, the licence
is treated as OK, which is the current OSS behaviour.

## Behaviour change (breaking)

- A licence on its own no longer starts the cloud usage reporter.
Deployments that relied on it must set `HEADROOM_USAGE_REPORTING=1`.
- `HEADROOM_LICENSE` is the variable. `HEADROOM_LICENSE_KEY` still works
for one release with a deprecation warning; rename it. The alias is
removed in the next release.

## Tests

`tests/test_license_env_and_usage_reporting.py`:
- Variable rules: precedence, the alias warning, the conflict warning,
the same value under both names staying quiet, blank values treated as
unset.
- Opt-in values: which values count as on and off.
- The real app lifespan with every outbound `httpx.AsyncClient.send`
captured: a licence alone sends nothing; with the opt-in, the first
outbound call is `/v1/license/validate`; offline mode beats the opt-in.
The first test fails on `main`, and the other two cannot run there
because the opt-in does not exist.

Manual runs of `headroom proxy` with nothing set, with
`HEADROOM_LICENSE`, with only `HEADROOM_LICENSE_KEY`, and with licence
plus opt-in plus `HEADROOM_OFFLINE=1` all served `/livez`, showed the
expected banner, did not start the reporter and printed no licence text.

Not tested live: opted in and online against the real cloud. That path
is covered by the in-process test with the network captured.

## Follow-ups, not in this repo

Several licensed extensions still fall back to core's
`config.license_key`. It now carries the same `HEADROOM_LICENSE` token,
so the fallback is redundant and can be removed.

Revised 2026-10-01 after self-review: merged main to refresh CI; no code
change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-01 15:57:32 -05:00
Tejas ChopraandClaude Opus 5 f624d3a00a perf(proxy): bound upstream calls and hot-path costs (#2852)
Seven commits from one week of load testing: one hang, two request-path
correctness fixes, and four hot-path costs that only show up in
production.

## Reliability

**Bound every upstream call.** The litellm backend had no timeout at
all, so a
request the upstream never answered blocked its caller forever. Observed
under
load on 2026-08-07: four agent workers on ESTABLISHED connections for
36+
minutes while `/readyz` answered in 0.11s. No error, no retry, no log
line —
indistinguishable from slow work, which is the worst shape a failure can
take.

A float rather than an `httpx.Timeout`, deliberately: litellm expands a
float
across all four httpx phases, so on a streaming call it becomes the
maximum gap
*between chunks*, not a cap on total generation. A long answer streaming
steadily is never cut off; a stalled one dies. Default 600s via
`HEADROOM_UPSTREAM_TIMEOUT`; 0, negative, and junk fall back to the
default
rather than meaning "no timeout".

**Keep the consistency re-count off the event loop.** It ran
`tokenizer.count_messages` twice directly on the loop. Since Claude
counting
moved to a real BPE that is CPU-bound work stalling every other
in-flight
request — ~1s on a 2.3 MB body, with `/healthz` gaps tracking body size.
Offloaded via `asyncio.to_thread` on the same tokenizer instance, so
reported
values are unchanged. (#2810)

**Survive a re-parse MemoryError.** `MemoryError` is not a `ValueError`,
so on
1M-context payloads the byte-faithful forwarder's verification re-parse
escaped
the handler and aborted an otherwise-fine request — 14 aborts across 8
days of
reporter logs. (#2768)

## Performance

All four are measured, not guessed. Each degrades with something a short
benchmark does not vary: uptime, content shape, or process age.

| fix | before | after |
|---|---|---|
| Cost-record walk per request (at 100k records) | 13.6 ms | bounded by
model count |
| JSON-block scan, JS-style object logs (1200 lines) | 4643 ms | 183 ms
|
| JSON-block scan, truncated JSONL | 3737 ms | 116 ms |
| Lazy imports inside user requests | multi-second | paid at startup |
| `count_text` (80% of local CPU) | — | memoised |

Two worth calling out:

- **The cost walk degrades with proxy *uptime*, not load.** A freshly
started
proxy pays ~0.01 ms; a month-old one pays 4–13 ms on every request, on
the
event loop, holding the metrics lock. Deliberately not a TTL cache over
`stats()`: those values feed `check_budget()` when `--budget` is set,
and a
stale reading under-enforces the budget. The fix is to stop computing
what
  the caller discards.
- **The JSON-block memo is built only *after* a scan fails to balance.**
That
ordering is load-bearing, not an optimisation — caching from the start
made
pretty-printed JSON ~2x slower, since content that balances on the first
scan
  has nothing to reuse and just pays the per-line dict traffic. Still a
  constant-factor fix, not an asymptotic one.

## Tests

+1202 lines, 20 files. Each fix is pinned by a test that fails on the
unmodified code: the re-count test asserts no `count_messages` pass runs
with a
live event loop in its thread; the re-parse test drives a `MemoryError`
through
the real request path and expects a 200; `totals()` equality with
`stats()` is
asserted across model counts, request volumes, and both pricing
branches. The
timeout test is structural rather than a mock — the failure mode is a
dispatch
path someone adds later without a guard, which mocking the existing four
cannot
catch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09 16:24:33 -07:00