mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b324efb1d4 |
fix(sql): stop granting anon read access to raw proxy telemetry (#4867)
> Replaces #4864: same change as one commit, because a commit message there failed commitlint and fixing it would need a force-push. ## Description `sql/create_proxy_telemetry_v2.sql` created an `anon_select` policy and `GRANT SELECT ... TO anon` on the raw telemetry table. The anon key is meant to be public, so anyone holding it could read every raw row. The Supabase beacon was removed in #1526, so the script is now marked legacy and keeps only the INSERT path that older releases still use. The SELECT policy and grant are gone. The script now runs `DROP POLICY IF EXISTS "anon_select"` and `REVOKE SELECT ... FROM anon`, and drops `anon_insert` before creating it, so re-running the whole script on a project created with the old version removes the read access. `run-all-plugins.sh` (dev launcher) silently exported `HEADROOM_LICENSE_DEV=1`, turning off licence verification, whenever `~/.headroom/plugins.env` was missing. It now keeps a `HEADROOM_LICENSE` already in the environment and otherwise exits with a message. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `sql/create_proxy_telemetry_v2.sql`: legacy header; anon INSERT only; idempotent policy section that drops the old read policy and revokes SELECT. - `run-all-plugins.sh`: no `HEADROOM_LICENSE_DEV` fallback. - `tests/test_sql_anon_access.py`: static checks (there is no SQL harness in the repo) that no policy or grant gives anon SELECT/ALL on the raw table, INSERT is still granted, the DROP POLICY / REVOKE run as live statements, and the dev script does not export `HEADROOM_LICENSE_DEV`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest -q tests/test_sql_anon_access.py 2 passed # same tests against origin/main files 2 failed ``` ## Real Behavior Proof - Not run against Postgres: a local Postgres container run was attempted but Docker was unavailable on the test machine. Not run against the live Supabase project; if it still exists, someone with access needs to re-run this script (or the DROP POLICY / REVOKE pair) once. This PR does not change any deployed database. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no runtime change; setup SQL and a dev script only. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4867" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
09b7e8c1e4 |
fix(packaging): rename the sandbox extra to lean and correct SECURITY.md claims (#3727)
**Revised 2026-10-05 after review:** tests now patch the deployment root
directly (never HOME) and assert POSIX modes only where enforced, the
supported-versions test follows the latest-release policy, and runner
scripts are recreated rather than truncated so a key is never written at
an old 0755 mode.
## Description
Security-remediation item A-10 (P-21, P-22). Renames the misleading
`sandbox` extra to `lean` (keeping `sandbox` as a deprecated alias for
one release, so `pip install "headroom-ai[sandbox]"` keeps working; not
a breaking change), corrects SECURITY.md's credential claims, and makes
the `headroom install --env` credential-persistence path owner-only and
documented. Details below.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
## Changes Made
- `pyproject.toml`: `sandbox` extra renamed to `lean`; `sandbox =
["headroom-ai[lean]"]` deprecated alias; `uv.lock` regenerated (`uv lock
--check` clean).
- CI `python-314-wheels`, `run-all-plugins.sh` and the installation
guide now use `[lean]`.
- SECURITY.md: latest-release support policy (from main); discloses the
Copilot OAuth token and the `install --env` persistence path with its
modes.
- `headroom/install/paths.py`: `chmod_owner_only` verifies and reports
the mode it set.
- `headroom/install/state.py`: profile dir 0700 / manifest 0600 on every
save; warns if the directory cannot be narrowed.
- `headroom/install/supervisors.py`: runner scripts 0700 (`.ps1` 0600),
created fresh (old file removed first, so `O_TRUNC` cannot keep a 0755
mode while secrets are written), fail closed on POSIX if the mode cannot
be set.
- Tests: deployment writes are routed through a monkeypatched deploy
root; POSIX-mode assertions are skipped where modes are not enforced
(Windows).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ LITELLM_LOCAL_MODEL_COST_MAP=True uv run --frozen --extra dev --extra proxy pytest -q \
tests/test_packaging_extras_and_security_docs.py tests/test_install/ tests/test_optional_dependencies.py
209 passed, 7 skipped
$ ruff check / ruff format --check (touched files) -> clean
$ uv lock --check -> Resolved 251 packages
```
Fail-before (2026-10-05 revision):
```text
test_rewriting_an_old_script_never_writes_secrets_at_its_old_mode (fix reverted) FAILED assert 493 == 448 (0755 != 0700)
test_security_policy_supports_the_latest_release_without_a_version_number
(pinned 0.39.x table restored) FAILED
test_unrestrictable_profile_dir_warns_and_still_saves with Path.home() forced to ignore $HOME (as on Windows):
previous version wrote <fake-home>/.headroom/deploy/default/manifest.json; new version writes nothing outside tmp_path
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.6), CPython 3.14 via `uv run --frozen`,
branch head after merging main.
- Exact command / steps: with `HEADROOM_WORKSPACE_DIR` pointed at a temp
dir containing a pre-existing 0755 `deploy/default/` and 0755
`run-headroom.sh` (simulating a profile from an earlier version), ran
the real `build_manifest(..., extra_env={"ANTHROPIC_API_KEY": ...})` ->
`save_manifest` -> `render_runner_scripts`, then `stat` on every file.
- Observed result: `default/` 0700, `manifest.json` 0600,
`run-headroom.sh` 0700, `ensure-headroom.sh` 0700. `uv.lock` records
`headroom-ai[lean]` under marker `extra == 'sandbox'`, so the alias
resolves.
- Not tested: Windows end-to-end (ACL behaviour; mode assertions are
skipped there by design), a real `pip install "headroom-ai[sandbox]"`
from a published wheel, supervisor registration (launchd/systemd).
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a (install-time file modes and packaging
metadata).
- Stable/default behavior changed: yes - deployment profile dirs and
runner scripts are owner-only after the next `headroom install apply`
(see release note below). `[sandbox]` keeps installing the same
dependencies.
- Kill switch / disable path: none; re-running an older version restores
old modes.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert the PR; existing files keep their narrowed
modes, which is harmless.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
History of the change and the earlier review rounds follows.
---
Security-remediation item **A-10** (P-21, P-22). Two documentation
defects, each of which leads a reader to a false premise about the
product.
## 1. The `sandbox` extra promises isolation it does not provide
`headroom-ai[sandbox]` is a **dependency profile** — a torch-free subset
of `[all]`. It provides no isolation, no confinement and no privilege
reduction of any kind. The name invites exactly the wrong assumption,
and "we run it in the sandbox extra" is the kind of sentence that ends
up in someone's control narrative.
Renamed to **`lean`**, which describes what it actually is. `sandbox` is
kept as a deprecated alias forwarding to `[lean]` for one release,
because our own `python-314-wheels` CI job installs it
(`.github/workflows/ci.yml:218`) and so may customers. The alias
forwards rather than duplicating the dependency list, so the two cannot
drift.
## 2. SECURITY.md is stale and overclaims
- **Supported-versions table sat at 0.27.x** while the code ships 0.38.0
— four minor releases behind. A lagging table is worse than none: it
tells a reporter their release is unsupported when it is the current
one.
- **"No credential storage: We never store or log API keys"** is broader
than the code supports. `headroom/copilot_auth.py:646` persists the
GitHub Copilot OAuth refresh token to `~/.headroom/copilot_auth.json`.
It does so correctly — `chmod 0600` at `:648` — but it *is* stored, and
a blanket denial is the sort of claim a customer's security reviewer
checks.
The replacement says what is actually true: provider API keys are read
from the environment and forwarded without being written to disk; the
Copilot OAuth token is the one deliberate exception; and
request/response bodies may reach the runtime log when logging is
enabled, so that log is sensitive.
## Tests (original revision)
`tests/test_packaging_extras_and_security_docs.py` — four tests, all
failing before and passing after:
```
FAILED test_sandbox_alias_still_resolves_to_the_lean_profile
FAILED test_lean_profile_stays_torch_free
FAILED test_security_policy_supported_versions_match_the_shipped_version
FAILED test_security_policy_does_not_claim_blanket_no_credential_storage
4 failed -> 4 passed
```
Method: commit, `git checkout origin/main -- pyproject.toml SECURITY.md`
(keeping the new tests), run, restore.
The version-table test is the one that matters most — main now documents
"latest release on PyPI" as the supported version, and the test (renamed
`test_security_policy_supports_the_latest_release_without_a_version_number`)
pins that and fails if a hard-coded version row returns, so **the table
cannot silently rot again**. That drift, not the wrong number, is the
actual defect.
`test_lean_profile_stays_torch_free` guards the profile's whole reason
for existing: adding a torch-pulling extra to it now fails the build by
name.
## Not done here
The backlog pairs this with a deprecation *warning* on the old extra
name. pip has no mechanism to warn on an extra at install time, so the
alias plus the comment is the available remedy; removal is noted for
after 0.39.x.
Ruff 0.16.7 `check` + `format --check` clean.
---
## Follow-up: the credential-persistence path (review of this PR)
@JerrettDavis blocked the first version because the SECURITY.md
*correction* was
itself inaccurate — it claimed provider API keys are not written to
disk, which
`headroom install --env KEY=VALUE` falsifies. Addressed in
|
||
|
|
746038991c |
fix(proxy)!: read HEADROOM_LICENSE and make usage reporting opt-in (#3857)
## Why Core read its licence from `HEADROOM_LICENSE_KEY`, while every licensed extension reads `HEADROOM_LICENSE`. Core's cloud usage reporter also started whenever a key was present: it posts the licence to the Headroom cloud for validation and then sends aggregate usage every 5 minutes. Renaming the variable on its own would have switched the reporter on for every deployment that already sets `HEADROOM_LICENSE` for its extensions. So this PR renames the variable and makes reporting opt-in in the same change. ## What changes - New `headroom/license_env.py` holds the rules: - `HEADROOM_LICENSE` is the variable. - `HEADROOM_LICENSE_KEY` is still read as a deprecated alias for one release and logs `event=license_env_deprecated`. - If both are set to different values, `HEADROOM_LICENSE` wins and `event=license_env_conflict` is logged. - `HEADROOM_USAGE_REPORTING=1` (also `true`, `yes`, `on`) is the only way to turn reporting on. - `ProxyConfig.usage_reporting` (default `False`) carries the opt-in. Multi-worker children get it automatically. - `HeadroomProxy` starts `UsageReporter` only when a licence is set, `usage_reporting` is true and offline mode is off. - The startup banner no longer prints the first 8 characters of the licence. It shows `OSS (no licence)` or `LICENSED (usage reporting off | ON | suppressed by HEADROOM_OFFLINE)`. - Docs: two new rows in the proxy environment table. The comment in `run-all-plugins.sh` is corrected. Compression gating is unchanged: with no reporter running, the licence is treated as OK, which is the current OSS behaviour. ## Behaviour change (breaking) - A licence on its own no longer starts the cloud usage reporter. Deployments that relied on it must set `HEADROOM_USAGE_REPORTING=1`. - `HEADROOM_LICENSE` is the variable. `HEADROOM_LICENSE_KEY` still works for one release with a deprecation warning; rename it. The alias is removed in the next release. ## Tests `tests/test_license_env_and_usage_reporting.py`: - Variable rules: precedence, the alias warning, the conflict warning, the same value under both names staying quiet, blank values treated as unset. - Opt-in values: which values count as on and off. - The real app lifespan with every outbound `httpx.AsyncClient.send` captured: a licence alone sends nothing; with the opt-in, the first outbound call is `/v1/license/validate`; offline mode beats the opt-in. The first test fails on `main`, and the other two cannot run there because the opt-in does not exist. Manual runs of `headroom proxy` with nothing set, with `HEADROOM_LICENSE`, with only `HEADROOM_LICENSE_KEY`, and with licence plus opt-in plus `HEADROOM_OFFLINE=1` all served `/livez`, showed the expected banner, did not start the reporter and printed no licence text. Not tested live: opted in and online against the real cloud. That path is covered by the in-process test with the network captured. ## Follow-ups, not in this repo Several licensed extensions still fall back to core's `config.license_key`. It now carries the same `HEADROOM_LICENSE` token, so the fallback is redundant and can be removed. Revised 2026-10-01 after self-review: merged main to refresh CI; no code change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
f624d3a00a |
perf(proxy): bound upstream calls and hot-path costs (#2852)
Seven commits from one week of load testing: one hang, two request-path correctness fixes, and four hot-path costs that only show up in production. ## Reliability **Bound every upstream call.** The litellm backend had no timeout at all, so a request the upstream never answered blocked its caller forever. Observed under load on 2026-08-07: four agent workers on ESTABLISHED connections for 36+ minutes while `/readyz` answered in 0.11s. No error, no retry, no log line — indistinguishable from slow work, which is the worst shape a failure can take. A float rather than an `httpx.Timeout`, deliberately: litellm expands a float across all four httpx phases, so on a streaming call it becomes the maximum gap *between chunks*, not a cap on total generation. A long answer streaming steadily is never cut off; a stalled one dies. Default 600s via `HEADROOM_UPSTREAM_TIMEOUT`; 0, negative, and junk fall back to the default rather than meaning "no timeout". **Keep the consistency re-count off the event loop.** It ran `tokenizer.count_messages` twice directly on the loop. Since Claude counting moved to a real BPE that is CPU-bound work stalling every other in-flight request — ~1s on a 2.3 MB body, with `/healthz` gaps tracking body size. Offloaded via `asyncio.to_thread` on the same tokenizer instance, so reported values are unchanged. (#2810) **Survive a re-parse MemoryError.** `MemoryError` is not a `ValueError`, so on 1M-context payloads the byte-faithful forwarder's verification re-parse escaped the handler and aborted an otherwise-fine request — 14 aborts across 8 days of reporter logs. (#2768) ## Performance All four are measured, not guessed. Each degrades with something a short benchmark does not vary: uptime, content shape, or process age. | fix | before | after | |---|---|---| | Cost-record walk per request (at 100k records) | 13.6 ms | bounded by model count | | JSON-block scan, JS-style object logs (1200 lines) | 4643 ms | 183 ms | | JSON-block scan, truncated JSONL | 3737 ms | 116 ms | | Lazy imports inside user requests | multi-second | paid at startup | | `count_text` (80% of local CPU) | — | memoised | Two worth calling out: - **The cost walk degrades with proxy *uptime*, not load.** A freshly started proxy pays ~0.01 ms; a month-old one pays 4–13 ms on every request, on the event loop, holding the metrics lock. Deliberately not a TTL cache over `stats()`: those values feed `check_budget()` when `--budget` is set, and a stale reading under-enforces the budget. The fix is to stop computing what the caller discards. - **The JSON-block memo is built only *after* a scan fails to balance.** That ordering is load-bearing, not an optimisation — caching from the start made pretty-printed JSON ~2x slower, since content that balances on the first scan has nothing to reuse and just pays the per-line dict traffic. Still a constant-factor fix, not an asymptotic one. ## Tests +1202 lines, 20 files. Each fix is pinned by a test that fails on the unmodified code: the re-count test asserts no `count_messages` pass runs with a live event loop in its thread; the re-parse test drives a `MemoryError` through the real request path and expects a 200; `totals()` equality with `stats()` is asserted across model counts, request volumes, and both pricing branches. The timeout test is structural rather than a mock — the failure mode is a dispatch path someone adds later without a guard, which mocking the existing four cannot catch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |