mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description
`/stats` and `/stats-lifetime` withhold the per-request tail
(`recent_requests`, `request_logs`) and project names from every
non-loopback caller, unless the request's `Host` is an IP literal
**and** the peer is inside
`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS`.
That rule cannot be met by a common deployment: the proxy runs on a
container platform with `HEADROOM_PROXY_TOKEN` set, and the dashboard
sits on its own host behind an authenticating reverse proxy (here Entra
ID sign-in in front of a small Caddy that adds `x-headroom-proxy-token`
after the user signed in). That reverse proxy reaches Headroom with a
domain `Host` (the platform routes internal calls by Host, so it cannot
be an IP) and from a peer address that is not stable enough to list. The
dashboard loads, the aggregate counters work, and the "Recent Requests"
table stays empty with no indication why.
This PR also admits callers whose proxy token the security gate has
already verified (`request.state.proxy_authenticated`, which the gate
sets only after the constant-time token comparison).
**This is a nice-to-have, not a bug fix.** The current behaviour is
documented and deliberate, and the existing trusted-CIDR path keeps
working unchanged. If you would rather not widen who sees this metadata,
closing the PR is fine; we carry the change on our fork either way.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py`: add `_request_can_view_stats_metadata`:
true when the gate verified the proxy token, otherwise the existing
`_request_can_view_dashboard_metadata` result. `/stats` and
`/stats-lifetime` use it. The helper is deliberately not used by
`/settings*` or `/transformations/feed`, which keep their loopback /
trusted-client guards.
- `tests/test_proxy_stats_recent_requests.py`: add a regression test
(domain `Host`, non-loopback peer, token sent) for `/stats` and
`/stats?cached=1`, asserting the tail is served with the token and that
the gate still answers 401 without it or with a wrong one.
- `docs/content/docs/security-model.mdx`: new section **What the proxy
token unlocks**. It lists exactly which fields a token holder now also
sees: `recent_requests`, `request_logs` and `config` on `/stats` (and
`?cached=1`), and `projects` and `persistence.error` on
`/stats-lifetime`. It says the settings routes and the
`require_loopback` routes stay closed to a token. It also warns that the
token is one shared secret, not a per-caller identity: every holder sees
every caller's request metadata, so a deployment that hands one token to
several clients that must not see each other's metadata should run one
proxy per client. The dashboard-CIDR table row now points to the
section.
- `docs/content/docs/proxy.mdx` (the `HEADROOM_PROXY_TOKEN` row) and
`SECURITY.md` (the local-control-surfaces bullet): one line each
pointing to the same consequence.
- `headroom/proxy/server.py` docstrings: the `/stats` handler and
`_request_can_view_stats_metadata` now describe the actual access rule.
The helper no longer claims the change widens nothing.
Who gains access: holders of `HEADROOM_PROXY_TOKEN`. This does widen
what the token grants. Before, it granted inference and the read-only
operator routes. Now it also grants the request metadata of every caller
on the proxy. With no token configured, nothing changes (the flag is
never set).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom/proxy/server.py`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
At `1c6666d` merged with current `main` (`4abbf07`):
```text
$ python -m pytest -q tests/test_proxy_stats_recent_requests.py tests/test_proxy_loopback_gating.py \
tests/test_dashboard_accessibility.py tests/test_proxy_hardening.py
214 passed, 1 skipped in 44.60s
$ python -m mypy headroom/proxy/server.py
Success: no issues found in 1 source file
$ ruff check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_stats_recent_requests.py
2 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13, this branch at `4abbf07`
(`1c6666d` merged with current `main`). A real `headroom proxy` with a
throwaway `HEADROOM_PROXY_TOKEN`, bound to a non-loopback LAN address,
with `HEADROOM_OFFLINE=1` and `--anthropic-api-url` pointed at a local
stub that answers `/v1/messages` with a fixed Anthropic message:
```text
headroom proxy --host 192.168.0.48 --port 18788 --no-optimize
--anthropic-api-url http://127.0.0.1:18999
```
- Exact command / steps: a client on `192.168.0.48` (not loopback, in no
trusted CIDR) sends what the dashboard's reverse proxy sends. That is
`Host: headroom-dashboard.example.test` (a domain, as the platform
routes by Host) plus `x-headroom-proxy-token`. It first makes one
token-authenticated `POST /v1/messages` so there is a request to list,
then reads the stats endpoints with the token, without it, and with a
wrong one. It also tries the settings and loopback-only routes with the
token. Same script against the same build with only
`headroom/proxy/server.py` swapped back to `main`'s, for the before
column.
- Observed result: verified token holders receive shared stats metadata,
while missing or incorrect tokens are rejected; exact before/after
responses follow.
```text
request (domain Host, non-loopback peer) before (main) after (this PR)
POST /v1/messages token 200 200
GET /stats token 200, recent_requests/request_logs/ 200,
recent_requests=1 request_logs=1
config absent config present, first={"provider":
"anthropic","model":"claude-sonnet-5-5"}
GET /stats?cached=1 token 200, same fields absent 200, same as /stats
GET /stats-lifetime token 200, projects absent 200, projects present
GET /stats no token 401 401
GET /stats wrong token 401 401
GET /stats-lifetime no token 401 401
GET /settings token 404
GET /settings/schema token 404
GET /dashboard/settings token 404
GET /transformations/feed token 404
```
The proxy log shows `event=proxy_auth_rejected path=/stats …
reason=missing_token`, `… reason=bad_token` and `path=/stats-lifetime …
reason=missing_token` for the three rejections.
- Not tested: the full suite (only the four files above). The deployed
dashboard end to end: the run above reproduces its request shape against
a real proxy, without the Entra/Caddy hop in front.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only when `HEADROOM_PROXY_TOKEN` is
set. Token-authenticated remote callers now receive `recent_requests`,
`request_logs` and `config` from `/stats`, and `projects` and
`persistence.error` from `/stats-lifetime`, for every caller of the
proxy. Loopback and trusted-CIDR behaviour is unchanged, and the
settings and loopback-only routes are not opened to the token.
- Kill switch / disable path: unset the token (then nothing changes), or
revert the commit
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- Documentation: done; see the security-model section above.
- A possible alternative that avoids widening the rule: an opt-in env
var (e.g. `HEADROOM_STATS_METADATA_FOR_TOKEN=1`). Happy to switch to
that if you prefer the default to stay strict.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_013MpfLG8sVzqPhD34teqdLD
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4168"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `09689b73ddeb8879a2f2c68726f9e885cd9bfdee`, upstream main
`e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`.
Mechanically merged upstream/main e4a60ed7 and re-reviewed
stats-metadata authorization plus the existing security/runtime
discussion. Only callers whose proxy token authentication succeeded gain
the additional per-request stats metadata; settings and loopback-only
routes retain their separate guards. Documentation explicitly states
that token holders share every caller's traffic metadata. No contributor
production logic was rewritten.
All 260 focused stats/recent-request, loopback-gating and hardening
tests pass; Ruff lint/format and whitespace checks pass. Existing
before/after real HTTP evidence remains explicitly scoped; local tests
use the existing native extension, not a fresh native build. Applied the
shared test-only Gemini streamed-request fixture repair for
independently established main failures; all ten cases pass. Formal
approval awaits exact-head hosted qualification.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
### Maintainer current-main requalification
Exact head `fa07187e666ca7895d1cc5708a9d930f38399fe1`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.
Clean mechanical merge with upstream main
c01cf07d94. Reviewed current author
discussion and Devin runtime/documentation findings. Additional stats
metadata is allowed only after successful proxy-token authentication or
the existing dashboard authorization; settings and loopback-only routes
retain separate guards. The security documentation explicitly states
that one shared token exposes every caller's traffic metadata to every
holder. No contributor authorization policy rewritten.
Fresh focused stats/recent-request, loopback-gating and hardening
verification: 260 tests passed with six existing deprecation/OpenAPI
warnings. Changed-file Ruff lint/format and main-relative whitespace
checks pass; owned tree clean. Existing non-loopback before/after HTTP
evidence remains scoped; no fresh live replay or branch-native build is
claimed. Python tests use the current-main native DLL. No new source
blocker found; approval awaits exact-head hosted checks. Nothing merged.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
### Maintainer current-main requalification
Exact head `20b62cc35f7a005b91a6c9054c41cfd51ac29c37`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.
Clean main integration against 0ddaa56c21.
315 stats/recent-request, loopback/security gating and incoming-main
tests passed, 11 optional skips, 6 warnings. Exact incoming-main
patches, lint/format, offline lock and whitespace checks pass. Complete
recorded native input-set/hash and artifact checksum verified before
existing DLL reuse; no fresh native build claimed.
Current author discussion and both resolved Devin threads/replies
reviewed. Verified token-authenticated state expands only stats metadata
authorization; settings and require_loopback guards remain separate.
Documentation explicitly lists fields and warns that the shared token
exposes every caller's request metadata to every holder, with separate
proxies required for isolation. This is an intentional documented
shared-trust policy, not a per-user identity. Prior author's
non-loopback real-proxy before/after evidence is retained with its
scope; this refresh adds local tests, not a fresh live replay or
browser/reverse-proxy proof. No new source blocker found.
Source/test/prose delta has no
workflow/setup/dependency/CLI/install/provider/OpenClaw/SDK path
changes. Inherited flagged workflow-validation and Init/Wrap bootstrap
paths are excluded. Docs build uses unchanged-main
workflow/package/lock/config and this PR changes only prose there.
Exact-head Actions and rollup remain pending before approval. Nothing
merged.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
181 lines
10 KiB
Markdown
181 lines
10 KiB
Markdown
# Security Policy
|
||
|
||
## Supported Versions
|
||
|
||
| Version | Supported |
|
||
| ------- | ------------------ |
|
||
| Latest release on PyPI | :white_check_mark: |
|
||
| Any earlier release | :x: |
|
||
|
||
Headroom releases frequently and only the most recent release is supported. Security
|
||
fixes ship in a new release rather than being backported to earlier ones, so upgrading is
|
||
the remediation path. Check what you are running with `headroom --version`, and compare it
|
||
against [the current release](https://pypi.org/project/headroom-ai/).
|
||
|
||
## Reporting a Vulnerability
|
||
|
||
We take security vulnerabilities seriously. If you discover a security issue, please report it responsibly.
|
||
|
||
### How to Report
|
||
|
||
**Please DO NOT open a public GitHub issue for security vulnerabilities.**
|
||
|
||
Instead, please email us at: **security@headroomlabs.ai**
|
||
|
||
Include the following information:
|
||
- Type of vulnerability (e.g., injection, data exposure, authentication bypass)
|
||
- Full path of the affected source file(s)
|
||
- Step-by-step instructions to reproduce the issue
|
||
- Proof-of-concept or exploit code (if possible)
|
||
- Impact assessment
|
||
|
||
### What to Expect
|
||
|
||
1. **Acknowledgment**: We will acknowledge receipt within 48 hours
|
||
2. **Assessment**: We will assess the vulnerability and determine its severity
|
||
3. **Updates**: We will keep you informed of our progress
|
||
4. **Resolution**: We aim to resolve critical issues within 7 days
|
||
5. **Credit**: With your permission, we will credit you in the security advisory
|
||
|
||
### Security Best Practices for Users
|
||
|
||
When using Headroom:
|
||
|
||
1. **API Keys**: Never commit API keys. Use environment variables.
|
||
2. **Proxy Exposure**: Don't expose the proxy server to the public internet without authentication
|
||
3. **Log Files**: Headroom always writes an operational log to `~/.headroom/logs/proxy.log`.
|
||
The opt-in `--log-file` request log, and especially `--log-messages`, additionally write
|
||
request and response *content* to disk. Be aware that both may contain sensitive information.
|
||
4. **Budget Limits**: Set budget limits to prevent unexpected costs
|
||
5. **Workspace directory**: `~/.headroom` holds both credential material and cached
|
||
request content. Treat it as sensitive — especially on shared or multi-user hosts.
|
||
Note that it is not the only location: with memory enabled, extracted memories default
|
||
to a project-local `.headroom/` directory beside the code you ran the agent in. See
|
||
[Security Model](https://headroom-docs.vercel.app/docs/security-model) for exactly what
|
||
is written where.
|
||
|
||
### Scope
|
||
|
||
The following are in scope for security reports:
|
||
- Headroom Python package (`pip install headroom-ai`)
|
||
- Headroom proxy server
|
||
- Official integrations (LangChain, Agno, Strands, LiteLLM, Vercel AI SDK, Anthropic/OpenAI SDK wrappers, MCP)
|
||
|
||
The following are out of scope:
|
||
- Third-party integrations not maintained by us
|
||
- Issues in dependencies (report these to the upstream project)
|
||
- Social engineering attacks
|
||
|
||
## Security Posture
|
||
|
||
Headroom is a local proxy that sits between your agents and your LLM providers, so
|
||
it necessarily handles both your traffic and your credentials. The
|
||
[Security Model](https://headroom-docs.vercel.app/docs/security-model) documents this in
|
||
full — what is written to disk, with what permissions and retention, what leaves the
|
||
host, and what the local admin surface does and does not allow. The summary:
|
||
|
||
- **Interactive provider API keys are forwarded.** Keys supplied via environment
|
||
variables or request headers are used to authenticate the upstream call and are not
|
||
written to disk or emitted to logs. Settings can also persist keys: the
|
||
`ANTHROPIC_TARGET_API_HEADERS` / `OPENAI_TARGET_API_HEADERS` header maps, if you set
|
||
them through the dashboard settings GUI, are persisted to `~/.headroom/settings.json`
|
||
in plaintext — and a header map is where a gateway key typically goes.
|
||
- **Some credentials *are* stored, by design.** `headroom copilot-auth login` persists a
|
||
GitHub Copilot OAuth refresh token to `~/.headroom/copilot_auth.json` (relocatable via
|
||
`$HEADROOM_COPILOT_AUTH_FILE`). It is plaintext JSON, written with `0600` permissions
|
||
on a best-effort basis.
|
||
- **Persistent installs store supplied environment values.** Every value passed
|
||
to `headroom install --env KEY=VALUE` is stored verbatim and in cleartext. Supervisors (launchd, systemd, Task
|
||
Scheduler, cron) start the proxy from a bare environment, so `--env` is the
|
||
only channel by which a provider API key reaches a supervised proxy — and
|
||
if you pass one, it is written to
|
||
`~/.headroom/deploy/<profile>/manifest.json` (mode `0600`) and `export`ed in
|
||
cleartext by the generated runner scripts `run-headroom.sh` /
|
||
`ensure-headroom.sh` (mode `0700`) in the same directory (mode `0700`).
|
||
The installer sets those modes and then reads them back: where the
|
||
platform enforces POSIX permission bits it refuses to leave behind a runner
|
||
script it could not restrict, and warns if the profile directory could not
|
||
be narrowed. On Windows the bits are advisory — access is governed by ACLs —
|
||
so the modes above are requested but not enforced there, and protection
|
||
comes from the account's own profile directory instead. Where they do hold,
|
||
the files are owner-only, not encrypted: anything that can read the account
|
||
— the user themself, root, a home-directory backup or sync — can read the
|
||
key, and rotating it requires re-running `headroom install apply`.
|
||
With the Docker runtime the same values are additionally passed as `docker
|
||
run --env NAME=VALUE`, which places them in the container process's command
|
||
line and so in `ps` output for other users on the host. Prefer a
|
||
supervisor-native secret facility (systemd `LoadCredential=` /
|
||
`EnvironmentFile=` on an owner-only file, a macOS keychain helper) over
|
||
`--env` for long-lived provider keys.
|
||
- **Cached request content is written to disk.** CCR stores pre-compression originals —
|
||
tool outputs, file contents, retrieved chunks — in a local SQLite database
|
||
(`~/.headroom/ccr_store.db`, `0600`) so they can be retrieved after compression. Entries
|
||
are plaintext and are not encrypted at rest. Set `HEADROOM_CCR_BACKEND=memory` if your
|
||
deployment cannot accept disk persistence.
|
||
- **Anonymous session summaries are uploaded by default.** The `HEADROOM_BEACON` telemetry
|
||
beacon is opt-*out*: it POSTs content-free session counters and distributions, a random per-install
|
||
UUID, version, OS, and architecture to Headroom Labs, plus the values of 15 allow-listed
|
||
`HEADROOM_*` settings and a bucketed structural descriptor of tool outputs (field count,
|
||
depth, shape flags). `headroom telemetry --show` prints the exact payload. No prompts, completions, file contents,
|
||
or paths are included. Disable with `HEADROOM_BEACON=off`, `DO_NOT_TRACK=1`, or
|
||
`HEADROOM_OFFLINE=1`. This is separate from `HEADROOM_TELEMETRY`, which is opt-in and
|
||
stays on the machine.
|
||
- **Passthrough mode**: Sensitive content passes through unchanged by default.
|
||
- **Operational logs are written unconditionally.** `~/.headroom/logs/proxy.log`
|
||
(10 MB × 5 rotations) is always on and also carries the admin audit stream. The
|
||
request log (`--log-file`) and full message logging (`--log-messages`) are opt-in and
|
||
write request/response content to a path you choose.
|
||
- **Local control surfaces are loopback-gated.** The `/admin/*`, `/debug/*`,
|
||
`/v1/retrieve*`, `/v1/telemetry*`, `/v1/toin/*`, and `/settings*` routes require a
|
||
loopback client address *and* a loopback `Host:` header, plus a same-origin check on
|
||
the mutating ones. This is a local-trust model: any process on the same host is
|
||
trusted. Setting `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` deliberately widens
|
||
the settings and stats surface beyond loopback. Setting `HEADROOM_PROXY_TOKEN` also
|
||
shows every token holder the per-request `/stats` metadata of all callers, but opens
|
||
none of these routes.
|
||
|
||
Thank you for helping keep Headroom and its users safe!
|
||
|
||
## Unpatched optional dependency advisories (reviewed 2026-09-10)
|
||
|
||
The following public upstream advisories remain unresolved. They are included in
|
||
`uv.lock` through optional extras; the presence of a package in that universal
|
||
lockfile does not mean it is installed with every Headroom installation.
|
||
|
||
### CrewAI / ChromaDB (no longer locked)
|
||
|
||
Headroom no longer provides a `crewai` extra, so CrewAI and ChromaDB are not in
|
||
`uv.lock` and are never installed by Headroom. The CrewAI integration uses
|
||
whichever CrewAI the user installs (`pip install headroom-ai crewai`), and that
|
||
installation owns its dependency exposure. Every current CrewAI release requires
|
||
ChromaDB, which has unpatched advisories in its server
|
||
([GHSA-f4j7-r4q5-qw2c](https://github.com/advisories/GHSA-f4j7-r4q5-qw2c),
|
||
[GHSA-36p7-vc44-83pf](https://github.com/advisories/GHSA-36p7-vc44-83pf),
|
||
[GHSA-2wm9-hf6c-p5cr](https://github.com/advisories/GHSA-2wm9-hf6c-p5cr),
|
||
[GHSA-xph7-9rjv-w5fr](https://github.com/advisories/GHSA-xph7-9rjv-w5fr)).
|
||
Headroom's integration only wraps tools and never starts a ChromaDB server. If you
|
||
run ChromaDB as a server, keep it inaccessible to untrusted clients and follow
|
||
those advisories.
|
||
|
||
### Voice training / Accelerate
|
||
|
||
The `voice-train` extra includes Accelerate (locked at 1.12.0).
|
||
[GHSA-4j2p-28q2-5m79](https://github.com/advisories/GHSA-4j2p-28q2-5m79)
|
||
describes path traversal and denial of service through unvalidated `weight_map`
|
||
entries in sharded checkpoint indexes. Use only trusted checkpoints, including
|
||
their index files and referenced shards, in training environments.
|
||
|
||
The advisory currently lists versions through 1.14.0, but an upgrade to 1.15.0
|
||
is not a verified fix: its
|
||
[checkpoint loader](https://github.com/huggingface/accelerate/blob/v1.15.0/src/accelerate/utils/modeling.py#L1941-L1944)
|
||
still joins index values to the checkpoint directory without containment or
|
||
file-type validation. The proposed fixes
|
||
[#4070](https://github.com/huggingface/accelerate/pull/4070) and
|
||
[#4138](https://github.com/huggingface/accelerate/pull/4138) were closed without
|
||
merging; the latter also explicitly leaves the named-pipe denial of service
|
||
unfixed. Keep the alert open until a released fix covers both cases.
|
||
|
||
Dependabot ignores only the reviewed unpatched Accelerate range (through 1.15.0).
|
||
Later releases remain eligible for review. These
|
||
update exceptions do not remediate the advisories or dismiss vulnerability alerts.
|