mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
feat/https-compose-caddy
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d603f038de |
harden: the deployment script interpolates the $bucket ... in query.sh (#2873)
## Description
Adds input validation for the \`R2_BUCKET\` environment variable in
\`deploy/beacon/query.sh\`, an operator-run local shell script (not a
web
service or request handler — it is never on a network path). The script
is
invoked manually or from a cron job by someone who already holds
Cloudflare R2
credentials on their workstation.
## Type of Change
- [x] Bug fix / security hardening
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Vulnerability
| Field | Value |
|-------|-------|
| ID | V-001 |
| Severity | HIGH |
| Scanner | multi_agent_ai |
| File | \`deploy/beacon/query.sh\` |
Description: Without this guard, a shell special character in
\`R2_BUCKET\`
(e.g. from a misconfigured env file or an untrusted CI/CD variable)
would
reach the DuckDB path string \`r2://\${BUCKET}/sessions//.json\`
unvalidated.
## Threat model
\`query.sh\` is an operator/admin script — it is not a CGI handler and
is not
reachable over a network. The threat scenario is: a workstation env file
that
has \`R2_BUCKET\` set to a value containing shell-special characters
(copied from
a non-R2 source), or a CI/CD pipeline that interpolates \`R2_BUCKET\`
from an
untrusted variable. The fix is defensive hardening appropriate to that
context.
## Changes Made
- Add an `R2_BUCKET` allowlist guard before environment loading,
credential checks, and DuckDB invocation.
- Add a dependency-free executable Bash regression covering invalid and
valid bucket names.
\`deploy/beacon/query.sh\` — one-line regex guard at line 23, running
before
env-file sourcing, credential checks, and DuckDB invocation:
\`\`\`bash
[[ "\$BUCKET" =~ ^[a-zA-Z0-9_-]+\$ ]] || {
echo "Invalid R2_BUCKET value — must contain only alphanumeric, hyphen,
or underscore characters" >&2
exit 1
}
\`\`\`
## Testing
- [x] Regression tests pass
- [x] New executable test added
### Test Output
```text
PASS: 7 adversarial R2_BUCKET values rejected
PASS: 5 representative R2_BUCKET values accepted
All 12 tests passed.
```
\`deploy/beacon/test_query_bucket_validation.sh\` — pure bash, no
external
dependencies, no DuckDB or Cloudflare credentials needed. Run:
\`\`\`bash
bash deploy/beacon/test_query_bucket_validation.sh
\`\`\`
Rejected values (exit 1 + \`"Invalid R2_BUCKET"\` in stderr):
| Value | Threat |
|---|---|
| \`../etc/passwd\` | path traversal |
| \`bucket; rm -rf /\` | shell injection |
| \`bucket\$(whoami)\` | command substitution |
| \`bucket with spaces\` | word-splitting |
| \`bucket!@#\` | special characters |
| \`bucket.name\` | dot (outside grammar) |
| \`bucket/subpath\` | slash (outside grammar) |
Accepted values (pass validation, then fail at credential check —
proving
the guard was cleared without real credentials):
| Value | Notes |
|---|---|
| \`headroom-telemetry\` | production default |
| \`headroom_backup\` | underscore variant |
| \`my-bucket-123\` | alphanumeric + hyphens |
| \`MyBucket\` | mixed case |
| \`a\` | single character |
Accepted cases use a temporary empty file for \`HEADROOM_ENV_FILE\`. The
script
clears bucket validation and then fails with \`"R2_ACCOUNT_ID not set"\`
— a
distinct error that confirms validation was passed.
## Behavior preservation
Valid bucket names (alphanumeric, hyphens, underscores) are unaffected.
The
guard rejects only values that contain characters outside that grammar.
## Real Behavior Proof
- Environment: pure Bash test on the current PR head; no DuckDB or
Cloudflare credentials required.
- Exact command / steps: ran bash
deploy/beacon/test_query_bucket_validation.sh, which invokes query.sh
with seven adversarial and five representative valid R2_BUCKET values.
- Observed result: all 12 cases passed; invalid values were rejected
before credential setup and valid values reached the expected
missing-credential check.
- Not tested: live DuckDB/R2 access with production credentials.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: OrbisAI Security <mediratta01.pally@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
|
e9a24f3ec1 |
fix(beacon): report all-layers savings, not context-compression only (#2796)
## Description `rates.saved_pct` and `rates.yield_pct` in the beacon payload divide `tokens_saved` by `original` / `attempted`. Tool-schema deferral never lands in either denominator — `outcome.py` says so explicitly, and `tokens.tool_saved` exists precisely because of it — so every beacon rate silently reports context compression only. On a tool-heavy fleet that is not a rounding difference. Across the first 516 sessions in the corpus the beacon reads **2.80%** where the dashboard headline for the same traffic reads **12.82%**: 157.6M context tokens vs 803.9M all-layers, with 646.3M of tool-schema deferral missing from the ratio. `headroom/proxy/server.py` already resolved this for the dashboard in #2737 — `savings_percent` is `all_layers_saved / (input + all_layers_saved)` and `active_savings_percent` puts tool savings on both sides of the ratio. The beacon was never brought along. This does that. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/telemetry/session.py`: add `rates.all_layers_saved_pct` and `rates.all_layers_yield_pct`, computed the way `server.py` builds `savings_percent` / `active_savings_percent` — tool savings added to **both** sides, since deferred schemas were attempted work that succeeded whole. - `headroom/telemetry/session.py`: extend the `demo()` self-test to assert both new rates against the existing tool-heavy fixture. - `deploy/beacon/query.sh`: add an `all_layers_pct` column to the fleet summary, so the reader stops showing the understated number too. **Kept alongside `saved_pct` rather than folded into it.** Every row already in the corpus means context-only under that name; redefining it would make old and new rows non-comparable with no field to tell them apart. **`SCHEMA_VERSION` deliberately stays at 1.** The change is purely additive, nothing reads the field, and the query uses `union_by_name = true`, so old and new rows mix cleanly. Happy to bump it if maintainers want the marker. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m headroom.telemetry.session ok $ python -m pytest tests/test_savings_tool_search_aggregation.py tests/test_outcome_dual_ruler_funnel.py -q FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted 2 failed, 5 passed, 3 warnings in 4.42s $ ruff check headroom/ All checks passed! $ mypy --python-version 3.12 headroom/telemetry/session.py Success: no issues found in 1 source file ``` The two `test_outcome_dual_ruler_funnel.py` failures are **pre-existing on `main`, not caused by this PR** — verified by `git stash`-ing the change and re-running: ```text $ git stash && python -m pytest tests/test_outcome_dual_ruler_funnel.py -q FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted 2 failed, 3 passed, 3 warnings in 3.71s ``` ## Real Behavior Proof - **Environment:** macOS 25.4.0 arm64, Python 3.12.6, repo `.venv`, branch rebased on `upstream/main` @ `d0a86d40`. - **Exact command / steps:** drive a real `SessionAggregator` with a tool-heavy outcome (`original=1000`, `attempted=400`, `tokens_saved=300`, plus `tool_search_deferred_tokens=800` and `turn_hook_tools_saved_tokens=200`) and print the emitted payload's `rates` block: ```text tokens: {"original": 1000, "attempted": 400, "saved": 300, "tool_saved": 1000} rates: { "saved_pct": 30.0, "eligible_pct": 40.0, "yield_pct": 75.0, "all_layers_saved_pct": 65.0, "all_layers_yield_pct": 92.86, "cache_read_pct": 50.0, "overhead_pct": 5.0 } ``` - **Observed result:** the pre-existing rates are byte-identical (30.0 / 40.0 / 75.0 / 50.0 / 5.0 — no regression), and the two new fields report the all-layers view: 1300 saved of 2000 sent = **65.0%**, 1300 of 1400 attempted = **92.86%**. Both denominators grow with the numerator, matching `server.py`. Cross-checked against the live corpus with DuckDB over the R2 bucket — restating all 516 sessions both ways reproduces the gap this PR closes: ```text ┌───────────┬────────────┬──────────────────┬──────────────────┬─────────────────────┐ │ ctx_saved │ tool_saved │ all_layers_saved │ beacon_saved_pct │ dashboard_saved_pct │ ├───────────┼────────────┼──────────────────┼──────────────────┼─────────────────────┤ │ 157561051 │ 646293457 │ 803854508 │ 2.8 │ 12.82 │ └───────────┴────────────┴──────────────────┴──────────────────┴─────────────────────┘ ``` - **Not tested:** no live proxy run was made against a real provider — the payload above comes from `SessionAggregator` driven directly, which is the same code path the proxy feeds. The 516 sessions already in R2 are **not backfilled**: they carry `tool_saved`, so the corrected rate is computable from them today, but their own `rates` block stays context-only. Only sessions emitted from the next release carry the new fields. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - **Documentation:** N/A — no doc references the beacon `rates` field names (`grep -rn "saved_pct\|yield_pct" docs/` returns nothing). The field semantics are documented inline in `session.py`, which this PR extends. - **Worth a maintainer opinion:** the dashboard formula adds `tool_saved` to the *denominator* as well. That is defensible — deferred schemas were attempted work that succeeded 100% — but it does mean a tool-heavy session's ratio is partly measuring a layer that is near-always at full yield. This PR matches the dashboard rather than inventing a third convention; if the convention should change, it should change in both places at once. - **Follow-up:** nothing here changes what the fleet actually saved, only what the beacon admits to. The 4.6x gap was reporting, not performance. |
||
|
|
9cfb00838a |
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|