mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ac932eb7c4 |
feat(telemetry): split beacon sessions by harness, model and day (#4881)
## Description
After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:
1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.
| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |
- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.
**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.
**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.
## Testing
- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```
Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.
## Real Behavior Proof
- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:
```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
{"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
{"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
{"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
{"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
{"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```
Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
- Linux, Windows, containers and CI.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
1f4c66f50a |
fix(beacon): roll up each hour in sixteen parallel queue slices (#4880)
## Description
The hourly rollup (`deploy/beacon/worker.js`) read every heartbeat
object of an hour inside one cron invocation.
- A Worker may have 6 requests in flight, so reads ran at about 12
objects/s regardless of `FANOUT = 100`.
- A cron invocation is stopped at 15 minutes, so one run could read
about 11k objects.
- Hours passed that size on 2026-08-19. A stopped run wrote nothing.
Result: 1,388 of 2,490 clock hours (23.8M of 33.3M heartbeat objects)
never got a rollup. Raw `sessions/` is complete; only `rollup/` had
gaps. The stats dashboard reads only rollups. The missing hours have
since been backfilled outside the Worker, so this PR is about not
falling behind again.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `scheduled()` no longer reads heartbeats. It lists hours with no
rollup and enqueues 16 messages per hour to a Cloudflare Queue, one per
first hex character of the object name (names are `crypto.randomUUID()`,
so slices are even and this works on objects already stored).
- `queue()` handles one slice per invocation: `rollupSlice()` reads it
(6 in flight) and writes `rollup-parts/<hour>/<slice>`; `finalizeHour()`
merges the 16 parts into `rollup/<hour>/data.ndjson` once all are
present, then deletes the parts.
- A failed slice is retried alone (60 s delay, 5 tries, then dead-letter
queue). An hour still unfinished after 3 hours is enqueued again.
`scheduled()` sweeps anything a finished hour left under
`rollup-parts/`.
- Rollup output is unchanged: same key, same winner per (install,
session).
- `wrangler.toml` gains the queue producer and consumer bindings.
## Testing
- [ ] Unit tests pass (`pytest`) — not applicable, no Python changed
- [ ] Linting passes (`ruff check .`) — not applicable
- [ ] Type checking passes (`mypy headroom`) — not applicable
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cd deploy/beacon && node test-rollup.mjs
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 104 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
nested allowlist: client fields kept, unknown nested keys and free text dropped
ok
```
New cases: cross-slice dedupe, a failed slice leaving the hour
unpublished, enqueue / skip / re-enqueue in `scheduled()`, retry with
delay, reuse of an existing part, and the sweep.
## Real Behavior Proof
- Environment: macOS 27.2.1 (arm64), Node v26.7.0, DuckDB v1.5.5 with
`httpfs`, this branch at `811e6cf1`. Input is production data read from
the R2 bucket `headroom-telemetry` with a read-only key. The Worker
module ran under Node, not on Cloudflare; the R2 binding was an
in-memory map that pages `list()` at 1000 keys like R2.
- Exact command / steps (both scripts are in full below):
1. `./fetch_hour.sh 2026-08-10 03 h0810` and `./fetch_hour.sh 2026-10-14
12 h1014` — download every raw object of the hour, and the rollup object
stored in R2 for it.
2. `node compare.mjs deploy/beacon/worker.js h0810 dt=2026-08-10/hh=03`
(and likewise for `h1014`) — deliver 16 slice messages concurrently to
this branch's `queue()` handler, then compare what it published with the
stored rollup, row by row.
3. `node deploy/beacon/test-rollup.mjs h0810/raw` — the repo's own
real-corpus check.
- Observed result:
```text
$ ./fetch_hour.sh 2026-08-10 03 h0810
3666 raw objects written to h0810/raw
$ ./fetch_hour.sh 2026-10-14 12 h1014
24850 raw objects written to h1014/raw
$ node compare.mjs deploy/beacon/worker.js h0810 dt=2026-08-10/hh=03
{ "part": "dt=2026-08-10/hh=03", "raw_objects": 3666, "raw_gets": 3666, "acked": 16, "retried": 0,
"sessions_published": 945, "sessions_in_reference": 945, "same_session_and_seq": 945,
"identical_rows": 945, "parts_left_behind": 0, "wall_ms": 39, "peak_rss_mb": 103 }
$ node compare.mjs deploy/beacon/worker.js h1014 dt=2026-10-14/hh=12
{ "part": "dt=2026-10-14/hh=12", "raw_objects": 24850, "raw_gets": 24850, "acked": 16, "retried": 0,
"sessions_published": 6903, "sessions_in_reference": 6903, "same_session_and_seq": 6903,
"identical_rows": 6903, "parts_left_behind": 0, "wall_ms": 706, "peak_rss_mb": 432 }
$ node deploy/beacon/test-rollup.mjs h0810/raw
real corpus: 3666 objects -> 945 sessions in 1 object
ok
```
- **`dt=2026-08-10/hh=03`** (3,666 objects): the reference is the rollup
the current production Worker wrote. All 945 rows are identical after
key-order normalisation.
- **`dt=2026-10-14/hh=12`** (24,850 objects, among the busiest hours):
the old Worker never completed this hour. The reference is the rollup
built by the offline backfill, a separate implementation, and uploaded
to R2 on 2026-11-15. All 6,903 rows are identical. The published object
is 15.7 MB.
- In both runs every raw object was read exactly once, all 16 messages
were acknowledged, none retried, and no parts were left behind.
- Not tested:
- **Anything on Cloudflare.** Not deployed, not run under `wrangler
dev`. Queue delivery, consumer concurrency and ramp-up, the 6-connection
limit, the 15-minute limit, retries, the dead-letter queue and real R2
latency are unobserved.
- **Memory on the Workers runtime.** `peak_rss_mb` above is the Node
process including all raw objects held by the stub, so it says nothing
about the 128 MB isolate. `finalizeHour` holds the merged hour in memory
and builds the output as one string; for the 15.7 MB hour that is a few
multiples of 15.7 MB, by estimate only.
- **`scheduled()` against a real bucket**, including the listing of
`rollup/` and `rollup-parts/` at production size and enqueueing a
backlog.
- Throughput per slice. The measured ~12 objects/s per invocation comes
from the old Worker's write timestamps, not from this code.
<details><summary><code>fetch_hour.sh</code></summary>
```bash
#!/usr/bin/env bash
# Download one hour of raw beacon objects from R2 into <out>/<object name>, plus
# the rollup object stored for that hour. ./fetch_hour.sh 2026-08-10 03 <out>
# Needs R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY (read-only is enough).
set -euo pipefail
DAY="$1"; HH="$2"; OUT="$3"; mkdir -p "$OUT/raw"
duckdb >/dev/null <<SQL
LOAD httpfs; SET threads=48; SET http_retries=8; SET http_timeout=120;
CREATE SECRET (TYPE r2, KEY_ID '${R2_ACCESS_KEY_ID}', SECRET '${R2_SECRET_ACCESS_KEY}', ACCOUNT_ID '${R2_ACCOUNT_ID}');
COPY (SELECT filename, json::VARCHAR line FROM read_ndjson_objects('r2://headroom-telemetry/sessions/dt=${DAY}/hh=${HH}/*.json', filename=true))
TO '${OUT}/raw.jsonl' (FORMAT json);
COPY (SELECT json::VARCHAR line FROM read_ndjson_objects('r2://headroom-telemetry/rollup/dt=${DAY}/hh=${HH}/data.ndjson', maximum_object_size=40000000))
TO '${OUT}/stored_rollup.ndjson' (FORMAT csv, HEADER false, QUOTE '', ESCAPE '', DELIMITER '\x1f');
SQL
python3 - "$OUT" <<'PY'
import json, os, sys, collections
out = sys.argv[1]; files = collections.defaultdict(list)
for l in open(f"{out}/raw.jsonl"):
r = json.loads(l); files[os.path.basename(r["filename"])].append(r["line"])
for name, lines in files.items():
open(f"{out}/raw/{name}", "w").write("\n".join(lines))
print(f"{len(files)} raw objects written to {out}/raw")
PY
```
</details>
<details><summary><code>compare.mjs</code></summary>
```js
// Run this branch's queue consumer over one real hour of raw objects and compare
// what it publishes with a reference rollup, row by row.
// node compare.mjs <worker.js> <dir with raw/ and stored_rollup.ndjson> <dt=…/hh=…>
import { readFileSync, readdirSync } from 'node:fs';
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const [workerPath, dir, part] = process.argv.slice(2);
const worker = (await import(pathToFileURL(resolve(workerPath)).href)).default;
// The slice of the R2 binding the Worker uses, over a Map. list() pages at 1000
// like R2 does.
const files = new Map();
for (const name of readdirSync(`${dir}/raw`)) {
files.set(`sessions/${part}/${name}`, readFileSync(`${dir}/raw/${name}`, 'utf8'));
}
const raw = files.size;
let gets = 0;
const CORPUS = {
list: async ({ prefix, cursor, limit = 1000 }) => {
const keys = [...files.keys()].filter((k) => k.startsWith(prefix)).sort();
const start = cursor ? keys.indexOf(cursor) : 0;
const page = keys.slice(start, start + limit);
const next = keys[start + limit];
return { objects: page.map((key) => ({ key, uploaded: new Date(0) })), truncated: !!next, cursor: next };
},
get: async (key) => { gets++; return files.has(key) ? { text: async () => files.get(key) } : null; },
head: async (key) => (files.has(key) ? { key } : null),
put: async (key, body) => { files.set(key, body); },
delete: async (keys) => { for (const key of keys) files.delete(key); },
};
const started = Date.now();
const outcome = { ack: 0, retry: 0 };
// Sixteen messages, one per slice, delivered concurrently as the queue would.
await Promise.all([...'0123456789abcdef'].map((slice) =>
worker.queue({ messages: [{ body: { part, slice }, ack: () => outcome.ack++, retry: () => outcome.retry++ }] }, { CORPUS })));
const ms = Date.now() - started;
const canon = (v) => Array.isArray(v) ? v.map(canon)
: v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, canon(v[k])])) : v;
const index = (text) => new Map(text.split('\n').filter(Boolean).map((line) => {
const rec = JSON.parse(line);
return [`${rec.resource?.['headroom.install_id']} ${rec.session?.id}`, { seq: rec.session?.seq, row: JSON.stringify(canon(rec)) }];
}));
const mine = index(files.get(`rollup/${part}/data.ndjson`) ?? '');
const reference = index(readFileSync(`${dir}/stored_rollup.ndjson`, 'utf8'));
let sameSeq = 0, sameRow = 0;
for (const [key, value] of mine) {
const other = reference.get(key);
if (other?.seq === value.seq) sameSeq++;
if (other?.row === value.row) sameRow++;
}
console.log(JSON.stringify({
part, raw_objects: raw, raw_gets: gets - 16, acked: outcome.ack, retried: outcome.retry,
sessions_published: mine.size, sessions_in_reference: reference.size,
same_session_and_seq: sameSeq, identical_rows: sameRow,
parts_left_behind: [...files.keys()].filter((k) => k.startsWith('rollup-parts/')).length,
wall_ms: ms, peak_rss_mb: Math.round(process.resourceUsage().maxRSS / 1024),
}, null, 1));
```
</details>
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the telemetry receiver's
cron, not the proxy; no client or request-path behaviour changes.
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on the Worker. Rollups are built
by queue consumers instead of inside the cron run; the rollup objects
readers consume are unchanged. A new `rollup-parts/` prefix appears in
the bucket while an hour is in flight.
- Kill switch / disable path: remove the cron trigger or the queue
consumer in `wrangler.toml` and redeploy. Raw `sessions/` is never
modified, so any rollup can be rebuilt.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: redeploy the previous Worker build. Hours it cannot
finish can be filled by the offline backfill, as was done for the
existing gap.
- Deploy steps (not done by this PR):
```
npx wrangler queues create beacon-rollup
npx wrangler queues create beacon-rollup-dlq
npx wrangler deploy
```
Queues need the Workers Paid plan, which this Worker already requires.
Deploying also ships the two merged-but-undeployed Worker changes (#4008
rate limit, #4691 nested allowlist); production is still the 2026-10-08
build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
needed that I know of
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — `node
deploy/beacon/test-rollup.mjs`
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- Nothing alerts on the dead-letter queue or on hours that stay
unrolled.
- The first run on Cloudflare should be watched: check that `rollup/`
gains the previous hour within a few minutes of :05 and that
`rollup-parts/` empties.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
456288da07 |
fix(beacon): enforce the allowlist below the top-level keys (#4691)
## Description
The header of `deploy/beacon/worker.js` promises to drop every field not
on the allowlist before anything is written, but `pick()` only filtered
top-level keys: whatever sat under an allowed key (`session`, `models`,
`config`, ...) was stored verbatim. This adds a nested schema (`NESTED`)
that mirrors `_Session.payload()` / `_payload_v2()` in
`headroom/telemetry/session.py`, so an unknown nested key, a non-scalar
value, or free text under an allowed key is dropped server-side, while
every field the current client sends is kept.
Fixes #4689
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `deploy/beacon/worker.js`: new `NESTED` table, one entry per
`ALLOWED_KEYS` key, describing the nested shape session.py emits. Leaves
are `num` (finite number), `bool`, or `str` (a token of at most 128
characters with no whitespace: slugs, hex ids, public model ids,
versions). `mapOf()` covers objects whose keys are data (`transforms`,
`skips`, `sources`, `failure_statuses`, `errors.by_status`, `hist`); map
keys are held to the same token rule. `trajectory.kinds` is checked
against the closed `_TURN_KINDS` run-length alphabet.
- `pick()` now runs each kept value through `clean()`. `ALLOWED_KEYS`
stays a plain array (the regex in
`tests/test_beacon_schema_v2.py::test_allowlist_covers_every_emitted_key`
still parses it); module load throws if `ALLOWED_KEYS` and `NESTED`
disagree, so a key added to one but not the other fails the deploy
instead of being silently dropped.
- Resource attributes are held to plain token strings (the client only
ever sends `stringValue` there).
- `routing` is emitted by an extension and its shape is not defined in
this repository, so it is bounded rather than described: numbers,
booleans and tokens only, at most 4 levels deep. Giving it a fixed
schema needs the extension's payload and is left to a maintainer.
- `deploy/beacon/test-rollup.mjs`: a new block drives `worker.fetch()`
with the repository's `sample-event.json` (empty lists filled with rows
of the shape session.py emits) and asserts it is stored unchanged; then
injects unknown nested keys, free text, an object where a slug belongs
and a non-scalar resource attribute, and asserts each is dropped.
- No size cap on list or map length was added: the existing 256 KB
body/decompression caps already bound those, and a count cap could drop
real rows (`compression.transforms` is not capped client-side).
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
pytest was run only for the two beacon test files that read `worker.js`
(below); ruff/mypy do not cover the changed JavaScript. The repository
has no JS linter configured for `deploy/beacon`.
### Test Output
Before the fix (new test block, unmodified `worker.js`), `node
test-rollup.mjs`:
```text
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 103 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
AssertionError [ERR_ASSERTION]: a nested field outside the allowlist was stored
+ actual - expected
...
+ prompt: 'secret',
...
+ value: 'https://internal.example/x?token=1 2'
...
+ 'Please summarise this customer email'
```
(The "client fields kept" assertion before it already passed on
unmodified code, as expected.)
After the fix:
```text
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 103 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
nested allowlist: client fields kept, unknown nested keys and free text dropped
ok
```
Revert check: with the `worker.js` change reverted (`git apply -R`) the
script exits 1 on the same assertion; restored, it exits 0.
Python tests that parse `worker.js` (WSL, Python venv, `pytest
tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py`):
```text
tests/test_beacon_schema_v2.py ........................................ [ 66%]
tests/test_beacon_gzip.py .................... [100%]
============================= 60 passed in 31.83s ==============================
```
## Real Behavior Proof
- Environment: Windows 11, Node v26.13.0 (`Request`/`Response` globals),
worker imported as an ES module and its `fetch()` handler called
directly with a stub R2 binding that records `put()`; Python client from
this branch in WSL Ubuntu for the second check.
- Exact command / steps: (1) `deploy/beacon/sample-event.json` with
`session.note = "customer: ACME, see ticket 4411"` and `"please
summarise the attached contract"` appended to `models`, POSTed to
`worker.fetch()`, stored NDJSON read back; run once on `main` and once
on this branch. (2) A busy session built by the real client
(`SessionAggregator` + `build_otlp_logs`, 60 outcomes with 4xx/5xx
statuses, strata labels, passthrough reasons, waste signals, 12
content/tool shape rows, 6 strategies, two `HEADROOM_*` config vars),
POSTed to `worker.fetch()`, stored row compared leaf by leaf against the
payload the client built.
- Observed result: (1) on `main` the stored row contains
`"note":"customer: ACME, see ticket 4411"` and
`"models":["claude-sonnet-5","please summarise the attached contract"]`;
on this branch it contains
`"session":{"id":"4996e02d2b7cafe7","seq":0,"duration_s":135,"turns":4,"ended":"shutdown","final":true}`
and `"models":["claude-sonnet-5"]`, status 204 in both. (2) `real client
session: status 204, 20 top-level keys, 4953 B of JSON, fields lost: 0`,
resource stored with all 7 attributes.
- Not tested: the Workers runtime (workerd) and a `wrangler deploy`; the
deployed worker; the `routing` payload of the real extension (not in
this repository, so the bounded rule was checked only against a made-up
shape); payloads from older released clients (I compared the emitted key
set of every `payload()` revision since
|
||
|
|
5698e91b2a |
chore(security): beacon rate limit, dev environment, weekly CodeQL triage (#4008)
SOC 2 follow-ups from the 2026-10-06 firewall review (R3, R4) and incident INC-2026-001. ## Beacon rate limit (R3) `deploy/beacon/wrangler.toml` said a WAF rule limited `/v1/logs` to 60 req/min/IP. That rule could not exist: headroomlabs.ai is not a Cloudflare zone, so the Worker is only on `workers.dev`. This adds a [Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/) checked in `fetch()` **before** the body is read. - Keyed on one constant, not the IP: the Worker still never reads `cf-connecting-ip` (privacy promise in the file header). Trade-off: a flood also sheds real events for that minute. - 3000 req/min **per Cloudflare location** (~10x today's ~300/min total). - Returns 429, which the client does **not** treat as a gzip refusal (`_GZIP_REFUSED_STATUSES` is 400/415), so rate-limited clients keep compressing. - Fails open if the binding is absent **or the limiter itself errors** (review fix): a rate-limit service outage lets beacons through rather than dropping all of them. `node deploy/beacon/test-rollup.mjs` covers 429 over the limit, fail-open on outage and when unbound, and the single shared key. ## Dev environment (R4) `[env.dev]` deploys `headroom-beacon-dev` bound to the new `headroom-telemetry-dev` bucket (600/min limit). Already deployed and tested: a sample event returns 204, and a 2,500-request burst got 384 × 429. ## Weekly CodeQL triage `.github/workflows/codeql-triage.yml` opens an issue every Monday, assigned to @chopratejas (tejas@headroomlabs.ai), listing open critical/high alerts with checkboxes. Alert #63 sat unowned for five months; this gives alerts an owner and leaves an audit trail. **One issue per week, however often it runs (review fix).** The logic is in `scripts/codeql_triage.sh`: - It keys the issue on the ISO week, using a hidden marker in the body. - It finds the existing issue by listing security-labelled issues, closed ones included. It uses a listing, not search, so an issue created seconds earlier is not missed. - It reuses that issue and never edits, reopens or retitles it, so ticked boxes and closure stand. Alerts opened since then are added once, as a comment. - The jq programs live in quoted heredocs, which clears ShellCheck SC2016 without any shell expansion. - `tests/test_codeql_triage_script.py` runs the real script against a fake `gh`. It covers same-day re-runs, a Friday dispatch on a closed issue with ticked boxes, the next week, and the ISO year boundary (2027-01-01 is in 2026-W53). ### Test output ```text $ pytest -q tests/test_codeql_triage_script.py 6 passed $ node deploy/beacon/test-rollup.mjs rate limit: 429 over, fail-open on outage and when unbound ok $ shellcheck scripts/codeql_triage.sh && actionlint .github/workflows/codeql-triage.yml (clean) ``` ## ⚠️ Before deploying production The live `headroom-beacon` Worker predates #3253 (gzip inflate, schema v2 allowlist). `wrangler deploy` from main will ship that too, not just this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4008" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
db983bacc6 |
fix(beacon): stop Cloudflare logging senders' IP addresses (#4018)
## Description
**In plain words:** our beacon code promises it never keeps anyone's IP
address, and it doesn't read it. But Cloudflare does. With per-request
logging switched on, Cloudflare saves a copy of every beacon request's
details in our Workers Logs, including:
- the sender's **IP address**
- their **city, postal code, and map coordinates**
- their internet provider
That's about 25,000 requests an hour. I confirmed it on 2026-10-07 by
opening one entry in the Cloudflare dashboard (Workers → headroom-beacon
→ Observability).
So the promise in `worker.js` ("never log, store, or forward the source
IP") isn't true right now. The logs expire after a few days, but they
shouldn't exist at all. Since the receiver is open source specifically
so people can check our privacy claims, the setup has to match the code.
## Changes Made
- `deploy/beacon/wrangler.toml`: adds `[observability.logs]
invocation_logs = false`. Cloudflare stops saving per-request details.
Observability stays on, so errors and anything the worker `console.log`s
still show up.
- `deploy/beacon/worker.js`: comment fixes only, no code change.
- Says the worker runs at `headroom-beacon.headroom-beacon.workers.dev`.
`otlp.headroomlabs.ai` is planned but doesn't resolve yet.
- Notes why invocation logs must stay off.
## Deployment Follow-up
1. Deploy: `npx wrangler deploy --config deploy/beacon/wrangler.toml`
2. Check: open a new log entry in Workers → headroom-beacon →
Observability. It should no longer show `cf-connecting-ip`, `x-real-ip`,
or location fields.
Validated with `wrangler deploy --dry-run` (wrangler 4.148.0): config
parses, R2 binding unchanged.
SOC 2: closes the gap in risk R-15 (telemetry privacy).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4018"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
## Type of Change
- [x] Bug fix
- [x] Documentation update
## Testing
Maintainer validation: the TOML parses with `observability.enabled =
true` and `observability.logs.invocation_logs = false`; `node
deploy/beacon/test-rollup.mjs` passes. Cloudflare documents this exact
setting:
https://developers.cloudflare.com/workers/observability/logs/workers-logs/#invocation-logs
## Real Behavior Proof
- Environment: isolated Windows maintainer worktree; contributor
production observation is described above.
- Exact command / steps: TOML parse and existing Worker checks;
production deployment steps are described above.
- Observed result: configuration disables invocation logs while
retaining observability; Worker checks pass.
- Not tested: no production deployment or post-deployment Cloudflare
dashboard verification performed by the maintainer. Merging alone does
not disable logging on the deployed Worker.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; Cloudflare Worker deployment
configuration.
- Minimum rollout channel: not applicable; takes effect on deployment.
- Stable/default behavior changed: automatic invocation logs are
disabled after deployment; application code is unchanged.
- Kill switch / disable path: retain invocation_logs=false to keep
request metadata out of Workers Logs.
- Unsafe override required: none.
- Qualification impact: verify the deployed Worker emits no invocation
entries; existing retained logs are not deleted by this change.
- Rollback path: roll back Worker code while retaining the privacy
setting; re-enabling invocation logs restores collection of request
metadata.
## Review Readiness
- [x] I performed self-review
- [x] This PR is ready for human review
The self-review checkbox records the maintainer's review of the diff and
configuration. Updated-head hosted CI is terminal with no failures;
post-deployment privacy verification remains a deployment follow-up.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
67eb910e38 |
feat(beacon): schema v2 — routing/compression training signal, gzip transport (#3253)
## What Nine new sections on the session heartbeat — `quality`, `hist`, `shapes`, `cache`, `trajectory`, `clients`, `errors`, `strata`, `config` — plus a gzip transport. Everything is a counter, a ratio, or a slug from a fixed list in the source. No prompts, code, file paths, tool names, project names, or error text can appear. ## No regression in v1 Every existing field keeps its name, type, and meaning. `tests/test_beacon_schema_v2.py` locks the v1 surface with literal key lists and a shape-stability invariant, so a v1 field that silently changes type fails there rather than in the corpus six weeks later. The v2 section is built in `_payload_v2()` behind `try/except`, and `_fold_extra()` likewise. A bug in new code can cost the new fields and nothing else. The first version built one dict literal — any raise dropped `tokens.saved` along with it. ## The STRUCT/MAP flip Four fields ship as lists of records rather than objects keyed by data (`clients`, `config`, `strata`, `waste`). DuckDB infers a JSON object as `STRUCT` when the keys are few and identifier-shaped, and as `MAP` otherwise — so a key set that grows with the fleet flips type mid-corpus and `BINDER ERROR`s the reader. Caught against the real 369k-session corpus, not in theory: `clients` inferred `STRUCT(claude_code, codex)` at two harnesses and `MAP(VARCHAR, BIGINT)` at 22, and `clients.cursor` errors on one of them. ## Transport Body is gzipped at level 6 (~6x). The receiver sniffs magic bytes rather than trusting `Content-Encoding`, enforces a decompression-bomb limit *during* the stream read, and leaves the plain path byte-identical for the existing fleet. A `400`/`415` disables gzip for the process and retries once. A `413` does not — an uncompressed retry is 6x larger and guaranteed to fail again. At current rates (~235k heartbeats/day): v1 today 892 MB/day, v2 uncompressed 1,802 MB/day, **v2 gzipped 295 MB/day** — a net reduction against what we send now. ## `headroom telemetry` Prints the real payload, built by the same `_Session.payload()` the beacon uses. \"Go read worker.js\" answers what happens to the data; it does not answer what leaves your machine. Uses `read_install_id()` and `resource_attributes(create_install_id=False)` — the default mints and persists the id, so building the report would itself be the act that identifies the machine. Filtering the id out of the *output* is not enough; by then the file exists. ## Receiver Allowlist gains the nine keys. `MAX_BODY_BYTES` 64KB → 256KB — that figure dated from \"a beacon event is ~2KB\", and worst-case v2 is 81,965 B uncompressed. ## Verification Against the production corpus (369,015 sessions): **14/14** `beacon.sh` reports pass, **18/18** dashboard queries pass, v1 totals byte-identical at `273,495,971,478` tokens saved. `sample-event.json` regenerated as a real v2 payload. The old v1-only version would have passed the post-deploy smoke test whether or not the new allowlist landed; the new one distinguishes the cases 9/9 vs 0/9. Suite: 11,655 passed, 1 failed — `test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`, which reproduces identically on clean `main`. ## Not verified here `node test-rollup.mjs` and `npx wrangler dev` have not been run — no `node` in the authoring environment. `DecompressionStream` is the one path with no fallback and has never been executed. **Both are required before deploy.** ## Deploy order **Worker first, client second.** The worker is backward-compatible with the v1 fleet (allowlist additive, plain path byte-identical, cap only raised). The reverse is not: the worker strips unknown keys at ingest, so v2 data arriving before the allowlist lands is gone, not recoverable by re-rolling. The corpus already carries 6 such rows from a dev proxy. Merging this PR does not ship the client — releases are separate commits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
e0870ef931 |
feat(beacon): hourly R2 compaction, per-strategy savings, and a stack that reports (#2853)
Three beacon changes bundled because they are one story: the corpus got
too
slow to query, and then too coarse to answer the question it was
collected for.
## 1. Hourly compaction (`deploy/beacon`)
The beacon writes one ~1 KB object per heartbeat — **64,987 on
2026-08-06** and
climbing. A full analysis `pull` was ~100k HTTPS round trips for 95 MB:
minutes
of pure per-object latency. Listing the bucket alone took 88 seconds.
Moving the query server-side does not help — R2 SQL reads only Iceberg
tables,
and a pile of tiny files is the pathological case for every query
engine.
Compaction is the fix, and it is Iceberg's own answer to the same
problem.
An hourly cron collapses each **complete** hour of `sessions/` into one
`rollup/dt=…/hh=…/data.ndjson`, keeping the highest-`seq` heartbeat per
`(install, session)`.
| measured on `dt=2026-08-06/hh=14` | before | after |
|---|---|---|
| objects | 3,938 | **1** |
| rows | 3,938 | **1,061** |
| analysis `pull` | minutes | **seconds** |
Hourly rather than daily because every R2 binding call is a subrequest:
a day
is ~65k, an hour is ~4k. Newest-first, so a backlog drains from the
present
backwards and live data never starves behind it; a failing hour is
logged and
skipped rather than blocking every older hour behind it. **Raw objects
are
never deleted**, so any rollup is rebuildable by deleting it.
Backfill runs to the **oldest surviving raw day**, not a fixed window. A
fixed
lookback strands everything older than it the moment analysis stops
reading
`sessions/`: the raw objects are still there, but nothing would ever
compact
them, so they disappear from every report. `oldestRawDay()` finds that
floor in
one delimited LIST, and the rollup listing starts from it — so the work
is
bounded by retention rather than by total history.
Three failure modes the tests pin down, because each one is silent:
- A **failed `get`** is transient, so the hour throws and writes
nothing. A
rollup is built once and trusted forever, so a short read would quietly
become the permanent record.
- A **corrupt record** loses only itself. This Worker wrote that content
with
`JSON.stringify`; it will never become valid, so blocking on it strands
the
hour instead of the record.
- An **empty hour** writes an `empty` marker. Without one the hour stays
"missing" and is re-listed on every run forever.
`test-rollup.mjs` asserts the Worker's dedup picks exactly the same rows
as the
analysis-side `QUALIFY`. If those two ever disagree the reports go
quietly
wrong rather than loudly broken, which is why that check exists. Its
stub
paginates at 3 keys so the list cursor loop — load-bearing at the real
~4,000
objects/hour — runs in every case.
## 2. Per-strategy savings (`compression.by_strategy`)
`compression.transforms` counts *invocations*, which cannot distinguish
a
compressor that saved 60% from one that ran constantly and saved
nothing. The
fleet's top transform by count contributes an unknown share of
`tokens.saved`.
It is worse than that in practice. Transform labels are slugged with
`split(":", 1)[0]`, so every `router:<strategy>:<detail>` label
collapses into a
single `router` bucket. On 2026-08-08 that bucket held **19.1 M of the
day's
transform counts, across 8,528 of 9,616 sessions** — the compressors
that do
most of the work are indistinguishable from each other, by name as well
as by
yield:
| transform | n | sessions |
|---|---|---|
| `router` | 19,108,118 | 8,528 |
| `anthropic` | 688,124 | 4,734 |
| `output_shaper` | 548,017 | 1,120 |
No question about which strategy is earning its keep can be answered
from that,
which is what this field is for.
The measurement already existed. `PrometheusMetrics.record_compression`
is the
configured `CompressionObserver` and already accumulates
`tokens_saved_by_strategy` on the hot path — the numbers just never left
the
process. This forwards from that one chokepoint rather than adding a
second
observer and a second measurement pass. The paths that have **no**
observer
configured (MCP server, LangGraph, Strands hooks, the transform
pipeline) get
`BeaconCompressionObserver` passed directly.
Compression runs on the executor thread *before* that request's outcome
reaches
`record()`, so events are **staged** into module state and drained by
the next
outcome. Staging is what makes two things true at once:
- The first turn of a session still reports its numbers — otherwise
every
session's opening turn, and any session short enough to be one turn,
would
report nothing.
- A compression event **never opens a session**. An abandoned request
would
otherwise emit a phantom `turns=0` row with all-zero tokens, inflating
fleet
session and install counts.
Staging takes a dedicated mutex the request path never touches, so the
fan-out
stays off the aggregator's lock and `record_compression` keeps its
"synchronous + lock-free" contract.
```json
"by_strategy": [
{"strategy": "code_aware", "n": 1, "tokens_in": 800, "tokens_out": 800},
{"strategy": "smart_crusher", "n": 2, "tokens_in": 1500, "tokens_out": 700}
]
```
A **list of records, sorted by strategy** — not an object keyed by
strategy.
Keyed shapes change type as keys accumulate: DuckDB infers a STRUCT
under ~24
keys and a MAP over it, so the analysis query breaks on the day the
fleet
picks up a 25th strategy. Sorted so heartbeats are byte-comparable.
**These do not sum to `tokens.saved`,** and the field comment says so:
strategies compose (the router routes, a strategy runs inside it) so the
same
text is measured more than once. A row means "of what this strategy was
handed,
it removed this much" — a per-strategy yield, not a share of the total.
A
strategy that saved nothing still appears; dropping it would make every
strategy look effective.
## 3. `headroom.stack`
`resource_attributes()` was called with no arguments at its one call
site, so
`headroom.stack` was absent from **all 24,040 sessions** in the corpus
while
`detect_stack` sat unused — dead code on both ends of a wire nobody
connected.
The fleet was unsegmentable by agent, which is the question the corpus
is asked
most often.
Environment detection alone is not enough. It answers `wrap_claude` only
under
`headroom wrap`; every install that points an agent at a persistent
proxy — the
common deployment — reports `proxy`, which segments nothing. The
per-request
`X-Headroom-Stack` slugs are the only signal that names the harness
there, so
`record_stack()` stages them the same way and feeds `detect_stack`'s
`by_stack`
branch:
```
9x wrap_claude, 1x wrap_cursor -> wrap_claude (dominant harness wins)
5x wrap_claude, 5x wrap_cursor -> mixed
no per-request signal -> proxy (environment fallback)
junk slug -> dropped before staging
```
## Privacy
The strategy string is slugged through the same `_safe_slug` as skip
reasons
and capped at `MAX_STRATEGIES`, because the observer protocol takes a
free
string and an extension could otherwise invent keys per request. Stack
slugs
are normalized and capped the same way.
**Deliberately not collected:** the tool names in
`smart_crush:<count>:<names>`.
Those are user-defined MCP identifiers and can name internal tooling
(`acme_deploy_prod`). They stay stripped by the existing `split(":",
1)[0]`,
and this PR does not widen it. No new key was needed in `worker.js` —
`by_strategy` nests under the already-allowlisted `compression`, and
`headroom.stack` was already in `ALLOWED_RESOURCE`.
Nothing here deletes or rewrites existing data: the Worker only ever
writes,
`sessions/` is never pruned by it, and readers merge old and new shapes
with
`union_by_name`, so pre-change heartbeats keep reading with the new
fields null.
## Verification
- `python -m headroom.telemetry.session` — self-check covers staging,
the
no-phantom-session case, drain exhaustiveness, a 0%-yield strategy
staying
visible, the cardinality cap, slug safety, and dominant/mixed/junk stack
resolution
- `node test-rollup.mjs /tmp/hr` — 3,938 real corpus objects → 1,061
sessions
in 1 object, plus the pagination, partial-read, corrupt-record,
empty-hour
and `oldestRawDay` cases
- 51 passing in `test_compression_observability`,
`test_prometheus_obs_counters`,
`test_telemetry_context`, `test_compression_strategy_outcomes`
- Consumer side exists and is checked: `beacon.sh by_strategy` in
headroom-beacon-stats reads the field end to end (sessions, installs,
invocations, tokens in/out, yield %), verified against a synthetic
parquet for
the cases that matter — aggregation across installs, a 0%-yield strategy
staying visible, pre-field sessions dropping out rather than erroring.
It is
guarded on the column, so it prints an instruction instead of a binder
error
until a release carrying this PR reaches the fleet.
- Deployed and running against the live corpus on the `5 * * * *`
trigger
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
7940c05ebf |
feat(beacon): allowlist the routing summary key (#2818)
One line in the receiver's allowlist. No client change; the proxy's own
payload is untouched.
## Why
A routing extension sees things the proxy alone cannot, and they are all
measurements rather than opinions:
- **Empirical `min_cacheable` per provider.** Fireworks, Together and
DeepInfra publish no minimum and litellm carries no value for them, so a
router has to guess. But the number is directly observable — send prefix
length L, see whether the repeat reports cached tokens. Across enough
installs the step function falls out.
- **TTL survival.** Currently modelled as a constant.
- **Conversation length distribution.** The horizon is the only free
parameter in a cache-aware cost model, and it decides the answer: at a
900-token prefix, 1 remaining turn and 20 remaining turns route to
different models.
- **Predicted vs actual cache hits.** Every response carries
`cache_read_input_tokens`. Comparing it to what was predicted is the
only way to find out when the cost model is lying.
## What lands here
`'routing'` added to `ALLOWED_KEYS`, and the comment above the list
corrected — it claimed the set mirrors `_Session.payload()`, which is no
longer the whole story now that an extension can emit its own event
carrying one of these keys.
The ordering constraint is the reason this is its own PR: **allowlisting
is a write-side gate**, so anything sent before the key exists is
dropped and unrecoverable. This has to be deployed before any client
starts emitting it, not alongside.
## Shape of the block
Same rule as every other key — counters and model ids, no free text:
```json
"routing": {
"harness": "claude-code",
"decisions": 47, "would_change": 12, "enforced": 9, "holdout": 3,
"at_free_boundary": 4, "cross_protocol": 0,
"picked": {"claude-haiku-4-5": 12, "claude-opus-5": 35},
"requested": {"claude-opus-5": 47},
"mean_prefix_tokens": 7514,
"measured_cost": 0.0236, "modelled_cost": 0.0376,
"cache_read_tokens": 3200, "cache_write_tokens": 0,
"predicted_hits": 4, "actual_hits": 4
}
```
`measured_cost` comes from the provider's own usage; `modelled_cost`
from the router's cost function. They stay separate because the
difference is the only thing that means anything.
The extension's `reason` string is deliberately absent. It is
code-generated, so it carries no user content, but it is unbounded — it
stays out rather than being reasoned about.
`holdout` is the count of turns deliberately left unrouted as a control.
Without it the rest is observational: once a router is acting on every
request, the corpus is entirely that router's own policy.
`sample-event.json` is unchanged on purpose — it mirrors
`_Session.payload()`, which does not produce this key, and adding it
there would suggest the proxy emits it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
2954e37048 |
fix(beacon): split session failures by status code (#2815)
## Description
The session beacon reports `failures` as a single count, incremented
whenever a turn ends `>= 500` (`headroom/telemetry/session.py`). Across
the current corpus that reads **3,969 failures on 595,445 turns
(0.67%)** — and the number cannot answer the only question anyone asks
of it: an Anthropic `529` is the provider shedding load and there is
nothing to fix; a `500` is usually ours. Today the two are
indistinguishable, so diagnosis falls back to inference from time-of-day
curves and per-install concentration.
This counts the status alongside the total.
```json
"failures": 3,
"failure_statuses": {"529": 2, "500": 1}
```
Motivating investigation on the live corpus (0.67% of turns, 6% of
sessions, 63% of all failures from 48 installs, a 2.5% plateau at 08–11
UTC decaying to 0.03% during the fleet's busiest hour) strongly suggests
provider-side 529 after retry exhaustion — but "strongly suggests" is
exactly the gap this field closes.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/telemetry/session.py`** — `_Session.failure_statuses`,
incremented next to `failures` in `record_outcome`. Keys are the bare
status string for the 5xx range, `"other"` beyond it. Emitted as a
sibling of `failures` in `payload()`.
- **`deploy/beacon/worker.js`** — `failure_statuses` added to
`ALLOWED_KEYS`. Without this the ingest allowlist silently drops it.
- **`deploy/beacon/sample-event.json`** — sample carries the new key in
OTLP `kvlistValue` form.
### Why no slug bounding
`skips` runs values through `_safe_slug` because they arrive as free
strings. A status code is an `int` the proxy itself produced; the `500
<= status < 600` check is what keeps a garbage value from inventing map
keys. Nothing here is user-derived, so the field stays content-free.
### Why `schema_version` stays 1
Additive, matching the precedent set by #2796, which added
`tokens.tool_saved` and the two `all_layers_*` rates without a bump.
Bumping signals a break to consumers when nothing about older rows
becomes invalid.
## Testing
- [x] Unit tests pass (`pytest`) — the module's own self-check, extended
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — see note
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m headroom.telemetry.session
ok
$ ruff check headroom/telemetry/session.py
All checks passed!
$ ruff format --check headroom/telemetry/session.py
1 file already formatted
$ mypy --python-version 3.12 headroom/telemetry/session.py
Success: no issues found in 1 source file
# --python-version 3.12 only to skip a pre-existing numpy-stub syntax error the
# repo's python_version = "3.10" triggers locally; unrelated to this diff.
$ node --check deploy/beacon/worker.js # ok
$ python -c "import json; json.load(open('deploy/beacon/sample-event.json'))" # parses
```
The self-check in `headroom/telemetry/session.py` now records two 529s
and one 500 and asserts both the total and the split:
```python
assert emitted[-1]["failures"] == 3
assert emitted[-1]["failure_statuses"] == {"529": 2, "500": 1}
```
plus `assert event["failure_statuses"] == {}` on the clean-session path.
## Real Behavior Proof
- **Environment:** macOS 25.4.0, Python 3.12 venv, this branch.
- **Exact command / steps:** drive `SessionAggregator` with three
failing outcomes and encode the payload through the same `_any_value`
the wire uses.
```text
payload: 3 {'529': 2, '500': 1}
otlp : {"kvlistValue": {"values": [{"key": "529", "value": {"intValue": "2"}},
{"key": "500", "value": {"intValue": "1"}}]}}
```
The OTLP form matches `deploy/beacon/sample-event.json` byte-for-byte in
shape, and `unwrap()` in `worker.js` turns `kvlistValue` back into a
plain object, so it lands in R2 as `{"529": 2, "500": 1}` — the same
shape as `skips`, which DuckDB reads as `MAP(VARCHAR, BIGINT)`.
- **Observed result:** as above. Verified against the live corpus that
schema evolution here is already routine — 3,836 of 3,884 existing rows
have `rates.all_layers_saved_pct = NULL` from #2796 landing mid-corpus,
and every report still runs.
- **Not tested:** the deployed Worker (no staging R2 binding locally);
`node --check` covers syntax only. The allowlist addition is one array
entry consumed by the existing `pick()`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Screenshots (if applicable)
N/A — wire-format change, covered by the output above.
## Additional Notes
**Deploy order matters.** The Worker allowlist drops unknown keys, so
`deploy/beacon/worker.js` must be deployed *before* a client release
that emits the field — otherwise it is discarded at the door. No
corruption either way, just missing data until the Worker catches up.
**Old data is unaffected.** R2 objects are immutable NDJSON written per
request; nothing rewrites history. The corpus reader already passes
`union_by_name = true`, which fills the column with NULL for rows
written before this ships.
|
||
|
|
9cfb00838a |
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|