mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:
1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.
| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |
- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.
**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.
**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.
## Testing
- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```
Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.
## Real Behavior Proof
- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:
```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
{"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
{"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
{"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
{"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
{"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```
Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
- Linux, Windows, containers and CI.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
778 lines
33 KiB
JavaScript
778 lines
33 KiB
JavaScript
/**
|
|
* Headroom telemetry beacon receiver.
|
|
*
|
|
* This file is open source on purpose. It is the other half of the promise
|
|
* made in headroom/telemetry/session.py: users can read exactly what the
|
|
* client sends AND exactly what happens to it on arrival. "Trust us" is not a
|
|
* privacy policy.
|
|
*
|
|
* Deployed at headroom-beacon.headroom-beacon.workers.dev (otlp.headroomlabs.ai
|
|
* is planned but not yet routed; see wrangler.toml). Three jobs:
|
|
*
|
|
* 1. Allowlist. Drop every field not on ALLOWED_KEYS, and every nested
|
|
* field not described by NESTED, before anything is written. This is the only privacy control that works retroactively —
|
|
* if a future client version ships a bug that leaks a field, we cannot
|
|
* patch the installs already in the wild, but we can stop storing it
|
|
* here in one deploy.
|
|
*
|
|
* 2. Flatten. OTLP AnyValue nesting is portable but miserable to query
|
|
* ({"kvlistValue":{"values":[{"key":"tokens",...}]}}). We keep OTLP on
|
|
* the wire so the backend stays vendor-swappable, and store plain JSON so
|
|
* DuckDB can read it without unwrapping anything.
|
|
*
|
|
* 3. Fan out. R2 for the durable corpus; optionally a metrics vendor for
|
|
* dashboards. Adding a destination is one more call here — never a
|
|
* client release.
|
|
*
|
|
* What this deliberately does NOT do: log, store, or forward the source IP.
|
|
* Cloudflare offers it as cf-connecting-ip; it is the one field that would
|
|
* deanonymise install_id, so it is never read. Cloudflare's own per-request
|
|
* invocation logs would record it regardless of this code, which is why
|
|
* wrangler.toml turns them off (observability.logs.invocation_logs = false).
|
|
*/
|
|
|
|
// Mostly mirrors the payload built by _Session.payload(); an extension may
|
|
// also emit its own event carrying one of these top-level keys. A key absent
|
|
// here is dropped, not stored. Adding a metric means adding it here first —
|
|
// that friction is the point, and it is also the only privacy control that
|
|
// works retroactively, so it must land BEFORE any client starts sending the
|
|
// key or that traffic is silently discarded and unrecoverable.
|
|
const ALLOWED_KEYS = [
|
|
'schema_version',
|
|
'session',
|
|
'tokens',
|
|
'rates',
|
|
'compression',
|
|
'skips',
|
|
'sources',
|
|
'providers',
|
|
'models',
|
|
'failures',
|
|
'failure_statuses',
|
|
// Model-routing summary. Emitted by a routing extension rather than by the
|
|
// proxy itself -- see proxy/route_advice.py for the decision seam. Same rule
|
|
// as everything above: counters and model ids, no free text. Allowlisted
|
|
// here so the corpus can answer what the proxy alone cannot -- a provider's
|
|
// real minimum cacheable prefix, how long a cache actually survives, and how
|
|
// far predicted cache hits are from the ones that happened.
|
|
'routing',
|
|
|
|
// ---- schema v2 -----------------------------------------------------------
|
|
//
|
|
// Additive. Every v1 key above is untouched, so a v1 client keeps storing
|
|
// exactly what it stores today and rows from the two versions union cleanly
|
|
// (`schema_version` separates them, `union_by_name` handles the rest).
|
|
//
|
|
// Read the ordering rule at the top of this list literally: this deploy must
|
|
// go out BEFORE the client release that starts sending these, or the first
|
|
// weeks of the new signals are dropped on arrival and cannot be recovered.
|
|
// Removing one line here is also the rollback -- it reverts that signal for
|
|
// every install in the wild without a client release, which is the whole
|
|
// reason the allowlist is server-side.
|
|
|
|
// Regret. `reread_compressed_tokens` is the counter-pressure on `saved_pct`:
|
|
// tokens the agent had to fetch again because compression removed them.
|
|
// Without it a compression ratio is unfalsifiable.
|
|
'quality',
|
|
// Fixed-bucket distributions for quantities otherwise reported only as
|
|
// session sums. Ships its own bucket edges, so no reader needs to know which
|
|
// client version wrote a row.
|
|
'hist',
|
|
// (content_type x strategy x yield), and the same for JSON tool output keyed
|
|
// by structural shape. The input term the corpus has never had.
|
|
// Structural descriptors only -- never TOIN's structure_hash, which is a
|
|
// digest of field names.
|
|
'shapes',
|
|
// Prompt-cache survival by gap since the previous turn: the empirical TTL
|
|
// and minimum-prefix curve per provider.
|
|
'cache',
|
|
// Session shape over time -- log-spaced turn buckets plus a run-length
|
|
// encoded turn-kind string from a closed seven-letter alphabet.
|
|
'trajectory',
|
|
// Identified client harness, per turn. `headroom.stack` answers this by
|
|
// detection and resolves to a literal "proxy" for most of the fleet.
|
|
'clients',
|
|
// 4xx, kept strictly separate from the 5xx-only `failures` above so neither
|
|
// metric changes meaning. 429 rate is the routing signal here.
|
|
'errors',
|
|
// output_shaper A/B strata as validated enums. Model family is not among
|
|
// them -- see the note in session.py's payload().
|
|
'strata',
|
|
// Allowlisted, slug-validated configuration, so the corpus can tell
|
|
// "compression underperformed" from "compression was switched off".
|
|
'config',
|
|
// The session's tokens split by harness and by model. `clients` and `models`
|
|
// only say who took part, so a session that mixed two could not be divided
|
|
// or priced. `client_usage` covers every turn: traffic with no identified
|
|
// harness is a row of its own instead of a gap.
|
|
'client_usage',
|
|
'model_usage',
|
|
// The harness's own release, numeric parts only.
|
|
'client_versions',
|
|
// Age of the install id in days, and whether it runs in a container or CI
|
|
// job: separates a returning user from an id minted on every run.
|
|
'install',
|
|
];
|
|
|
|
// Resource attributes we keep. Same rule: allowlist, not denylist.
|
|
const ALLOWED_RESOURCE = [
|
|
'service.name',
|
|
'service.version',
|
|
'headroom.install_id',
|
|
'headroom.install_mode',
|
|
'headroom.stack',
|
|
'os.type',
|
|
'host.arch',
|
|
];
|
|
|
|
// What may sit UNDER each allowed key. Without this the allowlist stops at the
|
|
// top level and a nested field a client bug adds is stored verbatim -- the very
|
|
// leak the header says one deploy here can stop. Mirrors _Session.payload() and
|
|
// _payload_v2() in headroom/telemetry/session.py. A nested key absent here is
|
|
// dropped, so the ordering rule on ALLOWED_KEYS applies to it too: teach it
|
|
// here before a client sends it.
|
|
//
|
|
// Leaves are 'num' (a finite number), 'bool', or 'str': a token of at most 128
|
|
// characters with no whitespace, which is every string the client sends (slugs,
|
|
// hex ids, public model ids, versions). Free text does not fit it. `[spec]` is
|
|
// a list of spec; mapOf(spec) is an object whose KEYS are data (transform
|
|
// names, skip reasons, status codes), each key itself held to 'str'.
|
|
const MAP = Symbol('map');
|
|
const mapOf = (spec) => ({ [MAP]: spec });
|
|
const nums = (...keys) => Object.fromEntries(keys.map((k) => [k, 'num']));
|
|
|
|
// A harness name: the client only sends ones from a closed list, so hold the
|
|
// field to that shape instead of the wider 'str'.
|
|
const SLUG = /^[a-z][a-z0-9_]{0,31}$/;
|
|
// One turn-and-token row of `client_usage` / `model_usage`: _USAGE_FIELDS in
|
|
// session.py.
|
|
const USAGE = nums(
|
|
'n', 'original', 'saved', 'tool_saved', 'output', 'cache_read', 'cache_write',
|
|
'cache_write_inferred', 'cache_write_5m', 'cache_write_1h', 'uncached'
|
|
);
|
|
|
|
const NESTED = {
|
|
schema_version: 'num',
|
|
session: {
|
|
id: 'str', seq: 'num', duration_s: 'num', turns: 'num', ended: 'str', final: 'bool',
|
|
},
|
|
tokens: nums(
|
|
'original', 'attempted', 'input', 'output', 'saved', 'tool_saved',
|
|
'cache_read', 'cache_write', 'uncached'
|
|
),
|
|
rates: nums(
|
|
'saved_pct', 'eligible_pct', 'yield_pct', 'all_layers_saved_pct',
|
|
'all_layers_yield_pct', 'cache_read_pct', 'overhead_pct'
|
|
),
|
|
compression: {
|
|
transforms: mapOf('num'),
|
|
by_strategy: [{ strategy: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
|
|
...nums(
|
|
'overhead_ms_total', 'latency_ms_total', 'overhead_ms_per_turn',
|
|
'latency_ms_per_turn', 'passthrough_turns', 'response_cache_hits'
|
|
),
|
|
},
|
|
skips: mapOf('num'),
|
|
sources: mapOf('num'),
|
|
providers: ['str'],
|
|
models: ['str'],
|
|
failures: 'num',
|
|
failure_statuses: mapOf('num'),
|
|
// Emitted by an extension, not by session.py, so this repository does not
|
|
// define its shape. Bounded rather than described: numbers, booleans and
|
|
// tokens only, at most ROUTING_DEPTH levels deep. It gets a real schema here
|
|
// once the extension's payload is written down.
|
|
routing: 'any',
|
|
quality: {
|
|
...nums('reread_tokens', 'reread_compressed_tokens', 'waste_turns'),
|
|
waste: [{ kind: 'str', tokens: 'num' }],
|
|
},
|
|
hist: mapOf({ edges: ['num'], counts: ['num'] }),
|
|
shapes: {
|
|
by_content: [{ content: 'str', strategy: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
|
|
by_tool: [{ shape: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
|
|
truncated: 'bool',
|
|
},
|
|
cache: {
|
|
gap_edges_s: ['num'], hits: ['num'], misses: ['num'],
|
|
...nums('write_5m', 'write_1h', 'inferred_turns'),
|
|
},
|
|
trajectory: {
|
|
buckets: 'num', turns: ['num'], tokens: ['num'], saved: ['num'],
|
|
// Runs over the closed _TURN_KINDS alphabet, at most MAX_TRAJECTORY_RUNS.
|
|
kinds: /^(?:[cfepzso]\d{1,9}){0,64}$/,
|
|
kinds_truncated: 'bool',
|
|
msgs_max: 'num',
|
|
},
|
|
clients: [{ client: 'str', n: 'num' }],
|
|
errors: { count: 'num', by_status: mapOf('num') },
|
|
strata: [{ dim: 'str', value: 'str', n: 'num' }],
|
|
config: [{ key: 'str', value: 'str' }],
|
|
client_usage: [{ client: SLUG, ...USAGE }],
|
|
model_usage: [{ model: 'str', ...USAGE }],
|
|
client_versions: [{ client: SLUG, version: /^\d{1,4}(?:\.\d{1,4}){0,2}$/, n: 'num' }],
|
|
install: { age_days: 'num', container: 'bool', ci: 'bool' },
|
|
};
|
|
|
|
// A key allowed at the top but given no shape would be dropped in full. Fail
|
|
// the deploy instead of shipping that.
|
|
if ([...ALLOWED_KEYS].sort().join() !== Object.keys(NESTED).sort().join()) {
|
|
throw new Error('beacon: ALLOWED_KEYS and NESTED disagree');
|
|
}
|
|
|
|
const TOKEN = /^[\w.:/@+|-]{0,128}$/;
|
|
// A slug-shaped string can still be a URL or a credential. Model ids and the
|
|
// other slugs the client sends never carry a scheme or a well-known key prefix,
|
|
// so those are dropped even though they pass TOKEN.
|
|
const SECRETISH = /:\/\/|^(?:sk|pk|rk)[-_]|^(?:ghp|gho|ghu|ghs|ghr)_|^github_pat_|^glpat-|^xox[abposr]-|^(?:AKIA|ASIA)[A-Z0-9]{12}|^AIza|^hf_|^eyJ/i;
|
|
const isToken = (s) => TOKEN.test(s) && !SECRETISH.test(s);
|
|
const ROUTING_DEPTH = 4;
|
|
|
|
/** `value` reduced to what `spec` allows, or undefined when none of it is. */
|
|
function clean(spec, value, depth = 0) {
|
|
if (spec === 'any') {
|
|
if (depth > ROUTING_DEPTH) return undefined;
|
|
spec = Array.isArray(value) ? ['any']
|
|
: value && typeof value === 'object' ? mapOf('any')
|
|
: typeof value === 'string' ? 'str'
|
|
: typeof value === 'number' ? 'num' : 'bool';
|
|
}
|
|
if (spec === 'num') return Number.isFinite(value) ? value : undefined;
|
|
if (spec === 'bool') return typeof value === 'boolean' ? value : undefined;
|
|
if (spec === 'str') return typeof value === 'string' && isToken(value) ? value : undefined;
|
|
if (spec instanceof RegExp) {
|
|
return typeof value === 'string' && spec.test(value) ? value : undefined;
|
|
}
|
|
if (Array.isArray(spec)) {
|
|
if (!Array.isArray(value)) return undefined;
|
|
return value.map((v) => clean(spec[0], v, depth + 1)).filter((v) => v !== undefined);
|
|
}
|
|
if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined;
|
|
const out = {};
|
|
if (spec[MAP]) {
|
|
for (const [key, v] of Object.entries(value)) {
|
|
const kept = isToken(key) ? clean(spec[MAP], v, depth + 1) : undefined;
|
|
if (kept !== undefined) out[key] = kept;
|
|
}
|
|
return out;
|
|
}
|
|
for (const [key, s] of Object.entries(spec)) {
|
|
if (!Object.hasOwn(value, key)) continue;
|
|
const kept = clean(s, value[key], depth + 1);
|
|
if (kept !== undefined) out[key] = kept;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
// Cap on what ARRIVES, compressed or not.
|
|
//
|
|
// Was 64KB, chosen when the comment above it read "a beacon event is ~2KB" --
|
|
// i.e. 32x the payload. Schema v2 made a typical event ~8KB uncompressed, and
|
|
// a session that fills every table (the shape cross product is 10 content
|
|
// types x 13 strategies) reaches ~82KB. That silently eroded the multiple to
|
|
// under 1x: a busy session sending uncompressed -- the kill switch is set, or
|
|
// the gzip fallback fired -- would have been answered 413 and, correctly, not
|
|
// retried, so the event would simply be lost.
|
|
//
|
|
// Restored to ~3x the worst legitimate payload. This is not the abuse control
|
|
// and never was: the endpoint is unauthenticated by design and the RATE_LIMIT
|
|
// binding (see fetch() and wrangler.toml) is what bounds a bad actor.
|
|
const MAX_BODY_BYTES = 256 * 1024;
|
|
|
|
// Cap on what an arriving body EXPANDS to. `raw.byteLength` cannot see this:
|
|
// a small gzip can expand to gigabytes, and a decompression bomb that is only
|
|
// noticed after it has been decompressed has already won. Enforced while the
|
|
// stream is read, not after. ~3x the worst legitimate payload, same as above.
|
|
const MAX_DECOMPRESSED_BYTES = 256 * 1024;
|
|
|
|
/**
|
|
* Decompress a gzip body, refusing anything that expands past `limit`.
|
|
*
|
|
* Counts bytes as chunks arrive and aborts mid-stream, so a bomb costs the
|
|
* limit rather than whatever it would have expanded to.
|
|
*/
|
|
export async function inflate(buffer, limit) {
|
|
const stream = new Response(buffer).body.pipeThrough(new DecompressionStream('gzip'));
|
|
const reader = stream.getReader();
|
|
const chunks = [];
|
|
let total = 0;
|
|
try {
|
|
for (;;) {
|
|
const { done, value } = await reader.read();
|
|
if (done) break;
|
|
total += value.byteLength;
|
|
if (total > limit) throw new Error('decompressed body too large');
|
|
chunks.push(value);
|
|
}
|
|
} finally {
|
|
// Releases the stream on the bomb path; a no-op once it has closed.
|
|
reader.cancel().catch(() => {});
|
|
}
|
|
const joined = new Uint8Array(total);
|
|
let offset = 0;
|
|
for (const chunk of chunks) {
|
|
joined.set(chunk, offset);
|
|
offset += chunk.byteLength;
|
|
}
|
|
return new TextDecoder().decode(joined);
|
|
}
|
|
|
|
/** OTLP AnyValue -> plain JS. The inverse of _any_value() in session.py. */
|
|
function unwrap(value) {
|
|
if (value == null) return null;
|
|
if ('stringValue' in value) return value.stringValue;
|
|
if ('boolValue' in value) return value.boolValue;
|
|
if ('intValue' in value) return Number(value.intValue);
|
|
if ('doubleValue' in value) return value.doubleValue;
|
|
if ('arrayValue' in value) return (value.arrayValue.values || []).map(unwrap);
|
|
if ('kvlistValue' in value) {
|
|
const out = {};
|
|
for (const kv of value.kvlistValue.values || []) out[kv.key] = unwrap(kv.value);
|
|
return out;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function pick(obj, allowed, specOf) {
|
|
const out = {};
|
|
if (!obj || typeof obj !== 'object') return out;
|
|
for (const key of allowed) {
|
|
if (!(key in obj)) continue;
|
|
const kept = clean(specOf(key), obj[key]);
|
|
if (kept !== undefined) out[key] = kept;
|
|
}
|
|
return out;
|
|
}
|
|
|
|
/** OTLP ExportLogsServiceRequest -> flat, allowlisted records. */
|
|
function extract(payload) {
|
|
const records = [];
|
|
for (const rl of payload.resourceLogs || []) {
|
|
const resource = {};
|
|
for (const attr of rl.resource?.attributes || []) {
|
|
resource[attr.key] = unwrap(attr.value);
|
|
}
|
|
// Every resource attribute the client sends is a plain string.
|
|
const cleanResource = pick(resource, ALLOWED_RESOURCE, () => 'str');
|
|
|
|
for (const sl of rl.scopeLogs || []) {
|
|
for (const rec of sl.logRecords || []) {
|
|
const body = unwrap(rec.body);
|
|
if (!body || typeof body !== 'object') continue;
|
|
records.push({
|
|
...pick(body, ALLOWED_KEYS, (key) => NESTED[key]),
|
|
resource: cleanResource,
|
|
// Server-stamped. A client clock can be wrong or forged; this is the
|
|
// timestamp partitioning and retention actually rely on.
|
|
received_at: new Date().toISOString(),
|
|
});
|
|
}
|
|
}
|
|
}
|
|
return records;
|
|
}
|
|
|
|
// ----------------------------------------------------------------- rollup --
|
|
//
|
|
// The corpus is one object per heartbeat. Analysis reads rollup/**, never
|
|
// sessions/**: each COMPLETE hour is collapsed into one object under rollup/,
|
|
// keeping only the highest-seq heartbeat per (install, session). Raw is left
|
|
// exactly as written, so any rollup can be rebuilt by deleting it.
|
|
//
|
|
// One invocation cannot do an hour on its own, and that is measured, not
|
|
// feared. A Worker may have six requests in flight, so reads ran at ~12
|
|
// objects/s whatever the code asked for, and a cron run is stopped at 15
|
|
// minutes: ~11k objects. The fleet passed that on 2026-08-19 and by October 98%
|
|
// of hours were being killed mid-read and writing nothing.
|
|
//
|
|
// So an hour is cut into slices and each slice is its own queue message, which
|
|
// means its own invocation with its own six connections and its own 15 minutes:
|
|
//
|
|
// scheduled() finds hours with no rollup and enqueues one message per slice
|
|
// rollupSlice() reads one slice, writes rollup-parts/<hour>/<slice>
|
|
// finalizeHour() once every slice is in, merges them into rollup/<hour>/
|
|
//
|
|
// Slices are the first hex character of the object name. Names are
|
|
// crypto.randomUUID() (see fetch()), so the sixteen slices are even, and the
|
|
// split needs nothing but a LIST prefix — which is also why it works on every
|
|
// object already in the bucket. Raise to two characters (256 slices) when a
|
|
// slice stops fitting; the per-slice cost falls sixteen-fold.
|
|
|
|
const SLICES = '0123456789abcdef';
|
|
// Requests in flight per invocation. Six is the platform's ceiling: asking for
|
|
// more only queues inside the runtime.
|
|
const POOL = 6;
|
|
// A queued hour that still has no rollup after this long is enqueued again:
|
|
// the safety net for a slice that ran out of retries. A second copy of a slice
|
|
// that did finish is cheap, because queue() reuses the part already written.
|
|
const REQUEUE_AFTER_MS = 3 * 3600_000;
|
|
// Hours enqueued per cron run, sized so they drain well inside REQUEUE_AFTER_MS
|
|
// (200 hours is 3,200 slices of a couple of minutes each, run in parallel). A
|
|
// bigger backlog is not stuck, the next run simply continues it.
|
|
const MAX_HOURS_PER_RUN = 200;
|
|
// Wait before a failed slice is tried again, so a short R2 outage does not burn
|
|
// every retry within seconds.
|
|
const RETRY_DELAY_S = 60;
|
|
// R2 deletes at most this many keys per call.
|
|
const DELETE_BATCH = 1000;
|
|
|
|
const partition = (d) =>
|
|
`dt=${d.toISOString().slice(0, 10)}/hh=${d.toISOString().slice(11, 13)}`;
|
|
|
|
/** Every object under a prefix, following the cursor. */
|
|
async function listAll(env, options) {
|
|
const objects = [];
|
|
let cursor;
|
|
do {
|
|
const page = await env.CORPUS.list({ ...options, cursor });
|
|
objects.push(...page.objects);
|
|
cursor = page.truncated ? page.cursor : undefined;
|
|
} while (cursor);
|
|
return objects;
|
|
}
|
|
|
|
/**
|
|
* Run `fn` over `items` with at most POOL calls in flight. Rejects on the first
|
|
* failure and starts nothing new after it: the slice is going to be retried
|
|
* whole, so further reads would be thrown away.
|
|
*/
|
|
async function pool(items, fn) {
|
|
let next = 0;
|
|
let failed = false;
|
|
const worker = async () => {
|
|
while (!failed && next < items.length) {
|
|
try {
|
|
await fn(items[next++]);
|
|
} catch (err) {
|
|
failed = true;
|
|
throw err;
|
|
}
|
|
}
|
|
};
|
|
await Promise.all(Array.from({ length: Math.min(POOL, items.length) }, worker));
|
|
}
|
|
|
|
/**
|
|
* Fold NDJSON `text` into `best`, keeping the highest-seq record per session.
|
|
* Returns how many lines would not parse.
|
|
*
|
|
* A session heartbeats every 5 minutes carrying CUMULATIVE totals, so the
|
|
* highest seq IS the whole session and every earlier row is a strict subset.
|
|
* Sessions straddle hours, so readers still dedupe across rollups on this same
|
|
* key — this only shrinks each hour.
|
|
*
|
|
* The winner is kept as freshly serialised text, not as a slice of `text`: a
|
|
* slice would pin the whole source string in memory, and the merge reads
|
|
* sixteen of them against a 128MB isolate.
|
|
*/
|
|
function keepBest(best, text) {
|
|
let corrupt = 0;
|
|
for (const line of text.split('\n')) {
|
|
if (!line) continue;
|
|
let rec;
|
|
try {
|
|
rec = JSON.parse(line);
|
|
} catch {
|
|
// Counted, but NOT a reason to abandon the slice. Content this Worker
|
|
// itself wrote with JSON.stringify does not become valid later, so
|
|
// blocking on it would strand the hour instead of losing one record.
|
|
corrupt++;
|
|
continue;
|
|
}
|
|
const id = `${rec.resource?.['headroom.install_id']} ${rec.session?.id}`;
|
|
const seq = rec.session?.seq ?? 0;
|
|
const prev = best.get(id);
|
|
if (!prev || seq > prev.seq) best.set(id, { seq, line: JSON.stringify(rec) });
|
|
}
|
|
return corrupt;
|
|
}
|
|
|
|
const ndjson = (best) => [...best.values()].map((r) => r.line).join('\n');
|
|
|
|
/** True once an hour has its rollup (or its `empty` marker). */
|
|
async function hourDone(env, part) {
|
|
const page = await env.CORPUS.list({ prefix: `rollup/${part}/`, limit: 1 });
|
|
return page.objects.length > 0;
|
|
}
|
|
|
|
/**
|
|
* One slice of one hour -> rollup-parts/<part>/<slice>.
|
|
*
|
|
* Throws if any listed object cannot be read, and writes nothing: the queue
|
|
* redelivers the message, and a part that was short would silently become the
|
|
* permanent record once merged. A slice with no objects still writes its (empty)
|
|
* part, because "all sixteen present" is what finalizeHour waits for.
|
|
*/
|
|
export async function rollupSlice(env, part, slice) {
|
|
const objects = await listAll(env, { prefix: `sessions/${part}/${slice}` });
|
|
const best = new Map();
|
|
let corrupt = 0;
|
|
await pool(objects, async (o) => {
|
|
const body = await env.CORPUS.get(o.key);
|
|
// The key came from a LIST, so the object existed; treating a miss as
|
|
// empty would quietly shrink the rollup.
|
|
if (!body) throw new Error(`${o.key}: listed but unreadable`);
|
|
// Folded as it arrives, so only the winners are held, never the slice.
|
|
corrupt += keepBest(best, await body.text());
|
|
});
|
|
if (corrupt) {
|
|
console.error(`rollup ${part}/${slice}: skipped ${corrupt} unparseable record(s)`);
|
|
}
|
|
await env.CORPUS.put(`rollup-parts/${part}/${slice}`, ndjson(best));
|
|
return { read: objects.length, kept: best.size };
|
|
}
|
|
|
|
/**
|
|
* Merge an hour's sixteen parts into rollup/<part>/, then remove the parts.
|
|
* Returns false, having written nothing, while any slice is still missing.
|
|
*
|
|
* Safe to run twice at once: both runs read the same parts and write the same
|
|
* bytes. The one that loses the race finds a part already deleted and stops.
|
|
*/
|
|
export async function finalizeHour(env, part) {
|
|
const parts = await listAll(env, { prefix: `rollup-parts/${part}/` });
|
|
const have = new Set(parts.map((o) => o.key.slice(o.key.lastIndexOf('/') + 1)));
|
|
if (![...SLICES].every((s) => have.has(s))) return false;
|
|
|
|
// One session's heartbeats are spread across slices, so the parts overlap and
|
|
// the merge is a real dedupe, not a concatenation. Read one at a time: the
|
|
// parts together are several times the size of the result.
|
|
const best = new Map();
|
|
for (const slice of SLICES) {
|
|
const body = await env.CORPUS.get(`rollup-parts/${part}/${slice}`);
|
|
if (!body) return false;
|
|
keepBest(best, await body.text());
|
|
}
|
|
|
|
// A genuinely empty hour gets a marker rather than a zero-byte NDJSON that
|
|
// every reader would have to special-case. Without it the hour stays
|
|
// "missing" and is enqueued again for the life of the bucket.
|
|
if (best.size === 0) {
|
|
await env.CORPUS.put(`rollup/${part}/empty`, '');
|
|
} else {
|
|
await env.CORPUS.put(`rollup/${part}/data.ndjson`, ndjson(best), {
|
|
httpMetadata: { contentType: 'application/x-ndjson' },
|
|
});
|
|
}
|
|
await env.CORPUS.delete(parts.map((o) => o.key));
|
|
return true;
|
|
}
|
|
|
|
/** Oldest `dt=` day still under sessions/, or null. One delimited LIST. */
|
|
export async function oldestRawDay(env) {
|
|
const page = await env.CORPUS.list({ prefix: 'sessions/', delimiter: '/' });
|
|
const days = (page.delimitedPrefixes || [])
|
|
.map((p) => p.slice('sessions/dt='.length).replace(/\/$/, ''))
|
|
.filter((d) => /^\d{4}-\d{2}-\d{2}$/.test(d))
|
|
.sort();
|
|
return days.length ? days[0] : null;
|
|
}
|
|
|
|
/** `dt=…/hh=…` of a key under `prefix`, i.e. everything up to its last slash. */
|
|
const partOf = (key, prefix) => key.slice(prefix.length, key.lastIndexOf('/'));
|
|
|
|
export default {
|
|
/**
|
|
* Hourly cron. Enqueues every complete hour that has no rollup, back to the
|
|
* oldest raw data. Does none of the reading itself — see the note above.
|
|
*/
|
|
async scheduled(event, env) {
|
|
// Backfill reaches all the way to the oldest surviving raw day, NOT a fixed
|
|
// window. A fixed window silently strands everything older than it the
|
|
// moment analysis stopped reading sessions/. Bounding by real data instead
|
|
// means the floor rises only when a lifecycle rule expires the raw objects.
|
|
const oldest = await oldestRawDay(env);
|
|
if (!oldest) return;
|
|
const floorMs = Date.parse(`${oldest}T00:00:00Z`);
|
|
if (Number.isNaN(floorMs)) return;
|
|
|
|
// Only list from the floor forward. Rollups older than the oldest raw day
|
|
// can never be rebuilt, so enumerating them answers nothing.
|
|
const done = new Set(
|
|
(await listAll(env, { prefix: 'rollup/', startAfter: `rollup/dt=${oldest}` })).map((o) =>
|
|
partOf(o.key, 'rollup/')
|
|
)
|
|
);
|
|
const inFlight = await listAll(env, { prefix: 'rollup-parts/' });
|
|
|
|
// Sweep what a finished hour left behind. finalizeHour removes its parts,
|
|
// but three things can outlive it: a duplicate slice that was still reading
|
|
// when the hour was merged, parts whose delete failed after the publish,
|
|
// and a `queued` marker written after a very small hour had already merged.
|
|
// Nothing else would ever remove them, and this prefix is listed every run.
|
|
const leftover = inFlight
|
|
.filter((o) => done.has(partOf(o.key, 'rollup-parts/')))
|
|
.map((o) => o.key);
|
|
for (let i = 0; i < leftover.length; i += DELETE_BATCH) {
|
|
await env.CORPUS.delete(leftover.slice(i, i + DELETE_BATCH));
|
|
}
|
|
|
|
// When each in-flight hour was enqueued, so an hour still being worked on
|
|
// is not enqueued again every time the cron fires.
|
|
const queuedAt = new Map(
|
|
inFlight
|
|
.filter((o) => o.key.endsWith('/queued'))
|
|
.map((o) => [partOf(o.key, 'rollup-parts/'), o.uploaded.getTime()])
|
|
);
|
|
|
|
// Newest first, so the freshest hour is never the one a backlog starves.
|
|
// Starts one hour back: the current hour is still being written to.
|
|
let enqueued = 0;
|
|
for (
|
|
let t = event.scheduledTime - 3600_000;
|
|
t >= floorMs && enqueued < MAX_HOURS_PER_RUN;
|
|
t -= 3600_000
|
|
) {
|
|
const part = partition(new Date(t));
|
|
if (done.has(part)) continue;
|
|
const at = queuedAt.get(part);
|
|
if (at !== undefined && event.scheduledTime - at < REQUEUE_AFTER_MS) continue;
|
|
try {
|
|
// Marker first: if the send then fails, the hour waits for the
|
|
// re-enqueue instead of being sent twice by the next run.
|
|
await env.CORPUS.put(`rollup-parts/${part}/queued`, '');
|
|
await env.ROLLUP_QUEUE.sendBatch([...SLICES].map((slice) => ({ body: { part, slice } })));
|
|
} catch (err) {
|
|
// One hour failing to enqueue must not end the run for the rest.
|
|
console.error(`rollup ${part}: enqueue failed: ${err}`);
|
|
}
|
|
enqueued++;
|
|
}
|
|
},
|
|
|
|
/**
|
|
* Queue consumer: one message is one slice of one hour. A slice that throws
|
|
* is retried by the queue on its own, so one bad read costs that slice a
|
|
* retry instead of costing the hour everything it had already read.
|
|
*/
|
|
async queue(batch, env) {
|
|
for (const message of batch.messages) {
|
|
const { part, slice } = message.body;
|
|
try {
|
|
// A redelivery, or a re-enqueue that raced the merge: nothing to do,
|
|
// and writing a part now would leave it behind for good.
|
|
if (!(await hourDone(env, part))) {
|
|
// The hour is closed, so a part that is already there is still
|
|
// right: a second copy of the message only has to try the merge.
|
|
if (!(await env.CORPUS.head(`rollup-parts/${part}/${slice}`))) {
|
|
await rollupSlice(env, part, slice);
|
|
}
|
|
await finalizeHour(env, part);
|
|
}
|
|
message.ack();
|
|
} catch (err) {
|
|
console.error(`rollup ${part}/${slice} failed: ${err}`);
|
|
message.retry({ delaySeconds: RETRY_DELAY_S });
|
|
}
|
|
}
|
|
},
|
|
|
|
async fetch(request, env, ctx) {
|
|
if (request.method !== 'POST') {
|
|
return new Response('beacon: POST OTLP logs to /v1/logs', { status: 405 });
|
|
}
|
|
const url = new URL(request.url);
|
|
if (url.pathname !== '/v1/logs') {
|
|
return new Response('not found', { status: 404 });
|
|
}
|
|
|
|
// Abuse bound, checked before the body is read so a flood costs neither
|
|
// the inflate nor an R2 write. Keyed on one constant rather than the
|
|
// sender: a per-IP limit would mean reading cf-connecting-ip, which this
|
|
// Worker never does (see the header). The price is that a flood also
|
|
// sheds real events until the window rolls over -- acceptable for
|
|
// fire-and-forget telemetry, and the cap sits ~10x above normal traffic.
|
|
//
|
|
// 429 is safe for old clients: only 400/415 make them drop gzip
|
|
// (_GZIP_REFUSED_STATUSES in headroom/telemetry/session.py).
|
|
//
|
|
// Fails open: with no binding (local tests, an old wrangler.toml) or when
|
|
// the rate-limit service itself errors, the upload proceeds. A limiter
|
|
// outage must not drop every legitimate beacon; the cost is that a flood
|
|
// during that outage is unbounded until the service recovers.
|
|
if (env.RATE_LIMIT) {
|
|
let limited = false;
|
|
try {
|
|
const { success } = await env.RATE_LIMIT.limit({ key: 'all' });
|
|
limited = !success;
|
|
} catch (err) {
|
|
console.error('beacon: rate limiter unavailable, failing open:', String(err));
|
|
}
|
|
if (limited) return new Response('rate limited', { status: 429 });
|
|
}
|
|
|
|
const raw = await request.arrayBuffer();
|
|
if (raw.byteLength > MAX_BODY_BYTES) {
|
|
return new Response('payload too large', { status: 413 });
|
|
}
|
|
|
|
let records;
|
|
try {
|
|
// Sniff the gzip magic number rather than trusting Content-Encoding.
|
|
// Three things can each independently decide whether a body arrives
|
|
// compressed -- the client, Cloudflare's edge, and any proxy between
|
|
// them -- and only the bytes know which of them acted. Keying off the
|
|
// header instead means a body that something already decompressed gets
|
|
// fed to DecompressionStream, or vice versa, and every upload 400s.
|
|
//
|
|
// It also means an old client that never sets the header keeps working
|
|
// through the identical code path it uses today: no header, no magic
|
|
// number, straight to TextDecoder.
|
|
const bytes = new Uint8Array(raw);
|
|
const gzipped = bytes.length >= 2 && bytes[0] === 0x1f && bytes[1] === 0x8b;
|
|
const text = gzipped
|
|
? await inflate(raw, MAX_DECOMPRESSED_BYTES)
|
|
: new TextDecoder().decode(raw);
|
|
records = extract(JSON.parse(text));
|
|
} catch {
|
|
// Malformed input is not worth a retry storm from clients.
|
|
//
|
|
// This 400 is also load-bearing in the other direction: it is what a
|
|
// client that gzipped against a Worker too old to inflate sees, and
|
|
// _post_blocking treats a 4xx on a compressed body as "this endpoint
|
|
// does not do gzip", falls back to sending it uncompressed, and stops
|
|
// compressing for the life of the process. So deploying the client
|
|
// before this Worker costs one retry per process, not the data.
|
|
return new Response('bad request', { status: 400 });
|
|
}
|
|
if (records.length === 0) return new Response(null, { status: 204 });
|
|
|
|
// Hive-style partitioning so DuckDB can prune by date without a catalog.
|
|
// Shares partition() with the rollup: the cron lists `sessions/<part>/`, so
|
|
// two independent spellings of this scheme would mean the writer and the
|
|
// compactor could drift apart and silently match zero objects.
|
|
// ponytail: one object per request. Compacted hourly into rollup/ by
|
|
// the queue consumer above — analysis reads that, never this. The random
|
|
// UUID is load-bearing there: its first character is the slice.
|
|
const key = `sessions/${partition(new Date())}/${crypto.randomUUID()}.json`;
|
|
const ndjson = records.map((r) => JSON.stringify(r)).join('\n');
|
|
|
|
// Respond immediately; durability work continues after the response.
|
|
// The client is fire-and-forget and ignores the status anyway — making it
|
|
// wait on R2 would only add latency to someone else's coding session.
|
|
ctx.waitUntil(
|
|
env.CORPUS.put(key, ndjson, {
|
|
httpMetadata: { contentType: 'application/x-ndjson' },
|
|
})
|
|
);
|
|
|
|
// Optional second lane: forward verbatim OTLP to a metrics backend for
|
|
// dashboards. Configured by secret, so it can be added or swapped with a
|
|
// `wrangler secret put` and no code change.
|
|
if (env.METRICS_OTLP_URL) {
|
|
ctx.waitUntil(
|
|
fetch(env.METRICS_OTLP_URL, {
|
|
method: 'POST',
|
|
headers: {
|
|
'content-type': 'application/json',
|
|
authorization: env.METRICS_OTLP_AUTH || '',
|
|
},
|
|
body: JSON.stringify({ resourceLogs: [{ scopeLogs: [{ logRecords: records.map((r) => ({ body: { stringValue: JSON.stringify(r) } })) }] }] }),
|
|
}).catch(() => {})
|
|
);
|
|
}
|
|
|
|
return new Response(null, { status: 204 });
|
|
},
|
|
};
|