Files
headroom/deploy/beacon/worker.js
ac932eb7c4 feat(telemetry): split beacon sessions by harness, model and day (#4881)
## Description

After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:

1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.

| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |

- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.

**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.

**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.

## Testing

- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
    tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```

Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.

## Real Behavior Proof

- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
  1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
  2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
  4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:

```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
  {"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
  {"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
  {"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
  {"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
  {"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```

Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
  - Linux, Windows, containers and CI.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:50:49 +05:30

778 lines
33 KiB
JavaScript

/**
* Headroom telemetry beacon receiver.
*
* This file is open source on purpose. It is the other half of the promise
* made in headroom/telemetry/session.py: users can read exactly what the
* client sends AND exactly what happens to it on arrival. "Trust us" is not a
* privacy policy.
*
* Deployed at headroom-beacon.headroom-beacon.workers.dev (otlp.headroomlabs.ai
* is planned but not yet routed; see wrangler.toml). Three jobs:
*
* 1. Allowlist. Drop every field not on ALLOWED_KEYS, and every nested
* field not described by NESTED, before anything is written. This is the only privacy control that works retroactively —
* if a future client version ships a bug that leaks a field, we cannot
* patch the installs already in the wild, but we can stop storing it
* here in one deploy.
*
* 2. Flatten. OTLP AnyValue nesting is portable but miserable to query
* ({"kvlistValue":{"values":[{"key":"tokens",...}]}}). We keep OTLP on
* the wire so the backend stays vendor-swappable, and store plain JSON so
* DuckDB can read it without unwrapping anything.
*
* 3. Fan out. R2 for the durable corpus; optionally a metrics vendor for
* dashboards. Adding a destination is one more call here — never a
* client release.
*
* What this deliberately does NOT do: log, store, or forward the source IP.
* Cloudflare offers it as cf-connecting-ip; it is the one field that would
* deanonymise install_id, so it is never read. Cloudflare's own per-request
* invocation logs would record it regardless of this code, which is why
* wrangler.toml turns them off (observability.logs.invocation_logs = false).
*/
// Mostly mirrors the payload built by _Session.payload(); an extension may
// also emit its own event carrying one of these top-level keys. A key absent
// here is dropped, not stored. Adding a metric means adding it here first —
// that friction is the point, and it is also the only privacy control that
// works retroactively, so it must land BEFORE any client starts sending the
// key or that traffic is silently discarded and unrecoverable.
const ALLOWED_KEYS = [
'schema_version',
'session',
'tokens',
'rates',
'compression',
'skips',
'sources',
'providers',
'models',
'failures',
'failure_statuses',
// Model-routing summary. Emitted by a routing extension rather than by the
// proxy itself -- see proxy/route_advice.py for the decision seam. Same rule
// as everything above: counters and model ids, no free text. Allowlisted
// here so the corpus can answer what the proxy alone cannot -- a provider's
// real minimum cacheable prefix, how long a cache actually survives, and how
// far predicted cache hits are from the ones that happened.
'routing',
// ---- schema v2 -----------------------------------------------------------
//
// Additive. Every v1 key above is untouched, so a v1 client keeps storing
// exactly what it stores today and rows from the two versions union cleanly
// (`schema_version` separates them, `union_by_name` handles the rest).
//
// Read the ordering rule at the top of this list literally: this deploy must
// go out BEFORE the client release that starts sending these, or the first
// weeks of the new signals are dropped on arrival and cannot be recovered.
// Removing one line here is also the rollback -- it reverts that signal for
// every install in the wild without a client release, which is the whole
// reason the allowlist is server-side.
// Regret. `reread_compressed_tokens` is the counter-pressure on `saved_pct`:
// tokens the agent had to fetch again because compression removed them.
// Without it a compression ratio is unfalsifiable.
'quality',
// Fixed-bucket distributions for quantities otherwise reported only as
// session sums. Ships its own bucket edges, so no reader needs to know which
// client version wrote a row.
'hist',
// (content_type x strategy x yield), and the same for JSON tool output keyed
// by structural shape. The input term the corpus has never had.
// Structural descriptors only -- never TOIN's structure_hash, which is a
// digest of field names.
'shapes',
// Prompt-cache survival by gap since the previous turn: the empirical TTL
// and minimum-prefix curve per provider.
'cache',
// Session shape over time -- log-spaced turn buckets plus a run-length
// encoded turn-kind string from a closed seven-letter alphabet.
'trajectory',
// Identified client harness, per turn. `headroom.stack` answers this by
// detection and resolves to a literal "proxy" for most of the fleet.
'clients',
// 4xx, kept strictly separate from the 5xx-only `failures` above so neither
// metric changes meaning. 429 rate is the routing signal here.
'errors',
// output_shaper A/B strata as validated enums. Model family is not among
// them -- see the note in session.py's payload().
'strata',
// Allowlisted, slug-validated configuration, so the corpus can tell
// "compression underperformed" from "compression was switched off".
'config',
// The session's tokens split by harness and by model. `clients` and `models`
// only say who took part, so a session that mixed two could not be divided
// or priced. `client_usage` covers every turn: traffic with no identified
// harness is a row of its own instead of a gap.
'client_usage',
'model_usage',
// The harness's own release, numeric parts only.
'client_versions',
// Age of the install id in days, and whether it runs in a container or CI
// job: separates a returning user from an id minted on every run.
'install',
];
// Resource attributes we keep. Same rule: allowlist, not denylist.
const ALLOWED_RESOURCE = [
'service.name',
'service.version',
'headroom.install_id',
'headroom.install_mode',
'headroom.stack',
'os.type',
'host.arch',
];
// What may sit UNDER each allowed key. Without this the allowlist stops at the
// top level and a nested field a client bug adds is stored verbatim -- the very
// leak the header says one deploy here can stop. Mirrors _Session.payload() and
// _payload_v2() in headroom/telemetry/session.py. A nested key absent here is
// dropped, so the ordering rule on ALLOWED_KEYS applies to it too: teach it
// here before a client sends it.
//
// Leaves are 'num' (a finite number), 'bool', or 'str': a token of at most 128
// characters with no whitespace, which is every string the client sends (slugs,
// hex ids, public model ids, versions). Free text does not fit it. `[spec]` is
// a list of spec; mapOf(spec) is an object whose KEYS are data (transform
// names, skip reasons, status codes), each key itself held to 'str'.
const MAP = Symbol('map');
const mapOf = (spec) => ({ [MAP]: spec });
const nums = (...keys) => Object.fromEntries(keys.map((k) => [k, 'num']));
// A harness name: the client only sends ones from a closed list, so hold the
// field to that shape instead of the wider 'str'.
const SLUG = /^[a-z][a-z0-9_]{0,31}$/;
// One turn-and-token row of `client_usage` / `model_usage`: _USAGE_FIELDS in
// session.py.
const USAGE = nums(
'n', 'original', 'saved', 'tool_saved', 'output', 'cache_read', 'cache_write',
'cache_write_inferred', 'cache_write_5m', 'cache_write_1h', 'uncached'
);
const NESTED = {
schema_version: 'num',
session: {
id: 'str', seq: 'num', duration_s: 'num', turns: 'num', ended: 'str', final: 'bool',
},
tokens: nums(
'original', 'attempted', 'input', 'output', 'saved', 'tool_saved',
'cache_read', 'cache_write', 'uncached'
),
rates: nums(
'saved_pct', 'eligible_pct', 'yield_pct', 'all_layers_saved_pct',
'all_layers_yield_pct', 'cache_read_pct', 'overhead_pct'
),
compression: {
transforms: mapOf('num'),
by_strategy: [{ strategy: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
...nums(
'overhead_ms_total', 'latency_ms_total', 'overhead_ms_per_turn',
'latency_ms_per_turn', 'passthrough_turns', 'response_cache_hits'
),
},
skips: mapOf('num'),
sources: mapOf('num'),
providers: ['str'],
models: ['str'],
failures: 'num',
failure_statuses: mapOf('num'),
// Emitted by an extension, not by session.py, so this repository does not
// define its shape. Bounded rather than described: numbers, booleans and
// tokens only, at most ROUTING_DEPTH levels deep. It gets a real schema here
// once the extension's payload is written down.
routing: 'any',
quality: {
...nums('reread_tokens', 'reread_compressed_tokens', 'waste_turns'),
waste: [{ kind: 'str', tokens: 'num' }],
},
hist: mapOf({ edges: ['num'], counts: ['num'] }),
shapes: {
by_content: [{ content: 'str', strategy: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
by_tool: [{ shape: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
truncated: 'bool',
},
cache: {
gap_edges_s: ['num'], hits: ['num'], misses: ['num'],
...nums('write_5m', 'write_1h', 'inferred_turns'),
},
trajectory: {
buckets: 'num', turns: ['num'], tokens: ['num'], saved: ['num'],
// Runs over the closed _TURN_KINDS alphabet, at most MAX_TRAJECTORY_RUNS.
kinds: /^(?:[cfepzso]\d{1,9}){0,64}$/,
kinds_truncated: 'bool',
msgs_max: 'num',
},
clients: [{ client: 'str', n: 'num' }],
errors: { count: 'num', by_status: mapOf('num') },
strata: [{ dim: 'str', value: 'str', n: 'num' }],
config: [{ key: 'str', value: 'str' }],
client_usage: [{ client: SLUG, ...USAGE }],
model_usage: [{ model: 'str', ...USAGE }],
client_versions: [{ client: SLUG, version: /^\d{1,4}(?:\.\d{1,4}){0,2}$/, n: 'num' }],
install: { age_days: 'num', container: 'bool', ci: 'bool' },
};
// A key allowed at the top but given no shape would be dropped in full. Fail
// the deploy instead of shipping that.
if ([...ALLOWED_KEYS].sort().join() !== Object.keys(NESTED).sort().join()) {
throw new Error('beacon: ALLOWED_KEYS and NESTED disagree');
}
const TOKEN = /^[\w.:/@+|-]{0,128}$/;
// A slug-shaped string can still be a URL or a credential. Model ids and the
// other slugs the client sends never carry a scheme or a well-known key prefix,
// so those are dropped even though they pass TOKEN.
const SECRETISH = /:\/\/|^(?:sk|pk|rk)[-_]|^(?:ghp|gho|ghu|ghs|ghr)_|^github_pat_|^glpat-|^xox[abposr]-|^(?:AKIA|ASIA)[A-Z0-9]{12}|^AIza|^hf_|^eyJ/i;
const isToken = (s) => TOKEN.test(s) && !SECRETISH.test(s);
const ROUTING_DEPTH = 4;
/** `value` reduced to what `spec` allows, or undefined when none of it is. */
function clean(spec, value, depth = 0) {
if (spec === 'any') {
if (depth > ROUTING_DEPTH) return undefined;
spec = Array.isArray(value) ? ['any']
: value && typeof value === 'object' ? mapOf('any')
: typeof value === 'string' ? 'str'
: typeof value === 'number' ? 'num' : 'bool';
}
if (spec === 'num') return Number.isFinite(value) ? value : undefined;
if (spec === 'bool') return typeof value === 'boolean' ? value : undefined;
if (spec === 'str') return typeof value === 'string' && isToken(value) ? value : undefined;
if (spec instanceof RegExp) {
return typeof value === 'string' && spec.test(value) ? value : undefined;
}
if (Array.isArray(spec)) {
if (!Array.isArray(value)) return undefined;
return value.map((v) => clean(spec[0], v, depth + 1)).filter((v) => v !== undefined);
}
if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined;
const out = {};
if (spec[MAP]) {
for (const [key, v] of Object.entries(value)) {
const kept = isToken(key) ? clean(spec[MAP], v, depth + 1) : undefined;
if (kept !== undefined) out[key] = kept;
}
return out;
}
for (const [key, s] of Object.entries(spec)) {
if (!Object.hasOwn(value, key)) continue;
const kept = clean(s, value[key], depth + 1);
if (kept !== undefined) out[key] = kept;
}
return out;
}
// Cap on what ARRIVES, compressed or not.
//
// Was 64KB, chosen when the comment above it read "a beacon event is ~2KB" --
// i.e. 32x the payload. Schema v2 made a typical event ~8KB uncompressed, and
// a session that fills every table (the shape cross product is 10 content
// types x 13 strategies) reaches ~82KB. That silently eroded the multiple to
// under 1x: a busy session sending uncompressed -- the kill switch is set, or
// the gzip fallback fired -- would have been answered 413 and, correctly, not
// retried, so the event would simply be lost.
//
// Restored to ~3x the worst legitimate payload. This is not the abuse control
// and never was: the endpoint is unauthenticated by design and the RATE_LIMIT
// binding (see fetch() and wrangler.toml) is what bounds a bad actor.
const MAX_BODY_BYTES = 256 * 1024;
// Cap on what an arriving body EXPANDS to. `raw.byteLength` cannot see this:
// a small gzip can expand to gigabytes, and a decompression bomb that is only
// noticed after it has been decompressed has already won. Enforced while the
// stream is read, not after. ~3x the worst legitimate payload, same as above.
const MAX_DECOMPRESSED_BYTES = 256 * 1024;
/**
* Decompress a gzip body, refusing anything that expands past `limit`.
*
* Counts bytes as chunks arrive and aborts mid-stream, so a bomb costs the
* limit rather than whatever it would have expanded to.
*/
export async function inflate(buffer, limit) {
const stream = new Response(buffer).body.pipeThrough(new DecompressionStream('gzip'));
const reader = stream.getReader();
const chunks = [];
let total = 0;
try {
for (;;) {
const { done, value } = await reader.read();
if (done) break;
total += value.byteLength;
if (total > limit) throw new Error('decompressed body too large');
chunks.push(value);
}
} finally {
// Releases the stream on the bomb path; a no-op once it has closed.
reader.cancel().catch(() => {});
}
const joined = new Uint8Array(total);
let offset = 0;
for (const chunk of chunks) {
joined.set(chunk, offset);
offset += chunk.byteLength;
}
return new TextDecoder().decode(joined);
}
/** OTLP AnyValue -> plain JS. The inverse of _any_value() in session.py. */
function unwrap(value) {
if (value == null) return null;
if ('stringValue' in value) return value.stringValue;
if ('boolValue' in value) return value.boolValue;
if ('intValue' in value) return Number(value.intValue);
if ('doubleValue' in value) return value.doubleValue;
if ('arrayValue' in value) return (value.arrayValue.values || []).map(unwrap);
if ('kvlistValue' in value) {
const out = {};
for (const kv of value.kvlistValue.values || []) out[kv.key] = unwrap(kv.value);
return out;
}
return null;
}
function pick(obj, allowed, specOf) {
const out = {};
if (!obj || typeof obj !== 'object') return out;
for (const key of allowed) {
if (!(key in obj)) continue;
const kept = clean(specOf(key), obj[key]);
if (kept !== undefined) out[key] = kept;
}
return out;
}
/** OTLP ExportLogsServiceRequest -> flat, allowlisted records. */
function extract(payload) {
const records = [];
for (const rl of payload.resourceLogs || []) {
const resource = {};
for (const attr of rl.resource?.attributes || []) {
resource[attr.key] = unwrap(attr.value);
}
// Every resource attribute the client sends is a plain string.
const cleanResource = pick(resource, ALLOWED_RESOURCE, () => 'str');
for (const sl of rl.scopeLogs || []) {
for (const rec of sl.logRecords || []) {
const body = unwrap(rec.body);
if (!body || typeof body !== 'object') continue;
records.push({
...pick(body, ALLOWED_KEYS, (key) => NESTED[key]),
resource: cleanResource,
// Server-stamped. A client clock can be wrong or forged; this is the
// timestamp partitioning and retention actually rely on.
received_at: new Date().toISOString(),
});
}
}
}
return records;
}
// ----------------------------------------------------------------- rollup --
//
// The corpus is one object per heartbeat. Analysis reads rollup/**, never
// sessions/**: each COMPLETE hour is collapsed into one object under rollup/,
// keeping only the highest-seq heartbeat per (install, session). Raw is left
// exactly as written, so any rollup can be rebuilt by deleting it.
//
// One invocation cannot do an hour on its own, and that is measured, not
// feared. A Worker may have six requests in flight, so reads ran at ~12
// objects/s whatever the code asked for, and a cron run is stopped at 15
// minutes: ~11k objects. The fleet passed that on 2026-08-19 and by October 98%
// of hours were being killed mid-read and writing nothing.
//
// So an hour is cut into slices and each slice is its own queue message, which
// means its own invocation with its own six connections and its own 15 minutes:
//
// scheduled() finds hours with no rollup and enqueues one message per slice
// rollupSlice() reads one slice, writes rollup-parts/<hour>/<slice>
// finalizeHour() once every slice is in, merges them into rollup/<hour>/
//
// Slices are the first hex character of the object name. Names are
// crypto.randomUUID() (see fetch()), so the sixteen slices are even, and the
// split needs nothing but a LIST prefix — which is also why it works on every
// object already in the bucket. Raise to two characters (256 slices) when a
// slice stops fitting; the per-slice cost falls sixteen-fold.
const SLICES = '0123456789abcdef';
// Requests in flight per invocation. Six is the platform's ceiling: asking for
// more only queues inside the runtime.
const POOL = 6;
// A queued hour that still has no rollup after this long is enqueued again:
// the safety net for a slice that ran out of retries. A second copy of a slice
// that did finish is cheap, because queue() reuses the part already written.
const REQUEUE_AFTER_MS = 3 * 3600_000;
// Hours enqueued per cron run, sized so they drain well inside REQUEUE_AFTER_MS
// (200 hours is 3,200 slices of a couple of minutes each, run in parallel). A
// bigger backlog is not stuck, the next run simply continues it.
const MAX_HOURS_PER_RUN = 200;
// Wait before a failed slice is tried again, so a short R2 outage does not burn
// every retry within seconds.
const RETRY_DELAY_S = 60;
// R2 deletes at most this many keys per call.
const DELETE_BATCH = 1000;
const partition = (d) =>
`dt=${d.toISOString().slice(0, 10)}/hh=${d.toISOString().slice(11, 13)}`;
/** Every object under a prefix, following the cursor. */
async function listAll(env, options) {
const objects = [];
let cursor;
do {
const page = await env.CORPUS.list({ ...options, cursor });
objects.push(...page.objects);
cursor = page.truncated ? page.cursor : undefined;
} while (cursor);
return objects;
}
/**
* Run `fn` over `items` with at most POOL calls in flight. Rejects on the first
* failure and starts nothing new after it: the slice is going to be retried
* whole, so further reads would be thrown away.
*/
async function pool(items, fn) {
let next = 0;
let failed = false;
const worker = async () => {
while (!failed && next < items.length) {
try {
await fn(items[next++]);
} catch (err) {
failed = true;
throw err;
}
}
};
await Promise.all(Array.from({ length: Math.min(POOL, items.length) }, worker));
}
/**
* Fold NDJSON `text` into `best`, keeping the highest-seq record per session.
* Returns how many lines would not parse.
*
* A session heartbeats every 5 minutes carrying CUMULATIVE totals, so the
* highest seq IS the whole session and every earlier row is a strict subset.
* Sessions straddle hours, so readers still dedupe across rollups on this same
* key — this only shrinks each hour.
*
* The winner is kept as freshly serialised text, not as a slice of `text`: a
* slice would pin the whole source string in memory, and the merge reads
* sixteen of them against a 128MB isolate.
*/
function keepBest(best, text) {
let corrupt = 0;
for (const line of text.split('\n')) {
if (!line) continue;
let rec;
try {
rec = JSON.parse(line);
} catch {
// Counted, but NOT a reason to abandon the slice. Content this Worker
// itself wrote with JSON.stringify does not become valid later, so
// blocking on it would strand the hour instead of losing one record.
corrupt++;
continue;
}
const id = `${rec.resource?.['headroom.install_id']} ${rec.session?.id}`;
const seq = rec.session?.seq ?? 0;
const prev = best.get(id);
if (!prev || seq > prev.seq) best.set(id, { seq, line: JSON.stringify(rec) });
}
return corrupt;
}
const ndjson = (best) => [...best.values()].map((r) => r.line).join('\n');
/** True once an hour has its rollup (or its `empty` marker). */
async function hourDone(env, part) {
const page = await env.CORPUS.list({ prefix: `rollup/${part}/`, limit: 1 });
return page.objects.length > 0;
}
/**
* One slice of one hour -> rollup-parts/<part>/<slice>.
*
* Throws if any listed object cannot be read, and writes nothing: the queue
* redelivers the message, and a part that was short would silently become the
* permanent record once merged. A slice with no objects still writes its (empty)
* part, because "all sixteen present" is what finalizeHour waits for.
*/
export async function rollupSlice(env, part, slice) {
const objects = await listAll(env, { prefix: `sessions/${part}/${slice}` });
const best = new Map();
let corrupt = 0;
await pool(objects, async (o) => {
const body = await env.CORPUS.get(o.key);
// The key came from a LIST, so the object existed; treating a miss as
// empty would quietly shrink the rollup.
if (!body) throw new Error(`${o.key}: listed but unreadable`);
// Folded as it arrives, so only the winners are held, never the slice.
corrupt += keepBest(best, await body.text());
});
if (corrupt) {
console.error(`rollup ${part}/${slice}: skipped ${corrupt} unparseable record(s)`);
}
await env.CORPUS.put(`rollup-parts/${part}/${slice}`, ndjson(best));
return { read: objects.length, kept: best.size };
}
/**
* Merge an hour's sixteen parts into rollup/<part>/, then remove the parts.
* Returns false, having written nothing, while any slice is still missing.
*
* Safe to run twice at once: both runs read the same parts and write the same
* bytes. The one that loses the race finds a part already deleted and stops.
*/
export async function finalizeHour(env, part) {
const parts = await listAll(env, { prefix: `rollup-parts/${part}/` });
const have = new Set(parts.map((o) => o.key.slice(o.key.lastIndexOf('/') + 1)));
if (![...SLICES].every((s) => have.has(s))) return false;
// One session's heartbeats are spread across slices, so the parts overlap and
// the merge is a real dedupe, not a concatenation. Read one at a time: the
// parts together are several times the size of the result.
const best = new Map();
for (const slice of SLICES) {
const body = await env.CORPUS.get(`rollup-parts/${part}/${slice}`);
if (!body) return false;
keepBest(best, await body.text());
}
// A genuinely empty hour gets a marker rather than a zero-byte NDJSON that
// every reader would have to special-case. Without it the hour stays
// "missing" and is enqueued again for the life of the bucket.
if (best.size === 0) {
await env.CORPUS.put(`rollup/${part}/empty`, '');
} else {
await env.CORPUS.put(`rollup/${part}/data.ndjson`, ndjson(best), {
httpMetadata: { contentType: 'application/x-ndjson' },
});
}
await env.CORPUS.delete(parts.map((o) => o.key));
return true;
}
/** Oldest `dt=` day still under sessions/, or null. One delimited LIST. */
export async function oldestRawDay(env) {
const page = await env.CORPUS.list({ prefix: 'sessions/', delimiter: '/' });
const days = (page.delimitedPrefixes || [])
.map((p) => p.slice('sessions/dt='.length).replace(/\/$/, ''))
.filter((d) => /^\d{4}-\d{2}-\d{2}$/.test(d))
.sort();
return days.length ? days[0] : null;
}
/** `dt=…/hh=…` of a key under `prefix`, i.e. everything up to its last slash. */
const partOf = (key, prefix) => key.slice(prefix.length, key.lastIndexOf('/'));
export default {
/**
* Hourly cron. Enqueues every complete hour that has no rollup, back to the
* oldest raw data. Does none of the reading itself — see the note above.
*/
async scheduled(event, env) {
// Backfill reaches all the way to the oldest surviving raw day, NOT a fixed
// window. A fixed window silently strands everything older than it the
// moment analysis stopped reading sessions/. Bounding by real data instead
// means the floor rises only when a lifecycle rule expires the raw objects.
const oldest = await oldestRawDay(env);
if (!oldest) return;
const floorMs = Date.parse(`${oldest}T00:00:00Z`);
if (Number.isNaN(floorMs)) return;
// Only list from the floor forward. Rollups older than the oldest raw day
// can never be rebuilt, so enumerating them answers nothing.
const done = new Set(
(await listAll(env, { prefix: 'rollup/', startAfter: `rollup/dt=${oldest}` })).map((o) =>
partOf(o.key, 'rollup/')
)
);
const inFlight = await listAll(env, { prefix: 'rollup-parts/' });
// Sweep what a finished hour left behind. finalizeHour removes its parts,
// but three things can outlive it: a duplicate slice that was still reading
// when the hour was merged, parts whose delete failed after the publish,
// and a `queued` marker written after a very small hour had already merged.
// Nothing else would ever remove them, and this prefix is listed every run.
const leftover = inFlight
.filter((o) => done.has(partOf(o.key, 'rollup-parts/')))
.map((o) => o.key);
for (let i = 0; i < leftover.length; i += DELETE_BATCH) {
await env.CORPUS.delete(leftover.slice(i, i + DELETE_BATCH));
}
// When each in-flight hour was enqueued, so an hour still being worked on
// is not enqueued again every time the cron fires.
const queuedAt = new Map(
inFlight
.filter((o) => o.key.endsWith('/queued'))
.map((o) => [partOf(o.key, 'rollup-parts/'), o.uploaded.getTime()])
);
// Newest first, so the freshest hour is never the one a backlog starves.
// Starts one hour back: the current hour is still being written to.
let enqueued = 0;
for (
let t = event.scheduledTime - 3600_000;
t >= floorMs && enqueued < MAX_HOURS_PER_RUN;
t -= 3600_000
) {
const part = partition(new Date(t));
if (done.has(part)) continue;
const at = queuedAt.get(part);
if (at !== undefined && event.scheduledTime - at < REQUEUE_AFTER_MS) continue;
try {
// Marker first: if the send then fails, the hour waits for the
// re-enqueue instead of being sent twice by the next run.
await env.CORPUS.put(`rollup-parts/${part}/queued`, '');
await env.ROLLUP_QUEUE.sendBatch([...SLICES].map((slice) => ({ body: { part, slice } })));
} catch (err) {
// One hour failing to enqueue must not end the run for the rest.
console.error(`rollup ${part}: enqueue failed: ${err}`);
}
enqueued++;
}
},
/**
* Queue consumer: one message is one slice of one hour. A slice that throws
* is retried by the queue on its own, so one bad read costs that slice a
* retry instead of costing the hour everything it had already read.
*/
async queue(batch, env) {
for (const message of batch.messages) {
const { part, slice } = message.body;
try {
// A redelivery, or a re-enqueue that raced the merge: nothing to do,
// and writing a part now would leave it behind for good.
if (!(await hourDone(env, part))) {
// The hour is closed, so a part that is already there is still
// right: a second copy of the message only has to try the merge.
if (!(await env.CORPUS.head(`rollup-parts/${part}/${slice}`))) {
await rollupSlice(env, part, slice);
}
await finalizeHour(env, part);
}
message.ack();
} catch (err) {
console.error(`rollup ${part}/${slice} failed: ${err}`);
message.retry({ delaySeconds: RETRY_DELAY_S });
}
}
},
async fetch(request, env, ctx) {
if (request.method !== 'POST') {
return new Response('beacon: POST OTLP logs to /v1/logs', { status: 405 });
}
const url = new URL(request.url);
if (url.pathname !== '/v1/logs') {
return new Response('not found', { status: 404 });
}
// Abuse bound, checked before the body is read so a flood costs neither
// the inflate nor an R2 write. Keyed on one constant rather than the
// sender: a per-IP limit would mean reading cf-connecting-ip, which this
// Worker never does (see the header). The price is that a flood also
// sheds real events until the window rolls over -- acceptable for
// fire-and-forget telemetry, and the cap sits ~10x above normal traffic.
//
// 429 is safe for old clients: only 400/415 make them drop gzip
// (_GZIP_REFUSED_STATUSES in headroom/telemetry/session.py).
//
// Fails open: with no binding (local tests, an old wrangler.toml) or when
// the rate-limit service itself errors, the upload proceeds. A limiter
// outage must not drop every legitimate beacon; the cost is that a flood
// during that outage is unbounded until the service recovers.
if (env.RATE_LIMIT) {
let limited = false;
try {
const { success } = await env.RATE_LIMIT.limit({ key: 'all' });
limited = !success;
} catch (err) {
console.error('beacon: rate limiter unavailable, failing open:', String(err));
}
if (limited) return new Response('rate limited', { status: 429 });
}
const raw = await request.arrayBuffer();
if (raw.byteLength > MAX_BODY_BYTES) {
return new Response('payload too large', { status: 413 });
}
let records;
try {
// Sniff the gzip magic number rather than trusting Content-Encoding.
// Three things can each independently decide whether a body arrives
// compressed -- the client, Cloudflare's edge, and any proxy between
// them -- and only the bytes know which of them acted. Keying off the
// header instead means a body that something already decompressed gets
// fed to DecompressionStream, or vice versa, and every upload 400s.
//
// It also means an old client that never sets the header keeps working
// through the identical code path it uses today: no header, no magic
// number, straight to TextDecoder.
const bytes = new Uint8Array(raw);
const gzipped = bytes.length >= 2 && bytes[0] === 0x1f && bytes[1] === 0x8b;
const text = gzipped
? await inflate(raw, MAX_DECOMPRESSED_BYTES)
: new TextDecoder().decode(raw);
records = extract(JSON.parse(text));
} catch {
// Malformed input is not worth a retry storm from clients.
//
// This 400 is also load-bearing in the other direction: it is what a
// client that gzipped against a Worker too old to inflate sees, and
// _post_blocking treats a 4xx on a compressed body as "this endpoint
// does not do gzip", falls back to sending it uncompressed, and stops
// compressing for the life of the process. So deploying the client
// before this Worker costs one retry per process, not the data.
return new Response('bad request', { status: 400 });
}
if (records.length === 0) return new Response(null, { status: 204 });
// Hive-style partitioning so DuckDB can prune by date without a catalog.
// Shares partition() with the rollup: the cron lists `sessions/<part>/`, so
// two independent spellings of this scheme would mean the writer and the
// compactor could drift apart and silently match zero objects.
// ponytail: one object per request. Compacted hourly into rollup/ by
// the queue consumer above — analysis reads that, never this. The random
// UUID is load-bearing there: its first character is the slice.
const key = `sessions/${partition(new Date())}/${crypto.randomUUID()}.json`;
const ndjson = records.map((r) => JSON.stringify(r)).join('\n');
// Respond immediately; durability work continues after the response.
// The client is fire-and-forget and ignores the status anyway — making it
// wait on R2 would only add latency to someone else's coding session.
ctx.waitUntil(
env.CORPUS.put(key, ndjson, {
httpMetadata: { contentType: 'application/x-ndjson' },
})
);
// Optional second lane: forward verbatim OTLP to a metrics backend for
// dashboards. Configured by secret, so it can be added or swapped with a
// `wrangler secret put` and no code change.
if (env.METRICS_OTLP_URL) {
ctx.waitUntil(
fetch(env.METRICS_OTLP_URL, {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: env.METRICS_OTLP_AUTH || '',
},
body: JSON.stringify({ resourceLogs: [{ scopeLogs: [{ logRecords: records.map((r) => ({ body: { stringValue: JSON.stringify(r) } })) }] }] }),
}).catch(() => {})
);
}
return new Response(null, { status: 204 });
},
};