fix(beacon): enforce the allowlist below the top-level keys (#4691)

## Description

The header of `deploy/beacon/worker.js` promises to drop every field not
on the allowlist before anything is written, but `pick()` only filtered
top-level keys: whatever sat under an allowed key (`session`, `models`,
`config`, ...) was stored verbatim. This adds a nested schema (`NESTED`)
that mirrors `_Session.payload()` / `_payload_v2()` in
`headroom/telemetry/session.py`, so an unknown nested key, a non-scalar
value, or free text under an allowed key is dropped server-side, while
every field the current client sends is kept.

Fixes #4689

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `deploy/beacon/worker.js`: new `NESTED` table, one entry per
`ALLOWED_KEYS` key, describing the nested shape session.py emits. Leaves
are `num` (finite number), `bool`, or `str` (a token of at most 128
characters with no whitespace: slugs, hex ids, public model ids,
versions). `mapOf()` covers objects whose keys are data (`transforms`,
`skips`, `sources`, `failure_statuses`, `errors.by_status`, `hist`); map
keys are held to the same token rule. `trajectory.kinds` is checked
against the closed `_TURN_KINDS` run-length alphabet.
- `pick()` now runs each kept value through `clean()`. `ALLOWED_KEYS`
stays a plain array (the regex in
`tests/test_beacon_schema_v2.py::test_allowlist_covers_every_emitted_key`
still parses it); module load throws if `ALLOWED_KEYS` and `NESTED`
disagree, so a key added to one but not the other fails the deploy
instead of being silently dropped.
- Resource attributes are held to plain token strings (the client only
ever sends `stringValue` there).
- `routing` is emitted by an extension and its shape is not defined in
this repository, so it is bounded rather than described: numbers,
booleans and tokens only, at most 4 levels deep. Giving it a fixed
schema needs the extension's payload and is left to a maintainer.
- `deploy/beacon/test-rollup.mjs`: a new block drives `worker.fetch()`
with the repository's `sample-event.json` (empty lists filled with rows
of the shape session.py emits) and asserts it is stored unchanged; then
injects unknown nested keys, free text, an object where a slug belongs
and a non-scalar resource attribute, and asserts each is dropped.
- No size cap on list or map length was added: the existing 256 KB
body/decompression caps already bound those, and a count cap could drop
real rows (`compression.transforms` is not capped client-side).

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

pytest was run only for the two beacon test files that read `worker.js`
(below); ruff/mypy do not cover the changed JavaScript. The repository
has no JS linter configured for `deploy/beacon`.

### Test Output

Before the fix (new test block, unmodified `worker.js`), `node
test-rollup.mjs`:

```text
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 103 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
AssertionError [ERR_ASSERTION]: a nested field outside the allowlist was stored
+ actual - expected
...
+         prompt: 'secret',
...
+       value: 'https://internal.example/x?token=1 2'
...
+     'Please summarise this customer email'
```

(The "client fields kept" assertion before it already passed on
unmodified code, as expected.)

After the fix:

```text
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 103 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
nested allowlist: client fields kept, unknown nested keys and free text dropped
ok
```

Revert check: with the `worker.js` change reverted (`git apply -R`) the
script exits 1 on the same assertion; restored, it exits 0.

Python tests that parse `worker.js` (WSL, Python venv, `pytest
tests/test_beacon_schema_v2.py tests/test_beacon_gzip.py`):

```text
tests/test_beacon_schema_v2.py ........................................  [ 66%]
tests/test_beacon_gzip.py ....................                           [100%]
============================= 60 passed in 31.83s ==============================
```

## Real Behavior Proof

- Environment: Windows 11, Node v26.13.0 (`Request`/`Response` globals),
worker imported as an ES module and its `fetch()` handler called
directly with a stub R2 binding that records `put()`; Python client from
this branch in WSL Ubuntu for the second check.
- Exact command / steps: (1) `deploy/beacon/sample-event.json` with
`session.note = "customer: ACME, see ticket 4411"` and `"please
summarise the attached contract"` appended to `models`, POSTed to
`worker.fetch()`, stored NDJSON read back; run once on `main` and once
on this branch. (2) A busy session built by the real client
(`SessionAggregator` + `build_otlp_logs`, 60 outcomes with 4xx/5xx
statuses, strata labels, passthrough reasons, waste signals, 12
content/tool shape rows, 6 strategies, two `HEADROOM_*` config vars),
POSTed to `worker.fetch()`, stored row compared leaf by leaf against the
payload the client built.
- Observed result: (1) on `main` the stored row contains
`"note":"customer: ACME, see ticket 4411"` and
`"models":["claude-sonnet-5","please summarise the attached contract"]`;
on this branch it contains
`"session":{"id":"4996e02d2b7cafe7","seq":0,"duration_s":135,"turns":4,"ended":"shutdown","final":true}`
and `"models":["claude-sonnet-5"]`, status 204 in both. (2) `real client
session: status 204, 20 top-level keys, 4953 B of JSON, fields lost: 0`,
resource stored with all 7 attributes.
- Not tested: the Workers runtime (workerd) and a `wrangler deploy`; the
deployed worker; the `routing` payload of the real extension (not in
this repository, so the bounded rule was checked only against a made-up
shape); payloads from older released clients (I compared the emitted key
set of every `payload()` revision since 9cfb0083 against main and found
no key removed, but did not replay old payloads); the optional
`METRICS_OTLP_URL` lane beyond the fact that it already forwards the
filtered `records`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; this is the beacon receiver Worker,
deployed separately from the client.
- Minimum rollout channel: n/a (server-side Worker deploy).
- Stable/default behavior changed: yes, server-side only: nested fields
not described in `NESTED`, and strings that are not short tokens, are no
longer stored. Every field the current client sends is kept (checked
above).
- Kill switch / disable path: redeploy the previous `worker.js`.
- Unsafe override required: no.
- Qualification impact: none for the client; a future client field
nested under an existing key now needs a `NESTED` entry first, the same
ordering rule `ALLOWED_KEYS` already has.
- Rollback path: `wrangler deploy` of the previous revision; nothing is
migrated or rewritten in R2.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no docs page describes the receiver's filtering; the
worker header comment was updated to mention `NESTED`.
- Design choice for a maintainer: `routing` is bounded, not described,
because its shape lives in an extension. If you would rather keep it
verbatim until that schema exists, it is one line (`routing: 'any'`) to
change.
- The token rule for strings (`^[\w.:/@+|-]{0,128}$`) was chosen to
cover litellm model ids such as
`us.anthropic.claude-3-5-sonnet-20241022-v2:0` and
`vertex_ai/gemini-1.5-pro@001`; a public model id longer than 128
characters or containing another character would be dropped.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4691"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
HetCreep
2026-11-07 20:35:01 -06:00
committed by GitHub
co-authored by Claude Opus 5.5
parent ced26852c6
commit 456288da07
2 changed files with 228 additions and 6 deletions
+94
View File
@@ -289,4 +289,98 @@ if (dir) {
console.log('rate limit: 429 over, fail-open on outage and when unbound');
}
// ---- fetch(): the nested allowlist -------------------------------------------
//
// The allowlist has to hold BELOW the top-level keys too: a nested field a
// client bug adds, or free text under an allowed key, must not be stored. And
// nothing the current client sends may be lost on the way in. The sample event
// is the real client's output; the lists it leaves empty are filled with rows
// of the shape session.py emits, so every nested field is exercised.
{
/** Plain JS -> OTLP AnyValue, as _any_value() in session.py builds it. */
const any = (v) =>
typeof v === 'boolean' ? { boolValue: v }
: Number.isInteger(v) ? { intValue: String(v) }
: typeof v === 'number' ? { doubleValue: v }
: typeof v === 'string' ? { stringValue: v }
: Array.isArray(v) ? { arrayValue: { values: v.map(any) } }
: { kvlistValue: { values: Object.entries(v).map(([key, x]) => ({ key, value: any(x) })) } };
const sample = JSON.parse(readFileSync(new URL('./sample-event.json', import.meta.url), 'utf8'));
const resourceAttrs = sample.resourceLogs[0].resource.attributes;
const plain = (v) =>
'kvlistValue' in v
? Object.fromEntries(v.kvlistValue.values.map((kv) => [kv.key, plain(kv.value)]))
: 'arrayValue' in v ? v.arrayValue.values.map(plain)
: 'intValue' in v ? Number(v.intValue)
: Object.values(v)[0];
const full = plain(sample.resourceLogs[0].scopeLogs[0].logRecords[0].body);
full.compression.transforms.extracted_3_dynamic_elements = 2;
full.compression.by_strategy = [{ strategy: 'smart_crusher', n: 2, tokens_in: 1500, tokens_out: 700 }];
full.skips = { 'passthrough:bypass_header': 1 };
full.failure_statuses = { 529: 1 };
full.models.push('us.anthropic.claude-3-5-sonnet-20241022-v2:0', 'vertex_ai/gemini-1.5-pro@001');
full.quality.waste = [{ kind: 'json_bloat', tokens: 40 }];
full.shapes.by_content = [{ content: 'json', strategy: 'smart_crusher', n: 1, tokens_in: 9, tokens_out: 3 }];
full.shapes.by_tool = [{ shape: 'f2d3_a', n: 1, tokens_in: 9, tokens_out: 3 }];
full.trajectory.kinds = 's12p3z40';
full.errors = { count: 1, by_status: { 429: 1 } };
full.strata = [{ dim: 'arm', value: 'treatment', n: 2 }];
full.config = [{ key: 'mode', value: 'token' }];
full.routing = { decisions: 3, by_model: [{ model: 'claude-haiku-4-5', n: 3 }], min_prefix: 1024 };
/** One OTLP record through fetch(); returns what would be written to R2. */
const store = async (body, resource = resourceAttrs) => {
const otlp = { resourceLogs: [{ resource: { attributes: resource },
scopeLogs: [{ logRecords: [{ body: any(body) }] }] }] };
const pending = [];
const written = {};
const CORPUS = { put: async (key, value) => { written[key] = value; } };
const res = await worker.fetch(
new Request('https://beacon.test/v1/logs', { method: 'POST', body: JSON.stringify(otlp) }),
{ CORPUS },
{ waitUntil: (p) => pending.push(p) }
);
await Promise.all(pending);
assert.equal(res.status, 204);
const [line] = Object.values(written);
const { received_at, ...rec } = JSON.parse(line);
assert.ok(received_at);
return rec;
};
const { resource, ...stored } = await store(full);
assert.deepEqual(stored, full, 'a field the current client sends was dropped');
assert.deepEqual(resource, plain({ kvlistValue: { values: resourceAttrs } }));
const injected = structuredClone(full);
injected.session.note = 'hello'; // unknown nested key
injected.tokens.prompt_sample = 12; // unknown nested counter
injected.compression.by_strategy[0].prompt = 'secret'; // unknown key in a record
injected.models.push('Please summarise this customer email'); // free text under a list
// Credential-shaped fixtures are assembled at run time so secret scanners do not
// flag this file; they are not real keys.
const fakeKey = ['sk', 'live', 'abc123'].join('_');
const fakeAwsId = 'AK' + 'IA' + 'X'.repeat(16);
injected.models.push(fakeKey, 'https://internal.example/admin'); // slug-shaped secret, URL
injected.strata.push({ dim: 'arm', value: fakeAwsId, n: 1 }); // key-shaped value
injected.config.push({ key: 'endpoint', value: 'https://internal.example/x?token=1 2' });
injected.session.ended = { nested: 'object where a slug belongs' };
injected.routing.note = 'free text with spaces';
injected.trajectory.kinds = 'see attached transcript';
const hostile = [...resourceAttrs.filter((a) => a.key !== 'headroom.stack'),
{ key: 'headroom.stack', value: any({ cwd: '/home/me' }) }];
const { resource: badResource, ...dropped } = await store(injected, hostile);
const expected = structuredClone(full);
delete expected.session.ended;
delete expected.trajectory.kinds;
expected.config.push({ key: 'endpoint' });
expected.strata.push({ dim: 'arm', n: 1 });
assert.deepEqual(dropped, expected, 'a nested field outside the allowlist was stored');
assert.equal(badResource['headroom.stack'], undefined, 'non-scalar resource value stored');
console.log('nested allowlist: client fields kept, unknown nested keys and free text dropped');
}
console.log('ok');
+134 -6
View File
@@ -9,8 +9,8 @@
* Deployed at headroom-beacon.headroom-beacon.workers.dev (otlp.headroomlabs.ai
* is planned but not yet routed; see wrangler.toml). Three jobs:
*
* 1. Allowlist. Drop every field not on ALLOWED_KEYS before anything is
* written. This is the only privacy control that works retroactively —
* 1. Allowlist. Drop every field not on ALLOWED_KEYS, and every nested
* field not described by NESTED, before anything is written. This is the only privacy control that works retroactively —
* if a future client version ships a bug that leaks a field, we cannot
* patch the installs already in the wild, but we can stop storing it
* here in one deploy.
@@ -114,6 +114,131 @@ const ALLOWED_RESOURCE = [
'host.arch',
];
// What may sit UNDER each allowed key. Without this the allowlist stops at the
// top level and a nested field a client bug adds is stored verbatim -- the very
// leak the header says one deploy here can stop. Mirrors _Session.payload() and
// _payload_v2() in headroom/telemetry/session.py. A nested key absent here is
// dropped, so the ordering rule on ALLOWED_KEYS applies to it too: teach it
// here before a client sends it.
//
// Leaves are 'num' (a finite number), 'bool', or 'str': a token of at most 128
// characters with no whitespace, which is every string the client sends (slugs,
// hex ids, public model ids, versions). Free text does not fit it. `[spec]` is
// a list of spec; mapOf(spec) is an object whose KEYS are data (transform
// names, skip reasons, status codes), each key itself held to 'str'.
const MAP = Symbol('map');
const mapOf = (spec) => ({ [MAP]: spec });
const nums = (...keys) => Object.fromEntries(keys.map((k) => [k, 'num']));
const NESTED = {
schema_version: 'num',
session: {
id: 'str', seq: 'num', duration_s: 'num', turns: 'num', ended: 'str', final: 'bool',
},
tokens: nums(
'original', 'attempted', 'input', 'output', 'saved', 'tool_saved',
'cache_read', 'cache_write', 'uncached'
),
rates: nums(
'saved_pct', 'eligible_pct', 'yield_pct', 'all_layers_saved_pct',
'all_layers_yield_pct', 'cache_read_pct', 'overhead_pct'
),
compression: {
transforms: mapOf('num'),
by_strategy: [{ strategy: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
...nums(
'overhead_ms_total', 'latency_ms_total', 'overhead_ms_per_turn',
'latency_ms_per_turn', 'passthrough_turns', 'response_cache_hits'
),
},
skips: mapOf('num'),
sources: mapOf('num'),
providers: ['str'],
models: ['str'],
failures: 'num',
failure_statuses: mapOf('num'),
// Emitted by an extension, not by session.py, so this repository does not
// define its shape. Bounded rather than described: numbers, booleans and
// tokens only, at most ROUTING_DEPTH levels deep. It gets a real schema here
// once the extension's payload is written down.
routing: 'any',
quality: {
...nums('reread_tokens', 'reread_compressed_tokens', 'waste_turns'),
waste: [{ kind: 'str', tokens: 'num' }],
},
hist: mapOf({ edges: ['num'], counts: ['num'] }),
shapes: {
by_content: [{ content: 'str', strategy: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
by_tool: [{ shape: 'str', ...nums('n', 'tokens_in', 'tokens_out') }],
truncated: 'bool',
},
cache: {
gap_edges_s: ['num'], hits: ['num'], misses: ['num'],
...nums('write_5m', 'write_1h', 'inferred_turns'),
},
trajectory: {
buckets: 'num', turns: ['num'], tokens: ['num'], saved: ['num'],
// Runs over the closed _TURN_KINDS alphabet, at most MAX_TRAJECTORY_RUNS.
kinds: /^(?:[cfepzso]\d{1,9}){0,64}$/,
kinds_truncated: 'bool',
msgs_max: 'num',
},
clients: [{ client: 'str', n: 'num' }],
errors: { count: 'num', by_status: mapOf('num') },
strata: [{ dim: 'str', value: 'str', n: 'num' }],
config: [{ key: 'str', value: 'str' }],
};
// A key allowed at the top but given no shape would be dropped in full. Fail
// the deploy instead of shipping that.
if ([...ALLOWED_KEYS].sort().join() !== Object.keys(NESTED).sort().join()) {
throw new Error('beacon: ALLOWED_KEYS and NESTED disagree');
}
const TOKEN = /^[\w.:/@+|-]{0,128}$/;
// A slug-shaped string can still be a URL or a credential. Model ids and the
// other slugs the client sends never carry a scheme or a well-known key prefix,
// so those are dropped even though they pass TOKEN.
const SECRETISH = /:\/\/|^(?:sk|pk|rk)[-_]|^(?:ghp|gho|ghu|ghs|ghr)_|^github_pat_|^glpat-|^xox[abposr]-|^(?:AKIA|ASIA)[A-Z0-9]{12}|^AIza|^hf_|^eyJ/i;
const isToken = (s) => TOKEN.test(s) && !SECRETISH.test(s);
const ROUTING_DEPTH = 4;
/** `value` reduced to what `spec` allows, or undefined when none of it is. */
function clean(spec, value, depth = 0) {
if (spec === 'any') {
if (depth > ROUTING_DEPTH) return undefined;
spec = Array.isArray(value) ? ['any']
: value && typeof value === 'object' ? mapOf('any')
: typeof value === 'string' ? 'str'
: typeof value === 'number' ? 'num' : 'bool';
}
if (spec === 'num') return Number.isFinite(value) ? value : undefined;
if (spec === 'bool') return typeof value === 'boolean' ? value : undefined;
if (spec === 'str') return typeof value === 'string' && isToken(value) ? value : undefined;
if (spec instanceof RegExp) {
return typeof value === 'string' && spec.test(value) ? value : undefined;
}
if (Array.isArray(spec)) {
if (!Array.isArray(value)) return undefined;
return value.map((v) => clean(spec[0], v, depth + 1)).filter((v) => v !== undefined);
}
if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined;
const out = {};
if (spec[MAP]) {
for (const [key, v] of Object.entries(value)) {
const kept = isToken(key) ? clean(spec[MAP], v, depth + 1) : undefined;
if (kept !== undefined) out[key] = kept;
}
return out;
}
for (const [key, s] of Object.entries(spec)) {
if (!Object.hasOwn(value, key)) continue;
const kept = clean(s, value[key], depth + 1);
if (kept !== undefined) out[key] = kept;
}
return out;
}
// Cap on what ARRIVES, compressed or not.
//
// Was 64KB, chosen when the comment above it read "a beacon event is ~2KB" --
@@ -183,11 +308,13 @@ function unwrap(value) {
return null;
}
function pick(obj, allowed) {
function pick(obj, allowed, specOf) {
const out = {};
if (!obj || typeof obj !== 'object') return out;
for (const key of allowed) {
if (key in obj) out[key] = obj[key];
if (!(key in obj)) continue;
const kept = clean(specOf(key), obj[key]);
if (kept !== undefined) out[key] = kept;
}
return out;
}
@@ -200,14 +327,15 @@ function extract(payload) {
for (const attr of rl.resource?.attributes || []) {
resource[attr.key] = unwrap(attr.value);
}
const cleanResource = pick(resource, ALLOWED_RESOURCE);
// Every resource attribute the client sends is a plain string.
const cleanResource = pick(resource, ALLOWED_RESOURCE, () => 'str');
for (const sl of rl.scopeLogs || []) {
for (const rec of sl.logRecords || []) {
const body = unwrap(rec.body);
if (!body || typeof body !== 'object') continue;
records.push({
...pick(body, ALLOWED_KEYS),
...pick(body, ALLOWED_KEYS, (key) => NESTED[key]),
resource: cleanResource,
// Server-stamped. A client clock can be wrong or forged; this is the
// timestamp partitioning and retention actually rely on.