feat(beacon): schema v2 — routing/compression training signal, gzip transport (#3253)

## What

Nine new sections on the session heartbeat — `quality`, `hist`,
`shapes`, `cache`, `trajectory`, `clients`, `errors`, `strata`, `config`
— plus a gzip transport. Everything is a counter, a ratio, or a slug
from a fixed list in the source. No prompts, code, file paths, tool
names, project names, or error text can appear.

## No regression in v1

Every existing field keeps its name, type, and meaning.
`tests/test_beacon_schema_v2.py` locks the v1 surface with literal key
lists and a shape-stability invariant, so a v1 field that silently
changes type fails there rather than in the corpus six weeks later.

The v2 section is built in `_payload_v2()` behind `try/except`, and
`_fold_extra()` likewise. A bug in new code can cost the new fields and
nothing else. The first version built one dict literal — any raise
dropped `tokens.saved` along with it.

## The STRUCT/MAP flip

Four fields ship as lists of records rather than objects keyed by data
(`clients`, `config`, `strata`, `waste`). DuckDB infers a JSON object as
`STRUCT` when the keys are few and identifier-shaped, and as `MAP`
otherwise — so a key set that grows with the fleet flips type mid-corpus
and `BINDER ERROR`s the reader.

Caught against the real 369k-session corpus, not in theory: `clients`
inferred `STRUCT(claude_code, codex)` at two harnesses and `MAP(VARCHAR,
BIGINT)` at 22, and `clients.cursor` errors on one of them.

## Transport

Body is gzipped at level 6 (~6x). The receiver sniffs magic bytes rather
than trusting `Content-Encoding`, enforces a decompression-bomb limit
*during* the stream read, and leaves the plain path byte-identical for
the existing fleet.

A `400`/`415` disables gzip for the process and retries once. A `413`
does not — an uncompressed retry is 6x larger and guaranteed to fail
again.

At current rates (~235k heartbeats/day): v1 today 892 MB/day, v2
uncompressed 1,802 MB/day, **v2 gzipped 295 MB/day** — a net reduction
against what we send now.

## `headroom telemetry`

Prints the real payload, built by the same `_Session.payload()` the
beacon uses. \"Go read worker.js\" answers what happens to the data; it
does not answer what leaves your machine.

Uses `read_install_id()` and
`resource_attributes(create_install_id=False)` — the default mints and
persists the id, so building the report would itself be the act that
identifies the machine. Filtering the id out of the *output* is not
enough; by then the file exists.

## Receiver

Allowlist gains the nine keys. `MAX_BODY_BYTES` 64KB → 256KB — that
figure dated from \"a beacon event is ~2KB\", and worst-case v2 is
81,965 B uncompressed.

## Verification

Against the production corpus (369,015 sessions): **14/14** `beacon.sh`
reports pass, **18/18** dashboard queries pass, v1 totals byte-identical
at `273,495,971,478` tokens saved.

`sample-event.json` regenerated as a real v2 payload. The old v1-only
version would have passed the post-deploy smoke test whether or not the
new allowlist landed; the new one distinguishes the cases 9/9 vs 0/9.

Suite: 11,655 passed, 1 failed —
`test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`,
which reproduces identically on clean `main`.

## Not verified here

`node test-rollup.mjs` and `npx wrangler dev` have not been run — no
`node` in the authoring environment. `DecompressionStream` is the one
path with no fallback and has never been executed. **Both are required
before deploy.**

## Deploy order

**Worker first, client second.** The worker is backward-compatible with
the v1 fleet (allowlist additive, plain path byte-identical, cap only
raised). The reverse is not: the worker strips unknown keys at ingest,
so v2 data arriving before the allowlist lands is gone, not recoverable
by re-rolling. The corpus already carries 6 such rows from a dev proxy.

Merging this PR does not ship the client — releases are separate
commits.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-09-19 11:58:29 -07:00
committed by GitHub
co-authored by Tejas Chopra Claude Opus 5 JD Davis
parent bc21c93707
commit 67eb910e38
12 changed files with 3859 additions and 71 deletions
File diff suppressed because it is too large Load Diff
+44 -1
View File
@@ -13,7 +13,7 @@
*/
import { readdirSync, readFileSync } from 'node:fs';
import assert from 'node:assert/strict';
import { oldestRawDay, rollupHour } from './worker.js';
import { inflate, oldestRawDay, rollupHour } from './worker.js';
// R2 returns at most 1000 keys per list page, so on a real hour (~4,000
// objects) the cursor loop in rollupHour is load-bearing. The stub paginates at
@@ -211,4 +211,47 @@ if (dir) {
);
}
// 8. Upload compression. A schema-v2 event is ~8KB of repetitive JSON and
// gzips ~6x, so fetch() sniffs the gzip magic number and inflates. Three
// things have to hold, and none of them can be checked in production without
// risking every upload:
//
// a) a real gzip body round-trips to the exact bytes that went in;
// b) DecompressionStream exists in this runtime at all (it is the one part
// of the ingest path with no fallback -- if it is missing, inflate throws,
// fetch answers 400, and every compressed upload is refused);
// c) the size cap is enforced WHILE the stream is read, not after, or a
// decompression bomb has already been decompressed by the time it is
// noticed.
{
const gzipBytes = async (text) => {
const stream = new Response(new TextEncoder().encode(text)).body.pipeThrough(
new CompressionStream('gzip')
);
return new Uint8Array(await new Response(stream).arrayBuffer());
};
const payload = JSON.stringify({
resourceLogs: [{ scopeLogs: [{ logRecords: [{ body: { stringValue: 'x'.repeat(4000) } }] }] }],
});
const packed = await gzipBytes(payload);
assert.equal(packed[0], 0x1f, 'gzip magic byte 0');
assert.equal(packed[1], 0x8b, 'gzip magic byte 1');
assert.ok(packed.length < payload.length / 3, 'gzip did not actually compress');
assert.equal(await inflate(packed.buffer, 256 * 1024), payload, 'round trip changed the body');
// A bomb: highly compressible input that expands past the cap. The cap must
// bite, and it must bite from the stream rather than from the final size.
const bomb = await gzipBytes('A'.repeat(2 * 1024 * 1024));
assert.ok(bomb.length < 64 * 1024, 'fixture is not actually a bomb');
await assert.rejects(
() => inflate(bomb.buffer, 256 * 1024),
/too large/,
'a decompression bomb was inflated in full'
);
console.log(`gzip: ${payload.length} B -> ${packed.length} B, bomb refused`);
}
console.log('ok');
+122 -3
View File
@@ -53,6 +53,51 @@ const ALLOWED_KEYS = [
// real minimum cacheable prefix, how long a cache actually survives, and how
// far predicted cache hits are from the ones that happened.
'routing',
// ---- schema v2 -----------------------------------------------------------
//
// Additive. Every v1 key above is untouched, so a v1 client keeps storing
// exactly what it stores today and rows from the two versions union cleanly
// (`schema_version` separates them, `union_by_name` handles the rest).
//
// Read the ordering rule at the top of this list literally: this deploy must
// go out BEFORE the client release that starts sending these, or the first
// weeks of the new signals are dropped on arrival and cannot be recovered.
// Removing one line here is also the rollback -- it reverts that signal for
// every install in the wild without a client release, which is the whole
// reason the allowlist is server-side.
// Regret. `reread_compressed_tokens` is the counter-pressure on `saved_pct`:
// tokens the agent had to fetch again because compression removed them.
// Without it a compression ratio is unfalsifiable.
'quality',
// Fixed-bucket distributions for quantities otherwise reported only as
// session sums. Ships its own bucket edges, so no reader needs to know which
// client version wrote a row.
'hist',
// (content_type x strategy x yield), and the same for JSON tool output keyed
// by structural shape. The input term the corpus has never had.
// Structural descriptors only -- never TOIN's structure_hash, which is a
// digest of field names.
'shapes',
// Prompt-cache survival by gap since the previous turn: the empirical TTL
// and minimum-prefix curve per provider.
'cache',
// Session shape over time -- log-spaced turn buckets plus a run-length
// encoded turn-kind string from a closed seven-letter alphabet.
'trajectory',
// Identified client harness, per turn. `headroom.stack` answers this by
// detection and resolves to a literal "proxy" for most of the fleet.
'clients',
// 4xx, kept strictly separate from the 5xx-only `failures` above so neither
// metric changes meaning. 429 rate is the routing signal here.
'errors',
// output_shaper A/B strata as validated enums. Model family is not among
// them -- see the note in session.py's payload().
'strata',
// Allowlisted, slug-validated configuration, so the corpus can tell
// "compression underperformed" from "compression was switched off".
'config',
];
// Resource attributes we keep. Same rule: allowlist, not denylist.
@@ -66,8 +111,60 @@ const ALLOWED_RESOURCE = [
'host.arch',
];
// A beacon event is ~2KB. Anything far past that is a bug or an attack.
const MAX_BODY_BYTES = 64 * 1024;
// Cap on what ARRIVES, compressed or not.
//
// Was 64KB, chosen when the comment above it read "a beacon event is ~2KB" --
// i.e. 32x the payload. Schema v2 made a typical event ~8KB uncompressed, and
// a session that fills every table (the shape cross product is 10 content
// types x 13 strategies) reaches ~82KB. That silently eroded the multiple to
// under 1x: a busy session sending uncompressed -- the kill switch is set, or
// the gzip fallback fired -- would have been answered 413 and, correctly, not
// retried, so the event would simply be lost.
//
// Restored to ~3x the worst legitimate payload. This is not the abuse control
// and never was: the endpoint is unauthenticated by design and the WAF rate
// limit (60 req/min/IP, see wrangler.toml) is what bounds a bad actor. At 60
// requests a minute the difference between these two ceilings is 3.8 MB/min
// and 15 MB/min, neither of which is interesting to Cloudflare.
const MAX_BODY_BYTES = 256 * 1024;
// Cap on what an arriving body EXPANDS to. `raw.byteLength` cannot see this:
// a small gzip can expand to gigabytes, and a decompression bomb that is only
// noticed after it has been decompressed has already won. Enforced while the
// stream is read, not after. ~3x the worst legitimate payload, same as above.
const MAX_DECOMPRESSED_BYTES = 256 * 1024;
/**
* Decompress a gzip body, refusing anything that expands past `limit`.
*
* Counts bytes as chunks arrive and aborts mid-stream, so a bomb costs the
* limit rather than whatever it would have expanded to.
*/
export async function inflate(buffer, limit) {
const stream = new Response(buffer).body.pipeThrough(new DecompressionStream('gzip'));
const reader = stream.getReader();
const chunks = [];
let total = 0;
try {
for (;;) {
const { done, value } = await reader.read();
if (done) break;
total += value.byteLength;
if (total > limit) throw new Error('decompressed body too large');
chunks.push(value);
}
} finally {
// Releases the stream on the bomb path; a no-op once it has closed.
reader.cancel().catch(() => {});
}
const joined = new Uint8Array(total);
let offset = 0;
for (const chunk of chunks) {
joined.set(chunk, offset);
offset += chunk.byteLength;
}
return new TextDecoder().decode(joined);
}
/** OTLP AnyValue -> plain JS. The inverse of _any_value() in session.py. */
function unwrap(value) {
@@ -315,9 +412,31 @@ export default {
let records;
try {
records = extract(JSON.parse(new TextDecoder().decode(raw)));
// Sniff the gzip magic number rather than trusting Content-Encoding.
// Three things can each independently decide whether a body arrives
// compressed -- the client, Cloudflare's edge, and any proxy between
// them -- and only the bytes know which of them acted. Keying off the
// header instead means a body that something already decompressed gets
// fed to DecompressionStream, or vice versa, and every upload 400s.
//
// It also means an old client that never sets the header keeps working
// through the identical code path it uses today: no header, no magic
// number, straight to TextDecoder.
const bytes = new Uint8Array(raw);
const gzipped = bytes.length >= 2 && bytes[0] === 0x1f && bytes[1] === 0x8b;
const text = gzipped
? await inflate(raw, MAX_DECOMPRESSED_BYTES)
: new TextDecoder().decode(raw);
records = extract(JSON.parse(text));
} catch {
// Malformed input is not worth a retry storm from clients.
//
// This 400 is also load-bearing in the other direction: it is what a
// client that gzipped against a Worker too old to inflate sees, and
// _post_blocking treats a 4xx on a compressed body as "this endpoint
// does not do gzip", falls back to sending it uncompressed, and stops
// compressing for the life of the process. So deploying the client
// before this Worker costs one retry per process, not the data.
return new Response('bad request', { status: 400 });
}
if (records.length === 0) return new Response(null, { status: 204 });
+1
View File
@@ -78,6 +78,7 @@ def _register_commands() -> None:
recover, # noqa: F401
rollout, # noqa: F401
savings, # noqa: F401
telemetry, # noqa: F401
tools, # noqa: F401
update, # noqa: F401
wrap, # noqa: F401
+147
View File
@@ -0,0 +1,147 @@
"""``headroom telemetry`` — show exactly what the beacon would send.
The beacon is opt-OUT and on by default. "The receiver is open source, go read
worker.js" is a good answer to what happens to the data, but it is a poor
answer to what LEAVES YOUR MACHINE — that requires reading two files, trusting
that the copy on GitHub matches the copy installed, and reconstructing a
payload by hand.
This prints the real thing: the current process's resource attributes and a
session payload built by the same `_Session.payload()` the beacon uses, with
representative counters folded in. Nothing here is a mock-up of the schema; if
a field can appear on the wire, it appears here.
"""
from __future__ import annotations
import json
from typing import Any
import click
from .main import main
def _sample_payload() -> dict[str, Any]:
"""One session payload, built through the real aggregator.
Uses the actual fold path rather than hand-writing a dict, so this cannot
drift from what is sent. The token counts are invented; the SHAPE — every
key, and the fact that every value is a counter, a ratio or a bounded slug
— is exactly what a real session produces.
"""
from headroom.telemetry import session as telemetry_session
class _Example:
provider = "anthropic"
model = "gpt-4o"
original_tokens = 120_000
attempted_input_tokens = 40_000
optimized_tokens = 90_000
provider_input_tokens = 90_000
output_tokens = 1_200
tokens_saved = 30_000
cache_read_tokens = 60_000
cache_write_tokens = 8_000
cache_write_5m_tokens = 6_000
cache_write_1h_tokens = 2_000
uncached_input_tokens = 22_000
status_code = 200
total_latency_ms = 4_200.0
overhead_ms = 48.0
ttfb_ms = 900.0
num_messages = 140
client = "claude_code"
transforms_applied = ("crush",)
tags: dict[str, Any] = {}
waste_signals = {"reread": 900, "reread_compressed": 400, "json_bloat": 50}
rows: list[dict[str, Any]] = []
aggregator = telemetry_session.SessionAggregator(rows.append)
for index in range(3):
aggregator.record(_Example(), now=1_000.0 + index * 45.0)
aggregator.flush_all()
return rows[-1]
@main.command()
@click.option(
"--show",
"show",
is_flag=True,
help="Print the exact payload the beacon would send, as JSON.",
)
@click.option(
"--json",
"as_json",
is_flag=True,
help="Print JSON only, with no explanatory footer. Implies --show.",
)
def telemetry(show: bool, as_json: bool) -> None:
"""Show what Headroom's telemetry beacon collects, and whether it is on.
\b
Examples:
headroom telemetry Current state and the two switches
headroom telemetry --show The exact payload, field by field
"""
from headroom.telemetry.beacon import is_beacon_enabled, is_telemetry_enabled
from headroom.telemetry.session import (
DEFAULT_ENDPOINT,
SCHEMA_VERSION,
_gzip_enabled,
read_install_id,
resource_attributes,
)
beacon_on = is_beacon_enabled()
local_on = is_telemetry_enabled()
known_id = read_install_id()
# Reported because it changes what goes on the wire, and because while the
# transport is staged this is the fastest way to answer "was that upload
# compressed?" without instrumenting anything.
gzip_on = _gzip_enabled()
if show or as_json:
report = {
"beacon_enabled": beacon_on,
"schema_version": SCHEMA_VERSION,
"endpoint": DEFAULT_ENDPOINT,
"gzip": gzip_on,
"install_id": known_id,
# create_install_id=False, and it is load-bearing: the default
# mints and persists an id, so building this report would be the
# act that identifies the machine. Filtering the id out of the
# OUTPUT is not enough -- by then the file exists.
"resource": resource_attributes(create_install_id=False),
"session_event": _sample_payload(),
}
click.echo(json.dumps(report, indent=2, sort_keys=True))
if as_json:
return
click.echo()
click.echo(
"The COUNTS above are made up. The FIELDS are not: this payload is built\n"
"by the same code the beacon uses, so every key that can ever be sent is\n"
"shown, and no other key can be.\n"
"\n"
"Every value is a counter, a ratio, or a slug from a fixed list in the\n"
"source. No prompts, code, file paths, tool names, project names or error\n"
"text can appear — see headroom/telemetry/session.py for what is built,\n"
"and deploy/beacon/worker.js for the allowlist it is filtered through on\n"
"arrival. Your install_id is a random UUID, not a machine fingerprint;\n"
"delete ~/.headroom/install_id to reset it. Your IP is never read."
)
return
click.echo(f"Upload beacon: {'ON (opt-out)' if beacon_on else 'OFF'}")
click.echo(f"Local stats: {'ON' if local_on else 'OFF (opt-in)'}")
click.echo(f"Endpoint: {DEFAULT_ENDPOINT}")
click.echo(
f"Compression: {'gzip' if gzip_on else 'off (plain JSON)'}"
f"{'' if gzip_on else ' — enable: HEADROOM_BEACON_GZIP=1'}"
)
click.echo(f"Install ID: {known_id or '(none yet — created on first upload)'}")
click.echo()
click.echo("Turn the beacon off: HEADROOM_BEACON=off (or DO_NOT_TRACK=1)")
click.echo("See the exact payload: headroom telemetry --show")
+11 -4
View File
@@ -124,17 +124,24 @@ def format_telemetry_notice(*, prefix: str = "") -> str:
# so this notice is the only place a user learns it is running at all —
# surprise is what turns anonymous telemetry into a trust incident.
#
# Wording: lead with what the data *is* (compression ratios) and why it is
# useful (understanding when compression works), not with the fact of
# Wording: lead with what the data *is* (counters) and why it is useful
# (understanding when compression works), not with the fact of
# transmission. "Stats are sent to Headroom Labs" is accurate but reads like
# extraction, which misrepresents a payload that is entirely counters. The
# reassurance has to stay concrete, though — "never prompts, code, or file
# paths" names the things people actually worry about, and every one of
# those is enforced by the allowlist, not just promised here.
#
# "compression stats" until schema v2, which widened the payload to cache
# behaviour, session shape, harness and configuration. All still counters,
# all still content-free — but this notice is the only place an opt-out
# beacon is disclosed at all, so it has to describe what is actually sent
# rather than the narrowest true thing. `headroom telemetry --show` prints
# the exact payload for anyone who wants the specifics.
if beacon:
return (
f"{prefix}Telemetry: anonymous compression stats — never prompts, code, or "
"file paths. Helps us improve compression | Off: HEADROOM_BEACON=off"
f"{prefix}Telemetry: anonymous usage counters — never prompts, code, or file "
"paths. See: headroom telemetry --show | Off: HEADROOM_BEACON=off"
)
return (
f"{prefix}Telemetry: ENABLED (local aggregate stats only — nothing sent externally) | "
File diff suppressed because it is too large Load Diff
+17
View File
@@ -565,6 +565,23 @@ class ToolIntelligenceNetwork:
model_family: Target model family (`claude-3-5`, `gpt-4o`, …).
Defaults to `DEFAULT_MODEL_FAMILY` when not provided.
"""
# The beacon's copy of this event is taken BEFORE the enabled check.
# TOIN's own store is gated on HEADROOM_TELEMETRY, which is opt-in and
# therefore off across almost the whole fleet -- so recording after the
# gate would mean the network in "Tool Output Intelligence Network"
# only ever sees the installs that least need it.
#
# Costs nothing extra: `tool_signature` was already built by the
# caller, and `record_tool_shape` reads a handful of its integer
# attributes and no hash at all. Off by default and never raises, like
# every other beacon entry point.
try:
from headroom.telemetry.session import record_tool_shape
record_tool_shape(tool_signature, original_tokens, compressed_tokens)
except Exception: # pragma: no cover - telemetry must never break a request
logger.debug("beacon: tool shape recording failed", exc_info=True)
# HIGH FIX: Check enabled FIRST to avoid computing structure_hash if disabled
# This saves CPU when TOIN is turned off
if not self._config.enabled:
+42
View File
@@ -1454,6 +1454,47 @@ class RoutingDecision:
return self.compressed_tokens / self.original_tokens
def _record_beacon_shapes(routing_log: list[RoutingDecision]) -> None:
"""Report (content_type -> strategy -> yield) for one compress() call.
`RoutingDecision` already carries the content type the detector assigned
alongside the strategy that was picked for it, so the joint table costs
nothing to measure -- it has simply never been reported anywhere. The
beacon's `by_strategy` sees the strategy and its yield but not the input,
which leaves it able to rank compressors and unable to say which one suits
a given piece of content.
Deliberately NOT routed through `CompressionObserver`: that protocol is
implemented outside this repo as well, and widening it would break every
such implementation. This is a direct call to a function that is off by
default, cheap when off, and cannot raise.
Runs whether or not an observer is installed, since the beacon path and the
Prometheus path are independent -- an install with no observer still
reports token totals, and would otherwise report an empty shape table
against them.
"""
if not routing_log:
return
try:
from ..telemetry.session import record_content_shapes
# One call, not one per decision: the beacon's staging lock is shared
# with `record_compression` on this same executor thread, and the note
# there is explicit that the contention is real.
record_content_shapes(
(
decision.content_type.value,
decision.strategy.value,
decision.original_tokens,
decision.compressed_tokens,
)
for decision in routing_log
)
except Exception as e: # pragma: no cover - defensive
logger.debug("beacon shape recording failed (non-fatal): %s", e)
@dataclass
class RouterCompressionResult:
"""Result from ContentRouter with routing metadata.
@@ -2561,6 +2602,7 @@ class ContentRouter(Transform):
anyway, swallow at debug level. Compression already succeeded;
a buggy observer must not turn a 200 into a 500.
"""
_record_beacon_shapes(result.routing_log)
if self._observer is None:
return
for d in result.routing_log:
+424
View File
@@ -0,0 +1,424 @@
"""Beacon upload compression, and the fallback that makes it safe to ship.
A schema-v2 event is ~8KB of repetitive JSON and gzips ~6x, which is worth more
than every schema trim available put together. The risk is not the compression;
it is that uploads are fire-and-forget, so an endpoint that cannot read a
gzipped body looks exactly like an endpoint that is working. These tests cover
the compression, and then the three ways it is allowed to stop.
The server here re-implements worker.js's sniff-and-inflate in Python. It is not
the Worker, and cannot prove `DecompressionStream` behaves — but it does pin the
wire contract the two halves have to agree on, which is the part that would
otherwise only be checked in production.
"""
from __future__ import annotations
import gzip
import itertools
import json
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from typing import Any
import pytest
from headroom.telemetry import session as S
class _Recorder(BaseHTTPRequestHandler):
"""Stands in for worker.js's fetch(): sniff magic bytes, inflate, parse."""
def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API
raw = self.rfile.read(int(self.headers.get("content-length", 0)))
box = self.server.box # type: ignore[attr-defined]
if box.get("too_large"):
self.send_response(413)
self.end_headers()
box["over"] += 1
return
if box.get("refuse_gzip") and raw[:2] == b"\x1f\x8b":
# A Worker that predates the inflate path: JSON.parse throws on the
# gzip bytes and it answers 400.
self.send_response(400)
self.end_headers()
box["refused"] += 1
return
if box.get("fail_5xx"):
self.send_response(503)
self.end_headers()
return
body = gzip.decompress(raw) if raw[:2] == b"\x1f\x8b" else raw
box["events"].append(json.loads(body))
box["encodings"].append(self.headers.get("content-encoding"))
box["wire_bytes"].append(len(raw))
self.send_response(204)
self.end_headers()
def log_message(self, *args: Any) -> None:
pass
@pytest.fixture
def collector(monkeypatch):
server = HTTPServer(("127.0.0.1", 0), _Recorder)
server.box = { # type: ignore[attr-defined]
"events": [],
"encodings": [],
"wire_bytes": [],
"refused": 0,
"over": 0,
}
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
host, port = server.server_address
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", f"http://{host}:{port}/v1/logs")
monkeypatch.setenv("HEADROOM_BEACON", "on")
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
# gzip ships opt-in (_GZIP_DEFAULT is False for the staged rollout), so the
# tests that exercise the transport have to turn it on explicitly. The
# default itself is locked by test_gzip_is_opt_in_by_default below, which
# deliberately does NOT use this fixture's environment.
monkeypatch.setenv("HEADROOM_BEACON_GZIP", "1")
# Compression disables itself process-wide on refusal; reset between tests.
monkeypatch.setattr(S, "_gzip_supported", True)
yield server.box # type: ignore[attr-defined]
server.shutdown()
server.server_close()
class Outcome:
provider = "anthropic"
model = "gpt-4o"
original_tokens = 120_000
attempted_input_tokens = 40_000
optimized_tokens = 90_000
output_tokens = 1_200
tokens_saved = 30_000
cache_read_tokens = 60_000
status_code = 200
total_latency_ms = 4_200.0
overhead_ms = 48.0
ttfb_ms = 900.0
num_messages = 140
client = "claude-code"
transforms_applied: tuple[str, ...] = ("crush",)
tags: dict[str, Any] = {}
waste_signals = {"reread": 900, "reread_compressed": 400}
def sample_payload() -> dict[str, Any]:
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
for index in range(40):
agg.record(Outcome(), now=1000.0 + index * 45)
agg.flush_all()
return rows[-1]
def test_payload_survives_the_round_trip_byte_for_byte(collector):
"""Compression must be invisible above the transport: what the collector
parses has to be exactly what the aggregator built."""
payload = sample_payload()
S._post_blocking(payload)
assert len(collector["events"]) == 1
received = collector["events"][0]
expected = S.build_otlp_logs(payload, S.resource_attributes())
body = received["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]["body"]
want = expected["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]["body"]
assert body == want, "the payload changed in transit"
def test_it_is_actually_compressed_and_declared(collector):
S._post_blocking(sample_payload())
assert collector["encodings"] == ["gzip"], "Content-Encoding was not declared"
plain = len(
json.dumps(
S.build_otlp_logs(sample_payload(), S.resource_attributes()), separators=(",", ":")
).encode()
)
sent = collector["wire_bytes"][0]
assert sent < plain / 3, f"only compressed {plain / sent:.1f}x ({sent} vs {plain} B)"
def test_a_worker_that_cannot_inflate_costs_one_retry_not_the_data(collector):
"""The rollout-order safety net. Shipping this client against a Worker that
predates the inflate path must not lose events — it must notice, fall back,
and stop compressing."""
collector["refuse_gzip"] = True
S._post_blocking(sample_payload())
assert collector["refused"] == 1, "the compressed attempt never happened"
assert len(collector["events"]) == 1, "the event was lost instead of retried"
assert collector["encodings"] == [None], "the retry was not uncompressed"
# ...and it must not keep paying that retry for the rest of the process.
S._post_blocking(sample_payload())
assert collector["refused"] == 1, "it tried gzip again after being refused"
assert len(collector["events"]) == 2
assert collector["encodings"] == [None, None]
def test_a_5xx_does_not_disable_compression(collector):
"""A sick endpoint says nothing about whether it understands gzip. Treating
a 503 as "no gzip here" would permanently give up the saving on the first
upstream blip."""
collector["fail_5xx"] = True
S._post_blocking(sample_payload())
assert S._gzip_enabled() is True
collector["fail_5xx"] = False
S._post_blocking(sample_payload())
assert collector["encodings"] == ["gzip"]
def test_the_kill_switch_works(collector, monkeypatch):
monkeypatch.setenv("HEADROOM_BEACON_GZIP", "off")
S._post_blocking(sample_payload())
assert collector["encodings"] == [None]
assert len(collector["events"]) == 1
def test_a_dead_endpoint_still_never_raises(monkeypatch):
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", "http://127.0.0.1:1/v1/logs")
monkeypatch.setenv("HEADROOM_BEACON", "on")
S._post_blocking(sample_payload(), timeout=0.25)
def test_the_wire_contract_worker_js_must_satisfy():
"""What worker.js sniffs for. If this changes, the Worker's `bytes[0] ===
0x1f && bytes[1] === 0x8b` check stops matching and every upload 400s."""
body = json.dumps(
S.build_otlp_logs(sample_payload(), S.resource_attributes()), separators=(",", ":")
).encode()
compressed = gzip.compress(body, S._GZIP_LEVEL)
assert compressed[0] == 0x1F and compressed[1] == 0x8B, "not gzip magic"
assert gzip.decompress(compressed) == body, "not standard gzip"
# And the Worker's arriving-body cap must still be comfortable.
assert len(compressed) < 64 * 1024
def test_a_413_must_not_trigger_the_uncompressed_retry(collector):
"""413 means the body was too big. Answering that by re-sending the SAME
payload uncompressed — six times larger — is guaranteed to 413 again, and
would permanently switch off the compression that was the only thing
keeping the request under the cap. Only "I cannot read this encoding"
(400/415) may fall back."""
collector["too_large"] = True
S._post_blocking(sample_payload())
assert collector["over"] == 1, "it should not have retried at all"
assert S._gzip_enabled() is True, "a 413 disabled compression"
def test_415_does_fall_back(collector, monkeypatch):
"""The other honest 'I cannot read this body' answer, from a collector that
rejects the encoding outright rather than failing to parse it."""
class Refuse415(_Recorder):
def do_POST(self): # noqa: N802
raw = self.rfile.read(int(self.headers.get("content-length", 0)))
box = self.server.box
if raw[:2] == b"\x1f\x8b":
self.send_response(415)
self.end_headers()
box["refused"] += 1
return
box["events"].append(json.loads(raw))
box["encodings"].append(self.headers.get("content-encoding"))
self.send_response(204)
self.end_headers()
collector_server = HTTPServer(("127.0.0.1", 0), Refuse415)
collector_server.box = {"events": [], "encodings": [], "refused": 0}
threading.Thread(target=collector_server.serve_forever, daemon=True).start()
host, port = collector_server.server_address
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", f"http://{host}:{port}/v1/logs")
try:
S._post_blocking(sample_payload())
assert collector_server.box["refused"] == 1
assert collector_server.box["encodings"] == [None], "no uncompressed retry"
assert S._gzip_enabled() is False
finally:
collector_server.shutdown()
collector_server.server_close()
def test_the_gzip_gate_never_raises(monkeypatch):
"""`_gzip_enabled` is called before _post_blocking's try, from a bare
daemon-thread target and from an atexit handler — both places where an
escaping exception reaches the user's terminal."""
import headroom.telemetry.beacon as beacon_module
monkeypatch.delattr(beacon_module, "_OFF_VALUES", raising=False)
assert S._gzip_enabled() is False, "it must degrade, not raise"
# ...and the upload path still completes.
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", "http://127.0.0.1:1/v1/logs")
S._post_blocking(sample_payload(), timeout=0.25)
def test_gzip_is_opt_in_by_default(monkeypatch):
"""The staged rollout: schema v2 ships without the new transport.
Locks the default so enabling gzip is a deliberate edit to `_GZIP_DEFAULT`
(or an operator setting the variable), never a side effect of touching this
module. While it holds, the Worker's `inflate()` is unreachable -- the
receiver sniffs magic bytes, and nothing produces them.
"""
monkeypatch.delenv("HEADROOM_BEACON_GZIP", raising=False)
monkeypatch.setattr(S, "_gzip_supported", True)
assert S._GZIP_DEFAULT is False, "gzip must stay staged until v2 is proven"
assert S._gzip_enabled() is False, "an unset variable must not compress"
def test_an_unrecognised_gzip_value_does_not_enable_it(monkeypatch):
"""A typo degrades to the proven transport, not to the new one."""
monkeypatch.setattr(S, "_gzip_supported", True)
for value in ("yess", "maybe", "2", " "):
monkeypatch.setenv("HEADROOM_BEACON_GZIP", value)
assert S._gzip_enabled() is False, f"{value!r} enabled compression"
def test_the_operator_can_opt_in(monkeypatch):
"""...and every documented on-value works, so the opt-in is discoverable."""
monkeypatch.setattr(S, "_gzip_supported", True)
for value in ("1", "on", "true", "yes", "enable", "enabled", "ON", " 1 "):
monkeypatch.setenv("HEADROOM_BEACON_GZIP", value)
assert S._gzip_enabled() is True, f"{value!r} did not enable compression"
# --------------------------------------------------------------- wire budget --
#
# The receiver answers an oversized body with 413, and 413 is deliberately not a
# reason to retry, so an event that exceeds the cap is lost outright -- v1
# counters included. These cover the client bounding itself instead.
_CONTENT = [f"ct{i:02d}" for i in range(10)]
_STRATEGIES = [f"st{i:02d}" for i in range(13)]
def _saturated(n_shape: int = 160, n_tool: int = 160, turns: int = 200) -> S._Session:
"""A session whose shape tables are at their caps."""
sess = S._Session(sid="a" * 32, started=0.0, last_seen=0.0)
sess.turns = turns
sess.tokens_saved = 987_654
combos = list(itertools.product(_CONTENT, _STRATEGIES))[:n_shape]
for i, (content, strategy) in enumerate(combos):
sess.shapes[(content, strategy)] = [i + 1, 222_222, 33_333]
for i in range(n_tool):
sess.tool_shapes[f"tool_shape_{i:03d}|d{i % 6}|w{i % 8}"] = [i + 1, 222_222, 33_333]
sess.models.add("claude_sonnet_5")
sess.providers.add("anthropic")
sess.clients["claude_code"] = turns
return sess
def _resource() -> dict[str, str]:
res = S.resource_attributes(create_install_id=False)
res.setdefault("headroom.install_id", "0" * 32)
return res
def _body_of(raw: bytes) -> dict[str, Any]:
"""Unwrap one OTLP body back to plain JSON -- the inverse of _any_value."""
def unwrap(value: dict[str, Any]) -> Any:
if "kvlistValue" in value:
return {kv["key"]: unwrap(kv["value"]) for kv in value["kvlistValue"]["values"]}
if "arrayValue" in value:
return [unwrap(v) for v in value["arrayValue"].get("values", [])]
if "intValue" in value:
return int(value["intValue"])
for key in ("stringValue", "boolValue", "doubleValue"):
if key in value:
return value[key]
return None
record = json.loads(raw)["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]
return unwrap(record["body"])
def test_an_oversized_plain_body_is_trimmed_to_fit():
"""Uncompressed schema v2 crosses 64KB at ~116 rows per table; caps are higher.
Without this the busiest sessions -- the most informative ones -- would be
exactly the ones the receiver refuses.
"""
payload = _saturated().payload("heartbeat")
raw = S._fit_to_wire(payload, _resource(), compress=False)
assert len(raw) <= 64 * 1024, f"{len(raw)} B exceeds the 64KB receiver cap"
assert _body_of(raw)["shapes"]["truncated"] is True
def test_trimming_never_costs_a_v1_counter():
"""The whole point of shedding shapes is that nothing else is shed."""
payload = _saturated().payload("heartbeat")
expected = payload["tokens"]["saved"]
body = _body_of(S._fit_to_wire(payload, _resource(), compress=False))
assert body["shapes"]["truncated"] is True, "nothing was shed; test proves nothing"
assert body["tokens"]["saved"] == expected
assert body["session"]["turns"] == 200
def test_the_trim_keeps_the_rows_carrying_the_most_evidence():
"""Rows are ranked by `n`, so a trimmed table loses resolution, not signal."""
payload = _saturated().payload("heartbeat")
kept = _body_of(S._fit_to_wire(payload, _resource(), compress=False))["shapes"]
counts = [row["n"] for row in kept["by_content"]]
assert counts, "everything was dropped"
# 130 (content x strategy) rows carry n = 1..130. Keeping the top means the
# thinnest survivor still beats what was dropped.
assert min(counts) > 1, "the trim kept the least-used rows"
def test_a_trimmed_table_keeps_the_canonical_row_order():
"""Ranking is a selection rule, not a wire format."""
payload = _saturated().payload("heartbeat")
kept = _body_of(S._fit_to_wire(payload, _resource(), compress=False))["shapes"]
assert kept["truncated"] is True, "nothing was shed; test proves nothing"
by_content = [(r["content"], r["strategy"]) for r in kept["by_content"]]
assert by_content == sorted(by_content), "row order depended on trimming"
def test_truncated_is_always_present_even_when_nothing_is_shed():
"""An optional key would give `shapes` two STRUCT layouts in the corpus."""
payload = _saturated(n_shape=2, n_tool=2).payload("heartbeat")
raw = S._fit_to_wire(payload, _resource(), compress=False)
shapes = _body_of(raw)["shapes"]
assert shapes["truncated"] is False
assert len(shapes["by_content"]) == 2, "a body under budget was trimmed anyway"
def test_compression_makes_the_budget_unreachable():
"""Self-cancelling: once gzip ships, a saturated payload is ~4KB."""
payload = _saturated().payload("heartbeat")
raw = S._fit_to_wire(payload, _resource(), compress=True)
shapes = _body_of(raw)["shapes"]
assert shapes["truncated"] is False, "gzip should leave the tables intact"
assert len(gzip.compress(raw, S._GZIP_LEVEL)) <= S._MAX_WIRE_BYTES
def test_the_gzip_fallback_rebudgets_before_resending(collector, monkeypatch):
"""The refusal path resends plain -- and plain is ~6x larger.
Without re-measuring, the fallback added to SAVE an event from a Worker that
cannot inflate would hand that Worker a body over its cap instead, turning a
400 into a 413 and losing the event after all.
"""
monkeypatch.setenv("HEADROOM_BEACON_GZIP", "1")
monkeypatch.setattr(S, "_gzip_supported", True)
collector["refuse_gzip"] = True
S._post_blocking(_saturated().payload("heartbeat"), timeout=5.0)
assert collector["refused"] == 1, "gzip was not attempted first"
assert collector["events"], "the fallback never arrived"
# 64KB is the cap on the OLDEST receiver still in service -- a literal on
# purpose. Asserting against _MAX_WIRE_BYTES would restate the code under
# test, and would keep passing if that budget were ever raised past what a
# deployed Worker accepts.
assert collector["wire_bytes"][-1] <= 64 * 1024, (
f"fell back with {collector['wire_bytes'][-1]} B, over the 64KB receiver cap"
)
# The collector keeps the OTLP envelope; unwrap it back to the payload.
sent = _body_of(json.dumps(collector["events"][-1]).encode())
assert sent["tokens"]["saved"], "the fallback lost the v1 counters"
assert sent["shapes"]["truncated"] is True, "the plain resend was not re-budgeted"
+906
View File
@@ -0,0 +1,906 @@
"""Beacon schema v2: the new signals, and proof the old ones did not move.
The corpus has years of rows whose meaning depends on the v1 payload staying
exactly what it has always been. Schema v2 is additive, and this file is where
that claim is mechanical rather than asserted: `test_v1_*` locks the shape and
the arithmetic of every pre-existing field, `test_v2_*` covers what was added,
and `test_allowlist_covers_every_emitted_key` enforces the deploy-ordering rule
that a key the worker does not allow is a key that is silently discarded.
"""
from __future__ import annotations
import gzip
import json
import re
from pathlib import Path
from typing import Any
import pytest
from headroom.telemetry import session as S
# ---------------------------------------------------------------------------
# The v1 payload, spelled out. Changing anything in this literal is a schema
# break for every consumer of the existing corpus, and should be very hard to
# do by accident.
# ---------------------------------------------------------------------------
V1_TOP = {
"schema_version",
"session",
"tokens",
"rates",
"compression",
"skips",
"sources",
"providers",
"models",
"failures",
"failure_statuses",
}
V1_NESTED = {
"session": {"id", "seq", "duration_s", "turns", "ended", "final"},
"tokens": {
"original",
"attempted",
"input",
"output",
"saved",
"tool_saved",
"cache_read",
"cache_write",
"uncached",
},
"rates": {
"saved_pct",
"eligible_pct",
"yield_pct",
"all_layers_saved_pct",
"all_layers_yield_pct",
"cache_read_pct",
"overhead_pct",
},
"compression": {
"transforms",
"by_strategy",
"overhead_ms_total",
"latency_ms_total",
"overhead_ms_per_turn",
"latency_ms_per_turn",
"passthrough_turns",
"response_cache_hits",
},
}
V2_TOP = {
"quality",
"hist",
"shapes",
"cache",
"trajectory",
"clients",
"errors",
"strata",
"config",
}
class Outcome:
"""A fully-populated RequestOutcome stand-in. `_fold` is duck-typed."""
provider = "anthropic"
model = "gpt-4o"
original_tokens = 1000
attempted_input_tokens = 400
optimized_tokens = 700
provider_input_tokens = 700
output_tokens = 20
tokens_saved = 300
cache_read_tokens = 500
cache_write_tokens = 100
cache_write_5m_tokens = 80
cache_write_1h_tokens = 20
uncached_input_tokens = 400
cache_inferred = False
from_response_cache = False
status_code = 200
total_latency_ms = 1000.0
overhead_ms = 50.0
ttfb_ms = 300.0
num_messages = 30
client = "claude-code" # real ids are hyphenated; see test_client_ids_survive
transforms_applied: tuple[str, ...] = ("crush",)
tags: dict[str, Any] = {}
waste_signals: dict[str, int] | None = None
def emit(
*outcomes: Any, start: float = 1000.0, step: float = 45.0, source: str = "proxy"
) -> dict[str, Any]:
"""Fold a sequence of outcomes into one session and return its final row."""
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
for index, outcome in enumerate(outcomes):
agg.record(outcome, now=start + index * step, source=source)
agg.flush_all()
return rows[-1]
@pytest.fixture(autouse=True)
def _clean_staging():
"""Staging is module state; a leak across tests would double-count."""
S._staged_strategies.clear()
S._staged_stacks.clear()
S._staged_shapes.clear()
S._staged_tool_shapes.clear()
yield
S._staged_strategies.clear()
S._staged_stacks.clear()
S._staged_shapes.clear()
S._staged_tool_shapes.clear()
@pytest.fixture
def beacon_on(monkeypatch):
"""conftest turns the beacon off for hermeticity; the record_* entry points
short-circuit on that, so anything testing them has to turn it back on."""
monkeypatch.setenv("HEADROOM_BEACON", "on")
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
# ---------------------------------------------------------------------------
# v1: nothing moved
# ---------------------------------------------------------------------------
def test_v1_keys_all_present_and_unchanged():
payload = emit(Outcome())
assert V1_TOP <= set(payload), f"v1 key(s) dropped: {V1_TOP - set(payload)}"
for section, expected in V1_NESTED.items():
assert expected <= set(payload[section]), (
f"{section} lost key(s): {expected - set(payload[section])}"
)
def test_v1_arithmetic_is_byte_for_byte_what_it_was():
"""The five ratios the product is judged on, plus the counters they derive
from. These values predate schema v2 and must survive it exactly."""
payload = emit(Outcome())
assert payload["tokens"] == {
"original": 1000,
"attempted": 400,
"input": 700,
"output": 20,
"saved": 300,
"tool_saved": 0,
"cache_read": 500,
"cache_write": 100,
"uncached": 400,
}
assert payload["rates"]["saved_pct"] == 30.0
assert payload["rates"]["eligible_pct"] == 40.0
assert payload["rates"]["yield_pct"] == 75.0
assert payload["rates"]["cache_read_pct"] == 50.0
assert payload["rates"]["overhead_pct"] == 5.0
assert payload["compression"]["transforms"] == {"crush": 1}
assert payload["sources"] == {"proxy": 1}
assert payload["providers"] == ["anthropic"]
def test_v1_failures_stay_5xx_only_when_4xx_is_present():
"""The single most dangerous change in v2: 4xx is now counted, and it must
be counted somewhere `failures` cannot see. Widening `failures` would have
silently redefined it — a session that rate-limited would start reading as
a session that failed.
"""
class RateLimited(Outcome):
status_code = 429
class Overloaded(Outcome):
status_code = 529
payload = emit(Outcome(), RateLimited(), RateLimited(), Overloaded())
assert payload["failures"] == 1, "a 4xx leaked into the 5xx failure count"
assert payload["failure_statuses"] == {"529": 1}
assert payload["errors"] == {"count": 2, "by_status": {"429": 2}}
def test_v1_transforms_still_collapse_the_stratum_suffix():
"""v2 reads the stratum label too, but the v1 `transforms` map must still
see only the prefix — the suffix carries a model family."""
class Shaped(Outcome):
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools", "crush")
payload = emit(Shaped())
assert payload["compression"]["transforms"] == {"output_shaper": 1, "crush": 1}
def test_v1_cumulative_dedupe_still_holds_with_v2_fields():
"""The corpus dedupes on max(seq) per (install, session) and never sums
across heartbeats. Every v2 field has to be cumulative for that to keep
working."""
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
for index in range(6):
agg.record(Outcome(), now=7000.0 + index * 50)
agg.flush_all()
assert len({row["session"]["id"] for row in rows}) == 1
assert [row["session"]["seq"] for row in rows] == list(range(len(rows)))
for field in ("turns",):
values = [row["session"][field] for row in rows]
assert values == sorted(values), f"session.{field} is not monotonic"
# Every v2 counter must climb or hold, never reset.
for row_a, row_b in zip(rows, rows[1:]):
assert row_b["quality"]["reread_tokens"] >= row_a["quality"]["reread_tokens"]
assert sum(row_b["trajectory"]["turns"]) >= sum(row_a["trajectory"]["turns"])
for name, hist in row_b["hist"].items():
assert sum(hist["counts"]) >= sum(row_a["hist"][name]["counts"]), name
# The last row alone reconstructs the session.
assert rows[-1]["session"]["turns"] == 6
assert sum(rows[-1]["trajectory"]["turns"]) == 6
def test_mcp_shim_still_folds_without_the_new_fields():
"""`_McpCompression` knows five fields. v2 reads a dozen more through the
same duck-typed `get`, and must not require any of them."""
payload = emit(
S._McpCompression(
original_tokens=800,
attempted_input_tokens=800,
optimized_tokens=200,
tokens_saved=600,
),
source="mcp",
)
assert payload["sources"] == {"mcp": 1}
assert payload["rates"]["yield_pct"] == 75.0
assert payload["trajectory"]["kinds"] == "s1"
assert payload["clients"] == []
def test_a_malformed_outcome_still_cannot_raise():
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
agg.record(object(), now=1.0)
agg.flush_all()
# ---------------------------------------------------------------------------
# v2: the new signals
# ---------------------------------------------------------------------------
def test_quality_reports_the_regret_label():
class Regretful(Outcome):
waste_signals = {
"reread": 900,
"reread_compressed": 400,
"json_bloat": 50,
"base64": 7,
}
payload = emit(Regretful(), Regretful(), Outcome())
assert payload["quality"]["reread_tokens"] == 1800
assert payload["quality"]["reread_compressed_tokens"] == 800
assert payload["quality"]["waste"] == [
{"kind": "base64", "tokens": 14},
{"kind": "json_bloat", "tokens": 100},
]
assert payload["quality"]["waste_turns"] == 2, "the clean turn was counted"
def test_waste_keys_are_allowlisted_not_copied():
class Sneaky(Outcome):
waste_signals = {"json_bloat": 10, "some_future_key": 99, "/etc/passwd": 5}
payload = emit(Sneaky())
assert payload["quality"]["waste"] == [{"kind": "json_bloat", "tokens": 10}]
def test_histograms_ship_their_own_edges_and_count_defined_samples():
class Small(Outcome):
original_tokens = 500
tokens_saved = 0
ttfb_ms = 0.0 # 0 means unmeasured, not instant
payload = emit(Outcome(), Outcome(), Small())
turn_tokens = payload["hist"]["turn_tokens"]
assert turn_tokens["edges"] == [1000, 4000, 16000, 64000, 128000, 256000]
assert len(turn_tokens["counts"]) == len(turn_tokens["edges"]) + 1
assert sum(turn_tokens["counts"]) == 3, "observed on every turn"
assert turn_tokens["counts"][0] == 1, "the 500-token turn is under the first edge"
assert sum(payload["hist"]["ttfb_ms"]["counts"]) == 2, "unmeasured ttfb excluded"
# 30% saved -> the [20, 40) bucket, which is index 4.
assert payload["hist"]["saved_pct"]["counts"][4] == 2
def test_cache_gap_curve_excludes_the_first_turn():
class Cold(Outcome):
cache_read_tokens = 0
payload = emit(Outcome(), Outcome(), Cold(), step=45.0)
hits = payload["cache"]["hits"]
misses = payload["cache"]["misses"]
assert sum(hits) + sum(misses) == 2, "first turn has no predecessor to measure"
assert payload["cache"]["gap_edges_s"] == [30.0, 60.0, 120.0, 300.0, 600.0, 1800.0]
# A 45s gap lands in the [30, 60) bucket.
assert hits[1] == 1 and misses[1] == 1
assert payload["cache"]["write_5m"] == 240
assert payload["cache"]["write_1h"] == 60
def test_trajectory_buckets_are_log_spaced_and_kinds_are_run_length_encoded():
class Bypassed(Outcome):
attempted_input_tokens = 0
tokens_saved = 0
tags = {"passthrough_reason": "bypass_header"}
payload = emit(*([Outcome()] * 4), Bypassed(), Bypassed(), *([Outcome()] * 3))
assert payload["trajectory"]["turns"] == [1, 2, 4, 2, 0, 0, 0, 0, 0, 0]
assert payload["trajectory"]["kinds"] == "s4p2s3"
assert payload["trajectory"]["kinds_truncated"] is False
assert payload["trajectory"]["msgs_max"] == 30
# And the v1 passthrough counter is untouched by any of it.
assert payload["compression"]["passthrough_turns"] == 2
def test_trajectory_kind_alphabet_is_closed():
class Cached(Outcome):
from_response_cache = True
class Barren(Outcome):
tokens_saved = 0
class Ineligible(Outcome):
attempted_input_tokens = 0
tokens_saved = 0
class Failed(Outcome):
status_code = 500
class Limited(Outcome):
status_code = 429
payload = emit(Cached(), Failed(), Limited(), Barren(), Outcome(), Ineligible())
kinds = payload["trajectory"]["kinds"]
assert kinds == "c1f1e1z1s1o1"
assert set(re.findall(r"[a-z]", kinds)) <= S._TURN_KINDS
def test_trajectory_string_is_capped():
outcomes = []
for index in range(S.MAX_TRAJECTORY_RUNS * 3):
cls = Outcome if index % 2 else type("Alt", (Outcome,), {"status_code": 500})
outcomes.append(cls())
payload = emit(*outcomes)
assert payload["trajectory"]["kinds_truncated"] is True
assert len(re.findall(r"[a-z]", payload["trajectory"]["kinds"])) == S.MAX_TRAJECTORY_RUNS
def test_clients_and_strata(beacon_on):
class Shaped(Outcome):
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
class Control(Outcome):
client = "claude-vscode"
transforms_applied = ("output_shaper:control:opus|user|s|notools",)
class Anonymous(Outcome):
client = None
payload = emit(Shaped(), Control(), Anonymous())
assert payload["clients"] == [
{"client": "claude_code", "n": 1},
{"client": "claude_vscode", "n": 1},
], "unknown client not bucketed"
strata = {(r["dim"], r["value"]): r["n"] for r in payload["strata"]}
assert strata[("arm", "treatment")] == 1 and strata[("arm", "control")] == 1
assert strata[("turn_kind", "tool_result")] == 1 and strata[("turn_kind", "user")] == 1
assert strata[("input_bucket", "l")] == 1 and strata[("input_bucket", "s")] == 1
assert strata[("tools", "tools")] == 1 and strata[("tools", "notools")] == 1
# Model family is the one stratum component that must never ship.
assert "sonnet" not in json.dumps(payload)
assert "opus" not in json.dumps(payload)
def test_client_ids_survive_slugging():
"""Every harness `classify_client` can return has to round-trip, or the
field reports "other" for the clients that matter. Asserted against the
real vocabulary rather than a value invented here — the first version of
this test used "claude_code", which is not an id the proxy ever emits, and
so passed while `claude-code` was being discarded.
"""
from headroom.proxy.auth_mode import CLIENT_UA_MAP
known = sorted({value for _, value in CLIENT_UA_MAP})
assert "claude-code" in known, "the vocabulary moved; this test needs updating"
outcomes = [type(f"C{i}", (Outcome,), {"client": name})() for i, name in enumerate(known)]
clients = {r["client"]: r["n"] for r in emit(*outcomes)["clients"]}
assert "other" not in clients, (
f"a real client id was discarded: {sorted(set(known) - set(clients))}"
)
assert len(clients) == len(known)
assert clients["claude_code"] == 1
def test_client_slug_still_rejects_junk():
payload = emit(type("Junk", (Outcome,), {"client": "../../etc/passwd"})())
assert payload["clients"] == [{"client": "other", "n": 1}]
def test_content_shape_table(beacon_on):
S.record_content_shape("search", "smart_crusher", 1000, 400)
S.record_content_shape("search", "smart_crusher", 500, 300)
S.record_content_shape("diff", "passthrough", 800, 800)
payload = emit(Outcome())
rows = {(r["content"], r["strategy"]): r for r in payload["shapes"]["by_content"]}
assert rows[("search", "smart_crusher")] == {
"content": "search",
"strategy": "smart_crusher",
"n": 2,
"tokens_in": 1500,
"tokens_out": 700,
}
assert rows[("diff", "passthrough")]["tokens_in"] == 800
assert isinstance(payload["shapes"]["by_content"], list), "a list, never a map"
def test_content_shape_cannot_open_a_session(beacon_on):
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
S.record_content_shape("search", "smart_crusher", 1000, 400)
agg.flush_all()
assert rows == [], "a compression event invented a session"
def test_content_shape_slugs_free_strings(beacon_on):
S.record_content_shape("../../etc/passwd", "SELECT * FROM x", 100, 50)
payload = emit(Outcome())
assert payload["shapes"]["by_content"][0]["content"] == "other"
assert payload["shapes"]["by_content"][0]["strategy"] == "other"
def test_tool_shape_is_a_descriptor_never_a_hash(beacon_on):
class Signature:
structure_hash = "deadbeefcafe0123456789ab"
field_count = 17
max_depth = 3
has_arrays = True
has_nested_objects = True
has_id_like_field = True
has_score_like_field = False
has_timestamp_like_field = True
has_status_like_field = False
has_error_like_field = False
has_message_like_field = False
S.record_tool_shape(Signature(), 5000, 1200)
payload = emit(Outcome())
assert payload["shapes"]["by_tool"] == [
{"shape": "f16d3_anit", "n": 1, "tokens_in": 5000, "tokens_out": 1200}
]
assert "deadbeef" not in json.dumps(payload), "structure_hash reached the wire"
def test_tool_shape_rejects_a_non_signature(beacon_on):
S.record_tool_shape(object(), 5000, 1200)
S.record_tool_shape(None, 5000, 1200)
payload = emit(Outcome())
assert payload["shapes"]["by_tool"] == []
def test_shape_tables_are_capped(beacon_on):
"""The cap is a guard against a caller inventing keys, not a budget — see
test_content_shape_cap_exceeds_its_vocabulary for the other side."""
for index in range(S.MAX_SHAPE_ROWS + 10):
S.record_content_shape(f"c{index}", "smart_crusher", 100, 50)
payload = emit(Outcome())
assert len(payload["shapes"]["by_content"]) == S.MAX_SHAPE_ROWS
def test_config_is_an_allowlist_and_every_value_is_slugged(monkeypatch):
monkeypatch.setenv("HEADROOM_MODE", "optimize")
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "https://secret.acme.internal/v1")
monkeypatch.setenv("HEADROOM_PROXY_TOKEN", "sk-live-abc123")
monkeypatch.setenv("HEADROOM_SAVINGS_PROFILE", "/Users/someone/private/path")
payload = emit(Outcome())
config = {r["key"]: r["value"] for r in payload["config"]}
assert config["mode"] == "optimize"
assert config["savings_profile"] == "other", "a path shipped verbatim"
assert "kompress_endpoint" not in config
assert "proxy_token" not in config
wire = json.dumps(payload)
assert "acme" not in wire and "sk-live" not in wire and "/Users/" not in wire
def test_config_omits_unset_variables(monkeypatch):
for name in S._CONFIG_ENV_KEYS:
monkeypatch.delenv(name, raising=False)
monkeypatch.setenv("HEADROOM_DEDUPE", " ")
assert emit(Outcome())["config"] == []
def test_a_v2_payload_failure_cannot_lose_the_v1_event():
"""The v2 sections claim they cannot cost a v1 number. Built inline in one
dict literal they could: any raise aborts the literal and drops the whole
event, `tokens.saved` included."""
from unittest.mock import patch
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
with patch.object(S, "_config_snapshot", side_effect=RuntimeError("boom")):
agg.record(Outcome(), now=1000.0)
agg.flush_all()
assert rows, "a v2 failure dropped the entire session event"
assert rows[0]["tokens"]["saved"] == 300, "v1 counters survived intact"
assert "config" not in rows[0], "the failed v2 section is absent, not partial"
def test_a_v2_fold_failure_cannot_cost_a_v1_counter_or_a_heartbeat():
from unittest.mock import patch
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
with patch.object(S, "_fold_extra", side_effect=RuntimeError("boom")):
for index in range(4):
agg.record(Outcome(), now=7000.0 + index * 60)
agg.flush_all()
assert len(rows) >= 2, "a v2 fold failure suppressed the heartbeat"
assert rows[-1]["session"]["turns"] == 4
assert rows[-1]["tokens"]["saved"] == 1200
# ---------------------------------------------------------------------------
# client and receiver must agree
# ---------------------------------------------------------------------------
def test_allowlist_covers_every_emitted_key():
"""The worker drops any top-level key not in ALLOWED_KEYS, permanently and
silently. A client that emits a key the worker has not been taught is a
signal that looks healthy and stores nothing, so the two files have to be
checked against each other rather than by eye.
"""
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
match = re.search(r"const ALLOWED_KEYS = \[(.*?)\n\];", worker, re.S)
assert match, "ALLOWED_KEYS not found in worker.js"
without_comments = re.sub(r"//[^\n]*", "", match.group(1))
allowed = set(re.findall(r"'([a-z_]+)'", without_comments))
emitted = set(emit(Outcome()))
missing = emitted - allowed
assert not missing, f"worker.js would silently discard: {sorted(missing)}"
assert V1_TOP <= allowed, "a v1 key was removed from the allowlist"
assert V2_TOP <= allowed
def test_schema_version_was_bumped():
assert S.SCHEMA_VERSION == 2
assert emit(Outcome())["schema_version"] == 2
def test_worst_case_payload_fits_the_receivers_body_limit():
"""The client must not be able to build a body the Worker will refuse.
Not a formality. The cap is checked on what ARRIVES, so an uncompressed
upload -- the kill switch is set, or the gzip fallback fired -- is what has
to fit. And a 413 correctly does NOT trigger the uncompressed retry (see
the gzip tests), so a body over the cap is simply lost.
Read from worker.js rather than hardcoded, so raising one and not the other
fails here instead of in production.
"""
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
assert match, "MAX_BODY_BYTES not found in worker.js"
cap = int(match.group(1)) * 1024
# Every bounded table driven to its cap at once.
for index in range(S.MAX_SHAPE_ROWS + 20):
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
S.record_tool_shape(
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
5000,
1200,
)
for index in range(S.MAX_STRATEGIES + 5):
S.record_compression(f"strategy{index}", 9000, 3000)
outcomes = []
for index in range(1200):
outcomes.append(
type(
f"O{index}",
(Outcome,),
{
"status_code": 400 + (index % 99),
"client": f"harness_{index % 70}",
"transforms_applied": (
f"output_shaper:stratum:m|kind{index % 40}|b{index % 9}|tools",
),
"tags": {"passthrough_reason": f"reason{index % 20}"},
"waste_signals": dict.fromkeys(
(*S._WASTE_KEYS, "reread", "reread_compressed"), 9
),
},
)()
)
payload = emit(*outcomes)
body = json.dumps(
S.build_otlp_logs(payload, S.resource_attributes()), separators=(",", ":")
).encode()
assert len(body) < cap, (
f"worst-case OTLP body is {len(body):,} B against a {cap:,} B cap — the "
"Worker would 413 it and it would not be retried"
)
# Gzipped it is a rounding error, which is the normal path.
assert len(gzip.compress(body, S._GZIP_LEVEL)) < cap // 10
# ---------------------------------------------------------------------------
# headroom telemetry --show
# ---------------------------------------------------------------------------
def test_telemetry_show_prints_the_real_schema_not_a_mockup():
"""The command exists so a user can verify what leaves their machine
without reading two source files and trusting they match the installed
copy. That only holds if it renders the same payload the beacon builds —
so it is asserted against the live key set, not against a fixture."""
from click.testing import CliRunner
from headroom.cli.main import main as cli
result = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
assert result.exit_code == 0, result.output
report = json.loads(result.output)
assert report["schema_version"] == S.SCHEMA_VERSION
assert V1_TOP <= set(report["session_event"])
assert V2_TOP <= set(report["session_event"])
assert set(report["session_event"]) == set(emit(Outcome())), (
"--show has drifted from the payload the beacon actually sends"
)
def test_telemetry_show_reports_the_beacon_state(monkeypatch):
from click.testing import CliRunner
from headroom.cli.main import main as cli
monkeypatch.setenv("HEADROOM_BEACON", "off")
off = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
assert json.loads(off.output)["beacon_enabled"] is False
monkeypatch.setenv("HEADROOM_BEACON", "on")
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
on = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
assert json.loads(on.output)["beacon_enabled"] is True
def test_telemetry_command_does_not_mint_an_install_id(tmp_path, monkeypatch):
"""Inspecting telemetry — very plausibly on the way to switching it off —
must not be the thing that gives a machine its identifier. `install_id()`
creates and persists one as a side effect, so this command reads the file
directly instead."""
from click.testing import CliRunner
from headroom.cli.main import main as cli
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
monkeypatch.setenv("HOME", str(tmp_path))
# The id is memoised in a module global once anything has read it, so the
# cache has to be cleared or this test reads the developer's real id and
# never exercises the create path at all.
monkeypatch.setattr(S, "_install_id", None)
# BOTH paths. `--show` is the one that matters and the one that regressed:
# filtering the id out of the rendered report is not enough, because
# resource_attributes() mints it while building that report.
for argv in (["telemetry"], ["telemetry", "--show"], ["telemetry", "--json"]):
monkeypatch.setattr(S, "_install_id", None)
result = CliRunner().invoke(cli, argv)
assert result.exit_code == 0, result.output
assert not list(tmp_path.rglob("install_id")), f"{argv} created an install id"
assert "none yet" in CliRunner().invoke(cli, ["telemetry"]).output
def test_clients_map_is_capped_against_a_hostile_header():
"""`classify_client` returns the `X-Client` header verbatim when a caller
sets one, so this field is attacker-influenced. Junk already collapses to
"other"; the cap is what stops a caller minting a new valid-looking slug
per request and growing the payload without bound."""
outcomes = [
type(f"C{i}", (Outcome,), {"client": f"harness_{i}"})() for i in range(S.MAX_SHAPES + 25)
]
clients = emit(*outcomes)["clients"]
assert len(clients) == S.MAX_SHAPES
# ---------------------------------------------------------------------------
# shape stability
# ---------------------------------------------------------------------------
def key_shape(node: Any, path: str = "") -> set[str]:
"""Every key path in a payload, with list contents collapsed.
Numeric key segments are normalised, because a map keyed by a number is
safe: DuckDB cannot read "429" as a struct field name, so it infers MAP —
which is what the v1 `failure_statuses` column has always been across the
whole corpus. Identifier-shaped keys are the dangerous ones.
"""
shape: set[str] = set()
if isinstance(node, dict):
for key, value in node.items():
name = "<num>" if str(key).isdigit() else key
shape.add(f"{path}.{name}")
shape |= key_shape(value, f"{path}.{name}")
elif isinstance(node, list):
for item in node:
shape |= key_shape(item, f"{path}[]")
return shape
def test_payload_shape_does_not_depend_on_the_data():
"""The invariant behind every "list of records, not an object" note in
payload(): two sessions with nothing in common must produce the SAME set of
key paths.
An object keyed by something that came from the data breaks this, and the
consequence is not cosmetic — DuckDB infers such an object as a STRUCT
while its keys are few and identifier-shaped and as a MAP once they are
not, so the COLUMN TYPE becomes a function of the contents. Measured on the
real corpus: two harnesses gave STRUCT(claude_code, codex) and a query for
`clients.cursor` failed to bind; twenty-two gave MAP(VARCHAR, BIGINT) and
the same query ran. That is the break that took out the transforms report.
This test is the general guard, so a future field cannot reintroduce it.
"""
class Left(Outcome):
client = "claude-code"
status_code = 200
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
waste_signals = {"json_bloat": 50, "reread": 10}
class Right(Outcome):
client = "aider"
status_code = 200
transforms_applied = ("output_shaper:control:opus|new_user_ask|xs|notools",)
waste_signals = {"base64": 7, "whitespace": 3, "repetition": 11}
import os
os.environ["HEADROOM_MODE"] = "optimize"
try:
left = key_shape(emit(Left()))
finally:
os.environ.pop("HEADROOM_MODE", None)
os.environ["HEADROOM_DEDUPE"] = "on"
os.environ["HEADROOM_LOSSLESS"] = "on"
try:
right = key_shape(emit(Right()))
finally:
os.environ.pop("HEADROOM_DEDUPE", None)
os.environ.pop("HEADROOM_LOSSLESS", None)
assert left == right, (
"payload shape varies with its contents — these paths differ, which "
f"means a column type will too: {sorted(left ^ right)}"
)
def test_the_fields_that_carry_open_vocabularies_are_lists():
"""Belt and braces on the test above, naming the four that were objects."""
payload = emit(Outcome())
for field in ("clients", "config", "strata"):
assert isinstance(payload[field], list), f"{field} must be a list of records"
assert isinstance(payload["quality"]["waste"], list)
# ...and the records have fixed field names, which is what makes the type
# stable no matter how the vocabulary grows.
assert isinstance(payload["shapes"]["by_content"], list)
def test_trajectory_string_is_a_faithful_prefix_once_truncated():
"""Past the cap the string must STOP, not keep growing its last run.
Letting a matching kind keep incrementing looks harmless and is not: once a
run has been dropped the remaining turns are no longer contiguous, so a run
of `s6` would claim six consecutive `s` turns where the session actually
did `s q s q s q`. Every run in the string has to be true.
"""
runs_at_cap = []
for index in range(S.MAX_TRAJECTORY_RUNS):
runs_at_cap.append(
type(f"A{index}", (Outcome,), {"status_code": 500 if index % 2 else 200})()
)
# Then a tail that alternates between the last run's kind and another.
tail = []
for index in range(20):
tail.append(type(f"B{index}", (Outcome,), {"status_code": 200 if index % 2 else 429})())
payload = emit(*runs_at_cap, *tail)
kinds = payload["trajectory"]["kinds"]
assert payload["trajectory"]["kinds_truncated"] is True
runs = re.findall(r"([a-z])(\d+)", kinds)
assert len(runs) == S.MAX_TRAJECTORY_RUNS
# The prefix is exactly the pre-cap history: alternating runs of length 1.
assert all(int(n) == 1 for _, n in runs), f"a run grew after truncation: {kinds}"
# And the uncapped turn counter still accounts for every turn.
assert sum(payload["trajectory"]["turns"]) == payload["session"]["turns"]
assert payload["session"]["turns"] == S.MAX_TRAJECTORY_RUNS + 20
def test_content_shape_cap_exceeds_its_vocabulary(beacon_on):
"""The cap must not be able to bite in normal operation. It drops rows
first-come-wins, which would bias the table toward whatever a session
routed early — and that table is what a routing policy gets fitted on."""
from headroom.transforms.content_detector import ContentType
from headroom.transforms.content_router import CompressionStrategy
# +1 each for the "other" bucket a free string collapses to.
cross_product = (len(list(ContentType)) + 1) * (len(list(CompressionStrategy)) + 1)
assert S.MAX_SHAPE_ROWS >= cross_product, (
f"{cross_product} legitimate (content, strategy) pairs exist but the cap "
f"is {S.MAX_SHAPE_ROWS}"
)
for content in ContentType:
for strategy in CompressionStrategy:
S.record_content_shape(content.value, strategy.value, 100, 50)
rows = emit(Outcome())["shapes"]["by_content"]
assert len(rows) == len(list(ContentType)) * len(list(CompressionStrategy)), (
"a legitimate content/strategy combination was dropped"
)
def test_the_client_budget_stays_under_every_deployed_receiver():
"""`_MAX_WIRE_BYTES` is only useful while it is below what receivers accept.
The current Worker allows 256KB, but an install upgrades on its own schedule
and may be pointed at an older deployment whose cap is 64KB. Raising the
client budget past that would silently reintroduce the 413 this bounds.
"""
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
assert match, "MAX_BODY_BYTES not found in worker.js"
assert S._MAX_WIRE_BYTES <= int(match.group(1)) * 1024
assert S._MAX_WIRE_BYTES <= 64 * 1024, (
f"the client budget is {S._MAX_WIRE_BYTES:,} B, over the 64KB cap of the "
"oldest Worker still in service"
)
def test_the_worst_case_body_actually_sent_fits_the_oldest_receiver():
"""The test above bounds what can be BUILT; this bounds what is SENT.
`_fit_to_wire` is the thing standing between the two, and uncompressed v2
crosses 64KB well before the shape caps are reached.
"""
for index in range(S.MAX_SHAPE_ROWS + 20):
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
S.record_tool_shape(
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
5000,
1200,
)
outcomes = [
type(
f"O{index}",
(Outcome,),
{"status_code": 400 + (index % 99), "client": f"harness_{index % 70}"},
)()
for index in range(600)
]
payload = emit(*outcomes)
saved = payload["tokens"]["saved"]
sent = S._fit_to_wire(payload, S.resource_attributes(), compress=False)
assert len(sent) <= 64 * 1024, f"sent {len(sent):,} B, over the 64KB receiver cap"
assert payload["tokens"]["saved"] == saved, "trimming cost a v1 counter"
+6 -2
View File
@@ -75,7 +75,11 @@ class TestFormatTelemetryNotice:
monkeypatch.delenv(var, raising=False)
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
notice = format_telemetry_notice()
assert "compression stats" in notice
# Substance, not the exact phrasing: what is sent (counters), what is
# not (prompts), and how to turn it off. The wording widened at schema
# v2 — "compression stats" stopped describing a payload that also
# carries cache behaviour, session shape and configuration.
assert "usage counters" in notice
assert "HEADROOM_BEACON=off" in notice
def test_beacon_notice_names_what_is_not_sent(self, monkeypatch):
@@ -233,7 +237,7 @@ class TestWrapCLITelemetryNotice:
_print_telemetry_notice()
captured = capsys.readouterr()
assert "compression stats" in captured.out
assert "usage counters" in captured.out
assert "HEADROOM_BEACON=off" in captured.out
def test_print_notice_silent_when_telemetry_off(self, monkeypatch, capsys):