mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(beacon): schema v2 — routing/compression training signal, gzip transport (#3253)
## What Nine new sections on the session heartbeat — `quality`, `hist`, `shapes`, `cache`, `trajectory`, `clients`, `errors`, `strata`, `config` — plus a gzip transport. Everything is a counter, a ratio, or a slug from a fixed list in the source. No prompts, code, file paths, tool names, project names, or error text can appear. ## No regression in v1 Every existing field keeps its name, type, and meaning. `tests/test_beacon_schema_v2.py` locks the v1 surface with literal key lists and a shape-stability invariant, so a v1 field that silently changes type fails there rather than in the corpus six weeks later. The v2 section is built in `_payload_v2()` behind `try/except`, and `_fold_extra()` likewise. A bug in new code can cost the new fields and nothing else. The first version built one dict literal — any raise dropped `tokens.saved` along with it. ## The STRUCT/MAP flip Four fields ship as lists of records rather than objects keyed by data (`clients`, `config`, `strata`, `waste`). DuckDB infers a JSON object as `STRUCT` when the keys are few and identifier-shaped, and as `MAP` otherwise — so a key set that grows with the fleet flips type mid-corpus and `BINDER ERROR`s the reader. Caught against the real 369k-session corpus, not in theory: `clients` inferred `STRUCT(claude_code, codex)` at two harnesses and `MAP(VARCHAR, BIGINT)` at 22, and `clients.cursor` errors on one of them. ## Transport Body is gzipped at level 6 (~6x). The receiver sniffs magic bytes rather than trusting `Content-Encoding`, enforces a decompression-bomb limit *during* the stream read, and leaves the plain path byte-identical for the existing fleet. A `400`/`415` disables gzip for the process and retries once. A `413` does not — an uncompressed retry is 6x larger and guaranteed to fail again. At current rates (~235k heartbeats/day): v1 today 892 MB/day, v2 uncompressed 1,802 MB/day, **v2 gzipped 295 MB/day** — a net reduction against what we send now. ## `headroom telemetry` Prints the real payload, built by the same `_Session.payload()` the beacon uses. \"Go read worker.js\" answers what happens to the data; it does not answer what leaves your machine. Uses `read_install_id()` and `resource_attributes(create_install_id=False)` — the default mints and persists the id, so building the report would itself be the act that identifies the machine. Filtering the id out of the *output* is not enough; by then the file exists. ## Receiver Allowlist gains the nine keys. `MAX_BODY_BYTES` 64KB → 256KB — that figure dated from \"a beacon event is ~2KB\", and worst-case v2 is 81,965 B uncompressed. ## Verification Against the production corpus (369,015 sessions): **14/14** `beacon.sh` reports pass, **18/18** dashboard queries pass, v1 totals byte-identical at `273,495,971,478` tokens saved. `sample-event.json` regenerated as a real v2 payload. The old v1-only version would have passed the post-deploy smoke test whether or not the new allowlist landed; the new one distinguishes the cases 9/9 vs 0/9. Suite: 11,655 passed, 1 failed — `test_mcp_reconcile.py::test_ordinary_install_does_not_adopt_serena`, which reproduces identically on clean `main`. ## Not verified here `node test-rollup.mjs` and `npx wrangler dev` have not been run — no `node` in the authoring environment. `DecompressionStream` is the one path with no fallback and has never been executed. **Both are required before deploy.** ## Deploy order **Worker first, client second.** The worker is backward-compatible with the v1 fleet (allowlist additive, plain path byte-identical, cap only raised). The reverse is not: the worker strips unknown keys at ingest, so v2 data arriving before the allowlist lands is gone, not recoverable by re-rolling. The corpus already carries 6 such rows from a dev proxy. Merging this PR does not ship the client — releases are separate commits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Tejas Chopra
Claude Opus 5
JD Davis
parent
bc21c93707
commit
67eb910e38
+817
-30
File diff suppressed because it is too large
Load Diff
@@ -13,7 +13,7 @@
|
||||
*/
|
||||
import { readdirSync, readFileSync } from 'node:fs';
|
||||
import assert from 'node:assert/strict';
|
||||
import { oldestRawDay, rollupHour } from './worker.js';
|
||||
import { inflate, oldestRawDay, rollupHour } from './worker.js';
|
||||
|
||||
// R2 returns at most 1000 keys per list page, so on a real hour (~4,000
|
||||
// objects) the cursor loop in rollupHour is load-bearing. The stub paginates at
|
||||
@@ -211,4 +211,47 @@ if (dir) {
|
||||
);
|
||||
}
|
||||
|
||||
// 8. Upload compression. A schema-v2 event is ~8KB of repetitive JSON and
|
||||
// gzips ~6x, so fetch() sniffs the gzip magic number and inflates. Three
|
||||
// things have to hold, and none of them can be checked in production without
|
||||
// risking every upload:
|
||||
//
|
||||
// a) a real gzip body round-trips to the exact bytes that went in;
|
||||
// b) DecompressionStream exists in this runtime at all (it is the one part
|
||||
// of the ingest path with no fallback -- if it is missing, inflate throws,
|
||||
// fetch answers 400, and every compressed upload is refused);
|
||||
// c) the size cap is enforced WHILE the stream is read, not after, or a
|
||||
// decompression bomb has already been decompressed by the time it is
|
||||
// noticed.
|
||||
{
|
||||
const gzipBytes = async (text) => {
|
||||
const stream = new Response(new TextEncoder().encode(text)).body.pipeThrough(
|
||||
new CompressionStream('gzip')
|
||||
);
|
||||
return new Uint8Array(await new Response(stream).arrayBuffer());
|
||||
};
|
||||
|
||||
const payload = JSON.stringify({
|
||||
resourceLogs: [{ scopeLogs: [{ logRecords: [{ body: { stringValue: 'x'.repeat(4000) } }] }] }],
|
||||
});
|
||||
const packed = await gzipBytes(payload);
|
||||
|
||||
assert.equal(packed[0], 0x1f, 'gzip magic byte 0');
|
||||
assert.equal(packed[1], 0x8b, 'gzip magic byte 1');
|
||||
assert.ok(packed.length < payload.length / 3, 'gzip did not actually compress');
|
||||
assert.equal(await inflate(packed.buffer, 256 * 1024), payload, 'round trip changed the body');
|
||||
|
||||
// A bomb: highly compressible input that expands past the cap. The cap must
|
||||
// bite, and it must bite from the stream rather than from the final size.
|
||||
const bomb = await gzipBytes('A'.repeat(2 * 1024 * 1024));
|
||||
assert.ok(bomb.length < 64 * 1024, 'fixture is not actually a bomb');
|
||||
await assert.rejects(
|
||||
() => inflate(bomb.buffer, 256 * 1024),
|
||||
/too large/,
|
||||
'a decompression bomb was inflated in full'
|
||||
);
|
||||
|
||||
console.log(`gzip: ${payload.length} B -> ${packed.length} B, bomb refused`);
|
||||
}
|
||||
|
||||
console.log('ok');
|
||||
|
||||
+122
-3
@@ -53,6 +53,51 @@ const ALLOWED_KEYS = [
|
||||
// real minimum cacheable prefix, how long a cache actually survives, and how
|
||||
// far predicted cache hits are from the ones that happened.
|
||||
'routing',
|
||||
|
||||
// ---- schema v2 -----------------------------------------------------------
|
||||
//
|
||||
// Additive. Every v1 key above is untouched, so a v1 client keeps storing
|
||||
// exactly what it stores today and rows from the two versions union cleanly
|
||||
// (`schema_version` separates them, `union_by_name` handles the rest).
|
||||
//
|
||||
// Read the ordering rule at the top of this list literally: this deploy must
|
||||
// go out BEFORE the client release that starts sending these, or the first
|
||||
// weeks of the new signals are dropped on arrival and cannot be recovered.
|
||||
// Removing one line here is also the rollback -- it reverts that signal for
|
||||
// every install in the wild without a client release, which is the whole
|
||||
// reason the allowlist is server-side.
|
||||
|
||||
// Regret. `reread_compressed_tokens` is the counter-pressure on `saved_pct`:
|
||||
// tokens the agent had to fetch again because compression removed them.
|
||||
// Without it a compression ratio is unfalsifiable.
|
||||
'quality',
|
||||
// Fixed-bucket distributions for quantities otherwise reported only as
|
||||
// session sums. Ships its own bucket edges, so no reader needs to know which
|
||||
// client version wrote a row.
|
||||
'hist',
|
||||
// (content_type x strategy x yield), and the same for JSON tool output keyed
|
||||
// by structural shape. The input term the corpus has never had.
|
||||
// Structural descriptors only -- never TOIN's structure_hash, which is a
|
||||
// digest of field names.
|
||||
'shapes',
|
||||
// Prompt-cache survival by gap since the previous turn: the empirical TTL
|
||||
// and minimum-prefix curve per provider.
|
||||
'cache',
|
||||
// Session shape over time -- log-spaced turn buckets plus a run-length
|
||||
// encoded turn-kind string from a closed seven-letter alphabet.
|
||||
'trajectory',
|
||||
// Identified client harness, per turn. `headroom.stack` answers this by
|
||||
// detection and resolves to a literal "proxy" for most of the fleet.
|
||||
'clients',
|
||||
// 4xx, kept strictly separate from the 5xx-only `failures` above so neither
|
||||
// metric changes meaning. 429 rate is the routing signal here.
|
||||
'errors',
|
||||
// output_shaper A/B strata as validated enums. Model family is not among
|
||||
// them -- see the note in session.py's payload().
|
||||
'strata',
|
||||
// Allowlisted, slug-validated configuration, so the corpus can tell
|
||||
// "compression underperformed" from "compression was switched off".
|
||||
'config',
|
||||
];
|
||||
|
||||
// Resource attributes we keep. Same rule: allowlist, not denylist.
|
||||
@@ -66,8 +111,60 @@ const ALLOWED_RESOURCE = [
|
||||
'host.arch',
|
||||
];
|
||||
|
||||
// A beacon event is ~2KB. Anything far past that is a bug or an attack.
|
||||
const MAX_BODY_BYTES = 64 * 1024;
|
||||
// Cap on what ARRIVES, compressed or not.
|
||||
//
|
||||
// Was 64KB, chosen when the comment above it read "a beacon event is ~2KB" --
|
||||
// i.e. 32x the payload. Schema v2 made a typical event ~8KB uncompressed, and
|
||||
// a session that fills every table (the shape cross product is 10 content
|
||||
// types x 13 strategies) reaches ~82KB. That silently eroded the multiple to
|
||||
// under 1x: a busy session sending uncompressed -- the kill switch is set, or
|
||||
// the gzip fallback fired -- would have been answered 413 and, correctly, not
|
||||
// retried, so the event would simply be lost.
|
||||
//
|
||||
// Restored to ~3x the worst legitimate payload. This is not the abuse control
|
||||
// and never was: the endpoint is unauthenticated by design and the WAF rate
|
||||
// limit (60 req/min/IP, see wrangler.toml) is what bounds a bad actor. At 60
|
||||
// requests a minute the difference between these two ceilings is 3.8 MB/min
|
||||
// and 15 MB/min, neither of which is interesting to Cloudflare.
|
||||
const MAX_BODY_BYTES = 256 * 1024;
|
||||
|
||||
// Cap on what an arriving body EXPANDS to. `raw.byteLength` cannot see this:
|
||||
// a small gzip can expand to gigabytes, and a decompression bomb that is only
|
||||
// noticed after it has been decompressed has already won. Enforced while the
|
||||
// stream is read, not after. ~3x the worst legitimate payload, same as above.
|
||||
const MAX_DECOMPRESSED_BYTES = 256 * 1024;
|
||||
|
||||
/**
|
||||
* Decompress a gzip body, refusing anything that expands past `limit`.
|
||||
*
|
||||
* Counts bytes as chunks arrive and aborts mid-stream, so a bomb costs the
|
||||
* limit rather than whatever it would have expanded to.
|
||||
*/
|
||||
export async function inflate(buffer, limit) {
|
||||
const stream = new Response(buffer).body.pipeThrough(new DecompressionStream('gzip'));
|
||||
const reader = stream.getReader();
|
||||
const chunks = [];
|
||||
let total = 0;
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
total += value.byteLength;
|
||||
if (total > limit) throw new Error('decompressed body too large');
|
||||
chunks.push(value);
|
||||
}
|
||||
} finally {
|
||||
// Releases the stream on the bomb path; a no-op once it has closed.
|
||||
reader.cancel().catch(() => {});
|
||||
}
|
||||
const joined = new Uint8Array(total);
|
||||
let offset = 0;
|
||||
for (const chunk of chunks) {
|
||||
joined.set(chunk, offset);
|
||||
offset += chunk.byteLength;
|
||||
}
|
||||
return new TextDecoder().decode(joined);
|
||||
}
|
||||
|
||||
/** OTLP AnyValue -> plain JS. The inverse of _any_value() in session.py. */
|
||||
function unwrap(value) {
|
||||
@@ -315,9 +412,31 @@ export default {
|
||||
|
||||
let records;
|
||||
try {
|
||||
records = extract(JSON.parse(new TextDecoder().decode(raw)));
|
||||
// Sniff the gzip magic number rather than trusting Content-Encoding.
|
||||
// Three things can each independently decide whether a body arrives
|
||||
// compressed -- the client, Cloudflare's edge, and any proxy between
|
||||
// them -- and only the bytes know which of them acted. Keying off the
|
||||
// header instead means a body that something already decompressed gets
|
||||
// fed to DecompressionStream, or vice versa, and every upload 400s.
|
||||
//
|
||||
// It also means an old client that never sets the header keeps working
|
||||
// through the identical code path it uses today: no header, no magic
|
||||
// number, straight to TextDecoder.
|
||||
const bytes = new Uint8Array(raw);
|
||||
const gzipped = bytes.length >= 2 && bytes[0] === 0x1f && bytes[1] === 0x8b;
|
||||
const text = gzipped
|
||||
? await inflate(raw, MAX_DECOMPRESSED_BYTES)
|
||||
: new TextDecoder().decode(raw);
|
||||
records = extract(JSON.parse(text));
|
||||
} catch {
|
||||
// Malformed input is not worth a retry storm from clients.
|
||||
//
|
||||
// This 400 is also load-bearing in the other direction: it is what a
|
||||
// client that gzipped against a Worker too old to inflate sees, and
|
||||
// _post_blocking treats a 4xx on a compressed body as "this endpoint
|
||||
// does not do gzip", falls back to sending it uncompressed, and stops
|
||||
// compressing for the life of the process. So deploying the client
|
||||
// before this Worker costs one retry per process, not the data.
|
||||
return new Response('bad request', { status: 400 });
|
||||
}
|
||||
if (records.length === 0) return new Response(null, { status: 204 });
|
||||
|
||||
@@ -78,6 +78,7 @@ def _register_commands() -> None:
|
||||
recover, # noqa: F401
|
||||
rollout, # noqa: F401
|
||||
savings, # noqa: F401
|
||||
telemetry, # noqa: F401
|
||||
tools, # noqa: F401
|
||||
update, # noqa: F401
|
||||
wrap, # noqa: F401
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
"""``headroom telemetry`` — show exactly what the beacon would send.
|
||||
|
||||
The beacon is opt-OUT and on by default. "The receiver is open source, go read
|
||||
worker.js" is a good answer to what happens to the data, but it is a poor
|
||||
answer to what LEAVES YOUR MACHINE — that requires reading two files, trusting
|
||||
that the copy on GitHub matches the copy installed, and reconstructing a
|
||||
payload by hand.
|
||||
|
||||
This prints the real thing: the current process's resource attributes and a
|
||||
session payload built by the same `_Session.payload()` the beacon uses, with
|
||||
representative counters folded in. Nothing here is a mock-up of the schema; if
|
||||
a field can appear on the wire, it appears here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import click
|
||||
|
||||
from .main import main
|
||||
|
||||
|
||||
def _sample_payload() -> dict[str, Any]:
|
||||
"""One session payload, built through the real aggregator.
|
||||
|
||||
Uses the actual fold path rather than hand-writing a dict, so this cannot
|
||||
drift from what is sent. The token counts are invented; the SHAPE — every
|
||||
key, and the fact that every value is a counter, a ratio or a bounded slug
|
||||
— is exactly what a real session produces.
|
||||
"""
|
||||
from headroom.telemetry import session as telemetry_session
|
||||
|
||||
class _Example:
|
||||
provider = "anthropic"
|
||||
model = "gpt-4o"
|
||||
original_tokens = 120_000
|
||||
attempted_input_tokens = 40_000
|
||||
optimized_tokens = 90_000
|
||||
provider_input_tokens = 90_000
|
||||
output_tokens = 1_200
|
||||
tokens_saved = 30_000
|
||||
cache_read_tokens = 60_000
|
||||
cache_write_tokens = 8_000
|
||||
cache_write_5m_tokens = 6_000
|
||||
cache_write_1h_tokens = 2_000
|
||||
uncached_input_tokens = 22_000
|
||||
status_code = 200
|
||||
total_latency_ms = 4_200.0
|
||||
overhead_ms = 48.0
|
||||
ttfb_ms = 900.0
|
||||
num_messages = 140
|
||||
client = "claude_code"
|
||||
transforms_applied = ("crush",)
|
||||
tags: dict[str, Any] = {}
|
||||
waste_signals = {"reread": 900, "reread_compressed": 400, "json_bloat": 50}
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
aggregator = telemetry_session.SessionAggregator(rows.append)
|
||||
for index in range(3):
|
||||
aggregator.record(_Example(), now=1_000.0 + index * 45.0)
|
||||
aggregator.flush_all()
|
||||
return rows[-1]
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.option(
|
||||
"--show",
|
||||
"show",
|
||||
is_flag=True,
|
||||
help="Print the exact payload the beacon would send, as JSON.",
|
||||
)
|
||||
@click.option(
|
||||
"--json",
|
||||
"as_json",
|
||||
is_flag=True,
|
||||
help="Print JSON only, with no explanatory footer. Implies --show.",
|
||||
)
|
||||
def telemetry(show: bool, as_json: bool) -> None:
|
||||
"""Show what Headroom's telemetry beacon collects, and whether it is on.
|
||||
|
||||
\b
|
||||
Examples:
|
||||
headroom telemetry Current state and the two switches
|
||||
headroom telemetry --show The exact payload, field by field
|
||||
"""
|
||||
from headroom.telemetry.beacon import is_beacon_enabled, is_telemetry_enabled
|
||||
from headroom.telemetry.session import (
|
||||
DEFAULT_ENDPOINT,
|
||||
SCHEMA_VERSION,
|
||||
_gzip_enabled,
|
||||
read_install_id,
|
||||
resource_attributes,
|
||||
)
|
||||
|
||||
beacon_on = is_beacon_enabled()
|
||||
local_on = is_telemetry_enabled()
|
||||
known_id = read_install_id()
|
||||
# Reported because it changes what goes on the wire, and because while the
|
||||
# transport is staged this is the fastest way to answer "was that upload
|
||||
# compressed?" without instrumenting anything.
|
||||
gzip_on = _gzip_enabled()
|
||||
|
||||
if show or as_json:
|
||||
report = {
|
||||
"beacon_enabled": beacon_on,
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"endpoint": DEFAULT_ENDPOINT,
|
||||
"gzip": gzip_on,
|
||||
"install_id": known_id,
|
||||
# create_install_id=False, and it is load-bearing: the default
|
||||
# mints and persists an id, so building this report would be the
|
||||
# act that identifies the machine. Filtering the id out of the
|
||||
# OUTPUT is not enough -- by then the file exists.
|
||||
"resource": resource_attributes(create_install_id=False),
|
||||
"session_event": _sample_payload(),
|
||||
}
|
||||
click.echo(json.dumps(report, indent=2, sort_keys=True))
|
||||
if as_json:
|
||||
return
|
||||
click.echo()
|
||||
click.echo(
|
||||
"The COUNTS above are made up. The FIELDS are not: this payload is built\n"
|
||||
"by the same code the beacon uses, so every key that can ever be sent is\n"
|
||||
"shown, and no other key can be.\n"
|
||||
"\n"
|
||||
"Every value is a counter, a ratio, or a slug from a fixed list in the\n"
|
||||
"source. No prompts, code, file paths, tool names, project names or error\n"
|
||||
"text can appear — see headroom/telemetry/session.py for what is built,\n"
|
||||
"and deploy/beacon/worker.js for the allowlist it is filtered through on\n"
|
||||
"arrival. Your install_id is a random UUID, not a machine fingerprint;\n"
|
||||
"delete ~/.headroom/install_id to reset it. Your IP is never read."
|
||||
)
|
||||
return
|
||||
|
||||
click.echo(f"Upload beacon: {'ON (opt-out)' if beacon_on else 'OFF'}")
|
||||
click.echo(f"Local stats: {'ON' if local_on else 'OFF (opt-in)'}")
|
||||
click.echo(f"Endpoint: {DEFAULT_ENDPOINT}")
|
||||
click.echo(
|
||||
f"Compression: {'gzip' if gzip_on else 'off (plain JSON)'}"
|
||||
f"{'' if gzip_on else ' — enable: HEADROOM_BEACON_GZIP=1'}"
|
||||
)
|
||||
click.echo(f"Install ID: {known_id or '(none yet — created on first upload)'}")
|
||||
click.echo()
|
||||
click.echo("Turn the beacon off: HEADROOM_BEACON=off (or DO_NOT_TRACK=1)")
|
||||
click.echo("See the exact payload: headroom telemetry --show")
|
||||
@@ -124,17 +124,24 @@ def format_telemetry_notice(*, prefix: str = "") -> str:
|
||||
# so this notice is the only place a user learns it is running at all —
|
||||
# surprise is what turns anonymous telemetry into a trust incident.
|
||||
#
|
||||
# Wording: lead with what the data *is* (compression ratios) and why it is
|
||||
# useful (understanding when compression works), not with the fact of
|
||||
# Wording: lead with what the data *is* (counters) and why it is useful
|
||||
# (understanding when compression works), not with the fact of
|
||||
# transmission. "Stats are sent to Headroom Labs" is accurate but reads like
|
||||
# extraction, which misrepresents a payload that is entirely counters. The
|
||||
# reassurance has to stay concrete, though — "never prompts, code, or file
|
||||
# paths" names the things people actually worry about, and every one of
|
||||
# those is enforced by the allowlist, not just promised here.
|
||||
#
|
||||
# "compression stats" until schema v2, which widened the payload to cache
|
||||
# behaviour, session shape, harness and configuration. All still counters,
|
||||
# all still content-free — but this notice is the only place an opt-out
|
||||
# beacon is disclosed at all, so it has to describe what is actually sent
|
||||
# rather than the narrowest true thing. `headroom telemetry --show` prints
|
||||
# the exact payload for anyone who wants the specifics.
|
||||
if beacon:
|
||||
return (
|
||||
f"{prefix}Telemetry: anonymous compression stats — never prompts, code, or "
|
||||
"file paths. Helps us improve compression | Off: HEADROOM_BEACON=off"
|
||||
f"{prefix}Telemetry: anonymous usage counters — never prompts, code, or file "
|
||||
"paths. See: headroom telemetry --show | Off: HEADROOM_BEACON=off"
|
||||
)
|
||||
return (
|
||||
f"{prefix}Telemetry: ENABLED (local aggregate stats only — nothing sent externally) | "
|
||||
|
||||
+1322
-31
File diff suppressed because it is too large
Load Diff
@@ -565,6 +565,23 @@ class ToolIntelligenceNetwork:
|
||||
model_family: Target model family (`claude-3-5`, `gpt-4o`, …).
|
||||
Defaults to `DEFAULT_MODEL_FAMILY` when not provided.
|
||||
"""
|
||||
# The beacon's copy of this event is taken BEFORE the enabled check.
|
||||
# TOIN's own store is gated on HEADROOM_TELEMETRY, which is opt-in and
|
||||
# therefore off across almost the whole fleet -- so recording after the
|
||||
# gate would mean the network in "Tool Output Intelligence Network"
|
||||
# only ever sees the installs that least need it.
|
||||
#
|
||||
# Costs nothing extra: `tool_signature` was already built by the
|
||||
# caller, and `record_tool_shape` reads a handful of its integer
|
||||
# attributes and no hash at all. Off by default and never raises, like
|
||||
# every other beacon entry point.
|
||||
try:
|
||||
from headroom.telemetry.session import record_tool_shape
|
||||
|
||||
record_tool_shape(tool_signature, original_tokens, compressed_tokens)
|
||||
except Exception: # pragma: no cover - telemetry must never break a request
|
||||
logger.debug("beacon: tool shape recording failed", exc_info=True)
|
||||
|
||||
# HIGH FIX: Check enabled FIRST to avoid computing structure_hash if disabled
|
||||
# This saves CPU when TOIN is turned off
|
||||
if not self._config.enabled:
|
||||
|
||||
@@ -1454,6 +1454,47 @@ class RoutingDecision:
|
||||
return self.compressed_tokens / self.original_tokens
|
||||
|
||||
|
||||
def _record_beacon_shapes(routing_log: list[RoutingDecision]) -> None:
|
||||
"""Report (content_type -> strategy -> yield) for one compress() call.
|
||||
|
||||
`RoutingDecision` already carries the content type the detector assigned
|
||||
alongside the strategy that was picked for it, so the joint table costs
|
||||
nothing to measure -- it has simply never been reported anywhere. The
|
||||
beacon's `by_strategy` sees the strategy and its yield but not the input,
|
||||
which leaves it able to rank compressors and unable to say which one suits
|
||||
a given piece of content.
|
||||
|
||||
Deliberately NOT routed through `CompressionObserver`: that protocol is
|
||||
implemented outside this repo as well, and widening it would break every
|
||||
such implementation. This is a direct call to a function that is off by
|
||||
default, cheap when off, and cannot raise.
|
||||
|
||||
Runs whether or not an observer is installed, since the beacon path and the
|
||||
Prometheus path are independent -- an install with no observer still
|
||||
reports token totals, and would otherwise report an empty shape table
|
||||
against them.
|
||||
"""
|
||||
if not routing_log:
|
||||
return
|
||||
try:
|
||||
from ..telemetry.session import record_content_shapes
|
||||
|
||||
# One call, not one per decision: the beacon's staging lock is shared
|
||||
# with `record_compression` on this same executor thread, and the note
|
||||
# there is explicit that the contention is real.
|
||||
record_content_shapes(
|
||||
(
|
||||
decision.content_type.value,
|
||||
decision.strategy.value,
|
||||
decision.original_tokens,
|
||||
decision.compressed_tokens,
|
||||
)
|
||||
for decision in routing_log
|
||||
)
|
||||
except Exception as e: # pragma: no cover - defensive
|
||||
logger.debug("beacon shape recording failed (non-fatal): %s", e)
|
||||
|
||||
|
||||
@dataclass
|
||||
class RouterCompressionResult:
|
||||
"""Result from ContentRouter with routing metadata.
|
||||
@@ -2561,6 +2602,7 @@ class ContentRouter(Transform):
|
||||
anyway, swallow at debug level. Compression already succeeded;
|
||||
a buggy observer must not turn a 200 into a 500.
|
||||
"""
|
||||
_record_beacon_shapes(result.routing_log)
|
||||
if self._observer is None:
|
||||
return
|
||||
for d in result.routing_log:
|
||||
|
||||
@@ -0,0 +1,424 @@
|
||||
"""Beacon upload compression, and the fallback that makes it safe to ship.
|
||||
|
||||
A schema-v2 event is ~8KB of repetitive JSON and gzips ~6x, which is worth more
|
||||
than every schema trim available put together. The risk is not the compression;
|
||||
it is that uploads are fire-and-forget, so an endpoint that cannot read a
|
||||
gzipped body looks exactly like an endpoint that is working. These tests cover
|
||||
the compression, and then the three ways it is allowed to stop.
|
||||
|
||||
The server here re-implements worker.js's sniff-and-inflate in Python. It is not
|
||||
the Worker, and cannot prove `DecompressionStream` behaves — but it does pin the
|
||||
wire contract the two halves have to agree on, which is the part that would
|
||||
otherwise only be checked in production.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import itertools
|
||||
import json
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.telemetry import session as S
|
||||
|
||||
|
||||
class _Recorder(BaseHTTPRequestHandler):
|
||||
"""Stands in for worker.js's fetch(): sniff magic bytes, inflate, parse."""
|
||||
|
||||
def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler API
|
||||
raw = self.rfile.read(int(self.headers.get("content-length", 0)))
|
||||
box = self.server.box # type: ignore[attr-defined]
|
||||
|
||||
if box.get("too_large"):
|
||||
self.send_response(413)
|
||||
self.end_headers()
|
||||
box["over"] += 1
|
||||
return
|
||||
if box.get("refuse_gzip") and raw[:2] == b"\x1f\x8b":
|
||||
# A Worker that predates the inflate path: JSON.parse throws on the
|
||||
# gzip bytes and it answers 400.
|
||||
self.send_response(400)
|
||||
self.end_headers()
|
||||
box["refused"] += 1
|
||||
return
|
||||
if box.get("fail_5xx"):
|
||||
self.send_response(503)
|
||||
self.end_headers()
|
||||
return
|
||||
|
||||
body = gzip.decompress(raw) if raw[:2] == b"\x1f\x8b" else raw
|
||||
box["events"].append(json.loads(body))
|
||||
box["encodings"].append(self.headers.get("content-encoding"))
|
||||
box["wire_bytes"].append(len(raw))
|
||||
self.send_response(204)
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *args: Any) -> None:
|
||||
pass
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def collector(monkeypatch):
|
||||
server = HTTPServer(("127.0.0.1", 0), _Recorder)
|
||||
server.box = { # type: ignore[attr-defined]
|
||||
"events": [],
|
||||
"encodings": [],
|
||||
"wire_bytes": [],
|
||||
"refused": 0,
|
||||
"over": 0,
|
||||
}
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
host, port = server.server_address
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", f"http://{host}:{port}/v1/logs")
|
||||
monkeypatch.setenv("HEADROOM_BEACON", "on")
|
||||
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
|
||||
# gzip ships opt-in (_GZIP_DEFAULT is False for the staged rollout), so the
|
||||
# tests that exercise the transport have to turn it on explicitly. The
|
||||
# default itself is locked by test_gzip_is_opt_in_by_default below, which
|
||||
# deliberately does NOT use this fixture's environment.
|
||||
monkeypatch.setenv("HEADROOM_BEACON_GZIP", "1")
|
||||
# Compression disables itself process-wide on refusal; reset between tests.
|
||||
monkeypatch.setattr(S, "_gzip_supported", True)
|
||||
yield server.box # type: ignore[attr-defined]
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
class Outcome:
|
||||
provider = "anthropic"
|
||||
model = "gpt-4o"
|
||||
original_tokens = 120_000
|
||||
attempted_input_tokens = 40_000
|
||||
optimized_tokens = 90_000
|
||||
output_tokens = 1_200
|
||||
tokens_saved = 30_000
|
||||
cache_read_tokens = 60_000
|
||||
status_code = 200
|
||||
total_latency_ms = 4_200.0
|
||||
overhead_ms = 48.0
|
||||
ttfb_ms = 900.0
|
||||
num_messages = 140
|
||||
client = "claude-code"
|
||||
transforms_applied: tuple[str, ...] = ("crush",)
|
||||
tags: dict[str, Any] = {}
|
||||
waste_signals = {"reread": 900, "reread_compressed": 400}
|
||||
|
||||
|
||||
def sample_payload() -> dict[str, Any]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append)
|
||||
for index in range(40):
|
||||
agg.record(Outcome(), now=1000.0 + index * 45)
|
||||
agg.flush_all()
|
||||
return rows[-1]
|
||||
|
||||
|
||||
def test_payload_survives_the_round_trip_byte_for_byte(collector):
|
||||
"""Compression must be invisible above the transport: what the collector
|
||||
parses has to be exactly what the aggregator built."""
|
||||
payload = sample_payload()
|
||||
S._post_blocking(payload)
|
||||
assert len(collector["events"]) == 1
|
||||
received = collector["events"][0]
|
||||
expected = S.build_otlp_logs(payload, S.resource_attributes())
|
||||
body = received["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]["body"]
|
||||
want = expected["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]["body"]
|
||||
assert body == want, "the payload changed in transit"
|
||||
|
||||
|
||||
def test_it_is_actually_compressed_and_declared(collector):
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector["encodings"] == ["gzip"], "Content-Encoding was not declared"
|
||||
plain = len(
|
||||
json.dumps(
|
||||
S.build_otlp_logs(sample_payload(), S.resource_attributes()), separators=(",", ":")
|
||||
).encode()
|
||||
)
|
||||
sent = collector["wire_bytes"][0]
|
||||
assert sent < plain / 3, f"only compressed {plain / sent:.1f}x ({sent} vs {plain} B)"
|
||||
|
||||
|
||||
def test_a_worker_that_cannot_inflate_costs_one_retry_not_the_data(collector):
|
||||
"""The rollout-order safety net. Shipping this client against a Worker that
|
||||
predates the inflate path must not lose events — it must notice, fall back,
|
||||
and stop compressing."""
|
||||
collector["refuse_gzip"] = True
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector["refused"] == 1, "the compressed attempt never happened"
|
||||
assert len(collector["events"]) == 1, "the event was lost instead of retried"
|
||||
assert collector["encodings"] == [None], "the retry was not uncompressed"
|
||||
|
||||
# ...and it must not keep paying that retry for the rest of the process.
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector["refused"] == 1, "it tried gzip again after being refused"
|
||||
assert len(collector["events"]) == 2
|
||||
assert collector["encodings"] == [None, None]
|
||||
|
||||
|
||||
def test_a_5xx_does_not_disable_compression(collector):
|
||||
"""A sick endpoint says nothing about whether it understands gzip. Treating
|
||||
a 503 as "no gzip here" would permanently give up the saving on the first
|
||||
upstream blip."""
|
||||
collector["fail_5xx"] = True
|
||||
S._post_blocking(sample_payload())
|
||||
assert S._gzip_enabled() is True
|
||||
|
||||
collector["fail_5xx"] = False
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector["encodings"] == ["gzip"]
|
||||
|
||||
|
||||
def test_the_kill_switch_works(collector, monkeypatch):
|
||||
monkeypatch.setenv("HEADROOM_BEACON_GZIP", "off")
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector["encodings"] == [None]
|
||||
assert len(collector["events"]) == 1
|
||||
|
||||
|
||||
def test_a_dead_endpoint_still_never_raises(monkeypatch):
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", "http://127.0.0.1:1/v1/logs")
|
||||
monkeypatch.setenv("HEADROOM_BEACON", "on")
|
||||
S._post_blocking(sample_payload(), timeout=0.25)
|
||||
|
||||
|
||||
def test_the_wire_contract_worker_js_must_satisfy():
|
||||
"""What worker.js sniffs for. If this changes, the Worker's `bytes[0] ===
|
||||
0x1f && bytes[1] === 0x8b` check stops matching and every upload 400s."""
|
||||
body = json.dumps(
|
||||
S.build_otlp_logs(sample_payload(), S.resource_attributes()), separators=(",", ":")
|
||||
).encode()
|
||||
compressed = gzip.compress(body, S._GZIP_LEVEL)
|
||||
assert compressed[0] == 0x1F and compressed[1] == 0x8B, "not gzip magic"
|
||||
assert gzip.decompress(compressed) == body, "not standard gzip"
|
||||
# And the Worker's arriving-body cap must still be comfortable.
|
||||
assert len(compressed) < 64 * 1024
|
||||
|
||||
|
||||
def test_a_413_must_not_trigger_the_uncompressed_retry(collector):
|
||||
"""413 means the body was too big. Answering that by re-sending the SAME
|
||||
payload uncompressed — six times larger — is guaranteed to 413 again, and
|
||||
would permanently switch off the compression that was the only thing
|
||||
keeping the request under the cap. Only "I cannot read this encoding"
|
||||
(400/415) may fall back."""
|
||||
collector["too_large"] = True
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector["over"] == 1, "it should not have retried at all"
|
||||
assert S._gzip_enabled() is True, "a 413 disabled compression"
|
||||
|
||||
|
||||
def test_415_does_fall_back(collector, monkeypatch):
|
||||
"""The other honest 'I cannot read this body' answer, from a collector that
|
||||
rejects the encoding outright rather than failing to parse it."""
|
||||
|
||||
class Refuse415(_Recorder):
|
||||
def do_POST(self): # noqa: N802
|
||||
raw = self.rfile.read(int(self.headers.get("content-length", 0)))
|
||||
box = self.server.box
|
||||
if raw[:2] == b"\x1f\x8b":
|
||||
self.send_response(415)
|
||||
self.end_headers()
|
||||
box["refused"] += 1
|
||||
return
|
||||
box["events"].append(json.loads(raw))
|
||||
box["encodings"].append(self.headers.get("content-encoding"))
|
||||
self.send_response(204)
|
||||
self.end_headers()
|
||||
|
||||
collector_server = HTTPServer(("127.0.0.1", 0), Refuse415)
|
||||
collector_server.box = {"events": [], "encodings": [], "refused": 0}
|
||||
threading.Thread(target=collector_server.serve_forever, daemon=True).start()
|
||||
host, port = collector_server.server_address
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", f"http://{host}:{port}/v1/logs")
|
||||
try:
|
||||
S._post_blocking(sample_payload())
|
||||
assert collector_server.box["refused"] == 1
|
||||
assert collector_server.box["encodings"] == [None], "no uncompressed retry"
|
||||
assert S._gzip_enabled() is False
|
||||
finally:
|
||||
collector_server.shutdown()
|
||||
collector_server.server_close()
|
||||
|
||||
|
||||
def test_the_gzip_gate_never_raises(monkeypatch):
|
||||
"""`_gzip_enabled` is called before _post_blocking's try, from a bare
|
||||
daemon-thread target and from an atexit handler — both places where an
|
||||
escaping exception reaches the user's terminal."""
|
||||
import headroom.telemetry.beacon as beacon_module
|
||||
|
||||
monkeypatch.delattr(beacon_module, "_OFF_VALUES", raising=False)
|
||||
assert S._gzip_enabled() is False, "it must degrade, not raise"
|
||||
# ...and the upload path still completes.
|
||||
monkeypatch.setenv("HEADROOM_TELEMETRY_ENDPOINT", "http://127.0.0.1:1/v1/logs")
|
||||
S._post_blocking(sample_payload(), timeout=0.25)
|
||||
|
||||
|
||||
def test_gzip_is_opt_in_by_default(monkeypatch):
|
||||
"""The staged rollout: schema v2 ships without the new transport.
|
||||
|
||||
Locks the default so enabling gzip is a deliberate edit to `_GZIP_DEFAULT`
|
||||
(or an operator setting the variable), never a side effect of touching this
|
||||
module. While it holds, the Worker's `inflate()` is unreachable -- the
|
||||
receiver sniffs magic bytes, and nothing produces them.
|
||||
"""
|
||||
monkeypatch.delenv("HEADROOM_BEACON_GZIP", raising=False)
|
||||
monkeypatch.setattr(S, "_gzip_supported", True)
|
||||
assert S._GZIP_DEFAULT is False, "gzip must stay staged until v2 is proven"
|
||||
assert S._gzip_enabled() is False, "an unset variable must not compress"
|
||||
|
||||
|
||||
def test_an_unrecognised_gzip_value_does_not_enable_it(monkeypatch):
|
||||
"""A typo degrades to the proven transport, not to the new one."""
|
||||
monkeypatch.setattr(S, "_gzip_supported", True)
|
||||
for value in ("yess", "maybe", "2", " "):
|
||||
monkeypatch.setenv("HEADROOM_BEACON_GZIP", value)
|
||||
assert S._gzip_enabled() is False, f"{value!r} enabled compression"
|
||||
|
||||
|
||||
def test_the_operator_can_opt_in(monkeypatch):
|
||||
"""...and every documented on-value works, so the opt-in is discoverable."""
|
||||
monkeypatch.setattr(S, "_gzip_supported", True)
|
||||
for value in ("1", "on", "true", "yes", "enable", "enabled", "ON", " 1 "):
|
||||
monkeypatch.setenv("HEADROOM_BEACON_GZIP", value)
|
||||
assert S._gzip_enabled() is True, f"{value!r} did not enable compression"
|
||||
|
||||
|
||||
# --------------------------------------------------------------- wire budget --
|
||||
#
|
||||
# The receiver answers an oversized body with 413, and 413 is deliberately not a
|
||||
# reason to retry, so an event that exceeds the cap is lost outright -- v1
|
||||
# counters included. These cover the client bounding itself instead.
|
||||
|
||||
_CONTENT = [f"ct{i:02d}" for i in range(10)]
|
||||
_STRATEGIES = [f"st{i:02d}" for i in range(13)]
|
||||
|
||||
|
||||
def _saturated(n_shape: int = 160, n_tool: int = 160, turns: int = 200) -> S._Session:
|
||||
"""A session whose shape tables are at their caps."""
|
||||
sess = S._Session(sid="a" * 32, started=0.0, last_seen=0.0)
|
||||
sess.turns = turns
|
||||
sess.tokens_saved = 987_654
|
||||
combos = list(itertools.product(_CONTENT, _STRATEGIES))[:n_shape]
|
||||
for i, (content, strategy) in enumerate(combos):
|
||||
sess.shapes[(content, strategy)] = [i + 1, 222_222, 33_333]
|
||||
for i in range(n_tool):
|
||||
sess.tool_shapes[f"tool_shape_{i:03d}|d{i % 6}|w{i % 8}"] = [i + 1, 222_222, 33_333]
|
||||
sess.models.add("claude_sonnet_5")
|
||||
sess.providers.add("anthropic")
|
||||
sess.clients["claude_code"] = turns
|
||||
return sess
|
||||
|
||||
|
||||
def _resource() -> dict[str, str]:
|
||||
res = S.resource_attributes(create_install_id=False)
|
||||
res.setdefault("headroom.install_id", "0" * 32)
|
||||
return res
|
||||
|
||||
|
||||
def _body_of(raw: bytes) -> dict[str, Any]:
|
||||
"""Unwrap one OTLP body back to plain JSON -- the inverse of _any_value."""
|
||||
|
||||
def unwrap(value: dict[str, Any]) -> Any:
|
||||
if "kvlistValue" in value:
|
||||
return {kv["key"]: unwrap(kv["value"]) for kv in value["kvlistValue"]["values"]}
|
||||
if "arrayValue" in value:
|
||||
return [unwrap(v) for v in value["arrayValue"].get("values", [])]
|
||||
if "intValue" in value:
|
||||
return int(value["intValue"])
|
||||
for key in ("stringValue", "boolValue", "doubleValue"):
|
||||
if key in value:
|
||||
return value[key]
|
||||
return None
|
||||
|
||||
record = json.loads(raw)["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0]
|
||||
return unwrap(record["body"])
|
||||
|
||||
|
||||
def test_an_oversized_plain_body_is_trimmed_to_fit():
|
||||
"""Uncompressed schema v2 crosses 64KB at ~116 rows per table; caps are higher.
|
||||
|
||||
Without this the busiest sessions -- the most informative ones -- would be
|
||||
exactly the ones the receiver refuses.
|
||||
"""
|
||||
payload = _saturated().payload("heartbeat")
|
||||
raw = S._fit_to_wire(payload, _resource(), compress=False)
|
||||
assert len(raw) <= 64 * 1024, f"{len(raw)} B exceeds the 64KB receiver cap"
|
||||
assert _body_of(raw)["shapes"]["truncated"] is True
|
||||
|
||||
|
||||
def test_trimming_never_costs_a_v1_counter():
|
||||
"""The whole point of shedding shapes is that nothing else is shed."""
|
||||
payload = _saturated().payload("heartbeat")
|
||||
expected = payload["tokens"]["saved"]
|
||||
body = _body_of(S._fit_to_wire(payload, _resource(), compress=False))
|
||||
assert body["shapes"]["truncated"] is True, "nothing was shed; test proves nothing"
|
||||
assert body["tokens"]["saved"] == expected
|
||||
assert body["session"]["turns"] == 200
|
||||
|
||||
|
||||
def test_the_trim_keeps_the_rows_carrying_the_most_evidence():
|
||||
"""Rows are ranked by `n`, so a trimmed table loses resolution, not signal."""
|
||||
payload = _saturated().payload("heartbeat")
|
||||
kept = _body_of(S._fit_to_wire(payload, _resource(), compress=False))["shapes"]
|
||||
counts = [row["n"] for row in kept["by_content"]]
|
||||
assert counts, "everything was dropped"
|
||||
# 130 (content x strategy) rows carry n = 1..130. Keeping the top means the
|
||||
# thinnest survivor still beats what was dropped.
|
||||
assert min(counts) > 1, "the trim kept the least-used rows"
|
||||
|
||||
|
||||
def test_a_trimmed_table_keeps_the_canonical_row_order():
|
||||
"""Ranking is a selection rule, not a wire format."""
|
||||
payload = _saturated().payload("heartbeat")
|
||||
kept = _body_of(S._fit_to_wire(payload, _resource(), compress=False))["shapes"]
|
||||
assert kept["truncated"] is True, "nothing was shed; test proves nothing"
|
||||
by_content = [(r["content"], r["strategy"]) for r in kept["by_content"]]
|
||||
assert by_content == sorted(by_content), "row order depended on trimming"
|
||||
|
||||
|
||||
def test_truncated_is_always_present_even_when_nothing_is_shed():
|
||||
"""An optional key would give `shapes` two STRUCT layouts in the corpus."""
|
||||
payload = _saturated(n_shape=2, n_tool=2).payload("heartbeat")
|
||||
raw = S._fit_to_wire(payload, _resource(), compress=False)
|
||||
shapes = _body_of(raw)["shapes"]
|
||||
assert shapes["truncated"] is False
|
||||
assert len(shapes["by_content"]) == 2, "a body under budget was trimmed anyway"
|
||||
|
||||
|
||||
def test_compression_makes_the_budget_unreachable():
|
||||
"""Self-cancelling: once gzip ships, a saturated payload is ~4KB."""
|
||||
payload = _saturated().payload("heartbeat")
|
||||
raw = S._fit_to_wire(payload, _resource(), compress=True)
|
||||
shapes = _body_of(raw)["shapes"]
|
||||
assert shapes["truncated"] is False, "gzip should leave the tables intact"
|
||||
assert len(gzip.compress(raw, S._GZIP_LEVEL)) <= S._MAX_WIRE_BYTES
|
||||
|
||||
|
||||
def test_the_gzip_fallback_rebudgets_before_resending(collector, monkeypatch):
|
||||
"""The refusal path resends plain -- and plain is ~6x larger.
|
||||
|
||||
Without re-measuring, the fallback added to SAVE an event from a Worker that
|
||||
cannot inflate would hand that Worker a body over its cap instead, turning a
|
||||
400 into a 413 and losing the event after all.
|
||||
"""
|
||||
monkeypatch.setenv("HEADROOM_BEACON_GZIP", "1")
|
||||
monkeypatch.setattr(S, "_gzip_supported", True)
|
||||
collector["refuse_gzip"] = True
|
||||
S._post_blocking(_saturated().payload("heartbeat"), timeout=5.0)
|
||||
assert collector["refused"] == 1, "gzip was not attempted first"
|
||||
assert collector["events"], "the fallback never arrived"
|
||||
# 64KB is the cap on the OLDEST receiver still in service -- a literal on
|
||||
# purpose. Asserting against _MAX_WIRE_BYTES would restate the code under
|
||||
# test, and would keep passing if that budget were ever raised past what a
|
||||
# deployed Worker accepts.
|
||||
assert collector["wire_bytes"][-1] <= 64 * 1024, (
|
||||
f"fell back with {collector['wire_bytes'][-1]} B, over the 64KB receiver cap"
|
||||
)
|
||||
# The collector keeps the OTLP envelope; unwrap it back to the payload.
|
||||
sent = _body_of(json.dumps(collector["events"][-1]).encode())
|
||||
assert sent["tokens"]["saved"], "the fallback lost the v1 counters"
|
||||
assert sent["shapes"]["truncated"] is True, "the plain resend was not re-budgeted"
|
||||
@@ -0,0 +1,906 @@
|
||||
"""Beacon schema v2: the new signals, and proof the old ones did not move.
|
||||
|
||||
The corpus has years of rows whose meaning depends on the v1 payload staying
|
||||
exactly what it has always been. Schema v2 is additive, and this file is where
|
||||
that claim is mechanical rather than asserted: `test_v1_*` locks the shape and
|
||||
the arithmetic of every pre-existing field, `test_v2_*` covers what was added,
|
||||
and `test_allowlist_covers_every_emitted_key` enforces the deploy-ordering rule
|
||||
that a key the worker does not allow is a key that is silently discarded.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.telemetry import session as S
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The v1 payload, spelled out. Changing anything in this literal is a schema
|
||||
# break for every consumer of the existing corpus, and should be very hard to
|
||||
# do by accident.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
V1_TOP = {
|
||||
"schema_version",
|
||||
"session",
|
||||
"tokens",
|
||||
"rates",
|
||||
"compression",
|
||||
"skips",
|
||||
"sources",
|
||||
"providers",
|
||||
"models",
|
||||
"failures",
|
||||
"failure_statuses",
|
||||
}
|
||||
|
||||
V1_NESTED = {
|
||||
"session": {"id", "seq", "duration_s", "turns", "ended", "final"},
|
||||
"tokens": {
|
||||
"original",
|
||||
"attempted",
|
||||
"input",
|
||||
"output",
|
||||
"saved",
|
||||
"tool_saved",
|
||||
"cache_read",
|
||||
"cache_write",
|
||||
"uncached",
|
||||
},
|
||||
"rates": {
|
||||
"saved_pct",
|
||||
"eligible_pct",
|
||||
"yield_pct",
|
||||
"all_layers_saved_pct",
|
||||
"all_layers_yield_pct",
|
||||
"cache_read_pct",
|
||||
"overhead_pct",
|
||||
},
|
||||
"compression": {
|
||||
"transforms",
|
||||
"by_strategy",
|
||||
"overhead_ms_total",
|
||||
"latency_ms_total",
|
||||
"overhead_ms_per_turn",
|
||||
"latency_ms_per_turn",
|
||||
"passthrough_turns",
|
||||
"response_cache_hits",
|
||||
},
|
||||
}
|
||||
|
||||
V2_TOP = {
|
||||
"quality",
|
||||
"hist",
|
||||
"shapes",
|
||||
"cache",
|
||||
"trajectory",
|
||||
"clients",
|
||||
"errors",
|
||||
"strata",
|
||||
"config",
|
||||
}
|
||||
|
||||
|
||||
class Outcome:
|
||||
"""A fully-populated RequestOutcome stand-in. `_fold` is duck-typed."""
|
||||
|
||||
provider = "anthropic"
|
||||
model = "gpt-4o"
|
||||
original_tokens = 1000
|
||||
attempted_input_tokens = 400
|
||||
optimized_tokens = 700
|
||||
provider_input_tokens = 700
|
||||
output_tokens = 20
|
||||
tokens_saved = 300
|
||||
cache_read_tokens = 500
|
||||
cache_write_tokens = 100
|
||||
cache_write_5m_tokens = 80
|
||||
cache_write_1h_tokens = 20
|
||||
uncached_input_tokens = 400
|
||||
cache_inferred = False
|
||||
from_response_cache = False
|
||||
status_code = 200
|
||||
total_latency_ms = 1000.0
|
||||
overhead_ms = 50.0
|
||||
ttfb_ms = 300.0
|
||||
num_messages = 30
|
||||
client = "claude-code" # real ids are hyphenated; see test_client_ids_survive
|
||||
transforms_applied: tuple[str, ...] = ("crush",)
|
||||
tags: dict[str, Any] = {}
|
||||
waste_signals: dict[str, int] | None = None
|
||||
|
||||
|
||||
def emit(
|
||||
*outcomes: Any, start: float = 1000.0, step: float = 45.0, source: str = "proxy"
|
||||
) -> dict[str, Any]:
|
||||
"""Fold a sequence of outcomes into one session and return its final row."""
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append)
|
||||
for index, outcome in enumerate(outcomes):
|
||||
agg.record(outcome, now=start + index * step, source=source)
|
||||
agg.flush_all()
|
||||
return rows[-1]
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_staging():
|
||||
"""Staging is module state; a leak across tests would double-count."""
|
||||
S._staged_strategies.clear()
|
||||
S._staged_stacks.clear()
|
||||
S._staged_shapes.clear()
|
||||
S._staged_tool_shapes.clear()
|
||||
yield
|
||||
S._staged_strategies.clear()
|
||||
S._staged_stacks.clear()
|
||||
S._staged_shapes.clear()
|
||||
S._staged_tool_shapes.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def beacon_on(monkeypatch):
|
||||
"""conftest turns the beacon off for hermeticity; the record_* entry points
|
||||
short-circuit on that, so anything testing them has to turn it back on."""
|
||||
monkeypatch.setenv("HEADROOM_BEACON", "on")
|
||||
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# v1: nothing moved
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_v1_keys_all_present_and_unchanged():
|
||||
payload = emit(Outcome())
|
||||
assert V1_TOP <= set(payload), f"v1 key(s) dropped: {V1_TOP - set(payload)}"
|
||||
for section, expected in V1_NESTED.items():
|
||||
assert expected <= set(payload[section]), (
|
||||
f"{section} lost key(s): {expected - set(payload[section])}"
|
||||
)
|
||||
|
||||
|
||||
def test_v1_arithmetic_is_byte_for_byte_what_it_was():
|
||||
"""The five ratios the product is judged on, plus the counters they derive
|
||||
from. These values predate schema v2 and must survive it exactly."""
|
||||
payload = emit(Outcome())
|
||||
assert payload["tokens"] == {
|
||||
"original": 1000,
|
||||
"attempted": 400,
|
||||
"input": 700,
|
||||
"output": 20,
|
||||
"saved": 300,
|
||||
"tool_saved": 0,
|
||||
"cache_read": 500,
|
||||
"cache_write": 100,
|
||||
"uncached": 400,
|
||||
}
|
||||
assert payload["rates"]["saved_pct"] == 30.0
|
||||
assert payload["rates"]["eligible_pct"] == 40.0
|
||||
assert payload["rates"]["yield_pct"] == 75.0
|
||||
assert payload["rates"]["cache_read_pct"] == 50.0
|
||||
assert payload["rates"]["overhead_pct"] == 5.0
|
||||
assert payload["compression"]["transforms"] == {"crush": 1}
|
||||
assert payload["sources"] == {"proxy": 1}
|
||||
assert payload["providers"] == ["anthropic"]
|
||||
|
||||
|
||||
def test_v1_failures_stay_5xx_only_when_4xx_is_present():
|
||||
"""The single most dangerous change in v2: 4xx is now counted, and it must
|
||||
be counted somewhere `failures` cannot see. Widening `failures` would have
|
||||
silently redefined it — a session that rate-limited would start reading as
|
||||
a session that failed.
|
||||
"""
|
||||
|
||||
class RateLimited(Outcome):
|
||||
status_code = 429
|
||||
|
||||
class Overloaded(Outcome):
|
||||
status_code = 529
|
||||
|
||||
payload = emit(Outcome(), RateLimited(), RateLimited(), Overloaded())
|
||||
assert payload["failures"] == 1, "a 4xx leaked into the 5xx failure count"
|
||||
assert payload["failure_statuses"] == {"529": 1}
|
||||
assert payload["errors"] == {"count": 2, "by_status": {"429": 2}}
|
||||
|
||||
|
||||
def test_v1_transforms_still_collapse_the_stratum_suffix():
|
||||
"""v2 reads the stratum label too, but the v1 `transforms` map must still
|
||||
see only the prefix — the suffix carries a model family."""
|
||||
|
||||
class Shaped(Outcome):
|
||||
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools", "crush")
|
||||
|
||||
payload = emit(Shaped())
|
||||
assert payload["compression"]["transforms"] == {"output_shaper": 1, "crush": 1}
|
||||
|
||||
|
||||
def test_v1_cumulative_dedupe_still_holds_with_v2_fields():
|
||||
"""The corpus dedupes on max(seq) per (install, session) and never sums
|
||||
across heartbeats. Every v2 field has to be cumulative for that to keep
|
||||
working."""
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
|
||||
for index in range(6):
|
||||
agg.record(Outcome(), now=7000.0 + index * 50)
|
||||
agg.flush_all()
|
||||
|
||||
assert len({row["session"]["id"] for row in rows}) == 1
|
||||
assert [row["session"]["seq"] for row in rows] == list(range(len(rows)))
|
||||
for field in ("turns",):
|
||||
values = [row["session"][field] for row in rows]
|
||||
assert values == sorted(values), f"session.{field} is not monotonic"
|
||||
# Every v2 counter must climb or hold, never reset.
|
||||
for row_a, row_b in zip(rows, rows[1:]):
|
||||
assert row_b["quality"]["reread_tokens"] >= row_a["quality"]["reread_tokens"]
|
||||
assert sum(row_b["trajectory"]["turns"]) >= sum(row_a["trajectory"]["turns"])
|
||||
for name, hist in row_b["hist"].items():
|
||||
assert sum(hist["counts"]) >= sum(row_a["hist"][name]["counts"]), name
|
||||
# The last row alone reconstructs the session.
|
||||
assert rows[-1]["session"]["turns"] == 6
|
||||
assert sum(rows[-1]["trajectory"]["turns"]) == 6
|
||||
|
||||
|
||||
def test_mcp_shim_still_folds_without_the_new_fields():
|
||||
"""`_McpCompression` knows five fields. v2 reads a dozen more through the
|
||||
same duck-typed `get`, and must not require any of them."""
|
||||
payload = emit(
|
||||
S._McpCompression(
|
||||
original_tokens=800,
|
||||
attempted_input_tokens=800,
|
||||
optimized_tokens=200,
|
||||
tokens_saved=600,
|
||||
),
|
||||
source="mcp",
|
||||
)
|
||||
assert payload["sources"] == {"mcp": 1}
|
||||
assert payload["rates"]["yield_pct"] == 75.0
|
||||
assert payload["trajectory"]["kinds"] == "s1"
|
||||
assert payload["clients"] == []
|
||||
|
||||
|
||||
def test_a_malformed_outcome_still_cannot_raise():
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append)
|
||||
agg.record(object(), now=1.0)
|
||||
agg.flush_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# v2: the new signals
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_quality_reports_the_regret_label():
|
||||
class Regretful(Outcome):
|
||||
waste_signals = {
|
||||
"reread": 900,
|
||||
"reread_compressed": 400,
|
||||
"json_bloat": 50,
|
||||
"base64": 7,
|
||||
}
|
||||
|
||||
payload = emit(Regretful(), Regretful(), Outcome())
|
||||
assert payload["quality"]["reread_tokens"] == 1800
|
||||
assert payload["quality"]["reread_compressed_tokens"] == 800
|
||||
assert payload["quality"]["waste"] == [
|
||||
{"kind": "base64", "tokens": 14},
|
||||
{"kind": "json_bloat", "tokens": 100},
|
||||
]
|
||||
assert payload["quality"]["waste_turns"] == 2, "the clean turn was counted"
|
||||
|
||||
|
||||
def test_waste_keys_are_allowlisted_not_copied():
|
||||
class Sneaky(Outcome):
|
||||
waste_signals = {"json_bloat": 10, "some_future_key": 99, "/etc/passwd": 5}
|
||||
|
||||
payload = emit(Sneaky())
|
||||
assert payload["quality"]["waste"] == [{"kind": "json_bloat", "tokens": 10}]
|
||||
|
||||
|
||||
def test_histograms_ship_their_own_edges_and_count_defined_samples():
|
||||
class Small(Outcome):
|
||||
original_tokens = 500
|
||||
tokens_saved = 0
|
||||
ttfb_ms = 0.0 # 0 means unmeasured, not instant
|
||||
|
||||
payload = emit(Outcome(), Outcome(), Small())
|
||||
turn_tokens = payload["hist"]["turn_tokens"]
|
||||
assert turn_tokens["edges"] == [1000, 4000, 16000, 64000, 128000, 256000]
|
||||
assert len(turn_tokens["counts"]) == len(turn_tokens["edges"]) + 1
|
||||
assert sum(turn_tokens["counts"]) == 3, "observed on every turn"
|
||||
assert turn_tokens["counts"][0] == 1, "the 500-token turn is under the first edge"
|
||||
assert sum(payload["hist"]["ttfb_ms"]["counts"]) == 2, "unmeasured ttfb excluded"
|
||||
# 30% saved -> the [20, 40) bucket, which is index 4.
|
||||
assert payload["hist"]["saved_pct"]["counts"][4] == 2
|
||||
|
||||
|
||||
def test_cache_gap_curve_excludes_the_first_turn():
|
||||
class Cold(Outcome):
|
||||
cache_read_tokens = 0
|
||||
|
||||
payload = emit(Outcome(), Outcome(), Cold(), step=45.0)
|
||||
hits = payload["cache"]["hits"]
|
||||
misses = payload["cache"]["misses"]
|
||||
assert sum(hits) + sum(misses) == 2, "first turn has no predecessor to measure"
|
||||
assert payload["cache"]["gap_edges_s"] == [30.0, 60.0, 120.0, 300.0, 600.0, 1800.0]
|
||||
# A 45s gap lands in the [30, 60) bucket.
|
||||
assert hits[1] == 1 and misses[1] == 1
|
||||
assert payload["cache"]["write_5m"] == 240
|
||||
assert payload["cache"]["write_1h"] == 60
|
||||
|
||||
|
||||
def test_trajectory_buckets_are_log_spaced_and_kinds_are_run_length_encoded():
|
||||
class Bypassed(Outcome):
|
||||
attempted_input_tokens = 0
|
||||
tokens_saved = 0
|
||||
tags = {"passthrough_reason": "bypass_header"}
|
||||
|
||||
payload = emit(*([Outcome()] * 4), Bypassed(), Bypassed(), *([Outcome()] * 3))
|
||||
assert payload["trajectory"]["turns"] == [1, 2, 4, 2, 0, 0, 0, 0, 0, 0]
|
||||
assert payload["trajectory"]["kinds"] == "s4p2s3"
|
||||
assert payload["trajectory"]["kinds_truncated"] is False
|
||||
assert payload["trajectory"]["msgs_max"] == 30
|
||||
# And the v1 passthrough counter is untouched by any of it.
|
||||
assert payload["compression"]["passthrough_turns"] == 2
|
||||
|
||||
|
||||
def test_trajectory_kind_alphabet_is_closed():
|
||||
class Cached(Outcome):
|
||||
from_response_cache = True
|
||||
|
||||
class Barren(Outcome):
|
||||
tokens_saved = 0
|
||||
|
||||
class Ineligible(Outcome):
|
||||
attempted_input_tokens = 0
|
||||
tokens_saved = 0
|
||||
|
||||
class Failed(Outcome):
|
||||
status_code = 500
|
||||
|
||||
class Limited(Outcome):
|
||||
status_code = 429
|
||||
|
||||
payload = emit(Cached(), Failed(), Limited(), Barren(), Outcome(), Ineligible())
|
||||
kinds = payload["trajectory"]["kinds"]
|
||||
assert kinds == "c1f1e1z1s1o1"
|
||||
assert set(re.findall(r"[a-z]", kinds)) <= S._TURN_KINDS
|
||||
|
||||
|
||||
def test_trajectory_string_is_capped():
|
||||
outcomes = []
|
||||
for index in range(S.MAX_TRAJECTORY_RUNS * 3):
|
||||
cls = Outcome if index % 2 else type("Alt", (Outcome,), {"status_code": 500})
|
||||
outcomes.append(cls())
|
||||
payload = emit(*outcomes)
|
||||
assert payload["trajectory"]["kinds_truncated"] is True
|
||||
assert len(re.findall(r"[a-z]", payload["trajectory"]["kinds"])) == S.MAX_TRAJECTORY_RUNS
|
||||
|
||||
|
||||
def test_clients_and_strata(beacon_on):
|
||||
class Shaped(Outcome):
|
||||
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
|
||||
|
||||
class Control(Outcome):
|
||||
client = "claude-vscode"
|
||||
transforms_applied = ("output_shaper:control:opus|user|s|notools",)
|
||||
|
||||
class Anonymous(Outcome):
|
||||
client = None
|
||||
|
||||
payload = emit(Shaped(), Control(), Anonymous())
|
||||
assert payload["clients"] == [
|
||||
{"client": "claude_code", "n": 1},
|
||||
{"client": "claude_vscode", "n": 1},
|
||||
], "unknown client not bucketed"
|
||||
strata = {(r["dim"], r["value"]): r["n"] for r in payload["strata"]}
|
||||
assert strata[("arm", "treatment")] == 1 and strata[("arm", "control")] == 1
|
||||
assert strata[("turn_kind", "tool_result")] == 1 and strata[("turn_kind", "user")] == 1
|
||||
assert strata[("input_bucket", "l")] == 1 and strata[("input_bucket", "s")] == 1
|
||||
assert strata[("tools", "tools")] == 1 and strata[("tools", "notools")] == 1
|
||||
# Model family is the one stratum component that must never ship.
|
||||
assert "sonnet" not in json.dumps(payload)
|
||||
assert "opus" not in json.dumps(payload)
|
||||
|
||||
|
||||
def test_client_ids_survive_slugging():
|
||||
"""Every harness `classify_client` can return has to round-trip, or the
|
||||
field reports "other" for the clients that matter. Asserted against the
|
||||
real vocabulary rather than a value invented here — the first version of
|
||||
this test used "claude_code", which is not an id the proxy ever emits, and
|
||||
so passed while `claude-code` was being discarded.
|
||||
"""
|
||||
from headroom.proxy.auth_mode import CLIENT_UA_MAP
|
||||
|
||||
known = sorted({value for _, value in CLIENT_UA_MAP})
|
||||
assert "claude-code" in known, "the vocabulary moved; this test needs updating"
|
||||
|
||||
outcomes = [type(f"C{i}", (Outcome,), {"client": name})() for i, name in enumerate(known)]
|
||||
clients = {r["client"]: r["n"] for r in emit(*outcomes)["clients"]}
|
||||
assert "other" not in clients, (
|
||||
f"a real client id was discarded: {sorted(set(known) - set(clients))}"
|
||||
)
|
||||
assert len(clients) == len(known)
|
||||
assert clients["claude_code"] == 1
|
||||
|
||||
|
||||
def test_client_slug_still_rejects_junk():
|
||||
payload = emit(type("Junk", (Outcome,), {"client": "../../etc/passwd"})())
|
||||
assert payload["clients"] == [{"client": "other", "n": 1}]
|
||||
|
||||
|
||||
def test_content_shape_table(beacon_on):
|
||||
S.record_content_shape("search", "smart_crusher", 1000, 400)
|
||||
S.record_content_shape("search", "smart_crusher", 500, 300)
|
||||
S.record_content_shape("diff", "passthrough", 800, 800)
|
||||
payload = emit(Outcome())
|
||||
rows = {(r["content"], r["strategy"]): r for r in payload["shapes"]["by_content"]}
|
||||
assert rows[("search", "smart_crusher")] == {
|
||||
"content": "search",
|
||||
"strategy": "smart_crusher",
|
||||
"n": 2,
|
||||
"tokens_in": 1500,
|
||||
"tokens_out": 700,
|
||||
}
|
||||
assert rows[("diff", "passthrough")]["tokens_in"] == 800
|
||||
assert isinstance(payload["shapes"]["by_content"], list), "a list, never a map"
|
||||
|
||||
|
||||
def test_content_shape_cannot_open_a_session(beacon_on):
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append)
|
||||
S.record_content_shape("search", "smart_crusher", 1000, 400)
|
||||
agg.flush_all()
|
||||
assert rows == [], "a compression event invented a session"
|
||||
|
||||
|
||||
def test_content_shape_slugs_free_strings(beacon_on):
|
||||
S.record_content_shape("../../etc/passwd", "SELECT * FROM x", 100, 50)
|
||||
payload = emit(Outcome())
|
||||
assert payload["shapes"]["by_content"][0]["content"] == "other"
|
||||
assert payload["shapes"]["by_content"][0]["strategy"] == "other"
|
||||
|
||||
|
||||
def test_tool_shape_is_a_descriptor_never_a_hash(beacon_on):
|
||||
class Signature:
|
||||
structure_hash = "deadbeefcafe0123456789ab"
|
||||
field_count = 17
|
||||
max_depth = 3
|
||||
has_arrays = True
|
||||
has_nested_objects = True
|
||||
has_id_like_field = True
|
||||
has_score_like_field = False
|
||||
has_timestamp_like_field = True
|
||||
has_status_like_field = False
|
||||
has_error_like_field = False
|
||||
has_message_like_field = False
|
||||
|
||||
S.record_tool_shape(Signature(), 5000, 1200)
|
||||
payload = emit(Outcome())
|
||||
assert payload["shapes"]["by_tool"] == [
|
||||
{"shape": "f16d3_anit", "n": 1, "tokens_in": 5000, "tokens_out": 1200}
|
||||
]
|
||||
assert "deadbeef" not in json.dumps(payload), "structure_hash reached the wire"
|
||||
|
||||
|
||||
def test_tool_shape_rejects_a_non_signature(beacon_on):
|
||||
S.record_tool_shape(object(), 5000, 1200)
|
||||
S.record_tool_shape(None, 5000, 1200)
|
||||
payload = emit(Outcome())
|
||||
assert payload["shapes"]["by_tool"] == []
|
||||
|
||||
|
||||
def test_shape_tables_are_capped(beacon_on):
|
||||
"""The cap is a guard against a caller inventing keys, not a budget — see
|
||||
test_content_shape_cap_exceeds_its_vocabulary for the other side."""
|
||||
for index in range(S.MAX_SHAPE_ROWS + 10):
|
||||
S.record_content_shape(f"c{index}", "smart_crusher", 100, 50)
|
||||
payload = emit(Outcome())
|
||||
assert len(payload["shapes"]["by_content"]) == S.MAX_SHAPE_ROWS
|
||||
|
||||
|
||||
def test_config_is_an_allowlist_and_every_value_is_slugged(monkeypatch):
|
||||
monkeypatch.setenv("HEADROOM_MODE", "optimize")
|
||||
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "https://secret.acme.internal/v1")
|
||||
monkeypatch.setenv("HEADROOM_PROXY_TOKEN", "sk-live-abc123")
|
||||
monkeypatch.setenv("HEADROOM_SAVINGS_PROFILE", "/Users/someone/private/path")
|
||||
payload = emit(Outcome())
|
||||
config = {r["key"]: r["value"] for r in payload["config"]}
|
||||
assert config["mode"] == "optimize"
|
||||
assert config["savings_profile"] == "other", "a path shipped verbatim"
|
||||
assert "kompress_endpoint" not in config
|
||||
assert "proxy_token" not in config
|
||||
wire = json.dumps(payload)
|
||||
assert "acme" not in wire and "sk-live" not in wire and "/Users/" not in wire
|
||||
|
||||
|
||||
def test_config_omits_unset_variables(monkeypatch):
|
||||
for name in S._CONFIG_ENV_KEYS:
|
||||
monkeypatch.delenv(name, raising=False)
|
||||
monkeypatch.setenv("HEADROOM_DEDUPE", " ")
|
||||
assert emit(Outcome())["config"] == []
|
||||
|
||||
|
||||
def test_a_v2_payload_failure_cannot_lose_the_v1_event():
|
||||
"""The v2 sections claim they cannot cost a v1 number. Built inline in one
|
||||
dict literal they could: any raise aborts the literal and drops the whole
|
||||
event, `tokens.saved` included."""
|
||||
from unittest.mock import patch
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append)
|
||||
with patch.object(S, "_config_snapshot", side_effect=RuntimeError("boom")):
|
||||
agg.record(Outcome(), now=1000.0)
|
||||
agg.flush_all()
|
||||
assert rows, "a v2 failure dropped the entire session event"
|
||||
assert rows[0]["tokens"]["saved"] == 300, "v1 counters survived intact"
|
||||
assert "config" not in rows[0], "the failed v2 section is absent, not partial"
|
||||
|
||||
|
||||
def test_a_v2_fold_failure_cannot_cost_a_v1_counter_or_a_heartbeat():
|
||||
from unittest.mock import patch
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
|
||||
with patch.object(S, "_fold_extra", side_effect=RuntimeError("boom")):
|
||||
for index in range(4):
|
||||
agg.record(Outcome(), now=7000.0 + index * 60)
|
||||
agg.flush_all()
|
||||
assert len(rows) >= 2, "a v2 fold failure suppressed the heartbeat"
|
||||
assert rows[-1]["session"]["turns"] == 4
|
||||
assert rows[-1]["tokens"]["saved"] == 1200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# client and receiver must agree
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_allowlist_covers_every_emitted_key():
|
||||
"""The worker drops any top-level key not in ALLOWED_KEYS, permanently and
|
||||
silently. A client that emits a key the worker has not been taught is a
|
||||
signal that looks healthy and stores nothing, so the two files have to be
|
||||
checked against each other rather than by eye.
|
||||
"""
|
||||
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
|
||||
match = re.search(r"const ALLOWED_KEYS = \[(.*?)\n\];", worker, re.S)
|
||||
assert match, "ALLOWED_KEYS not found in worker.js"
|
||||
without_comments = re.sub(r"//[^\n]*", "", match.group(1))
|
||||
allowed = set(re.findall(r"'([a-z_]+)'", without_comments))
|
||||
|
||||
emitted = set(emit(Outcome()))
|
||||
missing = emitted - allowed
|
||||
assert not missing, f"worker.js would silently discard: {sorted(missing)}"
|
||||
assert V1_TOP <= allowed, "a v1 key was removed from the allowlist"
|
||||
assert V2_TOP <= allowed
|
||||
|
||||
|
||||
def test_schema_version_was_bumped():
|
||||
assert S.SCHEMA_VERSION == 2
|
||||
assert emit(Outcome())["schema_version"] == 2
|
||||
|
||||
|
||||
def test_worst_case_payload_fits_the_receivers_body_limit():
|
||||
"""The client must not be able to build a body the Worker will refuse.
|
||||
|
||||
Not a formality. The cap is checked on what ARRIVES, so an uncompressed
|
||||
upload -- the kill switch is set, or the gzip fallback fired -- is what has
|
||||
to fit. And a 413 correctly does NOT trigger the uncompressed retry (see
|
||||
the gzip tests), so a body over the cap is simply lost.
|
||||
|
||||
Read from worker.js rather than hardcoded, so raising one and not the other
|
||||
fails here instead of in production.
|
||||
"""
|
||||
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
|
||||
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
|
||||
assert match, "MAX_BODY_BYTES not found in worker.js"
|
||||
cap = int(match.group(1)) * 1024
|
||||
|
||||
# Every bounded table driven to its cap at once.
|
||||
for index in range(S.MAX_SHAPE_ROWS + 20):
|
||||
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
|
||||
S.record_tool_shape(
|
||||
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
|
||||
5000,
|
||||
1200,
|
||||
)
|
||||
for index in range(S.MAX_STRATEGIES + 5):
|
||||
S.record_compression(f"strategy{index}", 9000, 3000)
|
||||
|
||||
outcomes = []
|
||||
for index in range(1200):
|
||||
outcomes.append(
|
||||
type(
|
||||
f"O{index}",
|
||||
(Outcome,),
|
||||
{
|
||||
"status_code": 400 + (index % 99),
|
||||
"client": f"harness_{index % 70}",
|
||||
"transforms_applied": (
|
||||
f"output_shaper:stratum:m|kind{index % 40}|b{index % 9}|tools",
|
||||
),
|
||||
"tags": {"passthrough_reason": f"reason{index % 20}"},
|
||||
"waste_signals": dict.fromkeys(
|
||||
(*S._WASTE_KEYS, "reread", "reread_compressed"), 9
|
||||
),
|
||||
},
|
||||
)()
|
||||
)
|
||||
payload = emit(*outcomes)
|
||||
body = json.dumps(
|
||||
S.build_otlp_logs(payload, S.resource_attributes()), separators=(",", ":")
|
||||
).encode()
|
||||
assert len(body) < cap, (
|
||||
f"worst-case OTLP body is {len(body):,} B against a {cap:,} B cap — the "
|
||||
"Worker would 413 it and it would not be retried"
|
||||
)
|
||||
# Gzipped it is a rounding error, which is the normal path.
|
||||
assert len(gzip.compress(body, S._GZIP_LEVEL)) < cap // 10
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# headroom telemetry --show
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_telemetry_show_prints_the_real_schema_not_a_mockup():
|
||||
"""The command exists so a user can verify what leaves their machine
|
||||
without reading two source files and trusting they match the installed
|
||||
copy. That only holds if it renders the same payload the beacon builds —
|
||||
so it is asserted against the live key set, not against a fixture."""
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli.main import main as cli
|
||||
|
||||
result = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
|
||||
assert result.exit_code == 0, result.output
|
||||
report = json.loads(result.output)
|
||||
assert report["schema_version"] == S.SCHEMA_VERSION
|
||||
assert V1_TOP <= set(report["session_event"])
|
||||
assert V2_TOP <= set(report["session_event"])
|
||||
assert set(report["session_event"]) == set(emit(Outcome())), (
|
||||
"--show has drifted from the payload the beacon actually sends"
|
||||
)
|
||||
|
||||
|
||||
def test_telemetry_show_reports_the_beacon_state(monkeypatch):
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli.main import main as cli
|
||||
|
||||
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
||||
off = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
|
||||
assert json.loads(off.output)["beacon_enabled"] is False
|
||||
|
||||
monkeypatch.setenv("HEADROOM_BEACON", "on")
|
||||
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
|
||||
on = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
|
||||
assert json.loads(on.output)["beacon_enabled"] is True
|
||||
|
||||
|
||||
def test_telemetry_command_does_not_mint_an_install_id(tmp_path, monkeypatch):
|
||||
"""Inspecting telemetry — very plausibly on the way to switching it off —
|
||||
must not be the thing that gives a machine its identifier. `install_id()`
|
||||
creates and persists one as a side effect, so this command reads the file
|
||||
directly instead."""
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli.main import main as cli
|
||||
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
||||
monkeypatch.setenv("HOME", str(tmp_path))
|
||||
# The id is memoised in a module global once anything has read it, so the
|
||||
# cache has to be cleared or this test reads the developer's real id and
|
||||
# never exercises the create path at all.
|
||||
monkeypatch.setattr(S, "_install_id", None)
|
||||
|
||||
# BOTH paths. `--show` is the one that matters and the one that regressed:
|
||||
# filtering the id out of the rendered report is not enough, because
|
||||
# resource_attributes() mints it while building that report.
|
||||
for argv in (["telemetry"], ["telemetry", "--show"], ["telemetry", "--json"]):
|
||||
monkeypatch.setattr(S, "_install_id", None)
|
||||
result = CliRunner().invoke(cli, argv)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert not list(tmp_path.rglob("install_id")), f"{argv} created an install id"
|
||||
assert "none yet" in CliRunner().invoke(cli, ["telemetry"]).output
|
||||
|
||||
|
||||
def test_clients_map_is_capped_against_a_hostile_header():
|
||||
"""`classify_client` returns the `X-Client` header verbatim when a caller
|
||||
sets one, so this field is attacker-influenced. Junk already collapses to
|
||||
"other"; the cap is what stops a caller minting a new valid-looking slug
|
||||
per request and growing the payload without bound."""
|
||||
outcomes = [
|
||||
type(f"C{i}", (Outcome,), {"client": f"harness_{i}"})() for i in range(S.MAX_SHAPES + 25)
|
||||
]
|
||||
clients = emit(*outcomes)["clients"]
|
||||
assert len(clients) == S.MAX_SHAPES
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# shape stability
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def key_shape(node: Any, path: str = "") -> set[str]:
|
||||
"""Every key path in a payload, with list contents collapsed.
|
||||
|
||||
Numeric key segments are normalised, because a map keyed by a number is
|
||||
safe: DuckDB cannot read "429" as a struct field name, so it infers MAP —
|
||||
which is what the v1 `failure_statuses` column has always been across the
|
||||
whole corpus. Identifier-shaped keys are the dangerous ones.
|
||||
"""
|
||||
shape: set[str] = set()
|
||||
if isinstance(node, dict):
|
||||
for key, value in node.items():
|
||||
name = "<num>" if str(key).isdigit() else key
|
||||
shape.add(f"{path}.{name}")
|
||||
shape |= key_shape(value, f"{path}.{name}")
|
||||
elif isinstance(node, list):
|
||||
for item in node:
|
||||
shape |= key_shape(item, f"{path}[]")
|
||||
return shape
|
||||
|
||||
|
||||
def test_payload_shape_does_not_depend_on_the_data():
|
||||
"""The invariant behind every "list of records, not an object" note in
|
||||
payload(): two sessions with nothing in common must produce the SAME set of
|
||||
key paths.
|
||||
|
||||
An object keyed by something that came from the data breaks this, and the
|
||||
consequence is not cosmetic — DuckDB infers such an object as a STRUCT
|
||||
while its keys are few and identifier-shaped and as a MAP once they are
|
||||
not, so the COLUMN TYPE becomes a function of the contents. Measured on the
|
||||
real corpus: two harnesses gave STRUCT(claude_code, codex) and a query for
|
||||
`clients.cursor` failed to bind; twenty-two gave MAP(VARCHAR, BIGINT) and
|
||||
the same query ran. That is the break that took out the transforms report.
|
||||
|
||||
This test is the general guard, so a future field cannot reintroduce it.
|
||||
"""
|
||||
|
||||
class Left(Outcome):
|
||||
client = "claude-code"
|
||||
status_code = 200
|
||||
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
|
||||
waste_signals = {"json_bloat": 50, "reread": 10}
|
||||
|
||||
class Right(Outcome):
|
||||
client = "aider"
|
||||
status_code = 200
|
||||
transforms_applied = ("output_shaper:control:opus|new_user_ask|xs|notools",)
|
||||
waste_signals = {"base64": 7, "whitespace": 3, "repetition": 11}
|
||||
|
||||
import os
|
||||
|
||||
os.environ["HEADROOM_MODE"] = "optimize"
|
||||
try:
|
||||
left = key_shape(emit(Left()))
|
||||
finally:
|
||||
os.environ.pop("HEADROOM_MODE", None)
|
||||
os.environ["HEADROOM_DEDUPE"] = "on"
|
||||
os.environ["HEADROOM_LOSSLESS"] = "on"
|
||||
try:
|
||||
right = key_shape(emit(Right()))
|
||||
finally:
|
||||
os.environ.pop("HEADROOM_DEDUPE", None)
|
||||
os.environ.pop("HEADROOM_LOSSLESS", None)
|
||||
|
||||
assert left == right, (
|
||||
"payload shape varies with its contents — these paths differ, which "
|
||||
f"means a column type will too: {sorted(left ^ right)}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_fields_that_carry_open_vocabularies_are_lists():
|
||||
"""Belt and braces on the test above, naming the four that were objects."""
|
||||
payload = emit(Outcome())
|
||||
for field in ("clients", "config", "strata"):
|
||||
assert isinstance(payload[field], list), f"{field} must be a list of records"
|
||||
assert isinstance(payload["quality"]["waste"], list)
|
||||
# ...and the records have fixed field names, which is what makes the type
|
||||
# stable no matter how the vocabulary grows.
|
||||
assert isinstance(payload["shapes"]["by_content"], list)
|
||||
|
||||
|
||||
def test_trajectory_string_is_a_faithful_prefix_once_truncated():
|
||||
"""Past the cap the string must STOP, not keep growing its last run.
|
||||
|
||||
Letting a matching kind keep incrementing looks harmless and is not: once a
|
||||
run has been dropped the remaining turns are no longer contiguous, so a run
|
||||
of `s6` would claim six consecutive `s` turns where the session actually
|
||||
did `s q s q s q`. Every run in the string has to be true.
|
||||
"""
|
||||
runs_at_cap = []
|
||||
for index in range(S.MAX_TRAJECTORY_RUNS):
|
||||
runs_at_cap.append(
|
||||
type(f"A{index}", (Outcome,), {"status_code": 500 if index % 2 else 200})()
|
||||
)
|
||||
# Then a tail that alternates between the last run's kind and another.
|
||||
tail = []
|
||||
for index in range(20):
|
||||
tail.append(type(f"B{index}", (Outcome,), {"status_code": 200 if index % 2 else 429})())
|
||||
|
||||
payload = emit(*runs_at_cap, *tail)
|
||||
kinds = payload["trajectory"]["kinds"]
|
||||
assert payload["trajectory"]["kinds_truncated"] is True
|
||||
runs = re.findall(r"([a-z])(\d+)", kinds)
|
||||
assert len(runs) == S.MAX_TRAJECTORY_RUNS
|
||||
# The prefix is exactly the pre-cap history: alternating runs of length 1.
|
||||
assert all(int(n) == 1 for _, n in runs), f"a run grew after truncation: {kinds}"
|
||||
# And the uncapped turn counter still accounts for every turn.
|
||||
assert sum(payload["trajectory"]["turns"]) == payload["session"]["turns"]
|
||||
assert payload["session"]["turns"] == S.MAX_TRAJECTORY_RUNS + 20
|
||||
|
||||
|
||||
def test_content_shape_cap_exceeds_its_vocabulary(beacon_on):
|
||||
"""The cap must not be able to bite in normal operation. It drops rows
|
||||
first-come-wins, which would bias the table toward whatever a session
|
||||
routed early — and that table is what a routing policy gets fitted on."""
|
||||
from headroom.transforms.content_detector import ContentType
|
||||
from headroom.transforms.content_router import CompressionStrategy
|
||||
|
||||
# +1 each for the "other" bucket a free string collapses to.
|
||||
cross_product = (len(list(ContentType)) + 1) * (len(list(CompressionStrategy)) + 1)
|
||||
assert S.MAX_SHAPE_ROWS >= cross_product, (
|
||||
f"{cross_product} legitimate (content, strategy) pairs exist but the cap "
|
||||
f"is {S.MAX_SHAPE_ROWS}"
|
||||
)
|
||||
|
||||
for content in ContentType:
|
||||
for strategy in CompressionStrategy:
|
||||
S.record_content_shape(content.value, strategy.value, 100, 50)
|
||||
rows = emit(Outcome())["shapes"]["by_content"]
|
||||
assert len(rows) == len(list(ContentType)) * len(list(CompressionStrategy)), (
|
||||
"a legitimate content/strategy combination was dropped"
|
||||
)
|
||||
|
||||
|
||||
def test_the_client_budget_stays_under_every_deployed_receiver():
|
||||
"""`_MAX_WIRE_BYTES` is only useful while it is below what receivers accept.
|
||||
|
||||
The current Worker allows 256KB, but an install upgrades on its own schedule
|
||||
and may be pointed at an older deployment whose cap is 64KB. Raising the
|
||||
client budget past that would silently reintroduce the 413 this bounds.
|
||||
"""
|
||||
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
|
||||
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
|
||||
assert match, "MAX_BODY_BYTES not found in worker.js"
|
||||
assert S._MAX_WIRE_BYTES <= int(match.group(1)) * 1024
|
||||
assert S._MAX_WIRE_BYTES <= 64 * 1024, (
|
||||
f"the client budget is {S._MAX_WIRE_BYTES:,} B, over the 64KB cap of the "
|
||||
"oldest Worker still in service"
|
||||
)
|
||||
|
||||
|
||||
def test_the_worst_case_body_actually_sent_fits_the_oldest_receiver():
|
||||
"""The test above bounds what can be BUILT; this bounds what is SENT.
|
||||
|
||||
`_fit_to_wire` is the thing standing between the two, and uncompressed v2
|
||||
crosses 64KB well before the shape caps are reached.
|
||||
"""
|
||||
for index in range(S.MAX_SHAPE_ROWS + 20):
|
||||
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
|
||||
S.record_tool_shape(
|
||||
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
|
||||
5000,
|
||||
1200,
|
||||
)
|
||||
outcomes = [
|
||||
type(
|
||||
f"O{index}",
|
||||
(Outcome,),
|
||||
{"status_code": 400 + (index % 99), "client": f"harness_{index % 70}"},
|
||||
)()
|
||||
for index in range(600)
|
||||
]
|
||||
payload = emit(*outcomes)
|
||||
saved = payload["tokens"]["saved"]
|
||||
sent = S._fit_to_wire(payload, S.resource_attributes(), compress=False)
|
||||
assert len(sent) <= 64 * 1024, f"sent {len(sent):,} B, over the 64KB receiver cap"
|
||||
assert payload["tokens"]["saved"] == saved, "trimming cost a v1 counter"
|
||||
@@ -75,7 +75,11 @@ class TestFormatTelemetryNotice:
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
monkeypatch.delenv("HEADROOM_TELEMETRY_WARN", raising=False)
|
||||
notice = format_telemetry_notice()
|
||||
assert "compression stats" in notice
|
||||
# Substance, not the exact phrasing: what is sent (counters), what is
|
||||
# not (prompts), and how to turn it off. The wording widened at schema
|
||||
# v2 — "compression stats" stopped describing a payload that also
|
||||
# carries cache behaviour, session shape and configuration.
|
||||
assert "usage counters" in notice
|
||||
assert "HEADROOM_BEACON=off" in notice
|
||||
|
||||
def test_beacon_notice_names_what_is_not_sent(self, monkeypatch):
|
||||
@@ -233,7 +237,7 @@ class TestWrapCLITelemetryNotice:
|
||||
|
||||
_print_telemetry_notice()
|
||||
captured = capsys.readouterr()
|
||||
assert "compression stats" in captured.out
|
||||
assert "usage counters" in captured.out
|
||||
assert "HEADROOM_BEACON=off" in captured.out
|
||||
|
||||
def test_print_notice_silent_when_telemetry_off(self, monkeypatch, capsys):
|
||||
|
||||
Reference in New Issue
Block a user