Files
headroom/tests/test_beacon_schema_v2.py
ac932eb7c4 feat(telemetry): split beacon sessions by harness, model and day (#4881)
## Description

After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:

1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.

| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |

- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.

**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.

**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.

## Testing

- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
    tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```

Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.

## Real Behavior Proof

- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
  1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
  2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
  4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:

```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
  {"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
  {"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
  {"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
  {"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
  {"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```

Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
  - Linux, Windows, containers and CI.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:50:49 +05:30

1160 lines
45 KiB
Python

"""Beacon schema v2: the new signals, and proof the old ones did not move.
The corpus has years of rows whose meaning depends on the v1 payload staying
exactly what it has always been. Schema v2 is additive, and this file is where
that claim is mechanical rather than asserted: `test_v1_*` locks the shape and
the arithmetic of every pre-existing field, `test_v2_*` covers what was added,
and `test_allowlist_covers_every_emitted_key` enforces the deploy-ordering rule
that a key the worker does not allow is a key that is silently discarded.
"""
from __future__ import annotations
import gzip
import json
import os
import re
import time
from pathlib import Path
from typing import Any
import pytest
from headroom.telemetry import session as S
# ---------------------------------------------------------------------------
# The v1 payload, spelled out. Changing anything in this literal is a schema
# break for every consumer of the existing corpus, and should be very hard to
# do by accident.
# ---------------------------------------------------------------------------
V1_TOP = {
"schema_version",
"session",
"tokens",
"rates",
"compression",
"skips",
"sources",
"providers",
"models",
"failures",
"failure_statuses",
}
V1_NESTED = {
"session": {"id", "seq", "duration_s", "turns", "ended", "final"},
"tokens": {
"original",
"attempted",
"input",
"output",
"saved",
"tool_saved",
"cache_read",
"cache_write",
"uncached",
},
"rates": {
"saved_pct",
"eligible_pct",
"yield_pct",
"all_layers_saved_pct",
"all_layers_yield_pct",
"cache_read_pct",
"overhead_pct",
},
"compression": {
"transforms",
"by_strategy",
"overhead_ms_total",
"latency_ms_total",
"overhead_ms_per_turn",
"latency_ms_per_turn",
"passthrough_turns",
"response_cache_hits",
},
}
V2_TOP = {
"quality",
"hist",
"shapes",
"cache",
"trajectory",
"clients",
"errors",
"strata",
"config",
"client_usage",
"model_usage",
"client_versions",
"install",
}
class Outcome:
"""A fully-populated RequestOutcome stand-in. `_fold` is duck-typed."""
provider = "anthropic"
model = "gpt-4o"
original_tokens = 1000
attempted_input_tokens = 400
optimized_tokens = 700
provider_input_tokens = 700
output_tokens = 20
tokens_saved = 300
cache_read_tokens = 500
cache_write_tokens = 100
cache_write_5m_tokens = 80
cache_write_1h_tokens = 20
uncached_input_tokens = 400
cache_inferred = False
from_response_cache = False
status_code = 200
total_latency_ms = 1000.0
overhead_ms = 50.0
ttfb_ms = 300.0
num_messages = 30
client = "claude-code" # real ids are hyphenated; see test_client_ids_survive
transforms_applied: tuple[str, ...] = ("crush",)
tags: dict[str, Any] = {}
waste_signals: dict[str, int] | None = None
def emit(
*outcomes: Any, start: float = 1000.0, step: float = 45.0, source: str = "proxy"
) -> dict[str, Any]:
"""Fold a sequence of outcomes into one session and return its final row."""
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
for index, outcome in enumerate(outcomes):
agg.record(outcome, now=start + index * step, source=source)
agg.flush_all()
return rows[-1]
@pytest.fixture(autouse=True)
def _clean_staging():
"""Staging is module state; a leak across tests would double-count."""
S._staged_strategies.clear()
S._staged_stacks.clear()
S._staged_shapes.clear()
S._staged_tool_shapes.clear()
S._staged_client_versions.clear()
yield
S._staged_strategies.clear()
S._staged_stacks.clear()
S._staged_shapes.clear()
S._staged_tool_shapes.clear()
S._staged_client_versions.clear()
@pytest.fixture
def beacon_on(monkeypatch):
"""conftest turns the beacon off for hermeticity; the record_* entry points
short-circuit on that, so anything testing them has to turn it back on."""
monkeypatch.setenv("HEADROOM_BEACON", "on")
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
# ---------------------------------------------------------------------------
# v1: nothing moved
# ---------------------------------------------------------------------------
def test_v1_keys_all_present_and_unchanged():
payload = emit(Outcome())
assert V1_TOP <= set(payload), f"v1 key(s) dropped: {V1_TOP - set(payload)}"
for section, expected in V1_NESTED.items():
assert expected <= set(payload[section]), (
f"{section} lost key(s): {expected - set(payload[section])}"
)
def test_v1_arithmetic_is_byte_for_byte_what_it_was():
"""The five ratios the product is judged on, plus the counters they derive
from. These values predate schema v2 and must survive it exactly."""
payload = emit(Outcome())
assert payload["tokens"] == {
"original": 1000,
"attempted": 400,
"input": 700,
"output": 20,
"saved": 300,
"tool_saved": 0,
"cache_read": 500,
"cache_write": 100,
"uncached": 400,
}
assert payload["rates"]["saved_pct"] == 30.0
assert payload["rates"]["eligible_pct"] == 40.0
assert payload["rates"]["yield_pct"] == 75.0
assert payload["rates"]["cache_read_pct"] == 50.0
assert payload["rates"]["overhead_pct"] == 5.0
assert payload["compression"]["transforms"] == {"crush": 1}
assert payload["sources"] == {"proxy": 1}
assert payload["providers"] == ["anthropic"]
def test_v1_failures_stay_5xx_only_when_4xx_is_present():
"""The single most dangerous change in v2: 4xx is now counted, and it must
be counted somewhere `failures` cannot see. Widening `failures` would have
silently redefined it — a session that rate-limited would start reading as
a session that failed.
"""
class RateLimited(Outcome):
status_code = 429
class Overloaded(Outcome):
status_code = 529
payload = emit(Outcome(), RateLimited(), RateLimited(), Overloaded())
assert payload["failures"] == 1, "a 4xx leaked into the 5xx failure count"
assert payload["failure_statuses"] == {"529": 1}
assert payload["errors"] == {"count": 2, "by_status": {"429": 2}}
def test_v1_transforms_still_collapse_the_stratum_suffix():
"""v2 reads the stratum label too, but the v1 `transforms` map must still
see only the prefix — the suffix carries a model family."""
class Shaped(Outcome):
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools", "crush")
payload = emit(Shaped())
assert payload["compression"]["transforms"] == {"output_shaper": 1, "crush": 1}
def test_v1_cumulative_dedupe_still_holds_with_v2_fields():
"""The corpus dedupes on max(seq) per (install, session) and never sums
across heartbeats. Every v2 field has to be cumulative for that to keep
working."""
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
for index in range(6):
agg.record(Outcome(), now=7000.0 + index * 50)
agg.flush_all()
assert len({row["session"]["id"] for row in rows}) == 1
assert [row["session"]["seq"] for row in rows] == list(range(len(rows)))
for field in ("turns",):
values = [row["session"][field] for row in rows]
assert values == sorted(values), f"session.{field} is not monotonic"
# Every v2 counter must climb or hold, never reset.
for row_a, row_b in zip(rows, rows[1:]):
assert row_b["quality"]["reread_tokens"] >= row_a["quality"]["reread_tokens"]
assert sum(row_b["trajectory"]["turns"]) >= sum(row_a["trajectory"]["turns"])
for name, hist in row_b["hist"].items():
assert sum(hist["counts"]) >= sum(row_a["hist"][name]["counts"]), name
# The last row alone reconstructs the session.
assert rows[-1]["session"]["turns"] == 6
assert sum(rows[-1]["trajectory"]["turns"]) == 6
def test_mcp_shim_still_folds_without_the_new_fields():
"""`_McpCompression` knows five fields. v2 reads a dozen more through the
same duck-typed `get`, and must not require any of them."""
payload = emit(
S._McpCompression(
original_tokens=800,
attempted_input_tokens=800,
optimized_tokens=200,
tokens_saved=600,
),
source="mcp",
)
assert payload["sources"] == {"mcp": 1}
assert payload["rates"]["yield_pct"] == 75.0
assert payload["trajectory"]["kinds"] == "s1"
assert payload["clients"] == []
def test_a_malformed_outcome_still_cannot_raise():
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
agg.record(object(), now=1.0)
agg.flush_all()
# ---------------------------------------------------------------------------
# v2: the new signals
# ---------------------------------------------------------------------------
def test_quality_reports_the_regret_label():
class Regretful(Outcome):
waste_signals = {
"reread": 900,
"reread_compressed": 400,
"json_bloat": 50,
"base64": 7,
}
payload = emit(Regretful(), Regretful(), Outcome())
assert payload["quality"]["reread_tokens"] == 1800
assert payload["quality"]["reread_compressed_tokens"] == 800
assert payload["quality"]["waste"] == [
{"kind": "base64", "tokens": 14},
{"kind": "json_bloat", "tokens": 100},
]
assert payload["quality"]["waste_turns"] == 2, "the clean turn was counted"
def test_waste_keys_are_allowlisted_not_copied():
class Sneaky(Outcome):
waste_signals = {"json_bloat": 10, "some_future_key": 99, "/etc/passwd": 5}
payload = emit(Sneaky())
assert payload["quality"]["waste"] == [{"kind": "json_bloat", "tokens": 10}]
def test_histograms_ship_their_own_edges_and_count_defined_samples():
class Small(Outcome):
original_tokens = 500
tokens_saved = 0
ttfb_ms = 0.0 # 0 means unmeasured, not instant
payload = emit(Outcome(), Outcome(), Small())
turn_tokens = payload["hist"]["turn_tokens"]
assert turn_tokens["edges"] == [1000, 4000, 16000, 64000, 128000, 256000]
assert len(turn_tokens["counts"]) == len(turn_tokens["edges"]) + 1
assert sum(turn_tokens["counts"]) == 3, "observed on every turn"
assert turn_tokens["counts"][0] == 1, "the 500-token turn is under the first edge"
assert sum(payload["hist"]["ttfb_ms"]["counts"]) == 2, "unmeasured ttfb excluded"
# 30% saved -> the [20, 40) bucket, which is index 4.
assert payload["hist"]["saved_pct"]["counts"][4] == 2
def test_cache_gap_curve_excludes_the_first_turn():
class Cold(Outcome):
cache_read_tokens = 0
payload = emit(Outcome(), Outcome(), Cold(), step=45.0)
hits = payload["cache"]["hits"]
misses = payload["cache"]["misses"]
assert sum(hits) + sum(misses) == 2, "first turn has no predecessor to measure"
assert payload["cache"]["gap_edges_s"] == [30.0, 60.0, 120.0, 300.0, 600.0, 1800.0]
# A 45s gap lands in the [30, 60) bucket.
assert hits[1] == 1 and misses[1] == 1
assert payload["cache"]["write_5m"] == 240
assert payload["cache"]["write_1h"] == 60
def test_trajectory_buckets_are_log_spaced_and_kinds_are_run_length_encoded():
class Bypassed(Outcome):
attempted_input_tokens = 0
tokens_saved = 0
tags = {"passthrough_reason": "bypass_header"}
payload = emit(*([Outcome()] * 4), Bypassed(), Bypassed(), *([Outcome()] * 3))
assert payload["trajectory"]["turns"] == [1, 2, 4, 2, 0, 0, 0, 0, 0, 0]
assert payload["trajectory"]["kinds"] == "s4p2s3"
assert payload["trajectory"]["kinds_truncated"] is False
assert payload["trajectory"]["msgs_max"] == 30
# And the v1 passthrough counter is untouched by any of it.
assert payload["compression"]["passthrough_turns"] == 2
def test_trajectory_kind_alphabet_is_closed():
class Cached(Outcome):
from_response_cache = True
class Barren(Outcome):
tokens_saved = 0
class Ineligible(Outcome):
attempted_input_tokens = 0
tokens_saved = 0
class Failed(Outcome):
status_code = 500
class Limited(Outcome):
status_code = 429
payload = emit(Cached(), Failed(), Limited(), Barren(), Outcome(), Ineligible())
kinds = payload["trajectory"]["kinds"]
assert kinds == "c1f1e1z1s1o1"
assert set(re.findall(r"[a-z]", kinds)) <= S._TURN_KINDS
def test_trajectory_string_is_capped():
outcomes = []
for index in range(S.MAX_TRAJECTORY_RUNS * 3):
cls = Outcome if index % 2 else type("Alt", (Outcome,), {"status_code": 500})
outcomes.append(cls())
payload = emit(*outcomes)
assert payload["trajectory"]["kinds_truncated"] is True
assert len(re.findall(r"[a-z]", payload["trajectory"]["kinds"])) == S.MAX_TRAJECTORY_RUNS
def test_clients_and_strata(beacon_on):
class Shaped(Outcome):
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
class Control(Outcome):
client = "claude-vscode"
transforms_applied = ("output_shaper:control:opus|user|s|notools",)
class Anonymous(Outcome):
client = None
payload = emit(Shaped(), Control(), Anonymous())
assert payload["clients"] == [
{"client": "claude_code", "n": 1},
{"client": "claude_vscode", "n": 1},
], "unknown client not bucketed"
strata = {(r["dim"], r["value"]): r["n"] for r in payload["strata"]}
assert strata[("arm", "treatment")] == 1 and strata[("arm", "control")] == 1
assert strata[("turn_kind", "tool_result")] == 1 and strata[("turn_kind", "user")] == 1
assert strata[("input_bucket", "l")] == 1 and strata[("input_bucket", "s")] == 1
assert strata[("tools", "tools")] == 1 and strata[("tools", "notools")] == 1
# Model family is the one stratum component that must never ship.
assert "sonnet" not in json.dumps(payload)
assert "opus" not in json.dumps(payload)
def test_client_ids_survive_slugging():
"""Every harness `classify_client` can return has to round-trip, or the
field reports "other" for the clients that matter. Asserted against the
real vocabulary rather than a value invented here — the first version of
this test used "claude_code", which is not an id the proxy ever emits, and
so passed while `claude-code` was being discarded.
"""
from headroom.proxy.auth_mode import CLIENT_UA_MAP
known = sorted({value for _, value in CLIENT_UA_MAP})
assert "claude-code" in known, "the vocabulary moved; this test needs updating"
outcomes = [type(f"C{i}", (Outcome,), {"client": name})() for i, name in enumerate(known)]
clients = {r["client"]: r["n"] for r in emit(*outcomes)["clients"]}
assert "other" not in clients, (
f"a real client id was discarded: {sorted(set(known) - set(clients))}"
)
assert len(clients) == len(known)
assert clients["claude_code"] == 1
def test_client_slug_still_rejects_junk():
payload = emit(type("Junk", (Outcome,), {"client": "../../etc/passwd"})())
assert payload["clients"] == [{"client": "other", "n": 1}]
def test_content_shape_table(beacon_on):
S.record_content_shape("search", "smart_crusher", 1000, 400)
S.record_content_shape("search", "smart_crusher", 500, 300)
S.record_content_shape("diff", "passthrough", 800, 800)
payload = emit(Outcome())
rows = {(r["content"], r["strategy"]): r for r in payload["shapes"]["by_content"]}
assert rows[("search", "smart_crusher")] == {
"content": "search",
"strategy": "smart_crusher",
"n": 2,
"tokens_in": 1500,
"tokens_out": 700,
}
assert rows[("diff", "passthrough")]["tokens_in"] == 800
assert isinstance(payload["shapes"]["by_content"], list), "a list, never a map"
def test_content_shape_cannot_open_a_session(beacon_on):
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
S.record_content_shape("search", "smart_crusher", 1000, 400)
agg.flush_all()
assert rows == [], "a compression event invented a session"
def test_content_shape_slugs_free_strings(beacon_on):
S.record_content_shape("../../etc/passwd", "SELECT * FROM x", 100, 50)
payload = emit(Outcome())
assert payload["shapes"]["by_content"][0]["content"] == "other"
assert payload["shapes"]["by_content"][0]["strategy"] == "other"
def test_tool_shape_is_a_descriptor_never_a_hash(beacon_on):
class Signature:
structure_hash = "deadbeefcafe0123456789ab"
field_count = 17
max_depth = 3
has_arrays = True
has_nested_objects = True
has_id_like_field = True
has_score_like_field = False
has_timestamp_like_field = True
has_status_like_field = False
has_error_like_field = False
has_message_like_field = False
S.record_tool_shape(Signature(), 5000, 1200)
payload = emit(Outcome())
assert payload["shapes"]["by_tool"] == [
{"shape": "f16d3_anit", "n": 1, "tokens_in": 5000, "tokens_out": 1200}
]
assert "deadbeef" not in json.dumps(payload), "structure_hash reached the wire"
def test_tool_shape_rejects_a_non_signature(beacon_on):
S.record_tool_shape(object(), 5000, 1200)
S.record_tool_shape(None, 5000, 1200)
payload = emit(Outcome())
assert payload["shapes"]["by_tool"] == []
def test_shape_tables_are_capped(beacon_on):
"""The cap is a guard against a caller inventing keys, not a budget — see
test_content_shape_cap_exceeds_its_vocabulary for the other side."""
for index in range(S.MAX_SHAPE_ROWS + 10):
S.record_content_shape(f"c{index}", "smart_crusher", 100, 50)
payload = emit(Outcome())
assert len(payload["shapes"]["by_content"]) == S.MAX_SHAPE_ROWS
def test_config_is_an_allowlist_and_every_value_is_slugged(monkeypatch):
monkeypatch.setenv("HEADROOM_MODE", "optimize")
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "https://secret.acme.internal/v1")
monkeypatch.setenv("HEADROOM_PROXY_TOKEN", "sk-live-abc123")
monkeypatch.setenv("HEADROOM_SAVINGS_PROFILE", "/Users/someone/private/path")
payload = emit(Outcome())
config = {r["key"]: r["value"] for r in payload["config"]}
assert config["mode"] == "optimize"
assert config["savings_profile"] == "other", "a path shipped verbatim"
assert "kompress_endpoint" not in config
assert "proxy_token" not in config
wire = json.dumps(payload)
assert "acme" not in wire and "sk-live" not in wire and "/Users/" not in wire
def test_config_omits_unset_variables(monkeypatch):
for name in S._CONFIG_ENV_KEYS:
monkeypatch.delenv(name, raising=False)
monkeypatch.setenv("HEADROOM_DEDUPE", " ")
assert emit(Outcome())["config"] == []
def test_a_v2_payload_failure_cannot_lose_the_v1_event():
"""The v2 sections claim they cannot cost a v1 number. Built inline in one
dict literal they could: any raise aborts the literal and drops the whole
event, `tokens.saved` included."""
from unittest.mock import patch
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append)
with patch.object(S, "_config_snapshot", side_effect=RuntimeError("boom")):
agg.record(Outcome(), now=1000.0)
agg.flush_all()
assert rows, "a v2 failure dropped the entire session event"
assert rows[0]["tokens"]["saved"] == 300, "v1 counters survived intact"
assert "config" not in rows[0], "the failed v2 section is absent, not partial"
def test_a_v2_fold_failure_cannot_cost_a_v1_counter_or_a_heartbeat():
from unittest.mock import patch
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
with patch.object(S, "_fold_extra", side_effect=RuntimeError("boom")):
for index in range(4):
agg.record(Outcome(), now=7000.0 + index * 60)
agg.flush_all()
assert len(rows) >= 2, "a v2 fold failure suppressed the heartbeat"
assert rows[-1]["session"]["turns"] == 4
assert rows[-1]["tokens"]["saved"] == 1200
# ---------------------------------------------------------------------------
# client and receiver must agree
# ---------------------------------------------------------------------------
def test_allowlist_covers_every_emitted_key():
"""The worker drops any top-level key not in ALLOWED_KEYS, permanently and
silently. A client that emits a key the worker has not been taught is a
signal that looks healthy and stores nothing, so the two files have to be
checked against each other rather than by eye.
"""
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
match = re.search(r"const ALLOWED_KEYS = \[(.*?)\n\];", worker, re.S)
assert match, "ALLOWED_KEYS not found in worker.js"
without_comments = re.sub(r"//[^\n]*", "", match.group(1))
allowed = set(re.findall(r"'([a-z_]+)'", without_comments))
emitted = set(emit(Outcome()))
missing = emitted - allowed
assert not missing, f"worker.js would silently discard: {sorted(missing)}"
assert V1_TOP <= allowed, "a v1 key was removed from the allowlist"
assert V2_TOP <= allowed
def test_schema_version_was_bumped():
assert S.SCHEMA_VERSION == 2
assert emit(Outcome())["schema_version"] == 2
def test_worst_case_payload_fits_the_receivers_body_limit():
"""The client must not be able to build a body the Worker will refuse.
Not a formality. The cap is checked on what ARRIVES, so an uncompressed
upload -- the kill switch is set, or the gzip fallback fired -- is what has
to fit. And a 413 correctly does NOT trigger the uncompressed retry (see
the gzip tests), so a body over the cap is simply lost.
Read from worker.js rather than hardcoded, so raising one and not the other
fails here instead of in production.
"""
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
assert match, "MAX_BODY_BYTES not found in worker.js"
cap = int(match.group(1)) * 1024
# Every bounded table driven to its cap at once.
for index in range(S.MAX_SHAPE_ROWS + 20):
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
S.record_tool_shape(
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
5000,
1200,
)
for index in range(S.MAX_STRATEGIES + 5):
S.record_compression(f"strategy{index}", 9000, 3000)
outcomes = []
for index in range(1200):
outcomes.append(
type(
f"O{index}",
(Outcome,),
{
"status_code": 400 + (index % 99),
"client": f"harness_{index % 70}",
"transforms_applied": (
f"output_shaper:stratum:m|kind{index % 40}|b{index % 9}|tools",
),
"tags": {"passthrough_reason": f"reason{index % 20}"},
"waste_signals": dict.fromkeys(
(*S._WASTE_KEYS, "reread", "reread_compressed"), 9
),
},
)()
)
payload = emit(*outcomes)
body = json.dumps(
S.build_otlp_logs(payload, S.resource_attributes()), separators=(",", ":")
).encode()
assert len(body) < cap, (
f"worst-case OTLP body is {len(body):,} B against a {cap:,} B cap — the "
"Worker would 413 it and it would not be retried"
)
# Gzipped it is a rounding error, which is the normal path.
assert len(gzip.compress(body, S._GZIP_LEVEL)) < cap // 10
# ---------------------------------------------------------------------------
# headroom telemetry --show
# ---------------------------------------------------------------------------
def test_telemetry_show_prints_the_real_schema_not_a_mockup():
"""The command exists so a user can verify what leaves their machine
without reading two source files and trusting they match the installed
copy. That only holds if it renders the same payload the beacon builds —
so it is asserted against the live key set, not against a fixture."""
from click.testing import CliRunner
from headroom.cli.main import main as cli
result = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
assert result.exit_code == 0, result.output
report = json.loads(result.output)
assert report["schema_version"] == S.SCHEMA_VERSION
assert V1_TOP <= set(report["session_event"])
assert V2_TOP <= set(report["session_event"])
assert set(report["session_event"]) == set(emit(Outcome())), (
"--show has drifted from the payload the beacon actually sends"
)
def test_telemetry_show_reports_the_beacon_state(monkeypatch):
from click.testing import CliRunner
from headroom.cli.main import main as cli
monkeypatch.setenv("HEADROOM_BEACON", "off")
off = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
assert json.loads(off.output)["beacon_enabled"] is False
monkeypatch.setenv("HEADROOM_BEACON", "on")
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
on = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
assert json.loads(on.output)["beacon_enabled"] is True
def test_telemetry_command_does_not_mint_an_install_id(tmp_path, monkeypatch):
"""Inspecting telemetry — very plausibly on the way to switching it off —
must not be the thing that gives a machine its identifier. `install_id()`
creates and persists one as a side effect, so this command reads the file
directly instead."""
from click.testing import CliRunner
from headroom.cli.main import main as cli
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
monkeypatch.setenv("HOME", str(tmp_path))
# The id is memoised in a module global once anything has read it, so the
# cache has to be cleared or this test reads the developer's real id and
# never exercises the create path at all.
monkeypatch.setattr(S, "_install_id", None)
# BOTH paths. `--show` is the one that matters and the one that regressed:
# filtering the id out of the rendered report is not enough, because
# resource_attributes() mints it while building that report.
for argv in (["telemetry"], ["telemetry", "--show"], ["telemetry", "--json"]):
monkeypatch.setattr(S, "_install_id", None)
result = CliRunner().invoke(cli, argv)
assert result.exit_code == 0, result.output
assert not list(tmp_path.rglob("install_id")), f"{argv} created an install id"
assert "none yet" in CliRunner().invoke(cli, ["telemetry"]).output
def test_clients_map_is_capped_against_a_hostile_header():
"""`classify_client` returns the `X-Client` header verbatim when a caller
sets one, so this field is attacker-influenced. Junk already collapses to
"other"; the cap is what stops a caller minting a new valid-looking slug
per request and growing the payload without bound."""
outcomes = [
type(f"C{i}", (Outcome,), {"client": f"harness_{i}"})() for i in range(S.MAX_SHAPES + 25)
]
clients = emit(*outcomes)["clients"]
assert len(clients) == S.MAX_SHAPES
# ---------------------------------------------------------------------------
# shape stability
# ---------------------------------------------------------------------------
def key_shape(node: Any, path: str = "") -> set[str]:
"""Every key path in a payload, with list contents collapsed.
Numeric key segments are normalised, because a map keyed by a number is
safe: DuckDB cannot read "429" as a struct field name, so it infers MAP —
which is what the v1 `failure_statuses` column has always been across the
whole corpus. Identifier-shaped keys are the dangerous ones.
"""
shape: set[str] = set()
if isinstance(node, dict):
for key, value in node.items():
name = "<num>" if str(key).isdigit() else key
shape.add(f"{path}.{name}")
shape |= key_shape(value, f"{path}.{name}")
elif isinstance(node, list):
for item in node:
shape |= key_shape(item, f"{path}[]")
return shape
def test_payload_shape_does_not_depend_on_the_data():
"""The invariant behind every "list of records, not an object" note in
payload(): two sessions with nothing in common must produce the SAME set of
key paths.
An object keyed by something that came from the data breaks this, and the
consequence is not cosmetic — DuckDB infers such an object as a STRUCT
while its keys are few and identifier-shaped and as a MAP once they are
not, so the COLUMN TYPE becomes a function of the contents. Measured on the
real corpus: two harnesses gave STRUCT(claude_code, codex) and a query for
`clients.cursor` failed to bind; twenty-two gave MAP(VARCHAR, BIGINT) and
the same query ran. That is the break that took out the transforms report.
This test is the general guard, so a future field cannot reintroduce it.
"""
class Left(Outcome):
client = "claude-code"
status_code = 200
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
waste_signals = {"json_bloat": 50, "reread": 10}
class Right(Outcome):
client = "aider"
status_code = 200
transforms_applied = ("output_shaper:control:opus|new_user_ask|xs|notools",)
waste_signals = {"base64": 7, "whitespace": 3, "repetition": 11}
import os
os.environ["HEADROOM_MODE"] = "optimize"
try:
left = key_shape(emit(Left()))
finally:
os.environ.pop("HEADROOM_MODE", None)
os.environ["HEADROOM_DEDUPE"] = "on"
os.environ["HEADROOM_LOSSLESS"] = "on"
try:
right = key_shape(emit(Right()))
finally:
os.environ.pop("HEADROOM_DEDUPE", None)
os.environ.pop("HEADROOM_LOSSLESS", None)
assert left == right, (
"payload shape varies with its contents — these paths differ, which "
f"means a column type will too: {sorted(left ^ right)}"
)
def test_the_fields_that_carry_open_vocabularies_are_lists():
"""Belt and braces on the test above, naming the four that were objects."""
payload = emit(Outcome())
for field in ("clients", "config", "strata"):
assert isinstance(payload[field], list), f"{field} must be a list of records"
assert isinstance(payload["quality"]["waste"], list)
# ...and the records have fixed field names, which is what makes the type
# stable no matter how the vocabulary grows.
assert isinstance(payload["shapes"]["by_content"], list)
def test_trajectory_string_is_a_faithful_prefix_once_truncated():
"""Past the cap the string must STOP, not keep growing its last run.
Letting a matching kind keep incrementing looks harmless and is not: once a
run has been dropped the remaining turns are no longer contiguous, so a run
of `s6` would claim six consecutive `s` turns where the session actually
did `s q s q s q`. Every run in the string has to be true.
"""
runs_at_cap = []
for index in range(S.MAX_TRAJECTORY_RUNS):
runs_at_cap.append(
type(f"A{index}", (Outcome,), {"status_code": 500 if index % 2 else 200})()
)
# Then a tail that alternates between the last run's kind and another.
tail = []
for index in range(20):
tail.append(type(f"B{index}", (Outcome,), {"status_code": 200 if index % 2 else 429})())
payload = emit(*runs_at_cap, *tail)
kinds = payload["trajectory"]["kinds"]
assert payload["trajectory"]["kinds_truncated"] is True
runs = re.findall(r"([a-z])(\d+)", kinds)
assert len(runs) == S.MAX_TRAJECTORY_RUNS
# The prefix is exactly the pre-cap history: alternating runs of length 1.
assert all(int(n) == 1 for _, n in runs), f"a run grew after truncation: {kinds}"
# And the uncapped turn counter still accounts for every turn.
assert sum(payload["trajectory"]["turns"]) == payload["session"]["turns"]
assert payload["session"]["turns"] == S.MAX_TRAJECTORY_RUNS + 20
def test_content_shape_cap_exceeds_its_vocabulary(beacon_on):
"""The cap must not be able to bite in normal operation. It drops rows
first-come-wins, which would bias the table toward whatever a session
routed early — and that table is what a routing policy gets fitted on."""
from headroom.transforms.content_detector import ContentType
from headroom.transforms.content_router import CompressionStrategy
# +1 each for the "other" bucket a free string collapses to.
cross_product = (len(list(ContentType)) + 1) * (len(list(CompressionStrategy)) + 1)
assert S.MAX_SHAPE_ROWS >= cross_product, (
f"{cross_product} legitimate (content, strategy) pairs exist but the cap "
f"is {S.MAX_SHAPE_ROWS}"
)
for content in ContentType:
for strategy in CompressionStrategy:
S.record_content_shape(content.value, strategy.value, 100, 50)
rows = emit(Outcome())["shapes"]["by_content"]
assert len(rows) == len(list(ContentType)) * len(list(CompressionStrategy)), (
"a legitimate content/strategy combination was dropped"
)
def test_the_client_budget_stays_under_every_deployed_receiver():
"""`_MAX_WIRE_BYTES` is only useful while it is below what receivers accept.
The current Worker allows 256KB, but an install upgrades on its own schedule
and may be pointed at an older deployment whose cap is 64KB. Raising the
client budget past that would silently reintroduce the 413 this bounds.
"""
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
assert match, "MAX_BODY_BYTES not found in worker.js"
assert S._MAX_WIRE_BYTES <= int(match.group(1)) * 1024
assert S._MAX_WIRE_BYTES <= 64 * 1024, (
f"the client budget is {S._MAX_WIRE_BYTES:,} B, over the 64KB cap of the "
"oldest Worker still in service"
)
def test_the_worst_case_body_actually_sent_fits_the_oldest_receiver():
"""The test above bounds what can be BUILT; this bounds what is SENT.
`_fit_to_wire` is the thing standing between the two, and uncompressed v2
crosses 64KB well before the shape caps are reached.
"""
for index in range(S.MAX_SHAPE_ROWS + 20):
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
S.record_tool_shape(
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
5000,
1200,
)
outcomes = [
type(
f"O{index}",
(Outcome,),
{"status_code": 400 + (index % 99), "client": f"harness_{index % 70}"},
)()
for index in range(600)
]
payload = emit(*outcomes)
# The usage tables and versions cannot be shed, so fill them to their caps
# with nine-digit counters: the body must still fit.
big = dict.fromkeys(S._USAGE_FIELDS, 987_654_321)
names = sorted(S._KNOWN_CLIENTS)
payload["client_usage"] = [{"client": names[i], **big} for i in range(S.MAX_USAGE_ROWS + 1)]
payload["model_usage"] = [
{"model": f"claude-opus-4-8-2026010{i}", **big} for i in range(S.MAX_USAGE_ROWS + 1)
]
payload["client_versions"] = [
{"client": names[i % len(names)], "version": f"1234.1234.{1000 + i}", "n": 987_654_321}
for i in range(S.MAX_CLIENT_VERSIONS)
]
saved = payload["tokens"]["saved"]
sent = S._fit_to_wire(payload, S.resource_attributes(), compress=False)
assert len(sent) <= 64 * 1024, f"sent {len(sent):,} B, over the 64KB receiver cap"
assert payload["tokens"]["saved"] == saved, "trimming cost a v1 counter"
# ---------------------------------------------------------------------------
# usage by harness and model, harness versions, install facts, daily rollover
# ---------------------------------------------------------------------------
def test_usage_tables_split_the_session_totals_without_residue():
"""`client_usage` and `model_usage` exist to divide a mixed session, so each
must add back up to the session: every turn lands in exactly one row,
including the turns no harness claimed."""
class Codex(Outcome):
client = "codex"
model = "gpt-4o-mini"
tags = {"tool_search_deferred_tokens": 70}
class Gateway(Outcome):
client = None
provider = "compress"
model = "ft:gpt-4o:acme-corp:internal-bot:abc123"
class Anonymous(Outcome):
client = None
payload = emit(Outcome(), Outcome(), Codex(), Gateway(), Anonymous())
by_client = {r["client"]: r for r in payload["client_usage"]}
assert {name: r["n"] for name, r in by_client.items()} == {
"claude_code": 2,
"codex": 1,
"via_compress_api": 1,
"unidentified": 1,
}
assert by_client["codex"]["tool_saved"] == 70
for table in ("client_usage", "model_usage"):
rows = payload[table]
assert sum(r["n"] for r in rows) == payload["session"]["turns"], table
for column, total in (
("original", "original"),
("saved", "saved"),
("tool_saved", "tool_saved"),
("output", "output"),
("cache_read", "cache_read"),
("cache_write", "cache_write"),
("uncached", "uncached"),
):
assert sum(r[column] for r in rows) == payload["tokens"][total], (table, column)
assert sum(r["cache_write_5m"] for r in rows) == payload["cache"]["write_5m"], table
assert sum(r["cache_write_1h"] for r in rows) == payload["cache"]["write_1h"], table
# A fine-tune id carries an org name: pooled, never sent.
assert "acme" not in json.dumps(payload)
assert {r["model"] for r in payload["model_usage"]} == {"gpt-4o", "gpt-4o-mini", "other"}
def test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does():
"""Where a provider reports no write count, Headroom infers one as input
minus cached. Those are the same tokens as `uncached` and are never billed
as a write, so a row has to say how much of its `cache_write` is inferred
or a reader double-counts them: 1000 billed tokens would price as 2000."""
class InferredWrite(Outcome):
provider = "openai"
model = "gpt-4o"
provider_input_tokens = 1000
cache_read_tokens = 0
cache_write_tokens = 1000
cache_write_5m_tokens = 0
cache_write_1h_tokens = 0
uncached_input_tokens = 1000
cache_inferred = True
class ReportedWrite(Outcome):
model = "gpt-4o-mini" # cache_write 100 = 80 at 5m + 20 at 1h, all billed
def billed_input(row: dict[str, Any]) -> int:
return (
row["uncached"] + row["cache_read"] + row["cache_write"] - row["cache_write_inferred"]
)
rows = {r["model"]: r for r in emit(InferredWrite(), ReportedWrite())["model_usage"]}
assert billed_input(rows["gpt-4o"]) == 1000
assert rows["gpt-4o"]["cache_write_inferred"] == 1000
assert billed_input(rows["gpt-4o-mini"]) == 400 + 500 + 100
assert (rows["gpt-4o-mini"]["cache_write_5m"], rows["gpt-4o-mini"]["cache_write_1h"]) == (
80,
20,
)
def test_an_mcp_turn_is_labelled_not_dropped():
payload = emit(
S._McpCompression(
original_tokens=800, attempted_input_tokens=800, optimized_tokens=200, tokens_saved=600
),
source="mcp",
)
assert [(r["client"], r["n"], r["saved"]) for r in payload["client_usage"]] == [
("via_mcp", 1, 600)
]
def test_usage_tables_pool_overflow_instead_of_losing_turns():
table: dict[str, list[int]] = {}
for index in range(S.MAX_USAGE_ROWS + 9):
S._add_usage(table, f"model{index}", (1, 10))
assert len(table) == S.MAX_USAGE_ROWS + 1
assert table["other"] == [9, 90]
assert sum(row[0] for row in table.values()) == S.MAX_USAGE_ROWS + 9
def test_a_caller_supplied_client_name_never_reaches_the_usage_tables(beacon_on):
"""`classify_client` returns the caller's `X-Client` header as typed, so the
name is checked against a closed list, not just for shape."""
class Named(Outcome):
client = "acme-corp-billing"
S.record_client_version("acme-corp-billing", "1.2.3")
payload = emit(Named(), Outcome())
assert [(r["client"], r["n"]) for r in payload["client_usage"]] == [
("claude_code", 1),
("other", 1),
]
assert payload["client_versions"] == []
def test_the_known_client_list_covers_every_harness_the_proxy_can_name():
from headroom.proxy.auth_policy import CLIENT_UA_MAP
from headroom.telemetry.context import _KNOWN_WRAP_AGENTS
named = {S._client_slug(name) for _, name in CLIENT_UA_MAP} | set(_KNOWN_WRAP_AGENTS)
assert named <= S._KNOWN_CLIENTS, sorted(named - S._KNOWN_CLIENTS)
@pytest.mark.parametrize(
("user_agent", "expected"),
[
("claude-cli/2.1.311 (external, cli)", ("claude-code", "2.1.311")),
("Corp-Wrapper/9 codex-cli/0.45.0-alpha.3", ("codex", "0.45.0")),
("claude-cli/dev-build", None),
("curl/8.4.0", None),
# Over-long or four-part numbers are not releases; never cut one down.
("claude-cli/20261115", None),
("claude-cli/1.2.3456789", None),
("claude-cli/10.0.3.7", None),
],
)
def test_user_agent_version_parser(user_agent, expected):
from headroom.proxy.auth_policy import client_version_from_user_agent
assert client_version_from_user_agent(user_agent) == expected
def test_only_a_numeric_release_of_a_known_harness_is_staged(beacon_on):
S.record_client_version("claude-code", "2.1.311")
S.record_client_version("claude-code", "2.1.311")
S.record_client_version("claude-code", "2.1.311-internal.example.com")
payload = emit(Outcome())
assert payload["client_versions"] == [{"client": "claude_code", "version": "2.1.311", "n": 2}]
def test_install_facts_report_age_and_ephemerality(tmp_path, monkeypatch):
id_file = tmp_path / "install_id"
id_file.write_text("a" * 32)
ten_days_ago = time.time() - 10 * 86400 - 60
os.utime(id_file, (ten_days_ago, ten_days_ago))
monkeypatch.setattr(S, "_install_id_path", lambda: id_file)
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
monkeypatch.setenv("CI", "true")
facts = emit(Outcome())["install"]
assert facts["age_days"] == 10 and facts["ci"] is True
assert set(facts) == {"age_days", "container", "ci"}
# Stateless: the id is random per process, so there is no age to report.
monkeypatch.setenv("HEADROOM_STATELESS", "1")
facts = emit(Outcome())["install"]
assert facts["age_days"] == 0
def test_a_long_running_session_is_closed_at_the_utc_day_boundary():
"""An always-on proxy never idles, so its session used to grow for weeks.
It now ends at the day boundary; a session that merely runs past midnight
is left whole."""
day = 86400.0
rows: list[dict[str, Any]] = []
agg = S.SessionAggregator(rows.append, idle_s=10 * day, flush_s=100 * day)
start = 20 * day + 3600.0 # 01:00 UTC
agg.record(Outcome(), now=start)
agg.record(Outcome(), now=start + 22 * 3600.0) # 23:00, same day
assert rows == []
agg.record(Outcome(), now=21 * day + 5.0) # just past midnight, 23h old
assert [(r["session"]["ended"], r["session"]["turns"]) for r in rows] == [("rollover", 2)]
agg.record(Outcome(), now=22 * day + 5.0) # the next piece closes a day later
assert [r["session"]["ended"] for r in rows] == ["rollover", "rollover"]
assert rows[0]["session"]["id"] != rows[1]["session"]["id"]
# A clock that steps backwards must not close anything.
agg.record(Outcome(), now=21 * day + 5.0)
assert len(rows) == 2
# Started at 20:00 UTC and still going at 01:00: five hours old, untouched.
rows.clear()
evening = S.SessionAggregator(rows.append, idle_s=10 * day, flush_s=100 * day)
evening.record(Outcome(), now=30 * day + 20 * 3600.0)
evening.record(Outcome(), now=31 * day + 3600.0)
assert rows == []
def test_the_proxy_reports_the_harness_release_it_was_called_by(beacon_on):
"""The wiring, end to end: a real request's User-Agent reaches the payload
as a harness and a release, and nothing else from the header does."""
from tests.test_header_isolation import _make_anthropic_app
client, _ = _make_anthropic_app()
response = client.post(
"/v1/messages",
headers={
"x-api-key": "test-key",
"anthropic-version": "2023-06-01",
"user-agent": "claude-cli/2.1.311 (external, cli) host-a.internal",
},
json={
"model": "claude-sonnet-4-6",
"max_tokens": 16,
"messages": [{"role": "user", "content": "hi"}],
},
)
assert response.status_code == 200, response.text
rows: list[dict[str, Any]] = []
S.get_session_aggregator().flush_all(emit=rows.append)
assert rows[-1]["client_versions"] == [{"client": "claude_code", "version": "2.1.311", "n": 1}]
assert rows[-1]["client_usage"][0]["client"] == "claude_code"
assert "host-a" not in json.dumps(rows[-1])