mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description
After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:
1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.
| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |
- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.
**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.
**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.
## Testing
- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```
Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.
## Real Behavior Proof
- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:
```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
{"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
{"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
{"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
{"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
{"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```
Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
- Linux, Windows, containers and CI.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
1160 lines
45 KiB
Python
1160 lines
45 KiB
Python
"""Beacon schema v2: the new signals, and proof the old ones did not move.
|
|
|
|
The corpus has years of rows whose meaning depends on the v1 payload staying
|
|
exactly what it has always been. Schema v2 is additive, and this file is where
|
|
that claim is mechanical rather than asserted: `test_v1_*` locks the shape and
|
|
the arithmetic of every pre-existing field, `test_v2_*` covers what was added,
|
|
and `test_allowlist_covers_every_emitted_key` enforces the deploy-ordering rule
|
|
that a key the worker does not allow is a key that is silently discarded.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import gzip
|
|
import json
|
|
import os
|
|
import re
|
|
import time
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from headroom.telemetry import session as S
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The v1 payload, spelled out. Changing anything in this literal is a schema
|
|
# break for every consumer of the existing corpus, and should be very hard to
|
|
# do by accident.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
V1_TOP = {
|
|
"schema_version",
|
|
"session",
|
|
"tokens",
|
|
"rates",
|
|
"compression",
|
|
"skips",
|
|
"sources",
|
|
"providers",
|
|
"models",
|
|
"failures",
|
|
"failure_statuses",
|
|
}
|
|
|
|
V1_NESTED = {
|
|
"session": {"id", "seq", "duration_s", "turns", "ended", "final"},
|
|
"tokens": {
|
|
"original",
|
|
"attempted",
|
|
"input",
|
|
"output",
|
|
"saved",
|
|
"tool_saved",
|
|
"cache_read",
|
|
"cache_write",
|
|
"uncached",
|
|
},
|
|
"rates": {
|
|
"saved_pct",
|
|
"eligible_pct",
|
|
"yield_pct",
|
|
"all_layers_saved_pct",
|
|
"all_layers_yield_pct",
|
|
"cache_read_pct",
|
|
"overhead_pct",
|
|
},
|
|
"compression": {
|
|
"transforms",
|
|
"by_strategy",
|
|
"overhead_ms_total",
|
|
"latency_ms_total",
|
|
"overhead_ms_per_turn",
|
|
"latency_ms_per_turn",
|
|
"passthrough_turns",
|
|
"response_cache_hits",
|
|
},
|
|
}
|
|
|
|
V2_TOP = {
|
|
"quality",
|
|
"hist",
|
|
"shapes",
|
|
"cache",
|
|
"trajectory",
|
|
"clients",
|
|
"errors",
|
|
"strata",
|
|
"config",
|
|
"client_usage",
|
|
"model_usage",
|
|
"client_versions",
|
|
"install",
|
|
}
|
|
|
|
|
|
class Outcome:
|
|
"""A fully-populated RequestOutcome stand-in. `_fold` is duck-typed."""
|
|
|
|
provider = "anthropic"
|
|
model = "gpt-4o"
|
|
original_tokens = 1000
|
|
attempted_input_tokens = 400
|
|
optimized_tokens = 700
|
|
provider_input_tokens = 700
|
|
output_tokens = 20
|
|
tokens_saved = 300
|
|
cache_read_tokens = 500
|
|
cache_write_tokens = 100
|
|
cache_write_5m_tokens = 80
|
|
cache_write_1h_tokens = 20
|
|
uncached_input_tokens = 400
|
|
cache_inferred = False
|
|
from_response_cache = False
|
|
status_code = 200
|
|
total_latency_ms = 1000.0
|
|
overhead_ms = 50.0
|
|
ttfb_ms = 300.0
|
|
num_messages = 30
|
|
client = "claude-code" # real ids are hyphenated; see test_client_ids_survive
|
|
transforms_applied: tuple[str, ...] = ("crush",)
|
|
tags: dict[str, Any] = {}
|
|
waste_signals: dict[str, int] | None = None
|
|
|
|
|
|
def emit(
|
|
*outcomes: Any, start: float = 1000.0, step: float = 45.0, source: str = "proxy"
|
|
) -> dict[str, Any]:
|
|
"""Fold a sequence of outcomes into one session and return its final row."""
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append)
|
|
for index, outcome in enumerate(outcomes):
|
|
agg.record(outcome, now=start + index * step, source=source)
|
|
agg.flush_all()
|
|
return rows[-1]
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean_staging():
|
|
"""Staging is module state; a leak across tests would double-count."""
|
|
S._staged_strategies.clear()
|
|
S._staged_stacks.clear()
|
|
S._staged_shapes.clear()
|
|
S._staged_tool_shapes.clear()
|
|
S._staged_client_versions.clear()
|
|
yield
|
|
S._staged_strategies.clear()
|
|
S._staged_stacks.clear()
|
|
S._staged_shapes.clear()
|
|
S._staged_tool_shapes.clear()
|
|
S._staged_client_versions.clear()
|
|
|
|
|
|
@pytest.fixture
|
|
def beacon_on(monkeypatch):
|
|
"""conftest turns the beacon off for hermeticity; the record_* entry points
|
|
short-circuit on that, so anything testing them has to turn it back on."""
|
|
monkeypatch.setenv("HEADROOM_BEACON", "on")
|
|
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
|
|
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# v1: nothing moved
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_v1_keys_all_present_and_unchanged():
|
|
payload = emit(Outcome())
|
|
assert V1_TOP <= set(payload), f"v1 key(s) dropped: {V1_TOP - set(payload)}"
|
|
for section, expected in V1_NESTED.items():
|
|
assert expected <= set(payload[section]), (
|
|
f"{section} lost key(s): {expected - set(payload[section])}"
|
|
)
|
|
|
|
|
|
def test_v1_arithmetic_is_byte_for_byte_what_it_was():
|
|
"""The five ratios the product is judged on, plus the counters they derive
|
|
from. These values predate schema v2 and must survive it exactly."""
|
|
payload = emit(Outcome())
|
|
assert payload["tokens"] == {
|
|
"original": 1000,
|
|
"attempted": 400,
|
|
"input": 700,
|
|
"output": 20,
|
|
"saved": 300,
|
|
"tool_saved": 0,
|
|
"cache_read": 500,
|
|
"cache_write": 100,
|
|
"uncached": 400,
|
|
}
|
|
assert payload["rates"]["saved_pct"] == 30.0
|
|
assert payload["rates"]["eligible_pct"] == 40.0
|
|
assert payload["rates"]["yield_pct"] == 75.0
|
|
assert payload["rates"]["cache_read_pct"] == 50.0
|
|
assert payload["rates"]["overhead_pct"] == 5.0
|
|
assert payload["compression"]["transforms"] == {"crush": 1}
|
|
assert payload["sources"] == {"proxy": 1}
|
|
assert payload["providers"] == ["anthropic"]
|
|
|
|
|
|
def test_v1_failures_stay_5xx_only_when_4xx_is_present():
|
|
"""The single most dangerous change in v2: 4xx is now counted, and it must
|
|
be counted somewhere `failures` cannot see. Widening `failures` would have
|
|
silently redefined it — a session that rate-limited would start reading as
|
|
a session that failed.
|
|
"""
|
|
|
|
class RateLimited(Outcome):
|
|
status_code = 429
|
|
|
|
class Overloaded(Outcome):
|
|
status_code = 529
|
|
|
|
payload = emit(Outcome(), RateLimited(), RateLimited(), Overloaded())
|
|
assert payload["failures"] == 1, "a 4xx leaked into the 5xx failure count"
|
|
assert payload["failure_statuses"] == {"529": 1}
|
|
assert payload["errors"] == {"count": 2, "by_status": {"429": 2}}
|
|
|
|
|
|
def test_v1_transforms_still_collapse_the_stratum_suffix():
|
|
"""v2 reads the stratum label too, but the v1 `transforms` map must still
|
|
see only the prefix — the suffix carries a model family."""
|
|
|
|
class Shaped(Outcome):
|
|
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools", "crush")
|
|
|
|
payload = emit(Shaped())
|
|
assert payload["compression"]["transforms"] == {"output_shaper": 1, "crush": 1}
|
|
|
|
|
|
def test_v1_cumulative_dedupe_still_holds_with_v2_fields():
|
|
"""The corpus dedupes on max(seq) per (install, session) and never sums
|
|
across heartbeats. Every v2 field has to be cumulative for that to keep
|
|
working."""
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
|
|
for index in range(6):
|
|
agg.record(Outcome(), now=7000.0 + index * 50)
|
|
agg.flush_all()
|
|
|
|
assert len({row["session"]["id"] for row in rows}) == 1
|
|
assert [row["session"]["seq"] for row in rows] == list(range(len(rows)))
|
|
for field in ("turns",):
|
|
values = [row["session"][field] for row in rows]
|
|
assert values == sorted(values), f"session.{field} is not monotonic"
|
|
# Every v2 counter must climb or hold, never reset.
|
|
for row_a, row_b in zip(rows, rows[1:]):
|
|
assert row_b["quality"]["reread_tokens"] >= row_a["quality"]["reread_tokens"]
|
|
assert sum(row_b["trajectory"]["turns"]) >= sum(row_a["trajectory"]["turns"])
|
|
for name, hist in row_b["hist"].items():
|
|
assert sum(hist["counts"]) >= sum(row_a["hist"][name]["counts"]), name
|
|
# The last row alone reconstructs the session.
|
|
assert rows[-1]["session"]["turns"] == 6
|
|
assert sum(rows[-1]["trajectory"]["turns"]) == 6
|
|
|
|
|
|
def test_mcp_shim_still_folds_without_the_new_fields():
|
|
"""`_McpCompression` knows five fields. v2 reads a dozen more through the
|
|
same duck-typed `get`, and must not require any of them."""
|
|
payload = emit(
|
|
S._McpCompression(
|
|
original_tokens=800,
|
|
attempted_input_tokens=800,
|
|
optimized_tokens=200,
|
|
tokens_saved=600,
|
|
),
|
|
source="mcp",
|
|
)
|
|
assert payload["sources"] == {"mcp": 1}
|
|
assert payload["rates"]["yield_pct"] == 75.0
|
|
assert payload["trajectory"]["kinds"] == "s1"
|
|
assert payload["clients"] == []
|
|
|
|
|
|
def test_a_malformed_outcome_still_cannot_raise():
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append)
|
|
agg.record(object(), now=1.0)
|
|
agg.flush_all()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# v2: the new signals
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_quality_reports_the_regret_label():
|
|
class Regretful(Outcome):
|
|
waste_signals = {
|
|
"reread": 900,
|
|
"reread_compressed": 400,
|
|
"json_bloat": 50,
|
|
"base64": 7,
|
|
}
|
|
|
|
payload = emit(Regretful(), Regretful(), Outcome())
|
|
assert payload["quality"]["reread_tokens"] == 1800
|
|
assert payload["quality"]["reread_compressed_tokens"] == 800
|
|
assert payload["quality"]["waste"] == [
|
|
{"kind": "base64", "tokens": 14},
|
|
{"kind": "json_bloat", "tokens": 100},
|
|
]
|
|
assert payload["quality"]["waste_turns"] == 2, "the clean turn was counted"
|
|
|
|
|
|
def test_waste_keys_are_allowlisted_not_copied():
|
|
class Sneaky(Outcome):
|
|
waste_signals = {"json_bloat": 10, "some_future_key": 99, "/etc/passwd": 5}
|
|
|
|
payload = emit(Sneaky())
|
|
assert payload["quality"]["waste"] == [{"kind": "json_bloat", "tokens": 10}]
|
|
|
|
|
|
def test_histograms_ship_their_own_edges_and_count_defined_samples():
|
|
class Small(Outcome):
|
|
original_tokens = 500
|
|
tokens_saved = 0
|
|
ttfb_ms = 0.0 # 0 means unmeasured, not instant
|
|
|
|
payload = emit(Outcome(), Outcome(), Small())
|
|
turn_tokens = payload["hist"]["turn_tokens"]
|
|
assert turn_tokens["edges"] == [1000, 4000, 16000, 64000, 128000, 256000]
|
|
assert len(turn_tokens["counts"]) == len(turn_tokens["edges"]) + 1
|
|
assert sum(turn_tokens["counts"]) == 3, "observed on every turn"
|
|
assert turn_tokens["counts"][0] == 1, "the 500-token turn is under the first edge"
|
|
assert sum(payload["hist"]["ttfb_ms"]["counts"]) == 2, "unmeasured ttfb excluded"
|
|
# 30% saved -> the [20, 40) bucket, which is index 4.
|
|
assert payload["hist"]["saved_pct"]["counts"][4] == 2
|
|
|
|
|
|
def test_cache_gap_curve_excludes_the_first_turn():
|
|
class Cold(Outcome):
|
|
cache_read_tokens = 0
|
|
|
|
payload = emit(Outcome(), Outcome(), Cold(), step=45.0)
|
|
hits = payload["cache"]["hits"]
|
|
misses = payload["cache"]["misses"]
|
|
assert sum(hits) + sum(misses) == 2, "first turn has no predecessor to measure"
|
|
assert payload["cache"]["gap_edges_s"] == [30.0, 60.0, 120.0, 300.0, 600.0, 1800.0]
|
|
# A 45s gap lands in the [30, 60) bucket.
|
|
assert hits[1] == 1 and misses[1] == 1
|
|
assert payload["cache"]["write_5m"] == 240
|
|
assert payload["cache"]["write_1h"] == 60
|
|
|
|
|
|
def test_trajectory_buckets_are_log_spaced_and_kinds_are_run_length_encoded():
|
|
class Bypassed(Outcome):
|
|
attempted_input_tokens = 0
|
|
tokens_saved = 0
|
|
tags = {"passthrough_reason": "bypass_header"}
|
|
|
|
payload = emit(*([Outcome()] * 4), Bypassed(), Bypassed(), *([Outcome()] * 3))
|
|
assert payload["trajectory"]["turns"] == [1, 2, 4, 2, 0, 0, 0, 0, 0, 0]
|
|
assert payload["trajectory"]["kinds"] == "s4p2s3"
|
|
assert payload["trajectory"]["kinds_truncated"] is False
|
|
assert payload["trajectory"]["msgs_max"] == 30
|
|
# And the v1 passthrough counter is untouched by any of it.
|
|
assert payload["compression"]["passthrough_turns"] == 2
|
|
|
|
|
|
def test_trajectory_kind_alphabet_is_closed():
|
|
class Cached(Outcome):
|
|
from_response_cache = True
|
|
|
|
class Barren(Outcome):
|
|
tokens_saved = 0
|
|
|
|
class Ineligible(Outcome):
|
|
attempted_input_tokens = 0
|
|
tokens_saved = 0
|
|
|
|
class Failed(Outcome):
|
|
status_code = 500
|
|
|
|
class Limited(Outcome):
|
|
status_code = 429
|
|
|
|
payload = emit(Cached(), Failed(), Limited(), Barren(), Outcome(), Ineligible())
|
|
kinds = payload["trajectory"]["kinds"]
|
|
assert kinds == "c1f1e1z1s1o1"
|
|
assert set(re.findall(r"[a-z]", kinds)) <= S._TURN_KINDS
|
|
|
|
|
|
def test_trajectory_string_is_capped():
|
|
outcomes = []
|
|
for index in range(S.MAX_TRAJECTORY_RUNS * 3):
|
|
cls = Outcome if index % 2 else type("Alt", (Outcome,), {"status_code": 500})
|
|
outcomes.append(cls())
|
|
payload = emit(*outcomes)
|
|
assert payload["trajectory"]["kinds_truncated"] is True
|
|
assert len(re.findall(r"[a-z]", payload["trajectory"]["kinds"])) == S.MAX_TRAJECTORY_RUNS
|
|
|
|
|
|
def test_clients_and_strata(beacon_on):
|
|
class Shaped(Outcome):
|
|
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
|
|
|
|
class Control(Outcome):
|
|
client = "claude-vscode"
|
|
transforms_applied = ("output_shaper:control:opus|user|s|notools",)
|
|
|
|
class Anonymous(Outcome):
|
|
client = None
|
|
|
|
payload = emit(Shaped(), Control(), Anonymous())
|
|
assert payload["clients"] == [
|
|
{"client": "claude_code", "n": 1},
|
|
{"client": "claude_vscode", "n": 1},
|
|
], "unknown client not bucketed"
|
|
strata = {(r["dim"], r["value"]): r["n"] for r in payload["strata"]}
|
|
assert strata[("arm", "treatment")] == 1 and strata[("arm", "control")] == 1
|
|
assert strata[("turn_kind", "tool_result")] == 1 and strata[("turn_kind", "user")] == 1
|
|
assert strata[("input_bucket", "l")] == 1 and strata[("input_bucket", "s")] == 1
|
|
assert strata[("tools", "tools")] == 1 and strata[("tools", "notools")] == 1
|
|
# Model family is the one stratum component that must never ship.
|
|
assert "sonnet" not in json.dumps(payload)
|
|
assert "opus" not in json.dumps(payload)
|
|
|
|
|
|
def test_client_ids_survive_slugging():
|
|
"""Every harness `classify_client` can return has to round-trip, or the
|
|
field reports "other" for the clients that matter. Asserted against the
|
|
real vocabulary rather than a value invented here — the first version of
|
|
this test used "claude_code", which is not an id the proxy ever emits, and
|
|
so passed while `claude-code` was being discarded.
|
|
"""
|
|
from headroom.proxy.auth_mode import CLIENT_UA_MAP
|
|
|
|
known = sorted({value for _, value in CLIENT_UA_MAP})
|
|
assert "claude-code" in known, "the vocabulary moved; this test needs updating"
|
|
|
|
outcomes = [type(f"C{i}", (Outcome,), {"client": name})() for i, name in enumerate(known)]
|
|
clients = {r["client"]: r["n"] for r in emit(*outcomes)["clients"]}
|
|
assert "other" not in clients, (
|
|
f"a real client id was discarded: {sorted(set(known) - set(clients))}"
|
|
)
|
|
assert len(clients) == len(known)
|
|
assert clients["claude_code"] == 1
|
|
|
|
|
|
def test_client_slug_still_rejects_junk():
|
|
payload = emit(type("Junk", (Outcome,), {"client": "../../etc/passwd"})())
|
|
assert payload["clients"] == [{"client": "other", "n": 1}]
|
|
|
|
|
|
def test_content_shape_table(beacon_on):
|
|
S.record_content_shape("search", "smart_crusher", 1000, 400)
|
|
S.record_content_shape("search", "smart_crusher", 500, 300)
|
|
S.record_content_shape("diff", "passthrough", 800, 800)
|
|
payload = emit(Outcome())
|
|
rows = {(r["content"], r["strategy"]): r for r in payload["shapes"]["by_content"]}
|
|
assert rows[("search", "smart_crusher")] == {
|
|
"content": "search",
|
|
"strategy": "smart_crusher",
|
|
"n": 2,
|
|
"tokens_in": 1500,
|
|
"tokens_out": 700,
|
|
}
|
|
assert rows[("diff", "passthrough")]["tokens_in"] == 800
|
|
assert isinstance(payload["shapes"]["by_content"], list), "a list, never a map"
|
|
|
|
|
|
def test_content_shape_cannot_open_a_session(beacon_on):
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append)
|
|
S.record_content_shape("search", "smart_crusher", 1000, 400)
|
|
agg.flush_all()
|
|
assert rows == [], "a compression event invented a session"
|
|
|
|
|
|
def test_content_shape_slugs_free_strings(beacon_on):
|
|
S.record_content_shape("../../etc/passwd", "SELECT * FROM x", 100, 50)
|
|
payload = emit(Outcome())
|
|
assert payload["shapes"]["by_content"][0]["content"] == "other"
|
|
assert payload["shapes"]["by_content"][0]["strategy"] == "other"
|
|
|
|
|
|
def test_tool_shape_is_a_descriptor_never_a_hash(beacon_on):
|
|
class Signature:
|
|
structure_hash = "deadbeefcafe0123456789ab"
|
|
field_count = 17
|
|
max_depth = 3
|
|
has_arrays = True
|
|
has_nested_objects = True
|
|
has_id_like_field = True
|
|
has_score_like_field = False
|
|
has_timestamp_like_field = True
|
|
has_status_like_field = False
|
|
has_error_like_field = False
|
|
has_message_like_field = False
|
|
|
|
S.record_tool_shape(Signature(), 5000, 1200)
|
|
payload = emit(Outcome())
|
|
assert payload["shapes"]["by_tool"] == [
|
|
{"shape": "f16d3_anit", "n": 1, "tokens_in": 5000, "tokens_out": 1200}
|
|
]
|
|
assert "deadbeef" not in json.dumps(payload), "structure_hash reached the wire"
|
|
|
|
|
|
def test_tool_shape_rejects_a_non_signature(beacon_on):
|
|
S.record_tool_shape(object(), 5000, 1200)
|
|
S.record_tool_shape(None, 5000, 1200)
|
|
payload = emit(Outcome())
|
|
assert payload["shapes"]["by_tool"] == []
|
|
|
|
|
|
def test_shape_tables_are_capped(beacon_on):
|
|
"""The cap is a guard against a caller inventing keys, not a budget — see
|
|
test_content_shape_cap_exceeds_its_vocabulary for the other side."""
|
|
for index in range(S.MAX_SHAPE_ROWS + 10):
|
|
S.record_content_shape(f"c{index}", "smart_crusher", 100, 50)
|
|
payload = emit(Outcome())
|
|
assert len(payload["shapes"]["by_content"]) == S.MAX_SHAPE_ROWS
|
|
|
|
|
|
def test_config_is_an_allowlist_and_every_value_is_slugged(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_MODE", "optimize")
|
|
monkeypatch.setenv("HEADROOM_KOMPRESS_ENDPOINT", "https://secret.acme.internal/v1")
|
|
monkeypatch.setenv("HEADROOM_PROXY_TOKEN", "sk-live-abc123")
|
|
monkeypatch.setenv("HEADROOM_SAVINGS_PROFILE", "/Users/someone/private/path")
|
|
payload = emit(Outcome())
|
|
config = {r["key"]: r["value"] for r in payload["config"]}
|
|
assert config["mode"] == "optimize"
|
|
assert config["savings_profile"] == "other", "a path shipped verbatim"
|
|
assert "kompress_endpoint" not in config
|
|
assert "proxy_token" not in config
|
|
wire = json.dumps(payload)
|
|
assert "acme" not in wire and "sk-live" not in wire and "/Users/" not in wire
|
|
|
|
|
|
def test_config_omits_unset_variables(monkeypatch):
|
|
for name in S._CONFIG_ENV_KEYS:
|
|
monkeypatch.delenv(name, raising=False)
|
|
monkeypatch.setenv("HEADROOM_DEDUPE", " ")
|
|
assert emit(Outcome())["config"] == []
|
|
|
|
|
|
def test_a_v2_payload_failure_cannot_lose_the_v1_event():
|
|
"""The v2 sections claim they cannot cost a v1 number. Built inline in one
|
|
dict literal they could: any raise aborts the literal and drops the whole
|
|
event, `tokens.saved` included."""
|
|
from unittest.mock import patch
|
|
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append)
|
|
with patch.object(S, "_config_snapshot", side_effect=RuntimeError("boom")):
|
|
agg.record(Outcome(), now=1000.0)
|
|
agg.flush_all()
|
|
assert rows, "a v2 failure dropped the entire session event"
|
|
assert rows[0]["tokens"]["saved"] == 300, "v1 counters survived intact"
|
|
assert "config" not in rows[0], "the failed v2 section is absent, not partial"
|
|
|
|
|
|
def test_a_v2_fold_failure_cannot_cost_a_v1_counter_or_a_heartbeat():
|
|
from unittest.mock import patch
|
|
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append, idle_s=900.0, flush_s=100.0)
|
|
with patch.object(S, "_fold_extra", side_effect=RuntimeError("boom")):
|
|
for index in range(4):
|
|
agg.record(Outcome(), now=7000.0 + index * 60)
|
|
agg.flush_all()
|
|
assert len(rows) >= 2, "a v2 fold failure suppressed the heartbeat"
|
|
assert rows[-1]["session"]["turns"] == 4
|
|
assert rows[-1]["tokens"]["saved"] == 1200
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# client and receiver must agree
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_allowlist_covers_every_emitted_key():
|
|
"""The worker drops any top-level key not in ALLOWED_KEYS, permanently and
|
|
silently. A client that emits a key the worker has not been taught is a
|
|
signal that looks healthy and stores nothing, so the two files have to be
|
|
checked against each other rather than by eye.
|
|
"""
|
|
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
|
|
match = re.search(r"const ALLOWED_KEYS = \[(.*?)\n\];", worker, re.S)
|
|
assert match, "ALLOWED_KEYS not found in worker.js"
|
|
without_comments = re.sub(r"//[^\n]*", "", match.group(1))
|
|
allowed = set(re.findall(r"'([a-z_]+)'", without_comments))
|
|
|
|
emitted = set(emit(Outcome()))
|
|
missing = emitted - allowed
|
|
assert not missing, f"worker.js would silently discard: {sorted(missing)}"
|
|
assert V1_TOP <= allowed, "a v1 key was removed from the allowlist"
|
|
assert V2_TOP <= allowed
|
|
|
|
|
|
def test_schema_version_was_bumped():
|
|
assert S.SCHEMA_VERSION == 2
|
|
assert emit(Outcome())["schema_version"] == 2
|
|
|
|
|
|
def test_worst_case_payload_fits_the_receivers_body_limit():
|
|
"""The client must not be able to build a body the Worker will refuse.
|
|
|
|
Not a formality. The cap is checked on what ARRIVES, so an uncompressed
|
|
upload -- the kill switch is set, or the gzip fallback fired -- is what has
|
|
to fit. And a 413 correctly does NOT trigger the uncompressed retry (see
|
|
the gzip tests), so a body over the cap is simply lost.
|
|
|
|
Read from worker.js rather than hardcoded, so raising one and not the other
|
|
fails here instead of in production.
|
|
"""
|
|
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
|
|
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
|
|
assert match, "MAX_BODY_BYTES not found in worker.js"
|
|
cap = int(match.group(1)) * 1024
|
|
|
|
# Every bounded table driven to its cap at once.
|
|
for index in range(S.MAX_SHAPE_ROWS + 20):
|
|
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
|
|
S.record_tool_shape(
|
|
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
|
|
5000,
|
|
1200,
|
|
)
|
|
for index in range(S.MAX_STRATEGIES + 5):
|
|
S.record_compression(f"strategy{index}", 9000, 3000)
|
|
|
|
outcomes = []
|
|
for index in range(1200):
|
|
outcomes.append(
|
|
type(
|
|
f"O{index}",
|
|
(Outcome,),
|
|
{
|
|
"status_code": 400 + (index % 99),
|
|
"client": f"harness_{index % 70}",
|
|
"transforms_applied": (
|
|
f"output_shaper:stratum:m|kind{index % 40}|b{index % 9}|tools",
|
|
),
|
|
"tags": {"passthrough_reason": f"reason{index % 20}"},
|
|
"waste_signals": dict.fromkeys(
|
|
(*S._WASTE_KEYS, "reread", "reread_compressed"), 9
|
|
),
|
|
},
|
|
)()
|
|
)
|
|
payload = emit(*outcomes)
|
|
body = json.dumps(
|
|
S.build_otlp_logs(payload, S.resource_attributes()), separators=(",", ":")
|
|
).encode()
|
|
assert len(body) < cap, (
|
|
f"worst-case OTLP body is {len(body):,} B against a {cap:,} B cap — the "
|
|
"Worker would 413 it and it would not be retried"
|
|
)
|
|
# Gzipped it is a rounding error, which is the normal path.
|
|
assert len(gzip.compress(body, S._GZIP_LEVEL)) < cap // 10
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# headroom telemetry --show
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_telemetry_show_prints_the_real_schema_not_a_mockup():
|
|
"""The command exists so a user can verify what leaves their machine
|
|
without reading two source files and trusting they match the installed
|
|
copy. That only holds if it renders the same payload the beacon builds —
|
|
so it is asserted against the live key set, not against a fixture."""
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main as cli
|
|
|
|
result = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
|
|
assert result.exit_code == 0, result.output
|
|
report = json.loads(result.output)
|
|
assert report["schema_version"] == S.SCHEMA_VERSION
|
|
assert V1_TOP <= set(report["session_event"])
|
|
assert V2_TOP <= set(report["session_event"])
|
|
assert set(report["session_event"]) == set(emit(Outcome())), (
|
|
"--show has drifted from the payload the beacon actually sends"
|
|
)
|
|
|
|
|
|
def test_telemetry_show_reports_the_beacon_state(monkeypatch):
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main as cli
|
|
|
|
monkeypatch.setenv("HEADROOM_BEACON", "off")
|
|
off = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
|
|
assert json.loads(off.output)["beacon_enabled"] is False
|
|
|
|
monkeypatch.setenv("HEADROOM_BEACON", "on")
|
|
monkeypatch.delenv("DO_NOT_TRACK", raising=False)
|
|
on = CliRunner().invoke(cli, ["telemetry", "--show", "--json"])
|
|
assert json.loads(on.output)["beacon_enabled"] is True
|
|
|
|
|
|
def test_telemetry_command_does_not_mint_an_install_id(tmp_path, monkeypatch):
|
|
"""Inspecting telemetry — very plausibly on the way to switching it off —
|
|
must not be the thing that gives a machine its identifier. `install_id()`
|
|
creates and persists one as a side effect, so this command reads the file
|
|
directly instead."""
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main as cli
|
|
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
|
monkeypatch.setenv("HOME", str(tmp_path))
|
|
# The id is memoised in a module global once anything has read it, so the
|
|
# cache has to be cleared or this test reads the developer's real id and
|
|
# never exercises the create path at all.
|
|
monkeypatch.setattr(S, "_install_id", None)
|
|
|
|
# BOTH paths. `--show` is the one that matters and the one that regressed:
|
|
# filtering the id out of the rendered report is not enough, because
|
|
# resource_attributes() mints it while building that report.
|
|
for argv in (["telemetry"], ["telemetry", "--show"], ["telemetry", "--json"]):
|
|
monkeypatch.setattr(S, "_install_id", None)
|
|
result = CliRunner().invoke(cli, argv)
|
|
assert result.exit_code == 0, result.output
|
|
assert not list(tmp_path.rglob("install_id")), f"{argv} created an install id"
|
|
assert "none yet" in CliRunner().invoke(cli, ["telemetry"]).output
|
|
|
|
|
|
def test_clients_map_is_capped_against_a_hostile_header():
|
|
"""`classify_client` returns the `X-Client` header verbatim when a caller
|
|
sets one, so this field is attacker-influenced. Junk already collapses to
|
|
"other"; the cap is what stops a caller minting a new valid-looking slug
|
|
per request and growing the payload without bound."""
|
|
outcomes = [
|
|
type(f"C{i}", (Outcome,), {"client": f"harness_{i}"})() for i in range(S.MAX_SHAPES + 25)
|
|
]
|
|
clients = emit(*outcomes)["clients"]
|
|
assert len(clients) == S.MAX_SHAPES
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# shape stability
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def key_shape(node: Any, path: str = "") -> set[str]:
|
|
"""Every key path in a payload, with list contents collapsed.
|
|
|
|
Numeric key segments are normalised, because a map keyed by a number is
|
|
safe: DuckDB cannot read "429" as a struct field name, so it infers MAP —
|
|
which is what the v1 `failure_statuses` column has always been across the
|
|
whole corpus. Identifier-shaped keys are the dangerous ones.
|
|
"""
|
|
shape: set[str] = set()
|
|
if isinstance(node, dict):
|
|
for key, value in node.items():
|
|
name = "<num>" if str(key).isdigit() else key
|
|
shape.add(f"{path}.{name}")
|
|
shape |= key_shape(value, f"{path}.{name}")
|
|
elif isinstance(node, list):
|
|
for item in node:
|
|
shape |= key_shape(item, f"{path}[]")
|
|
return shape
|
|
|
|
|
|
def test_payload_shape_does_not_depend_on_the_data():
|
|
"""The invariant behind every "list of records, not an object" note in
|
|
payload(): two sessions with nothing in common must produce the SAME set of
|
|
key paths.
|
|
|
|
An object keyed by something that came from the data breaks this, and the
|
|
consequence is not cosmetic — DuckDB infers such an object as a STRUCT
|
|
while its keys are few and identifier-shaped and as a MAP once they are
|
|
not, so the COLUMN TYPE becomes a function of the contents. Measured on the
|
|
real corpus: two harnesses gave STRUCT(claude_code, codex) and a query for
|
|
`clients.cursor` failed to bind; twenty-two gave MAP(VARCHAR, BIGINT) and
|
|
the same query ran. That is the break that took out the transforms report.
|
|
|
|
This test is the general guard, so a future field cannot reintroduce it.
|
|
"""
|
|
|
|
class Left(Outcome):
|
|
client = "claude-code"
|
|
status_code = 200
|
|
transforms_applied = ("output_shaper:stratum:sonnet|tool_result|l|tools",)
|
|
waste_signals = {"json_bloat": 50, "reread": 10}
|
|
|
|
class Right(Outcome):
|
|
client = "aider"
|
|
status_code = 200
|
|
transforms_applied = ("output_shaper:control:opus|new_user_ask|xs|notools",)
|
|
waste_signals = {"base64": 7, "whitespace": 3, "repetition": 11}
|
|
|
|
import os
|
|
|
|
os.environ["HEADROOM_MODE"] = "optimize"
|
|
try:
|
|
left = key_shape(emit(Left()))
|
|
finally:
|
|
os.environ.pop("HEADROOM_MODE", None)
|
|
os.environ["HEADROOM_DEDUPE"] = "on"
|
|
os.environ["HEADROOM_LOSSLESS"] = "on"
|
|
try:
|
|
right = key_shape(emit(Right()))
|
|
finally:
|
|
os.environ.pop("HEADROOM_DEDUPE", None)
|
|
os.environ.pop("HEADROOM_LOSSLESS", None)
|
|
|
|
assert left == right, (
|
|
"payload shape varies with its contents — these paths differ, which "
|
|
f"means a column type will too: {sorted(left ^ right)}"
|
|
)
|
|
|
|
|
|
def test_the_fields_that_carry_open_vocabularies_are_lists():
|
|
"""Belt and braces on the test above, naming the four that were objects."""
|
|
payload = emit(Outcome())
|
|
for field in ("clients", "config", "strata"):
|
|
assert isinstance(payload[field], list), f"{field} must be a list of records"
|
|
assert isinstance(payload["quality"]["waste"], list)
|
|
# ...and the records have fixed field names, which is what makes the type
|
|
# stable no matter how the vocabulary grows.
|
|
assert isinstance(payload["shapes"]["by_content"], list)
|
|
|
|
|
|
def test_trajectory_string_is_a_faithful_prefix_once_truncated():
|
|
"""Past the cap the string must STOP, not keep growing its last run.
|
|
|
|
Letting a matching kind keep incrementing looks harmless and is not: once a
|
|
run has been dropped the remaining turns are no longer contiguous, so a run
|
|
of `s6` would claim six consecutive `s` turns where the session actually
|
|
did `s q s q s q`. Every run in the string has to be true.
|
|
"""
|
|
runs_at_cap = []
|
|
for index in range(S.MAX_TRAJECTORY_RUNS):
|
|
runs_at_cap.append(
|
|
type(f"A{index}", (Outcome,), {"status_code": 500 if index % 2 else 200})()
|
|
)
|
|
# Then a tail that alternates between the last run's kind and another.
|
|
tail = []
|
|
for index in range(20):
|
|
tail.append(type(f"B{index}", (Outcome,), {"status_code": 200 if index % 2 else 429})())
|
|
|
|
payload = emit(*runs_at_cap, *tail)
|
|
kinds = payload["trajectory"]["kinds"]
|
|
assert payload["trajectory"]["kinds_truncated"] is True
|
|
runs = re.findall(r"([a-z])(\d+)", kinds)
|
|
assert len(runs) == S.MAX_TRAJECTORY_RUNS
|
|
# The prefix is exactly the pre-cap history: alternating runs of length 1.
|
|
assert all(int(n) == 1 for _, n in runs), f"a run grew after truncation: {kinds}"
|
|
# And the uncapped turn counter still accounts for every turn.
|
|
assert sum(payload["trajectory"]["turns"]) == payload["session"]["turns"]
|
|
assert payload["session"]["turns"] == S.MAX_TRAJECTORY_RUNS + 20
|
|
|
|
|
|
def test_content_shape_cap_exceeds_its_vocabulary(beacon_on):
|
|
"""The cap must not be able to bite in normal operation. It drops rows
|
|
first-come-wins, which would bias the table toward whatever a session
|
|
routed early — and that table is what a routing policy gets fitted on."""
|
|
from headroom.transforms.content_detector import ContentType
|
|
from headroom.transforms.content_router import CompressionStrategy
|
|
|
|
# +1 each for the "other" bucket a free string collapses to.
|
|
cross_product = (len(list(ContentType)) + 1) * (len(list(CompressionStrategy)) + 1)
|
|
assert S.MAX_SHAPE_ROWS >= cross_product, (
|
|
f"{cross_product} legitimate (content, strategy) pairs exist but the cap "
|
|
f"is {S.MAX_SHAPE_ROWS}"
|
|
)
|
|
|
|
for content in ContentType:
|
|
for strategy in CompressionStrategy:
|
|
S.record_content_shape(content.value, strategy.value, 100, 50)
|
|
rows = emit(Outcome())["shapes"]["by_content"]
|
|
assert len(rows) == len(list(ContentType)) * len(list(CompressionStrategy)), (
|
|
"a legitimate content/strategy combination was dropped"
|
|
)
|
|
|
|
|
|
def test_the_client_budget_stays_under_every_deployed_receiver():
|
|
"""`_MAX_WIRE_BYTES` is only useful while it is below what receivers accept.
|
|
|
|
The current Worker allows 256KB, but an install upgrades on its own schedule
|
|
and may be pointed at an older deployment whose cap is 64KB. Raising the
|
|
client budget past that would silently reintroduce the 413 this bounds.
|
|
"""
|
|
worker = (Path(__file__).resolve().parents[1] / "deploy/beacon/worker.js").read_text()
|
|
match = re.search(r"const MAX_BODY_BYTES = (\d+) \* 1024;", worker)
|
|
assert match, "MAX_BODY_BYTES not found in worker.js"
|
|
assert S._MAX_WIRE_BYTES <= int(match.group(1)) * 1024
|
|
assert S._MAX_WIRE_BYTES <= 64 * 1024, (
|
|
f"the client budget is {S._MAX_WIRE_BYTES:,} B, over the 64KB cap of the "
|
|
"oldest Worker still in service"
|
|
)
|
|
|
|
|
|
def test_the_worst_case_body_actually_sent_fits_the_oldest_receiver():
|
|
"""The test above bounds what can be BUILT; this bounds what is SENT.
|
|
|
|
`_fit_to_wire` is the thing standing between the two, and uncompressed v2
|
|
crosses 64KB well before the shape caps are reached.
|
|
"""
|
|
for index in range(S.MAX_SHAPE_ROWS + 20):
|
|
S.record_content_shape(f"content{index}", f"strategy{index}", 9000, 3000)
|
|
S.record_tool_shape(
|
|
type("Sig", (), {"field_count": 8, "max_depth": index % 10, "has_arrays": True})(),
|
|
5000,
|
|
1200,
|
|
)
|
|
outcomes = [
|
|
type(
|
|
f"O{index}",
|
|
(Outcome,),
|
|
{"status_code": 400 + (index % 99), "client": f"harness_{index % 70}"},
|
|
)()
|
|
for index in range(600)
|
|
]
|
|
payload = emit(*outcomes)
|
|
# The usage tables and versions cannot be shed, so fill them to their caps
|
|
# with nine-digit counters: the body must still fit.
|
|
big = dict.fromkeys(S._USAGE_FIELDS, 987_654_321)
|
|
names = sorted(S._KNOWN_CLIENTS)
|
|
payload["client_usage"] = [{"client": names[i], **big} for i in range(S.MAX_USAGE_ROWS + 1)]
|
|
payload["model_usage"] = [
|
|
{"model": f"claude-opus-4-8-2026010{i}", **big} for i in range(S.MAX_USAGE_ROWS + 1)
|
|
]
|
|
payload["client_versions"] = [
|
|
{"client": names[i % len(names)], "version": f"1234.1234.{1000 + i}", "n": 987_654_321}
|
|
for i in range(S.MAX_CLIENT_VERSIONS)
|
|
]
|
|
saved = payload["tokens"]["saved"]
|
|
sent = S._fit_to_wire(payload, S.resource_attributes(), compress=False)
|
|
assert len(sent) <= 64 * 1024, f"sent {len(sent):,} B, over the 64KB receiver cap"
|
|
assert payload["tokens"]["saved"] == saved, "trimming cost a v1 counter"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# usage by harness and model, harness versions, install facts, daily rollover
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_usage_tables_split_the_session_totals_without_residue():
|
|
"""`client_usage` and `model_usage` exist to divide a mixed session, so each
|
|
must add back up to the session: every turn lands in exactly one row,
|
|
including the turns no harness claimed."""
|
|
|
|
class Codex(Outcome):
|
|
client = "codex"
|
|
model = "gpt-4o-mini"
|
|
tags = {"tool_search_deferred_tokens": 70}
|
|
|
|
class Gateway(Outcome):
|
|
client = None
|
|
provider = "compress"
|
|
model = "ft:gpt-4o:acme-corp:internal-bot:abc123"
|
|
|
|
class Anonymous(Outcome):
|
|
client = None
|
|
|
|
payload = emit(Outcome(), Outcome(), Codex(), Gateway(), Anonymous())
|
|
by_client = {r["client"]: r for r in payload["client_usage"]}
|
|
assert {name: r["n"] for name, r in by_client.items()} == {
|
|
"claude_code": 2,
|
|
"codex": 1,
|
|
"via_compress_api": 1,
|
|
"unidentified": 1,
|
|
}
|
|
assert by_client["codex"]["tool_saved"] == 70
|
|
for table in ("client_usage", "model_usage"):
|
|
rows = payload[table]
|
|
assert sum(r["n"] for r in rows) == payload["session"]["turns"], table
|
|
for column, total in (
|
|
("original", "original"),
|
|
("saved", "saved"),
|
|
("tool_saved", "tool_saved"),
|
|
("output", "output"),
|
|
("cache_read", "cache_read"),
|
|
("cache_write", "cache_write"),
|
|
("uncached", "uncached"),
|
|
):
|
|
assert sum(r[column] for r in rows) == payload["tokens"][total], (table, column)
|
|
assert sum(r["cache_write_5m"] for r in rows) == payload["cache"]["write_5m"], table
|
|
assert sum(r["cache_write_1h"] for r in rows) == payload["cache"]["write_1h"], table
|
|
# A fine-tune id carries an org name: pooled, never sent.
|
|
assert "acme" not in json.dumps(payload)
|
|
assert {r["model"] for r in payload["model_usage"]} == {"gpt-4o", "gpt-4o-mini", "other"}
|
|
|
|
|
|
def test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does():
|
|
"""Where a provider reports no write count, Headroom infers one as input
|
|
minus cached. Those are the same tokens as `uncached` and are never billed
|
|
as a write, so a row has to say how much of its `cache_write` is inferred
|
|
or a reader double-counts them: 1000 billed tokens would price as 2000."""
|
|
|
|
class InferredWrite(Outcome):
|
|
provider = "openai"
|
|
model = "gpt-4o"
|
|
provider_input_tokens = 1000
|
|
cache_read_tokens = 0
|
|
cache_write_tokens = 1000
|
|
cache_write_5m_tokens = 0
|
|
cache_write_1h_tokens = 0
|
|
uncached_input_tokens = 1000
|
|
cache_inferred = True
|
|
|
|
class ReportedWrite(Outcome):
|
|
model = "gpt-4o-mini" # cache_write 100 = 80 at 5m + 20 at 1h, all billed
|
|
|
|
def billed_input(row: dict[str, Any]) -> int:
|
|
return (
|
|
row["uncached"] + row["cache_read"] + row["cache_write"] - row["cache_write_inferred"]
|
|
)
|
|
|
|
rows = {r["model"]: r for r in emit(InferredWrite(), ReportedWrite())["model_usage"]}
|
|
assert billed_input(rows["gpt-4o"]) == 1000
|
|
assert rows["gpt-4o"]["cache_write_inferred"] == 1000
|
|
assert billed_input(rows["gpt-4o-mini"]) == 400 + 500 + 100
|
|
assert (rows["gpt-4o-mini"]["cache_write_5m"], rows["gpt-4o-mini"]["cache_write_1h"]) == (
|
|
80,
|
|
20,
|
|
)
|
|
|
|
|
|
def test_an_mcp_turn_is_labelled_not_dropped():
|
|
payload = emit(
|
|
S._McpCompression(
|
|
original_tokens=800, attempted_input_tokens=800, optimized_tokens=200, tokens_saved=600
|
|
),
|
|
source="mcp",
|
|
)
|
|
assert [(r["client"], r["n"], r["saved"]) for r in payload["client_usage"]] == [
|
|
("via_mcp", 1, 600)
|
|
]
|
|
|
|
|
|
def test_usage_tables_pool_overflow_instead_of_losing_turns():
|
|
table: dict[str, list[int]] = {}
|
|
for index in range(S.MAX_USAGE_ROWS + 9):
|
|
S._add_usage(table, f"model{index}", (1, 10))
|
|
assert len(table) == S.MAX_USAGE_ROWS + 1
|
|
assert table["other"] == [9, 90]
|
|
assert sum(row[0] for row in table.values()) == S.MAX_USAGE_ROWS + 9
|
|
|
|
|
|
def test_a_caller_supplied_client_name_never_reaches_the_usage_tables(beacon_on):
|
|
"""`classify_client` returns the caller's `X-Client` header as typed, so the
|
|
name is checked against a closed list, not just for shape."""
|
|
|
|
class Named(Outcome):
|
|
client = "acme-corp-billing"
|
|
|
|
S.record_client_version("acme-corp-billing", "1.2.3")
|
|
payload = emit(Named(), Outcome())
|
|
assert [(r["client"], r["n"]) for r in payload["client_usage"]] == [
|
|
("claude_code", 1),
|
|
("other", 1),
|
|
]
|
|
assert payload["client_versions"] == []
|
|
|
|
|
|
def test_the_known_client_list_covers_every_harness_the_proxy_can_name():
|
|
from headroom.proxy.auth_policy import CLIENT_UA_MAP
|
|
from headroom.telemetry.context import _KNOWN_WRAP_AGENTS
|
|
|
|
named = {S._client_slug(name) for _, name in CLIENT_UA_MAP} | set(_KNOWN_WRAP_AGENTS)
|
|
assert named <= S._KNOWN_CLIENTS, sorted(named - S._KNOWN_CLIENTS)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("user_agent", "expected"),
|
|
[
|
|
("claude-cli/2.1.311 (external, cli)", ("claude-code", "2.1.311")),
|
|
("Corp-Wrapper/9 codex-cli/0.45.0-alpha.3", ("codex", "0.45.0")),
|
|
("claude-cli/dev-build", None),
|
|
("curl/8.4.0", None),
|
|
# Over-long or four-part numbers are not releases; never cut one down.
|
|
("claude-cli/20261115", None),
|
|
("claude-cli/1.2.3456789", None),
|
|
("claude-cli/10.0.3.7", None),
|
|
],
|
|
)
|
|
def test_user_agent_version_parser(user_agent, expected):
|
|
from headroom.proxy.auth_policy import client_version_from_user_agent
|
|
|
|
assert client_version_from_user_agent(user_agent) == expected
|
|
|
|
|
|
def test_only_a_numeric_release_of_a_known_harness_is_staged(beacon_on):
|
|
S.record_client_version("claude-code", "2.1.311")
|
|
S.record_client_version("claude-code", "2.1.311")
|
|
S.record_client_version("claude-code", "2.1.311-internal.example.com")
|
|
payload = emit(Outcome())
|
|
assert payload["client_versions"] == [{"client": "claude_code", "version": "2.1.311", "n": 2}]
|
|
|
|
|
|
def test_install_facts_report_age_and_ephemerality(tmp_path, monkeypatch):
|
|
id_file = tmp_path / "install_id"
|
|
id_file.write_text("a" * 32)
|
|
ten_days_ago = time.time() - 10 * 86400 - 60
|
|
os.utime(id_file, (ten_days_ago, ten_days_ago))
|
|
monkeypatch.setattr(S, "_install_id_path", lambda: id_file)
|
|
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
|
monkeypatch.setenv("CI", "true")
|
|
facts = emit(Outcome())["install"]
|
|
assert facts["age_days"] == 10 and facts["ci"] is True
|
|
assert set(facts) == {"age_days", "container", "ci"}
|
|
|
|
# Stateless: the id is random per process, so there is no age to report.
|
|
monkeypatch.setenv("HEADROOM_STATELESS", "1")
|
|
facts = emit(Outcome())["install"]
|
|
assert facts["age_days"] == 0
|
|
|
|
|
|
def test_a_long_running_session_is_closed_at_the_utc_day_boundary():
|
|
"""An always-on proxy never idles, so its session used to grow for weeks.
|
|
It now ends at the day boundary; a session that merely runs past midnight
|
|
is left whole."""
|
|
day = 86400.0
|
|
rows: list[dict[str, Any]] = []
|
|
agg = S.SessionAggregator(rows.append, idle_s=10 * day, flush_s=100 * day)
|
|
start = 20 * day + 3600.0 # 01:00 UTC
|
|
agg.record(Outcome(), now=start)
|
|
agg.record(Outcome(), now=start + 22 * 3600.0) # 23:00, same day
|
|
assert rows == []
|
|
agg.record(Outcome(), now=21 * day + 5.0) # just past midnight, 23h old
|
|
assert [(r["session"]["ended"], r["session"]["turns"]) for r in rows] == [("rollover", 2)]
|
|
agg.record(Outcome(), now=22 * day + 5.0) # the next piece closes a day later
|
|
assert [r["session"]["ended"] for r in rows] == ["rollover", "rollover"]
|
|
assert rows[0]["session"]["id"] != rows[1]["session"]["id"]
|
|
|
|
# A clock that steps backwards must not close anything.
|
|
agg.record(Outcome(), now=21 * day + 5.0)
|
|
assert len(rows) == 2
|
|
|
|
# Started at 20:00 UTC and still going at 01:00: five hours old, untouched.
|
|
rows.clear()
|
|
evening = S.SessionAggregator(rows.append, idle_s=10 * day, flush_s=100 * day)
|
|
evening.record(Outcome(), now=30 * day + 20 * 3600.0)
|
|
evening.record(Outcome(), now=31 * day + 3600.0)
|
|
assert rows == []
|
|
|
|
|
|
def test_the_proxy_reports_the_harness_release_it_was_called_by(beacon_on):
|
|
"""The wiring, end to end: a real request's User-Agent reaches the payload
|
|
as a harness and a release, and nothing else from the header does."""
|
|
from tests.test_header_isolation import _make_anthropic_app
|
|
|
|
client, _ = _make_anthropic_app()
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={
|
|
"x-api-key": "test-key",
|
|
"anthropic-version": "2023-06-01",
|
|
"user-agent": "claude-cli/2.1.311 (external, cli) host-a.internal",
|
|
},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 16,
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
},
|
|
)
|
|
assert response.status_code == 200, response.text
|
|
rows: list[dict[str, Any]] = []
|
|
S.get_session_aggregator().flush_all(emit=rows.append)
|
|
assert rows[-1]["client_versions"] == [{"client": "claude_code", "version": "2.1.311", "n": 1}]
|
|
assert rows[-1]["client_usage"][0]["client"] == "claude_code"
|
|
assert "host-a" not in json.dumps(rows[-1])
|