mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## What's wrong
**`HEADROOM_OUTPUT_SHAPER=1` reduced nothing.**
`HEADROOM_MODE` defaults to `cache` (`server.py:5707`) and the `coding`
profile pins it. In cache mode `resolve_verbosity_level` forced the
level to `0` unconditionally, deliberately outranking even an explicit
`HEADROOM_VERBOSITY_LEVEL`:
```python
if not settings.steering_enabled:
# It deliberately outranks the manual override below: a level set in the
# environment must not reintroduce a prefix mutation the mode exists
# to prevent.
return 0, "cache_mode"
```
So for effectively every deployment, the shaper turned on, labelled its
requests, and left the body untouched. Both documented knobs were
silently ignored, with no warning and no override.
## Why the ban was too broad
It guards the right hazard with the wrong instrument.
What costs a prefix cache is the steering block **changing** — appearing
on a conversation already in flight, or moving between levels — because
it lives in the `system` array, inside the cached prefix of every
message-level `cache_control` breakpoint a harness sets.
What costs nothing is the block **being there**. `shape_request` applies
it on every request with no turn-kind gating, and the text is
byte-stable per level and idempotent via the `<headroom_output_shaping>`
sentinel. A level fixed at startup lands in turn 1's prefix and is
byte-identical on every turn after.
The file already said this and then contradicted itself: the module
docstring claims the block is appended "after any `cache_control`
breakpoint, so the provider prefix cache is preserved", while
`steering_allowed_for` says steering "writes into that key".
## Measured
Anthropic, `claude-sonnet-4-5`, 9.1k-token cached system prompt, four
sequential turns, `cache_control` on both the system block and the
trailing message. Arms isolated by a nonce — without that the second arm
reads the first's cache and both cold-start numbers are meaningless.
| turn | control write / read | L2 write / read |
|---|---|---|
| 1 | 9124 / 0 | 9206 / 0 |
| 2 | 100 / 9124 | 157 / 9206 |
| 3 | 126 / 9224 | 137 / 9363 |
| 4 | 161 / 9350 | 132 / 9500 |
**Identical caching pattern. No bust on any turn.** The whole difference
is the block's own ~82 tokens riding in the cached prefix — about
$0.0003 once and $0.000025 per turn at Sonnet rates.
Against that, a separate paired A/B on a long explain-with-code turn (3
pairs, non-overlapping ranges): **2375 → 1663 output tokens, −30%**, ≈
$0.011/turn. Roughly 450x the cache cost.
## The change
`resolve_verbosity_level` allows through the two sources that **cannot
move**:
| source | when |
|---|---|
| `env_pinned` | `HEADROOM_VERBOSITY_LEVEL` set explicitly |
| `cache_mode_default` | otherwise — the settings level, fixed at
startup |
The second is what makes `HEADROOM_OUTPUT_SHAPER=1` sufficient alone.
The shaper is opt-in (the `proxy_output_shaper` rollout spec
deliberately does not default-enable it), so an enabled shaper is
already an explicit request.
The controller and learned profile stay out of cache mode entirely —
they rewrite their files while conversations are open. Skipping the
lookup also spares the default mode two filesystem stats per request.
Autotune being ignored is warned once, naming `HEADROOM_MODE=token`.
**Default level 3 → 2**, named `DEFAULT_VERBOSITY_LEVEL`. L3 instructs
the model to drop content ("give conclusions only; omit rationale") —
the same reasoning the rollout spec gives for not defaulting the shaper
on at all. L2 only forbids ceremony and restatement, and is what the
−30% above was measured at. This also settles an existing inconsistency:
`from_env` defaulted to 3 but fell back to **2** on a malformed value.
## ⚠️ Behaviour changes
Three, all disclosed:
1. **Cache mode with the shaper on now steers.** That is the point.
Blast radius is exactly "operators who set `HEADROOM_OUTPUT_SHAPER=1`" —
the people who asked for output shaping and were not getting it.
2. **A pinned `HEADROOM_VERBOSITY_LEVEL` is now honoured in cache
mode**, where it was previously discarded.
3. **The default level drops 3 → 2**, including in token mode. Existing
token-mode users with the shaper on and no level set get *less*
aggressive shaping: lower savings, lower risk of the terseness reading
as regression.
Five existing tests encoded the old policy and are inverted here
deliberately, including the gateway-contract one
(`test_cache_mode_labels_but_never_steers` →
`test_cache_mode_steers_at_the_default_level`), which verifies the
change end-to-end through the real `/v1/compress` path rather than only
at the unit level.
`/stats` needed no change: it builds the same settings the handlers do
and reads the resolved level, so it follows automatically.
## Tests
New coverage:
- `HEADROOM_OUTPUT_SHAPER=1` alone resolves `(2, "cache_mode_default")`
- a pinned level resolves `(4, "env_pinned")` in cache mode
- the learned profile and the controller are both ignored there
- the ignored-autotune warning fires exactly once and names
`HEADROOM_MODE=token`
- cache mode consults **no workspace files** (asserted by making
`workspace_dir()` raise)
- **ten turns of a growing conversation produce a byte-identical
`system` array**, with the client's own `cache_control` still on the
first block — the cache-safety claim asserted rather than argued
340 tests pass across shaper / verbosity / steering / output_reduction /
cache_stability / output_savings / gateway_turn. `ruff check`, `ruff
format --check` (0.16.7) and `mypy headroom --ignore-missing-imports`
clean.
Local full-suite runs error en masse in a way unrelated to this change
(files that error in the full run pass individually), so CI is the
arbiter on the complete suite.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
240 lines
9.6 KiB
Python
240 lines
9.6 KiB
Python
"""/stats must not publish output-shaping claims a deployment never performed.
|
|
|
|
The output-savings ledger persists across restarts, and its arms can hold
|
|
unshaped traffic from periods when the shaper was rollout-blocked or from
|
|
pre-fix conversation-key schemas. A stable install with the shaper
|
|
blocked_by_channel published a "measured" -147.9% output reduction straight
|
|
from that ledger — a claim about shaping it never did. The /stats seam now
|
|
gates the claim on the shaper being active; the ledger itself is untouched.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from headroom.proxy.output_savings import SavingsEstimate, SavingsLedger, SavingsRecorder
|
|
from headroom.proxy.server import _output_reduction_payload
|
|
|
|
|
|
def _poisoned_measured() -> SavingsEstimate:
|
|
# The audited real-world shape: blocked stable channel, lopsided legacy
|
|
# arms, "measured" -147.9% over 19,644 requests.
|
|
return SavingsEstimate(
|
|
tokens_saved=-7_522_490,
|
|
baseline_tokens=5_086_874,
|
|
pct=-147.9,
|
|
ci_low_pct=-160.2,
|
|
ci_high_pct=-135.6,
|
|
n_requests=19_644,
|
|
kind="measured",
|
|
)
|
|
|
|
|
|
def test_inactive_shaper_claims_zero_regardless_of_ledger():
|
|
payload = _output_reduction_payload(False, _poisoned_measured())
|
|
assert payload["active"] is False
|
|
assert payload["method"] == "inactive"
|
|
assert payload["tokens_saved"] == 0
|
|
assert payload["reduction_percent"] == 0.0
|
|
assert payload["requests"] == 0
|
|
# Still an available, well-formed layer entry, so consumers can render
|
|
# "inactive" instead of mistaking absence for zero measurement.
|
|
assert payload["available"] is True
|
|
|
|
|
|
def test_active_shaper_publishes_the_estimate_as_before():
|
|
payload = _output_reduction_payload(True, _poisoned_measured())
|
|
assert payload["active"] is True
|
|
assert payload["available"] is True
|
|
assert payload["method"] == "measured"
|
|
assert payload["band_is_ci"] is True
|
|
assert payload["tokens_saved"] == -7_522_490
|
|
assert payload["reduction_percent"] == -147.9
|
|
assert payload["requests"] == 19_644
|
|
|
|
|
|
def test_active_shaper_without_data_stays_unavailable():
|
|
empty = SavingsEstimate(
|
|
tokens_saved=0.0,
|
|
baseline_tokens=0.0,
|
|
pct=0.0,
|
|
ci_low_pct=0.0,
|
|
ci_high_pct=0.0,
|
|
n_requests=0,
|
|
kind="estimated",
|
|
)
|
|
payload = _output_reduction_payload(True, empty)
|
|
assert payload["available"] is False
|
|
assert payload["active"] is True
|
|
assert payload["method"] is None
|
|
|
|
|
|
def test_every_branch_emits_the_same_key_set():
|
|
"""A consumer reading ci_low_percent or band_is_ci must not KeyError on the
|
|
inactive or no-data paths — the desktop app and dashboard both read these."""
|
|
no_data = SavingsEstimate(
|
|
tokens_saved=0.0,
|
|
baseline_tokens=0.0,
|
|
pct=0.0,
|
|
ci_low_pct=0.0,
|
|
ci_high_pct=0.0,
|
|
n_requests=0,
|
|
kind="estimated",
|
|
)
|
|
branches = [
|
|
_output_reduction_payload(False, _poisoned_measured()),
|
|
_output_reduction_payload(True, no_data),
|
|
_output_reduction_payload(True, _poisoned_measured()),
|
|
_output_reduction_payload(False),
|
|
]
|
|
expected = set(branches[-1])
|
|
for payload in branches:
|
|
assert set(payload) == expected
|
|
# Numeric totals stay numeric on every branch; only the band is nullable.
|
|
assert isinstance(payload["tokens_saved"], int | float)
|
|
assert isinstance(payload["baseline_tokens"], int | float)
|
|
assert isinstance(payload["reduction_percent"], int | float)
|
|
assert isinstance(payload["requests"], int)
|
|
|
|
|
|
def test_unshaped_treatment_request_is_not_recorded(tmp_path):
|
|
"""The gate above fixes the install that reported it; this fixes the next
|
|
one. The stratum label is attached whenever the shaper is configured on,
|
|
but shaping can still be a no-op — and a treatment arm holding unshaped
|
|
output republishes the same bogus reduction the moment /stats unblocks."""
|
|
recorder = SavingsRecorder(path=tmp_path / "ledger.json", flush_every=1)
|
|
|
|
assert not recorder.record_from_labels(("output_shaper:stratum:opus|chat|m|tools",), 500)
|
|
assert recorder._ledger.treatment == {}
|
|
|
|
assert recorder.record_from_labels(
|
|
("output_shaper:stratum:opus|chat|m|tools", "output_shaper:verbosity:L2"), 500
|
|
)
|
|
assert recorder._ledger.treatment["opus|chat|m|tools"].n == 1
|
|
|
|
# Control is unshaped by definition and keeps recording unconditionally.
|
|
assert recorder.record_from_labels(("output_shaper:control:opus|chat|m|tools",), 900)
|
|
assert recorder._ledger.control["opus|chat|m|tools"].n == 1
|
|
|
|
|
|
def test_ledger_written_before_the_rule_drops_its_arms_but_keeps_the_baseline():
|
|
legacy = {
|
|
"baseline": {"strata": {"opus|chat|m|tools": {"mean": 800.0, "var": 100.0, "n": 40}}},
|
|
"treatment": {"opus|chat|m|tools": {"n": 19_644, "sum": 5_000_000.0, "sumsq": 1e12}},
|
|
"control": {"opus|chat|m|tools": {"n": 12, "sum": 4_000.0, "sumsq": 1e6}},
|
|
}
|
|
ledger = SavingsLedger.from_dict(legacy)
|
|
assert ledger.treatment == {}
|
|
assert ledger.control == {}
|
|
# The offline baseline costs a `learn --verbosity` run to rebuild and was
|
|
# never the poisoned part.
|
|
assert ledger.baseline.lookup("opus|chat|m|tools")[2] == 40
|
|
# Round-tripping now carries the marker, so it is kept next time.
|
|
assert SavingsLedger.from_dict(ledger.to_dict()).baseline.lookup("opus|chat|m|tools")[2] == 40
|
|
|
|
|
|
# --- endpoint-level: the boolean /stats actually passes ----------------------
|
|
#
|
|
# The payload helper above is only half the gate. The other half is which
|
|
# boolean the /stats route hands it, and that is derived from the shaper's
|
|
# EFFECTIVE steering configuration — the same one the request handlers build.
|
|
# An enabled shaper whose level resolves to 0 (cache mode, an explicit
|
|
# override, a learned or controller zero) shapes nothing, so it must not
|
|
# publish the persisted ledger either. Testing the helper alone cannot catch a
|
|
# wrong boolean at the call site.
|
|
|
|
|
|
def _live_estimate() -> SavingsEstimate:
|
|
return SavingsEstimate(
|
|
tokens_saved=200.0,
|
|
baseline_tokens=1000.0,
|
|
pct=20.0,
|
|
ci_low_pct=15.0,
|
|
ci_high_pct=25.0,
|
|
n_requests=5,
|
|
kind="measured",
|
|
)
|
|
|
|
|
|
def _stats_output_reduction(tmp_path, monkeypatch, *, mode: str = "token"):
|
|
"""Hit the real /stats route with a ledger that already holds an estimate."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
import headroom.proxy.output_savings as output_savings
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
|
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "proxy_savings.json"))
|
|
monkeypatch.setenv("HEADROOM_OUTPUT_SHAPER", "1")
|
|
|
|
class _Recorder:
|
|
def estimate(self, level=None):
|
|
return _live_estimate()
|
|
|
|
monkeypatch.setattr(output_savings, "get_recorder", lambda: _Recorder())
|
|
|
|
config = ProxyConfig(
|
|
mode=mode,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
log_requests=False,
|
|
)
|
|
with TestClient(create_app(config)) as client:
|
|
response = client.get("/stats")
|
|
assert response.status_code == 200
|
|
return response.json()["tokens"]["output_reduction"]
|
|
|
|
|
|
def test_stats_publishes_the_estimate_while_steering_is_live(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_VERBOSITY_LEVEL", "3")
|
|
payload = _stats_output_reduction(tmp_path, monkeypatch)
|
|
assert payload["active"] is True
|
|
assert payload["method"] == "measured"
|
|
assert payload["tokens_saved"] == 200
|
|
assert payload["reduction_percent"] == 20.0
|
|
|
|
|
|
def test_stats_gates_an_explicit_verbosity_zero(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_VERBOSITY_LEVEL", "0")
|
|
payload = _stats_output_reduction(tmp_path, monkeypatch)
|
|
assert payload["active"] is False
|
|
assert payload["method"] == "inactive"
|
|
assert payload["tokens_saved"] == 0
|
|
assert payload["reduction_percent"] == 0.0
|
|
|
|
|
|
def test_stats_gates_a_learned_verbosity_zero(tmp_path, monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_VERBOSITY_LEVEL", raising=False)
|
|
(tmp_path / "verbosity.json").write_text('{"verbosity_level": 0}')
|
|
payload = _stats_output_reduction(tmp_path, monkeypatch)
|
|
assert payload["active"] is False
|
|
assert payload["method"] == "inactive"
|
|
|
|
|
|
def test_stats_gates_a_controller_verbosity_zero(tmp_path, monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_VERBOSITY_LEVEL", raising=False)
|
|
monkeypatch.setenv("HEADROOM_VERBOSITY_AUTOTUNE", "1")
|
|
(tmp_path / "verbosity_controller.json").write_text('{"level": 0}')
|
|
payload = _stats_output_reduction(tmp_path, monkeypatch)
|
|
assert payload["active"] is False
|
|
assert payload["method"] == "inactive"
|
|
|
|
|
|
def test_stats_publishes_in_cache_mode_when_the_level_is_pinned(tmp_path, monkeypatch):
|
|
"""A pinned level runs in cache mode, so /stats must report it as live.
|
|
|
|
/stats builds the SAME settings the handlers do; when they steer, it
|
|
publishes. The inverse -- publishing a ledger no live steering produced --
|
|
is what the sibling tests below guard."""
|
|
monkeypatch.setenv("HEADROOM_VERBOSITY_LEVEL", "3")
|
|
payload = _stats_output_reduction(tmp_path, monkeypatch, mode="cache")
|
|
assert payload["active"] is True
|
|
assert payload["method"] == "measured"
|
|
|
|
|
|
def test_stats_publishes_in_cache_mode_with_the_shaper_alone(tmp_path, monkeypatch):
|
|
"""An enabled shaper steers at the default level in cache mode, so /stats
|
|
must report it live -- with no HEADROOM_VERBOSITY_LEVEL set at all."""
|
|
monkeypatch.delenv("HEADROOM_VERBOSITY_LEVEL", raising=False)
|
|
payload = _stats_output_reduction(tmp_path, monkeypatch, mode="cache")
|
|
assert payload["active"] is True
|
|
assert payload["method"] == "measured"
|