Files
headroom/tests/test_stats_output_reduction_gate.py
Tejas Chopra c02929846c fix(proxy): make the output shaper actually reduce output in the default mode (#3773)
## What's wrong

**`HEADROOM_OUTPUT_SHAPER=1` reduced nothing.**

`HEADROOM_MODE` defaults to `cache` (`server.py:5707`) and the `coding`
profile pins it. In cache mode `resolve_verbosity_level` forced the
level to `0` unconditionally, deliberately outranking even an explicit
`HEADROOM_VERBOSITY_LEVEL`:

```python
if not settings.steering_enabled:
    # It deliberately outranks the manual override below: a level set in the
    # environment must not reintroduce a prefix mutation the mode exists
    # to prevent.
    return 0, "cache_mode"
```

So for effectively every deployment, the shaper turned on, labelled its
requests, and left the body untouched. Both documented knobs were
silently ignored, with no warning and no override.

## Why the ban was too broad

It guards the right hazard with the wrong instrument.

What costs a prefix cache is the steering block **changing** — appearing
on a conversation already in flight, or moving between levels — because
it lives in the `system` array, inside the cached prefix of every
message-level `cache_control` breakpoint a harness sets.

What costs nothing is the block **being there**. `shape_request` applies
it on every request with no turn-kind gating, and the text is
byte-stable per level and idempotent via the `<headroom_output_shaping>`
sentinel. A level fixed at startup lands in turn 1's prefix and is
byte-identical on every turn after.

The file already said this and then contradicted itself: the module
docstring claims the block is appended "after any `cache_control`
breakpoint, so the provider prefix cache is preserved", while
`steering_allowed_for` says steering "writes into that key".

## Measured

Anthropic, `claude-sonnet-4-5`, 9.1k-token cached system prompt, four
sequential turns, `cache_control` on both the system block and the
trailing message. Arms isolated by a nonce — without that the second arm
reads the first's cache and both cold-start numbers are meaningless.

| turn | control write / read | L2 write / read |
|---|---|---|
| 1 | 9124 / 0 | 9206 / 0 |
| 2 | 100 / 9124 | 157 / 9206 |
| 3 | 126 / 9224 | 137 / 9363 |
| 4 | 161 / 9350 | 132 / 9500 |

**Identical caching pattern. No bust on any turn.** The whole difference
is the block's own ~82 tokens riding in the cached prefix — about
$0.0003 once and $0.000025 per turn at Sonnet rates.

Against that, a separate paired A/B on a long explain-with-code turn (3
pairs, non-overlapping ranges): **2375 → 1663 output tokens, −30%**, ≈
$0.011/turn. Roughly 450x the cache cost.

## The change

`resolve_verbosity_level` allows through the two sources that **cannot
move**:

| source | when |
|---|---|
| `env_pinned` | `HEADROOM_VERBOSITY_LEVEL` set explicitly |
| `cache_mode_default` | otherwise — the settings level, fixed at
startup |

The second is what makes `HEADROOM_OUTPUT_SHAPER=1` sufficient alone.
The shaper is opt-in (the `proxy_output_shaper` rollout spec
deliberately does not default-enable it), so an enabled shaper is
already an explicit request.

The controller and learned profile stay out of cache mode entirely —
they rewrite their files while conversations are open. Skipping the
lookup also spares the default mode two filesystem stats per request.
Autotune being ignored is warned once, naming `HEADROOM_MODE=token`.

**Default level 3 → 2**, named `DEFAULT_VERBOSITY_LEVEL`. L3 instructs
the model to drop content ("give conclusions only; omit rationale") —
the same reasoning the rollout spec gives for not defaulting the shaper
on at all. L2 only forbids ceremony and restatement, and is what the
−30% above was measured at. This also settles an existing inconsistency:
`from_env` defaulted to 3 but fell back to **2** on a malformed value.

## ⚠️ Behaviour changes

Three, all disclosed:

1. **Cache mode with the shaper on now steers.** That is the point.
Blast radius is exactly "operators who set `HEADROOM_OUTPUT_SHAPER=1`" —
the people who asked for output shaping and were not getting it.
2. **A pinned `HEADROOM_VERBOSITY_LEVEL` is now honoured in cache
mode**, where it was previously discarded.
3. **The default level drops 3 → 2**, including in token mode. Existing
token-mode users with the shaper on and no level set get *less*
aggressive shaping: lower savings, lower risk of the terseness reading
as regression.

Five existing tests encoded the old policy and are inverted here
deliberately, including the gateway-contract one
(`test_cache_mode_labels_but_never_steers` →
`test_cache_mode_steers_at_the_default_level`), which verifies the
change end-to-end through the real `/v1/compress` path rather than only
at the unit level.

`/stats` needed no change: it builds the same settings the handlers do
and reads the resolved level, so it follows automatically.

## Tests

New coverage:

- `HEADROOM_OUTPUT_SHAPER=1` alone resolves `(2, "cache_mode_default")`
- a pinned level resolves `(4, "env_pinned")` in cache mode
- the learned profile and the controller are both ignored there
- the ignored-autotune warning fires exactly once and names
`HEADROOM_MODE=token`
- cache mode consults **no workspace files** (asserted by making
`workspace_dir()` raise)
- **ten turns of a growing conversation produce a byte-identical
`system` array**, with the client's own `cache_control` still on the
first block — the cache-safety claim asserted rather than argued

340 tests pass across shaper / verbosity / steering / output_reduction /
cache_stability / output_savings / gateway_turn. `ruff check`, `ruff
format --check` (0.16.7) and `mypy headroom --ignore-missing-imports`
clean.

Local full-suite runs error en masse in a way unrelated to this change
(files that error in the full run pass individually), so CI is the
arbiter on the complete suite.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-09-24 14:18:17 +02:00

240 lines
9.6 KiB
Python

"""/stats must not publish output-shaping claims a deployment never performed.
The output-savings ledger persists across restarts, and its arms can hold
unshaped traffic from periods when the shaper was rollout-blocked or from
pre-fix conversation-key schemas. A stable install with the shaper
blocked_by_channel published a "measured" -147.9% output reduction straight
from that ledger — a claim about shaping it never did. The /stats seam now
gates the claim on the shaper being active; the ledger itself is untouched.
"""
from __future__ import annotations
from headroom.proxy.output_savings import SavingsEstimate, SavingsLedger, SavingsRecorder
from headroom.proxy.server import _output_reduction_payload
def _poisoned_measured() -> SavingsEstimate:
# The audited real-world shape: blocked stable channel, lopsided legacy
# arms, "measured" -147.9% over 19,644 requests.
return SavingsEstimate(
tokens_saved=-7_522_490,
baseline_tokens=5_086_874,
pct=-147.9,
ci_low_pct=-160.2,
ci_high_pct=-135.6,
n_requests=19_644,
kind="measured",
)
def test_inactive_shaper_claims_zero_regardless_of_ledger():
payload = _output_reduction_payload(False, _poisoned_measured())
assert payload["active"] is False
assert payload["method"] == "inactive"
assert payload["tokens_saved"] == 0
assert payload["reduction_percent"] == 0.0
assert payload["requests"] == 0
# Still an available, well-formed layer entry, so consumers can render
# "inactive" instead of mistaking absence for zero measurement.
assert payload["available"] is True
def test_active_shaper_publishes_the_estimate_as_before():
payload = _output_reduction_payload(True, _poisoned_measured())
assert payload["active"] is True
assert payload["available"] is True
assert payload["method"] == "measured"
assert payload["band_is_ci"] is True
assert payload["tokens_saved"] == -7_522_490
assert payload["reduction_percent"] == -147.9
assert payload["requests"] == 19_644
def test_active_shaper_without_data_stays_unavailable():
empty = SavingsEstimate(
tokens_saved=0.0,
baseline_tokens=0.0,
pct=0.0,
ci_low_pct=0.0,
ci_high_pct=0.0,
n_requests=0,
kind="estimated",
)
payload = _output_reduction_payload(True, empty)
assert payload["available"] is False
assert payload["active"] is True
assert payload["method"] is None
def test_every_branch_emits_the_same_key_set():
"""A consumer reading ci_low_percent or band_is_ci must not KeyError on the
inactive or no-data paths — the desktop app and dashboard both read these."""
no_data = SavingsEstimate(
tokens_saved=0.0,
baseline_tokens=0.0,
pct=0.0,
ci_low_pct=0.0,
ci_high_pct=0.0,
n_requests=0,
kind="estimated",
)
branches = [
_output_reduction_payload(False, _poisoned_measured()),
_output_reduction_payload(True, no_data),
_output_reduction_payload(True, _poisoned_measured()),
_output_reduction_payload(False),
]
expected = set(branches[-1])
for payload in branches:
assert set(payload) == expected
# Numeric totals stay numeric on every branch; only the band is nullable.
assert isinstance(payload["tokens_saved"], int | float)
assert isinstance(payload["baseline_tokens"], int | float)
assert isinstance(payload["reduction_percent"], int | float)
assert isinstance(payload["requests"], int)
def test_unshaped_treatment_request_is_not_recorded(tmp_path):
"""The gate above fixes the install that reported it; this fixes the next
one. The stratum label is attached whenever the shaper is configured on,
but shaping can still be a no-op — and a treatment arm holding unshaped
output republishes the same bogus reduction the moment /stats unblocks."""
recorder = SavingsRecorder(path=tmp_path / "ledger.json", flush_every=1)
assert not recorder.record_from_labels(("output_shaper:stratum:opus|chat|m|tools",), 500)
assert recorder._ledger.treatment == {}
assert recorder.record_from_labels(
("output_shaper:stratum:opus|chat|m|tools", "output_shaper:verbosity:L2"), 500
)
assert recorder._ledger.treatment["opus|chat|m|tools"].n == 1
# Control is unshaped by definition and keeps recording unconditionally.
assert recorder.record_from_labels(("output_shaper:control:opus|chat|m|tools",), 900)
assert recorder._ledger.control["opus|chat|m|tools"].n == 1
def test_ledger_written_before_the_rule_drops_its_arms_but_keeps_the_baseline():
legacy = {
"baseline": {"strata": {"opus|chat|m|tools": {"mean": 800.0, "var": 100.0, "n": 40}}},
"treatment": {"opus|chat|m|tools": {"n": 19_644, "sum": 5_000_000.0, "sumsq": 1e12}},
"control": {"opus|chat|m|tools": {"n": 12, "sum": 4_000.0, "sumsq": 1e6}},
}
ledger = SavingsLedger.from_dict(legacy)
assert ledger.treatment == {}
assert ledger.control == {}
# The offline baseline costs a `learn --verbosity` run to rebuild and was
# never the poisoned part.
assert ledger.baseline.lookup("opus|chat|m|tools")[2] == 40
# Round-tripping now carries the marker, so it is kept next time.
assert SavingsLedger.from_dict(ledger.to_dict()).baseline.lookup("opus|chat|m|tools")[2] == 40
# --- endpoint-level: the boolean /stats actually passes ----------------------
#
# The payload helper above is only half the gate. The other half is which
# boolean the /stats route hands it, and that is derived from the shaper's
# EFFECTIVE steering configuration — the same one the request handlers build.
# An enabled shaper whose level resolves to 0 (cache mode, an explicit
# override, a learned or controller zero) shapes nothing, so it must not
# publish the persisted ledger either. Testing the helper alone cannot catch a
# wrong boolean at the call site.
def _live_estimate() -> SavingsEstimate:
return SavingsEstimate(
tokens_saved=200.0,
baseline_tokens=1000.0,
pct=20.0,
ci_low_pct=15.0,
ci_high_pct=25.0,
n_requests=5,
kind="measured",
)
def _stats_output_reduction(tmp_path, monkeypatch, *, mode: str = "token"):
"""Hit the real /stats route with a ledger that already holds an estimate."""
from fastapi.testclient import TestClient
import headroom.proxy.output_savings as output_savings
from headroom.proxy.server import ProxyConfig, create_app
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(tmp_path / "proxy_savings.json"))
monkeypatch.setenv("HEADROOM_OUTPUT_SHAPER", "1")
class _Recorder:
def estimate(self, level=None):
return _live_estimate()
monkeypatch.setattr(output_savings, "get_recorder", lambda: _Recorder())
config = ProxyConfig(
mode=mode,
cache_enabled=False,
rate_limit_enabled=False,
log_requests=False,
)
with TestClient(create_app(config)) as client:
response = client.get("/stats")
assert response.status_code == 200
return response.json()["tokens"]["output_reduction"]
def test_stats_publishes_the_estimate_while_steering_is_live(tmp_path, monkeypatch):
monkeypatch.setenv("HEADROOM_VERBOSITY_LEVEL", "3")
payload = _stats_output_reduction(tmp_path, monkeypatch)
assert payload["active"] is True
assert payload["method"] == "measured"
assert payload["tokens_saved"] == 200
assert payload["reduction_percent"] == 20.0
def test_stats_gates_an_explicit_verbosity_zero(tmp_path, monkeypatch):
monkeypatch.setenv("HEADROOM_VERBOSITY_LEVEL", "0")
payload = _stats_output_reduction(tmp_path, monkeypatch)
assert payload["active"] is False
assert payload["method"] == "inactive"
assert payload["tokens_saved"] == 0
assert payload["reduction_percent"] == 0.0
def test_stats_gates_a_learned_verbosity_zero(tmp_path, monkeypatch):
monkeypatch.delenv("HEADROOM_VERBOSITY_LEVEL", raising=False)
(tmp_path / "verbosity.json").write_text('{"verbosity_level": 0}')
payload = _stats_output_reduction(tmp_path, monkeypatch)
assert payload["active"] is False
assert payload["method"] == "inactive"
def test_stats_gates_a_controller_verbosity_zero(tmp_path, monkeypatch):
monkeypatch.delenv("HEADROOM_VERBOSITY_LEVEL", raising=False)
monkeypatch.setenv("HEADROOM_VERBOSITY_AUTOTUNE", "1")
(tmp_path / "verbosity_controller.json").write_text('{"level": 0}')
payload = _stats_output_reduction(tmp_path, monkeypatch)
assert payload["active"] is False
assert payload["method"] == "inactive"
def test_stats_publishes_in_cache_mode_when_the_level_is_pinned(tmp_path, monkeypatch):
"""A pinned level runs in cache mode, so /stats must report it as live.
/stats builds the SAME settings the handlers do; when they steer, it
publishes. The inverse -- publishing a ledger no live steering produced --
is what the sibling tests below guard."""
monkeypatch.setenv("HEADROOM_VERBOSITY_LEVEL", "3")
payload = _stats_output_reduction(tmp_path, monkeypatch, mode="cache")
assert payload["active"] is True
assert payload["method"] == "measured"
def test_stats_publishes_in_cache_mode_with_the_shaper_alone(tmp_path, monkeypatch):
"""An enabled shaper steers at the default level in cache mode, so /stats
must report it live -- with no HEADROOM_VERBOSITY_LEVEL set at all."""
monkeypatch.delenv("HEADROOM_VERBOSITY_LEVEL", raising=False)
payload = _stats_output_reduction(tmp_path, monkeypatch, mode="cache")
assert payload["active"] is True
assert payload["method"] == "measured"