Files
headroom/tests/test_output_savings_cli.py
074f0ae9df fix(stats): gate the output-shaping claim on the shaper being active (#3386)
## Description

`/stats` builds its `savings.by_layer.output_shaping` entry from the
persisted output-savings ledger (`get_recorder().estimate(...)`) without
checking whether the shaper is actually running. The ledger survives
restarts and can span periods or key schemas where the arms accumulated
unshaped traffic (the pre-#3209 coarse conversation keys locked whole
projects into one arm). Result, observed in the field: a stable-channel
install with the shaper `blocked_by_channel` published a "measured"
-147.9% output reduction over 19,644 requests — a claim about shaping
that deployment never performed. A user audit quoted that line back at
us as evidence the numbers can't be trusted, and they were right to.

This PR fixes it at both ends. The /stats claim is gated: when the
shaper is disabled (rollout-blocked channel, kill switch, config), the
layer reports `active: false, method: "inactive"` with a zero claim,
while active deployments publish exactly what they did before (including
honest negatives, which #3127's holdout gating owns). And the ledger no
longer accumulates the data that made the claim bogus: a treatment
observation is recorded only when the request was actually shaped.

That second half came out of review. The gate alone fixes the install
that reported the -147.9% and leaves the next install exposed, because a
ledger entry carries no shaper state - so unshaped traffic accumulated
while blocked republishes the same number on the first request after the
channel unblocks. The stratum label is attached whenever the shaper is
*configured* on, but the shaping itself can still be a no-op, and a
treatment arm holding unshaped output is precisely what manufactures a
reduction the deployment never performed.

Same spirit as #3377: Headroom only claims what it measured on this
deployment's traffic.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- New module-level `_output_reduction_payload(shaper_active, est)` in
`headroom/proxy/server.py`; the /stats route now tracks `_shaper_active`
from `OutputShaperSettings.from_env(...).enabled` and delegates payload
construction to it.
- Inactive shaper: `{"available": true, "active": false, "method":
"inactive", "tokens_saved": 0, ...}` so consumers can render "inactive"
instead of mistaking absence for a zero measurement.
- Active shaper: unchanged payload plus an additive `active: true` key.
- `SavingsRecorder.record_from_labels` records a treatment observation
only when the shaper's own `output_shaper:verbosity:` label is present
on the same channel. That label is appended only when `shape_request`
actually changed the request (`output_shaper.py` sets `changed` and the
label together, and every handler extends labels only on `changed`), so
it is the per-request proof that shaping happened. Control is unshaped
by definition and records unconditionally.
- `SavingsLedger` persists a `shaped_only` marker; arms written before
that rule are dropped on load with a warning, since they cannot be told
apart entry by entry. The offline baseline is kept - it is learned from
pre-shaper history, costs a `learn --verbosity` run to rebuild, and was
never the poisoned part.
- Every payload branch emits the same key set, nulls where a value does
not exist (`method`, `band_is_ci`, `ci_low_percent`, `ci_high_percent`),
so a consumer reading the band cannot KeyError on the inactive or
no-data paths. Numeric totals stay 0 rather than null because callers
already sum them.
- New `tests/test_stats_output_reduction_gate.py` covering the
blocked-install shape (the real -147.9% figures), the active
passthrough, the no-data case, key-set stability across all four
branches, the unshaped-treatment drop, and the legacy-ledger arm drop.
- Existing tests that recorded treatment from a bare stratum label now
attach the shaper label (`tests/test_output_savings.py`,
`tests/test_output_savings_cli.py`) - they encoded the old contract.

## Testing

- [x] Unit tests pass (`pytest`)

### Test Output

```text
uv run --frozen --extra dev pytest tests/test_stats_output_reduction_gate.py tests/test_routing_stats_seam.py tests/test_dashboard_token_savings.py -q
3 passed; 8 passed
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.12 via uv, this branch's
worktree.
- Exact command / steps: Reproduced the field payload by calling
`_output_reduction_payload` with the exact `SavingsEstimate` a blocked
stable install published (tokens_saved=-7,522,490, pct=-147.9,
n_requests=19,644, kind="measured"), shaper_active=False, then
shaper_active=True.
- Observed result: inactive path returns `method="inactive",
tokens_saved=0, reduction_percent=0.0`; active path returns the estimate
byte-identical to today's payload plus `active: true`.
- Not tested: an end-to-end /stats call on a live rollout-blocked
install (no blocked-channel machine at hand); the desktop app renders
from its own ledger recompute since 0.8.7 and does not consume this
field for its headline.

## Runtime Rollout Safety

- Rollout-managed feature(s): Reads the output-shaper's rollout state;
changes no rollout behavior itself.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, for deployments where the shaper
is disabled: the /stats output_shaping layer reports inactive/zero
instead of the persisted ledger estimate. Deployments with the shaper
active are unchanged.
- Kill switch / disable path: Not applicable; the gate follows the
shaper's own enablement, which already has one.
- Unsafe override required: No.
- Qualification impact: Blocked-channel installs stop advertising
phantom (often negative) output savings in /stats and telemetry built on
it.
- Rollback path: Revert the commit; the payload shape addition
(`active`) is additive and the ledger format is untouched.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
2026-09-09 22:55:12 -05:00

62 lines
2.2 KiB
Python

"""Smoke tests for the output-savings CLI and the outcome→ledger wiring."""
from __future__ import annotations
import json
from click.testing import CliRunner
from headroom.cli.main import main
from headroom.proxy.output_savings import SavingsRecorder, stratum_label
# A treatment observation only counts when the request was actually shaped,
# evidenced by the shaper's own verbosity label on the same channel.
SHAPED = "output_shaper:verbosity:L2"
def test_output_savings_empty(tmp_path, monkeypatch):
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
result = CliRunner().invoke(main, ["output-savings"])
assert result.exit_code == 0
assert "No output-savings data" in result.output
def test_output_savings_reports_estimate(tmp_path, monkeypatch):
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
# Seed a baseline + treatment observations directly via the ledger.
from headroom.proxy.output_savings import SavingsLedger
ledger = SavingsLedger()
for _ in range(50):
ledger.baseline.observe("opus|new_user_ask|s|tools", 1000)
for _ in range(30):
ledger.record("treatment", "opus|new_user_ask|s|tools", 700)
ledger.save(tmp_path / "output_savings.json")
result = CliRunner().invoke(main, ["output-savings"])
assert result.exit_code == 0
assert "ESTIMATED" in result.output
assert "Reduction:" in result.output
assert "30.0%" in result.output
def test_recorder_round_trips_via_labels(tmp_path):
path = tmp_path / "savings.json"
rec = SavingsRecorder(path, flush_every=1)
# Baseline so the estimate has something to compare against.
rec._ledger.baseline.observe("opus|new_user_ask|s|tools", 1000)
labels = [
"compress:smartcrush",
stratum_label("treatment", "opus|new_user_ask|s|tools"),
SHAPED,
]
assert rec.record_from_labels(labels, output_tokens=600) is True
assert rec.record_from_labels(["no-shaper-label"], output_tokens=999) is False
est = rec.estimate()
assert est.n_requests == 1
assert est.tokens_saved == 400 # 1000 - 600
# Persisted to disk.
data = json.loads(path.read_text())
assert "treatment" in data