mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:07:29 +08:00
## Description `/stats` builds its `savings.by_layer.output_shaping` entry from the persisted output-savings ledger (`get_recorder().estimate(...)`) without checking whether the shaper is actually running. The ledger survives restarts and can span periods or key schemas where the arms accumulated unshaped traffic (the pre-#3209 coarse conversation keys locked whole projects into one arm). Result, observed in the field: a stable-channel install with the shaper `blocked_by_channel` published a "measured" -147.9% output reduction over 19,644 requests — a claim about shaping that deployment never performed. A user audit quoted that line back at us as evidence the numbers can't be trusted, and they were right to. This PR fixes it at both ends. The /stats claim is gated: when the shaper is disabled (rollout-blocked channel, kill switch, config), the layer reports `active: false, method: "inactive"` with a zero claim, while active deployments publish exactly what they did before (including honest negatives, which #3127's holdout gating owns). And the ledger no longer accumulates the data that made the claim bogus: a treatment observation is recorded only when the request was actually shaped. That second half came out of review. The gate alone fixes the install that reported the -147.9% and leaves the next install exposed, because a ledger entry carries no shaper state - so unshaped traffic accumulated while blocked republishes the same number on the first request after the channel unblocks. The stratum label is attached whenever the shaper is *configured* on, but the shaping itself can still be a no-op, and a treatment arm holding unshaped output is precisely what manufactures a reduction the deployment never performed. Same spirit as #3377: Headroom only claims what it measured on this deployment's traffic. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - New module-level `_output_reduction_payload(shaper_active, est)` in `headroom/proxy/server.py`; the /stats route now tracks `_shaper_active` from `OutputShaperSettings.from_env(...).enabled` and delegates payload construction to it. - Inactive shaper: `{"available": true, "active": false, "method": "inactive", "tokens_saved": 0, ...}` so consumers can render "inactive" instead of mistaking absence for a zero measurement. - Active shaper: unchanged payload plus an additive `active: true` key. - `SavingsRecorder.record_from_labels` records a treatment observation only when the shaper's own `output_shaper:verbosity:` label is present on the same channel. That label is appended only when `shape_request` actually changed the request (`output_shaper.py` sets `changed` and the label together, and every handler extends labels only on `changed`), so it is the per-request proof that shaping happened. Control is unshaped by definition and records unconditionally. - `SavingsLedger` persists a `shaped_only` marker; arms written before that rule are dropped on load with a warning, since they cannot be told apart entry by entry. The offline baseline is kept - it is learned from pre-shaper history, costs a `learn --verbosity` run to rebuild, and was never the poisoned part. - Every payload branch emits the same key set, nulls where a value does not exist (`method`, `band_is_ci`, `ci_low_percent`, `ci_high_percent`), so a consumer reading the band cannot KeyError on the inactive or no-data paths. Numeric totals stay 0 rather than null because callers already sum them. - New `tests/test_stats_output_reduction_gate.py` covering the blocked-install shape (the real -147.9% figures), the active passthrough, the no-data case, key-set stability across all four branches, the unshaped-treatment drop, and the legacy-ledger arm drop. - Existing tests that recorded treatment from a bare stratum label now attach the shaper label (`tests/test_output_savings.py`, `tests/test_output_savings_cli.py`) - they encoded the old contract. ## Testing - [x] Unit tests pass (`pytest`) ### Test Output ```text uv run --frozen --extra dev pytest tests/test_stats_output_reduction_gate.py tests/test_routing_stats_seam.py tests/test_dashboard_token_savings.py -q 3 passed; 8 passed ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.12 via uv, this branch's worktree. - Exact command / steps: Reproduced the field payload by calling `_output_reduction_payload` with the exact `SavingsEstimate` a blocked stable install published (tokens_saved=-7,522,490, pct=-147.9, n_requests=19,644, kind="measured"), shaper_active=False, then shaper_active=True. - Observed result: inactive path returns `method="inactive", tokens_saved=0, reduction_percent=0.0`; active path returns the estimate byte-identical to today's payload plus `active: true`. - Not tested: an end-to-end /stats call on a live rollout-blocked install (no blocked-channel machine at hand); the desktop app renders from its own ledger recompute since 0.8.7 and does not consume this field for its headline. ## Runtime Rollout Safety - Rollout-managed feature(s): Reads the output-shaper's rollout state; changes no rollout behavior itself. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, for deployments where the shaper is disabled: the /stats output_shaping layer reports inactive/zero instead of the persisted ledger estimate. Deployments with the shaper active are unchanged. - Kill switch / disable path: Not applicable; the gate follows the shaper's own enablement, which already has one. - Unsafe override required: No. - Qualification impact: Blocked-channel installs stop advertising phantom (often negative) output savings in /stats and telemetry built on it. - Rollback path: Revert the commit; the payload shape addition (`active`) is additive and the ledger format is untouched. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
62 lines
2.2 KiB
Python
62 lines
2.2 KiB
Python
"""Smoke tests for the output-savings CLI and the outcome→ledger wiring."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
from headroom.proxy.output_savings import SavingsRecorder, stratum_label
|
|
|
|
# A treatment observation only counts when the request was actually shaped,
|
|
# evidenced by the shaper's own verbosity label on the same channel.
|
|
SHAPED = "output_shaper:verbosity:L2"
|
|
|
|
|
|
def test_output_savings_empty(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
|
result = CliRunner().invoke(main, ["output-savings"])
|
|
assert result.exit_code == 0
|
|
assert "No output-savings data" in result.output
|
|
|
|
|
|
def test_output_savings_reports_estimate(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
|
# Seed a baseline + treatment observations directly via the ledger.
|
|
from headroom.proxy.output_savings import SavingsLedger
|
|
|
|
ledger = SavingsLedger()
|
|
for _ in range(50):
|
|
ledger.baseline.observe("opus|new_user_ask|s|tools", 1000)
|
|
for _ in range(30):
|
|
ledger.record("treatment", "opus|new_user_ask|s|tools", 700)
|
|
ledger.save(tmp_path / "output_savings.json")
|
|
|
|
result = CliRunner().invoke(main, ["output-savings"])
|
|
assert result.exit_code == 0
|
|
assert "ESTIMATED" in result.output
|
|
assert "Reduction:" in result.output
|
|
assert "30.0%" in result.output
|
|
|
|
|
|
def test_recorder_round_trips_via_labels(tmp_path):
|
|
path = tmp_path / "savings.json"
|
|
rec = SavingsRecorder(path, flush_every=1)
|
|
# Baseline so the estimate has something to compare against.
|
|
rec._ledger.baseline.observe("opus|new_user_ask|s|tools", 1000)
|
|
labels = [
|
|
"compress:smartcrush",
|
|
stratum_label("treatment", "opus|new_user_ask|s|tools"),
|
|
SHAPED,
|
|
]
|
|
assert rec.record_from_labels(labels, output_tokens=600) is True
|
|
assert rec.record_from_labels(["no-shaper-label"], output_tokens=999) is False
|
|
est = rec.estimate()
|
|
assert est.n_requests == 1
|
|
assert est.tokens_saved == 400 # 1000 - 600
|
|
|
|
# Persisted to disk.
|
|
data = json.loads(path.read_text())
|
|
assert "treatment" in data
|