fix(savings): don't let a thin holdout arm decide the headline number (#3127)

## Description

Closes #3993

`best_estimate` preferred the measured A/B number the moment a single
stratum held one sample in both arms:

```python
measured = self.estimate_from_holdout()
return measured if measured is not None else self.estimate_from_baseline()
```

A holdout arm starts empty and fills slowly. At a 3% holdout,
conversation-stable assignment means control samples arrive in clusters
over weeks, so for most of its life the arm is a handful of requests
deciding the reported reduction. Three stale control samples produced a
**-1439.9%** reduction that way.

That number leaves the estimator as a plain `reduction_percent` in
`/stats`, so every consumer has to either render it or invent its own
sanity clamp. Deciding when a measurement is worth believing belongs in
the estimator.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

`best_estimate` lets the measured (holdout) number replace the
synthetic-control estimate only when all three hold:

- Every contributing stratum holds `MEASURED_SUPERSEDE_MIN_CLUSTERS`
(30) distinct conversations in each arm. Upstream's
`MEASURED_MIN_CLUSTERS` (5) still decides whether a stratum is a sample
at all; 30 decides whether it may set the headline. The 95% band is
computed per request, so five long control conversations read as a tight
band (5 x 200 requests: about +/-4pp, where the honest band is tens of
points). Only capped cluster ids are stored, so a cluster-robust band
cannot be computed; this floor is the clustering guard instead. It stays
at or below `_CLUSTER_CAP` (32).
- The measured strata cover `MEASURED_MIN_COVERAGE` (0.5) of the
conversation-labelled treatment requests. Labelled only: requests
recorded before conversations were tracked can never be measured, so
they neither help nor block it.
- The 95% band is no wider than `MEASURED_MAX_CI_HALF_WIDTH_PCT`
(+/-10pp).

Below any bar the synthetic control stands, still labelled `estimated`.
A holdout-only deployment (no `learn --verbosity` baseline) still
reports the measurement once it clears the bars, since there is nothing
to fall back to.

Why it matters more now: #3209 seeds the holdout key on the whole
opener, so conversations are no longer collapsed into one key per client
and five conversations per arm fill quickly. On the bare cluster gate, a
corner of the traffic or a few long sessions would decide the headline.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Historical Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_output_savings.py tests/test_output_savings_cli.py -q
tests/test_output_savings.py ........................................... [ 89%]
tests/test_output_savings_cli.py ...                                     [100%]
48 passed in 0.39s

$ uvx ruff check headroom/proxy/output_savings.py tests/test_output_savings.py
All checks passed!

$ uv run --frozen --extra dev mypy headroom/proxy/output_savings.py
Success: no issues found in 1 source file
```

New regressions:
`test_a_single_control_observation_does_not_contribute`,
`test_repeated_identical_control_observations_do_not_contribute`, and
`test_a_wide_band_does_not_take_over_even_at_full_size` (which pins that
the width gate still bites on an arm that clears the sample floor).

Three pre-existing tests used 10-sample arms and now use the floor
instead. Their assertions are unchanged; only the loop bound moved.

## Real Behavior Proof

- Environment: historical run, from before the rebase onto #3460 that
replaced the per-arm floor with conversation-cluster admission. macOS
24.6.0, Python 3.10.18, uv, upstream/main at 17522fb0
- Exact command / steps: set `MEASURED_MIN_ARM_SAMPLES = 1`, which is
exactly the previous admission rule (`c.n == 0 or t.n == 0`), then ran
`uv run --frozen --extra dev pytest tests/test_output_savings.py -q -k
"single_control_observation or repeated_identical_control or
thin_control_arm"`
- Observed result: all three fail without the floor, each promoting a
200-request headline off an arm that cannot support one. One control
observation: `pct=20.0, ci_low_pct=20.0, ci_high_pct=20.0,
n_requests=200, kind='measured'` - a zero-width 95% band over 100%
coverage, so both aggregate gates pass it. Five identical control
observations: `pct=-300.0, ci_low_pct=-300.0, ci_high_pct=-300.0` - same
zero-width band, arbitrarily far from the treatment mean. Three
scattered observations: `pct=56.5, ci_low_pct=-84.3, ci_high_pct=197.3`.
All three return `None` from `estimate_from_holdout` with the floor in
place, and `best_estimate().kind` is `estimated`.
- Not tested: no live proxy run with a real holdout; the arms are
constructed directly on the ledger. The -1439.9% figure quoted above is
from a shipped desktop build, not reproduced here.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Output-savings estimation runs
wherever a ledger exists; this changes which of two already-computed
estimates is reported, not whether either is computed.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, deliberately. A deployment whose
holdout arm is thin now reports `estimated` where it previously reported
`measured`, and `reduction_percent` moves to the synthetic-control
number with it. Deployments with a well-populated holdout are
unaffected. No request accounting, no persisted state and no wire format
changes.
- Kill switch / disable path: No runtime flag. Revert this PR to remove
its coverage/interval-width promotion gates while retaining upstream
conversation-cluster admission. The historical
`MEASURED_MIN_ARM_SAMPLES` constant is no longer used.
- Unsafe override required: No
- Qualification impact: Three existing tests re-sized to the sample
floor with assertions unchanged; one existing test now asserts the thin
arm is excluded rather than merely unpromoted. No test was relaxed.
- Rollback path: Revert the commits. Both estimates are recomputed from
the ledger on every read, so the previous headline returns immediately
with no migration or backfill.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review


### Current validation (2026-10-06)

At `8738f3fc`, merged with upstream/main `855390d6` (0.40.0):
`tests/test_output_savings.py` and `tests/test_output_savings_cli.py`
pass (78), plus 185 across the 11 other test files that use the
output-savings ledger or `best_estimate`. ruff, ruff format and mypy are
clean. The repro in #3993 drives the ledger through the product's own
recording paths (`SavingsRecorder.record_from_labels`,
`SavingsRecorder.estimate()`): on main, five holdout conversations take
the headline at -1320.6% or +61.5% depending on which five landed in the
arm, each with a band under +/-16 points; on this branch both stay on
the synthetic-control 13.9%. No live provider billing or traffic run is
claimed.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3127"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->



### Current runtime evidence (2026-10-08, 3993a584b)

A standalone process records deterministic labelled outcomes through
`SavingsRecorder.record_from_labels`: 200 treatment conversations and
five control conversations with 200 requests each. On current main
3c418f6f, the same serialized ledger produces a measured -1438.5%
headline. On this branch, an actual uvicorn proxy serves GET /stats over
TCP with HTTP 200, method estimated, and reduction_percent 20.0 (40,000
saved / 200,000 baseline tokens). No endpoint or recorder method is
mocked. This is a seeded outcome replay, not live provider billing
traffic.

Fresh checks: 143
savings/CLI/rollup/stateless-writer/stats-gating/gateway tests pass,
Ruff and Linux-target mypy (566 files) pass, and npm audit reports zero
vulnerabilities after the main refresh. The request-level confidence
interval remains a limitation; the new gates restrict when it can
replace baseline evidence.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-08 10:50:58 -05:00
committed by GitHub
co-authored by Claude Opus 5 JD Davis
parent 71663b44e2
commit 2a12e035a3
2 changed files with 148 additions and 6 deletions
+61 -5
View File
@@ -354,6 +354,44 @@ class SavingsEstimate:
_SHAPED_LABEL_PREFIX = "output_shaper:verbosity:"
# :data:`MEASURED_MIN_CLUSTERS` decides whether a stratum is a real sample at
# all. The gates below decide something stricter: whether the measurement has
# earned the headline slot away from the synthetic control.
# Conversations per arm a stratum needs before it may count toward the
# headline. The 95% band is computed per REQUEST, so it reads one long
# conversation as many independent draws: five control conversations of 200
# requests each report about +/-4pp where the honest band is tens of points.
# Only capped per-stratum cluster ids are stored, so a cluster-robust band
# cannot be computed; this floor is the clustering guard instead. It must stay
# at or below ``_CLUSTER_CAP``, which ``n_clusters`` saturates at.
MEASURED_SUPERSEDE_MIN_CLUSTERS = 30
# Share of the conversation-labelled treatment requests the measured strata
# must cover. Below it the holdout describes a corner of the traffic, not the
# traffic. Labelled requests only: the measurement can only ever be built from
# those, so unlabelled legacy volume must neither help nor block it.
MEASURED_MIN_COVERAGE = 0.5
# ...and the 95% band must be at least this tight, in percentage points of
# reduction. A stratum can clear every cluster floor and still be too noisy to
# say anything.
MEASURED_MAX_CI_HALF_WIDTH_PCT = 10.0
def _measured_covers(measured: SavingsEstimate, labelled_requests: int) -> bool:
"""Whether the measured strata describe the traffic, not a corner of it."""
return measured.n_requests >= MEASURED_MIN_COVERAGE * labelled_requests
def _measured_supersedes(measured: SavingsEstimate, labelled_requests: int) -> bool:
"""Whether the A/B measurement has outgrown the synthetic-control estimate."""
half_width = (measured.ci_high_pct - measured.ci_low_pct) / 2.0
if half_width > MEASURED_MAX_CI_HALF_WIDTH_PCT:
return False
return _measured_covers(measured, labelled_requests)
@dataclass
class SavingsLedger:
"""Accumulates shaped (treatment) and unshaped (control) observations and
@@ -407,7 +445,9 @@ class SavingsLedger:
var += (n * n) * (mu_var / m)
return self._finalize(total_saved, total_baseline, var, n_requests, "estimated")
def estimate_from_holdout(self) -> SavingsEstimate | None:
def estimate_from_holdout(
self, min_clusters: int = MEASURED_MIN_CLUSTERS
) -> SavingsEstimate | None:
"""A/B measurement: per-stratum control mean minus treatment mean.
Only strata with conversation-labelled data in BOTH arms contribute,
@@ -439,7 +479,7 @@ class SavingsLedger:
c = self.control.get(key)
if c is None or c.qn == 0 or t.qn == 0:
continue
if c.n_clusters < MEASURED_MIN_CLUSTERS or t.n_clusters < MEASURED_MIN_CLUSTERS:
if c.n_clusters < min_clusters or t.n_clusters < min_clusters:
continue
contributing += 1
# Everything below reads the qualified subset only. The clusters
@@ -529,17 +569,33 @@ class SavingsLedger:
)
def best_estimate(self, level: int | None = None) -> SavingsEstimate:
"""Strongest available tier: measured > estimated > modelled.
"""Strongest believable tier: measured > estimated > modelled.
``level`` enables the modelled fallback; without it the behaviour is
unchanged from before, which keeps every existing caller honest.
The measured tier is preferred only once it is worth believing. With
conversation-stable keys five conversations per arm fill quickly, so on
the bare cluster gate a corner of the traffic, or a few long
conversations, would decide the headline. The measured number displaces
the synthetic control only when every stratum it is built from holds
:data:`MEASURED_SUPERSEDE_MIN_CLUSTERS` conversations per arm, those
strata cover :data:`MEASURED_MIN_COVERAGE` of the labelled treatment
requests, and its band is tight enough to mean something. With no
baseline to fall back on, the bare cluster gate's measurement is still
reported, since it displaces nothing, but only while it covers the
traffic: a corner of it is not a headline either way.
"""
measured = self.estimate_from_holdout()
if measured is not None:
measured = self.estimate_from_holdout(min_clusters=MEASURED_SUPERSEDE_MIN_CLUSTERS)
labelled = sum(t.qn for t in self.treatment.values())
if measured is not None and _measured_supersedes(measured, labelled):
return measured
estimated = self.estimate_from_baseline()
if estimated.n_requests > 0:
return estimated
measured = self.estimate_from_holdout()
if measured is not None and _measured_covers(measured, labelled):
return measured
if level is not None:
modelled = self.estimate_from_model(level)
if modelled is not None:
+87 -1
View File
@@ -8,6 +8,7 @@ import pytest
from headroom.proxy.output_savings import (
MEASURED_MIN_CLUSTERS,
MEASURED_SUPERSEDE_MIN_CLUSTERS,
BaselineModel,
SavingsLedger,
SavingsRecorder,
@@ -365,7 +366,7 @@ class TestEstimateFromHoldout:
def test_best_estimate_prefers_measured(self):
ledger = SavingsLedger()
for i in range(10):
for i in range(MEASURED_SUPERSEDE_MIN_CLUSTERS):
ledger.baseline.observe("opus|a|s|tools", 1000)
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
ledger.record("treatment", "opus|a|s|tools", 900, f"t{i}")
@@ -378,6 +379,91 @@ class TestEstimateFromHoldout:
ledger.record("treatment", "opus|a|s|tools", 900)
assert ledger.best_estimate().kind == "estimated"
def test_a_wide_band_does_not_take_over_even_at_full_size(self):
"""The width gate still bites once the cluster gate is cleared."""
ledger = SavingsLedger()
for i in range(200):
ledger.baseline.observe("opus|a|s|tools", 1000)
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
for i in range(30):
ledger.record("control", "opus|a|s|tools", (200, 1000, 5000)[i % 3], f"c{i}")
measured = ledger.estimate_from_holdout()
assert measured is not None, "the arm is spread over enough conversations"
assert (measured.ci_high_pct - measured.ci_low_pct) / 2 > 10, "and too noisy to believe"
assert ledger.best_estimate().kind == "estimated"
def test_a_holdout_covering_a_corner_of_traffic_does_not_take_over(self):
ledger = SavingsLedger()
# One stratum measured cleanly, but it is a fraction of the traffic.
for i in range(30):
ledger.baseline.observe("opus|a|s|tools", 1000)
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
ledger.record("treatment", "opus|a|s|tools", 900, f"t{i}")
for i in range(200):
ledger.baseline.observe("opus|b|xl|tools", 4000)
ledger.record("treatment", "opus|b|xl|tools", 3000, f"b{i}")
assert ledger.best_estimate().kind == "estimated"
def test_a_few_long_conversations_do_not_take_the_headline(self):
"""Five control conversations clear the cluster gate, and their
per-request band looks tight, but they are five draws, not 1,000."""
ledger = SavingsLedger()
for i in range(200):
ledger.baseline.observe("opus|a|s|tools", 1000)
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
for conv in range(5):
for j in range(200):
ledger.record("control", "opus|a|s|tools", 900 + 100 * conv + j % 3, f"c{conv}")
measured = ledger.estimate_from_holdout()
assert measured is not None, "a real arm by the cluster gate"
assert (measured.ci_high_pct - measured.ci_low_pct) / 2 < 10, "and a tight-looking band"
assert ledger.best_estimate().kind == "estimated"
def test_unlabelled_legacy_traffic_does_not_block_the_measurement(self):
"""Requests recorded before conversations were tracked can never be
measured, so they must not count against the measurement's coverage."""
ledger = SavingsLedger()
for _ in range(2000):
ledger.baseline.observe("opus|a|s|tools", 1000)
ledger.record("treatment", "opus|a|s|tools", 800)
for i in range(MEASURED_SUPERSEDE_MIN_CLUSTERS):
ledger.record("control", "opus|a|s|tools", 1000 + i % 3, f"c{i}")
ledger.record("treatment", "opus|a|s|tools", 900 + i % 3, f"t{i}")
assert ledger.estimate_from_baseline().n_requests == 2030
assert ledger.best_estimate().kind == "measured"
def test_a_young_holdout_still_reports_without_a_baseline(self):
"""Below the supersede floor there is nothing to displace, so the
cluster-gated measurement is reported rather than nothing."""
ledger = SavingsLedger()
for i in range(MEASURED_MIN_CLUSTERS):
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
assert ledger.best_estimate().kind == "measured"
def test_a_corner_holdout_does_not_set_the_headline_without_a_baseline(self):
ledger = SavingsLedger()
for i in range(MEASURED_MIN_CLUSTERS):
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
ledger.record("treatment", "opus|a|s|tools", 100, f"t{i}")
# The rest of the traffic has no control to compare against.
for i in range(200):
ledger.record("treatment", "opus|b|xl|tools", 3000, f"b{i}")
assert ledger.estimate_from_holdout() is not None
assert ledger.best_estimate().kind != "measured"
def test_measured_wins_without_a_baseline_to_fall_back_on(self):
"""A holdout-only deployment (no ``learn --verbosity`` run) still reports."""
ledger = SavingsLedger()
for i in range(50):
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
assert ledger.estimate_from_baseline().n_requests == 0
assert ledger.best_estimate().kind == "measured"
class TestHoldoutClusterGate:
"""A stratum needs distinct CONVERSATIONS in both arms, not requests.