mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(savings): don't let a thin holdout arm decide the headline number (#3127)
## Description Closes #3993 `best_estimate` preferred the measured A/B number the moment a single stratum held one sample in both arms: ```python measured = self.estimate_from_holdout() return measured if measured is not None else self.estimate_from_baseline() ``` A holdout arm starts empty and fills slowly. At a 3% holdout, conversation-stable assignment means control samples arrive in clusters over weeks, so for most of its life the arm is a handful of requests deciding the reported reduction. Three stale control samples produced a **-1439.9%** reduction that way. That number leaves the estimator as a plain `reduction_percent` in `/stats`, so every consumer has to either render it or invent its own sanity clamp. Deciding when a measurement is worth believing belongs in the estimator. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `best_estimate` lets the measured (holdout) number replace the synthetic-control estimate only when all three hold: - Every contributing stratum holds `MEASURED_SUPERSEDE_MIN_CLUSTERS` (30) distinct conversations in each arm. Upstream's `MEASURED_MIN_CLUSTERS` (5) still decides whether a stratum is a sample at all; 30 decides whether it may set the headline. The 95% band is computed per request, so five long control conversations read as a tight band (5 x 200 requests: about +/-4pp, where the honest band is tens of points). Only capped cluster ids are stored, so a cluster-robust band cannot be computed; this floor is the clustering guard instead. It stays at or below `_CLUSTER_CAP` (32). - The measured strata cover `MEASURED_MIN_COVERAGE` (0.5) of the conversation-labelled treatment requests. Labelled only: requests recorded before conversations were tracked can never be measured, so they neither help nor block it. - The 95% band is no wider than `MEASURED_MAX_CI_HALF_WIDTH_PCT` (+/-10pp). Below any bar the synthetic control stands, still labelled `estimated`. A holdout-only deployment (no `learn --verbosity` baseline) still reports the measurement once it clears the bars, since there is nothing to fall back to. Why it matters more now: #3209 seeds the holdout key on the whole opener, so conversations are no longer collapsed into one key per client and five conversations per arm fill quickly. On the bare cluster gate, a corner of the traffic or a few long sessions would decide the headline. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Historical Test Output ```text $ uv run --frozen --extra dev pytest tests/test_output_savings.py tests/test_output_savings_cli.py -q tests/test_output_savings.py ........................................... [ 89%] tests/test_output_savings_cli.py ... [100%] 48 passed in 0.39s $ uvx ruff check headroom/proxy/output_savings.py tests/test_output_savings.py All checks passed! $ uv run --frozen --extra dev mypy headroom/proxy/output_savings.py Success: no issues found in 1 source file ``` New regressions: `test_a_single_control_observation_does_not_contribute`, `test_repeated_identical_control_observations_do_not_contribute`, and `test_a_wide_band_does_not_take_over_even_at_full_size` (which pins that the width gate still bites on an arm that clears the sample floor). Three pre-existing tests used 10-sample arms and now use the floor instead. Their assertions are unchanged; only the loop bound moved. ## Real Behavior Proof - Environment: historical run, from before the rebase onto #3460 that replaced the per-arm floor with conversation-cluster admission. macOS 24.6.0, Python 3.10.18, uv, upstream/main at17522fb0- Exact command / steps: set `MEASURED_MIN_ARM_SAMPLES = 1`, which is exactly the previous admission rule (`c.n == 0 or t.n == 0`), then ran `uv run --frozen --extra dev pytest tests/test_output_savings.py -q -k "single_control_observation or repeated_identical_control or thin_control_arm"` - Observed result: all three fail without the floor, each promoting a 200-request headline off an arm that cannot support one. One control observation: `pct=20.0, ci_low_pct=20.0, ci_high_pct=20.0, n_requests=200, kind='measured'` - a zero-width 95% band over 100% coverage, so both aggregate gates pass it. Five identical control observations: `pct=-300.0, ci_low_pct=-300.0, ci_high_pct=-300.0` - same zero-width band, arbitrarily far from the treatment mean. Three scattered observations: `pct=56.5, ci_low_pct=-84.3, ci_high_pct=197.3`. All three return `None` from `estimate_from_holdout` with the floor in place, and `best_estimate().kind` is `estimated`. - Not tested: no live proxy run with a real holdout; the arms are constructed directly on the ledger. The -1439.9% figure quoted above is from a shipped desktop build, not reproduced here. ## Runtime Rollout Safety - Rollout-managed feature(s): None. Output-savings estimation runs wherever a ledger exists; this changes which of two already-computed estimates is reported, not whether either is computed. - Minimum rollout channel: N/A - Stable/default behavior changed: Yes, deliberately. A deployment whose holdout arm is thin now reports `estimated` where it previously reported `measured`, and `reduction_percent` moves to the synthetic-control number with it. Deployments with a well-populated holdout are unaffected. No request accounting, no persisted state and no wire format changes. - Kill switch / disable path: No runtime flag. Revert this PR to remove its coverage/interval-width promotion gates while retaining upstream conversation-cluster admission. The historical `MEASURED_MIN_ARM_SAMPLES` constant is no longer used. - Unsafe override required: No - Qualification impact: Three existing tests re-sized to the sample floor with assertions unchanged; one existing test now asserts the thin arm is excluded rather than merely unpromoted. No test was relaxed. - Rollback path: Revert the commits. Both estimates are recomputed from the ledger on every read, so the previous headline returns immediately with no migration or backfill. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ### Current validation (2026-10-06) At `8738f3fc`, merged with upstream/main `855390d6` (0.40.0): `tests/test_output_savings.py` and `tests/test_output_savings_cli.py` pass (78), plus 185 across the 11 other test files that use the output-savings ledger or `best_estimate`. ruff, ruff format and mypy are clean. The repro in #3993 drives the ledger through the product's own recording paths (`SavingsRecorder.record_from_labels`, `SavingsRecorder.estimate()`): on main, five holdout conversations take the headline at -1320.6% or +61.5% depending on which five landed in the arm, each with a band under +/-16 points; on this branch both stay on the synthetic-control 13.9%. No live provider billing or traffic run is claimed. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3127" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Current runtime evidence (2026-10-08,3993a584b) A standalone process records deterministic labelled outcomes through `SavingsRecorder.record_from_labels`: 200 treatment conversations and five control conversations with 200 requests each. On current main3c418f6f, the same serialized ledger produces a measured -1438.5% headline. On this branch, an actual uvicorn proxy serves GET /stats over TCP with HTTP 200, method estimated, and reduction_percent 20.0 (40,000 saved / 200,000 baseline tokens). No endpoint or recorder method is mocked. This is a seeded outcome replay, not live provider billing traffic. Fresh checks: 143 savings/CLI/rollup/stateless-writer/stats-gating/gateway tests pass, Ruff and Linux-target mypy (566 files) pass, and npm audit reports zero vulnerabilities after the main refresh. The request-level confidence interval remains a limitation; the new gates restrict when it can replace baseline evidence. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5
JD Davis
parent
71663b44e2
commit
2a12e035a3
@@ -354,6 +354,44 @@ class SavingsEstimate:
|
||||
_SHAPED_LABEL_PREFIX = "output_shaper:verbosity:"
|
||||
|
||||
|
||||
# :data:`MEASURED_MIN_CLUSTERS` decides whether a stratum is a real sample at
|
||||
# all. The gates below decide something stricter: whether the measurement has
|
||||
# earned the headline slot away from the synthetic control.
|
||||
|
||||
# Conversations per arm a stratum needs before it may count toward the
|
||||
# headline. The 95% band is computed per REQUEST, so it reads one long
|
||||
# conversation as many independent draws: five control conversations of 200
|
||||
# requests each report about +/-4pp where the honest band is tens of points.
|
||||
# Only capped per-stratum cluster ids are stored, so a cluster-robust band
|
||||
# cannot be computed; this floor is the clustering guard instead. It must stay
|
||||
# at or below ``_CLUSTER_CAP``, which ``n_clusters`` saturates at.
|
||||
MEASURED_SUPERSEDE_MIN_CLUSTERS = 30
|
||||
|
||||
# Share of the conversation-labelled treatment requests the measured strata
|
||||
# must cover. Below it the holdout describes a corner of the traffic, not the
|
||||
# traffic. Labelled requests only: the measurement can only ever be built from
|
||||
# those, so unlabelled legacy volume must neither help nor block it.
|
||||
MEASURED_MIN_COVERAGE = 0.5
|
||||
|
||||
# ...and the 95% band must be at least this tight, in percentage points of
|
||||
# reduction. A stratum can clear every cluster floor and still be too noisy to
|
||||
# say anything.
|
||||
MEASURED_MAX_CI_HALF_WIDTH_PCT = 10.0
|
||||
|
||||
|
||||
def _measured_covers(measured: SavingsEstimate, labelled_requests: int) -> bool:
|
||||
"""Whether the measured strata describe the traffic, not a corner of it."""
|
||||
return measured.n_requests >= MEASURED_MIN_COVERAGE * labelled_requests
|
||||
|
||||
|
||||
def _measured_supersedes(measured: SavingsEstimate, labelled_requests: int) -> bool:
|
||||
"""Whether the A/B measurement has outgrown the synthetic-control estimate."""
|
||||
half_width = (measured.ci_high_pct - measured.ci_low_pct) / 2.0
|
||||
if half_width > MEASURED_MAX_CI_HALF_WIDTH_PCT:
|
||||
return False
|
||||
return _measured_covers(measured, labelled_requests)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SavingsLedger:
|
||||
"""Accumulates shaped (treatment) and unshaped (control) observations and
|
||||
@@ -407,7 +445,9 @@ class SavingsLedger:
|
||||
var += (n * n) * (mu_var / m)
|
||||
return self._finalize(total_saved, total_baseline, var, n_requests, "estimated")
|
||||
|
||||
def estimate_from_holdout(self) -> SavingsEstimate | None:
|
||||
def estimate_from_holdout(
|
||||
self, min_clusters: int = MEASURED_MIN_CLUSTERS
|
||||
) -> SavingsEstimate | None:
|
||||
"""A/B measurement: per-stratum control mean minus treatment mean.
|
||||
|
||||
Only strata with conversation-labelled data in BOTH arms contribute,
|
||||
@@ -439,7 +479,7 @@ class SavingsLedger:
|
||||
c = self.control.get(key)
|
||||
if c is None or c.qn == 0 or t.qn == 0:
|
||||
continue
|
||||
if c.n_clusters < MEASURED_MIN_CLUSTERS or t.n_clusters < MEASURED_MIN_CLUSTERS:
|
||||
if c.n_clusters < min_clusters or t.n_clusters < min_clusters:
|
||||
continue
|
||||
contributing += 1
|
||||
# Everything below reads the qualified subset only. The clusters
|
||||
@@ -529,17 +569,33 @@ class SavingsLedger:
|
||||
)
|
||||
|
||||
def best_estimate(self, level: int | None = None) -> SavingsEstimate:
|
||||
"""Strongest available tier: measured > estimated > modelled.
|
||||
"""Strongest believable tier: measured > estimated > modelled.
|
||||
|
||||
``level`` enables the modelled fallback; without it the behaviour is
|
||||
unchanged from before, which keeps every existing caller honest.
|
||||
|
||||
The measured tier is preferred only once it is worth believing. With
|
||||
conversation-stable keys five conversations per arm fill quickly, so on
|
||||
the bare cluster gate a corner of the traffic, or a few long
|
||||
conversations, would decide the headline. The measured number displaces
|
||||
the synthetic control only when every stratum it is built from holds
|
||||
:data:`MEASURED_SUPERSEDE_MIN_CLUSTERS` conversations per arm, those
|
||||
strata cover :data:`MEASURED_MIN_COVERAGE` of the labelled treatment
|
||||
requests, and its band is tight enough to mean something. With no
|
||||
baseline to fall back on, the bare cluster gate's measurement is still
|
||||
reported, since it displaces nothing, but only while it covers the
|
||||
traffic: a corner of it is not a headline either way.
|
||||
"""
|
||||
measured = self.estimate_from_holdout()
|
||||
if measured is not None:
|
||||
measured = self.estimate_from_holdout(min_clusters=MEASURED_SUPERSEDE_MIN_CLUSTERS)
|
||||
labelled = sum(t.qn for t in self.treatment.values())
|
||||
if measured is not None and _measured_supersedes(measured, labelled):
|
||||
return measured
|
||||
estimated = self.estimate_from_baseline()
|
||||
if estimated.n_requests > 0:
|
||||
return estimated
|
||||
measured = self.estimate_from_holdout()
|
||||
if measured is not None and _measured_covers(measured, labelled):
|
||||
return measured
|
||||
if level is not None:
|
||||
modelled = self.estimate_from_model(level)
|
||||
if modelled is not None:
|
||||
|
||||
@@ -8,6 +8,7 @@ import pytest
|
||||
|
||||
from headroom.proxy.output_savings import (
|
||||
MEASURED_MIN_CLUSTERS,
|
||||
MEASURED_SUPERSEDE_MIN_CLUSTERS,
|
||||
BaselineModel,
|
||||
SavingsLedger,
|
||||
SavingsRecorder,
|
||||
@@ -365,7 +366,7 @@ class TestEstimateFromHoldout:
|
||||
|
||||
def test_best_estimate_prefers_measured(self):
|
||||
ledger = SavingsLedger()
|
||||
for i in range(10):
|
||||
for i in range(MEASURED_SUPERSEDE_MIN_CLUSTERS):
|
||||
ledger.baseline.observe("opus|a|s|tools", 1000)
|
||||
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
|
||||
ledger.record("treatment", "opus|a|s|tools", 900, f"t{i}")
|
||||
@@ -378,6 +379,91 @@ class TestEstimateFromHoldout:
|
||||
ledger.record("treatment", "opus|a|s|tools", 900)
|
||||
assert ledger.best_estimate().kind == "estimated"
|
||||
|
||||
def test_a_wide_band_does_not_take_over_even_at_full_size(self):
|
||||
"""The width gate still bites once the cluster gate is cleared."""
|
||||
ledger = SavingsLedger()
|
||||
for i in range(200):
|
||||
ledger.baseline.observe("opus|a|s|tools", 1000)
|
||||
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
|
||||
for i in range(30):
|
||||
ledger.record("control", "opus|a|s|tools", (200, 1000, 5000)[i % 3], f"c{i}")
|
||||
|
||||
measured = ledger.estimate_from_holdout()
|
||||
assert measured is not None, "the arm is spread over enough conversations"
|
||||
assert (measured.ci_high_pct - measured.ci_low_pct) / 2 > 10, "and too noisy to believe"
|
||||
assert ledger.best_estimate().kind == "estimated"
|
||||
|
||||
def test_a_holdout_covering_a_corner_of_traffic_does_not_take_over(self):
|
||||
ledger = SavingsLedger()
|
||||
# One stratum measured cleanly, but it is a fraction of the traffic.
|
||||
for i in range(30):
|
||||
ledger.baseline.observe("opus|a|s|tools", 1000)
|
||||
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
|
||||
ledger.record("treatment", "opus|a|s|tools", 900, f"t{i}")
|
||||
for i in range(200):
|
||||
ledger.baseline.observe("opus|b|xl|tools", 4000)
|
||||
ledger.record("treatment", "opus|b|xl|tools", 3000, f"b{i}")
|
||||
assert ledger.best_estimate().kind == "estimated"
|
||||
|
||||
def test_a_few_long_conversations_do_not_take_the_headline(self):
|
||||
"""Five control conversations clear the cluster gate, and their
|
||||
per-request band looks tight, but they are five draws, not 1,000."""
|
||||
ledger = SavingsLedger()
|
||||
for i in range(200):
|
||||
ledger.baseline.observe("opus|a|s|tools", 1000)
|
||||
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
|
||||
for conv in range(5):
|
||||
for j in range(200):
|
||||
ledger.record("control", "opus|a|s|tools", 900 + 100 * conv + j % 3, f"c{conv}")
|
||||
|
||||
measured = ledger.estimate_from_holdout()
|
||||
assert measured is not None, "a real arm by the cluster gate"
|
||||
assert (measured.ci_high_pct - measured.ci_low_pct) / 2 < 10, "and a tight-looking band"
|
||||
assert ledger.best_estimate().kind == "estimated"
|
||||
|
||||
def test_unlabelled_legacy_traffic_does_not_block_the_measurement(self):
|
||||
"""Requests recorded before conversations were tracked can never be
|
||||
measured, so they must not count against the measurement's coverage."""
|
||||
ledger = SavingsLedger()
|
||||
for _ in range(2000):
|
||||
ledger.baseline.observe("opus|a|s|tools", 1000)
|
||||
ledger.record("treatment", "opus|a|s|tools", 800)
|
||||
for i in range(MEASURED_SUPERSEDE_MIN_CLUSTERS):
|
||||
ledger.record("control", "opus|a|s|tools", 1000 + i % 3, f"c{i}")
|
||||
ledger.record("treatment", "opus|a|s|tools", 900 + i % 3, f"t{i}")
|
||||
|
||||
assert ledger.estimate_from_baseline().n_requests == 2030
|
||||
assert ledger.best_estimate().kind == "measured"
|
||||
|
||||
def test_a_young_holdout_still_reports_without_a_baseline(self):
|
||||
"""Below the supersede floor there is nothing to displace, so the
|
||||
cluster-gated measurement is reported rather than nothing."""
|
||||
ledger = SavingsLedger()
|
||||
for i in range(MEASURED_MIN_CLUSTERS):
|
||||
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
|
||||
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
|
||||
assert ledger.best_estimate().kind == "measured"
|
||||
|
||||
def test_a_corner_holdout_does_not_set_the_headline_without_a_baseline(self):
|
||||
ledger = SavingsLedger()
|
||||
for i in range(MEASURED_MIN_CLUSTERS):
|
||||
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
|
||||
ledger.record("treatment", "opus|a|s|tools", 100, f"t{i}")
|
||||
# The rest of the traffic has no control to compare against.
|
||||
for i in range(200):
|
||||
ledger.record("treatment", "opus|b|xl|tools", 3000, f"b{i}")
|
||||
assert ledger.estimate_from_holdout() is not None
|
||||
assert ledger.best_estimate().kind != "measured"
|
||||
|
||||
def test_measured_wins_without_a_baseline_to_fall_back_on(self):
|
||||
"""A holdout-only deployment (no ``learn --verbosity`` run) still reports."""
|
||||
ledger = SavingsLedger()
|
||||
for i in range(50):
|
||||
ledger.record("control", "opus|a|s|tools", 1000, f"c{i}")
|
||||
ledger.record("treatment", "opus|a|s|tools", 800, f"t{i}")
|
||||
assert ledger.estimate_from_baseline().n_requests == 0
|
||||
assert ledger.best_estimate().kind == "measured"
|
||||
|
||||
|
||||
class TestHoldoutClusterGate:
|
||||
"""A stratum needs distinct CONVERSATIONS in both arms, not requests.
|
||||
|
||||
Reference in New Issue
Block a user