fix(savings): stop scoring unobserved strata against the global mean (#3128)

## Description

`BaselineModel.lookup` fell back to the all-requests mean whenever
prefix back-off found nothing. That fallback is not a control for
anything.

The baseline is seeded once, by `learn --verbosity` reading session
history that predates the shaper. It therefore only ever covers the
model families the user ran *before* installing. It also cannot be
relearned: every transcript written since is already shaped, so a
re-learn would collapse the baseline onto the treatment mean. Each
family a user adopts later resolves to a single number derived from a
different population, permanently.

Measured on a real ledger (15,658 baseline samples over 16 strata, all
`opus`; 74,243 treatment requests over 83 strata):

| | |
|---|---|
| Requests scored against the global mean | 48% (every fable, sonnet,
haiku, gpt turn) |
| Global mean applied to them | 1,083 output tokens |
| `sonnet\|new_user_ask\|m\|notools` | 4,666 requests, replies average
**73** tokens, each credited ~1,010 saved |
| Share of total reported savings from the fallback | **74%** |
| Reported reduction | 42.6% |
| Reduction over strata the baseline actually observed | 32.1% |

Scoring a short no-tool ask against a long tool-calling turn is not a
synthetic control, it is a unit conversion.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

**Reported savings will drop** for any user whose traffic has moved
beyond the model family their baseline was seeded from. That is the
point of the change, but it is a visible number moving, so flagging it
explicitly.

## Changes Made

- `lookup` returns `(0.0, 0.0, 0)` when neither the exact stratum nor
prefix back-off finds evidence. Both estimators already treat `n == 0`
as "no evidence" and skip the request, so what changes is which requests
qualify, not how they are scored.
- The global fallback stays reachable via `lookup(key,
fall_back_to_global=True)` for callers that want it; nothing in the tree
passes it today.
- `glob` is still observed and still backs `total_samples`, so the
ledger format and the "has a baseline been seeded" check are unaffected.
- Also in this commit: prefix back-off merges every matching neighbour
instead of returning the first in dict order. The old loop made the
answer depend on insertion order, so a round-tripped ledger could score
the same request differently. Happy to split this out if you would
rather review it separately.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_output_savings.py tests/test_verbosity_learn.py tests/test_output_savings_cli.py -q
tests/test_output_savings.py ........................................... [ 66%]
tests/test_verbosity_learn.py ...................                        [ 95%]
tests/test_output_savings_cli.py ...                                     [100%]
65 passed in 1.12s

$ uvx ruff check headroom/proxy/output_savings.py tests/test_output_savings.py
All checks passed!

$ uv run --frozen --extra dev mypy headroom/proxy/output_savings.py
Success: no issues found in 1 source file
```

`test_lookup_falls_back_to_global` is rewritten into two tests: one
pinning the new default, one pinning that the fallback still works when
asked for.

## Real Behavior Proof

- Environment: macOS 24.6.0, Python 3.10.18, uv, upstream/main at
17522fb0. Ledger from a shipped desktop build,
`~/.headroom/output_savings.json`, 7.4KB.
- Exact command / steps: reimplemented both back-off policies against
that ledger and recomputed `estimate_from_baseline` under each.
Confirmed the current-policy recomputation reproduces what the running
proxy reports on `/stats` before comparing.
- Observed result: current policy reproduces the live `/stats` figures
exactly (`tokens_saved` 31,493,556, `baseline_tokens` 73,986,108, 42.6%,
74,297 requests), which validates the recomputation. Under the new
policy the same ledger gives 11,540,868 / 35,961,142 / 32.1% over 38,729
requests. Splitting the current-policy total by how each stratum
resolved: 8.29M from exact stratum hits, 2.76M from prefix back-off,
20.43M from the global mean -- 74% of the reported savings resting on
the fallback this PR removes.
- Not tested: one user's ledger. A baseline seeded from a corpus
spanning several model families would lose less; one seeded before a
model switch would lose more.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Baseline lookup is not behind a
rollout gate.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, and visibly. Reported savings
drop for any user whose traffic has moved past the model family their
baseline was seeded from: 42.6% -> 32.1% on the real ledger measured
above. Requests with no baseline evidence are now skipped rather than
scored against an unrelated mean, so `n_requests` in the estimate drops
with it. That is the fix, but it is a number users can see moving.
- Kill switch / disable path: No runtime flag. `lookup(key,
fall_back_to_global=True)` restores the previous behavior at the call
site, and `glob` is still accumulated, so the old number remains
computable from the same ledger.
- Unsafe override required: No
- Qualification impact: `test_lookup_falls_back_to_global` is split into
two tests, one pinning the new default and one pinning the opt-in
fallback. No assertion was relaxed.
- Rollback path: Revert the commit. The ledger format is unchanged and
every estimate is recomputed from it on read, so the previous figures
return immediately with no migration or backfill.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-09-30 23:58:06 -05:00
committed by GitHub
co-authored by Claude Opus 5
parent 76ef2c3bc9
commit f8645257c0
2 changed files with 72 additions and 10 deletions
+24 -8
View File
@@ -229,12 +229,23 @@ class BaselineModel:
self.strata.setdefault(key, _Accum()).merge(acc)
self.glob.merge(other.glob)
def lookup(self, key: str) -> tuple[float, float, int]:
def lookup(self, key: str, *, fall_back_to_global: bool = False) -> tuple[float, float, int]:
"""Return ``(mean, var, n)`` for *key* with hierarchical back-off.
Falls back by trimming trailing (least-specific) stratum fields, then
to the global mean. Back-off keeps the estimate defined for strata the
baseline never saw, at the cost of specificity.
Falls back by trimming trailing (least-specific) stratum fields, so a
stratum the baseline never saw is still scored against its nearest
observed neighbours. Returns ``(0.0, 0.0, 0)`` when even that finds
nothing -- callers already treat ``n == 0`` as "no evidence".
``fall_back_to_global`` restores the old last resort of the
all-requests mean. It is off by default because that mean is not a
control for anything: the baseline is seeded once, from whatever the
user ran *before* installing, so every model family they adopt later
resolves to it. On a real ledger that meant 48% of requests -- sonnet
and fable turns whose replies average 43-770 tokens -- being scored
against one opus-derived mean of 1,083, which alone produced 74% of the
reported savings. Scoring a short no-tool ask against a long
tool-calling turn is not a synthetic control, it is a unit conversion.
"""
acc = self.strata.get(key)
if acc is not None and acc.n > 0:
@@ -242,11 +253,16 @@ class BaselineModel:
parts = key.split("|")
while len(parts) > 1:
parts = parts[:-1]
prefix = "|".join(parts)
prefix = "|".join(parts) + "|"
neighbours = _Accum()
for k, a in self.strata.items():
if k.startswith(prefix + "|") and a.n > 0:
return a.mean, a.var, a.n
return self.glob.mean, self.glob.var, self.glob.n
if a.n > 0 and k.startswith(prefix):
neighbours.merge(a)
if neighbours.n > 0:
return neighbours.mean, neighbours.var, neighbours.n
if fall_back_to_global:
return self.glob.mean, self.glob.var, self.glob.n
return 0.0, 0.0, 0
def to_dict(self) -> dict[str, Any]:
return {
+48 -2
View File
@@ -189,14 +189,38 @@ class TestBaselineModel:
assert mean == 500.0
assert n == 1
def test_lookup_falls_back_to_global(self):
def test_an_unobserved_family_is_not_scored_against_the_global_mean(self):
# The baseline is seeded once, from what the user ran before installing.
# Every family adopted later lands here, and the all-requests mean is
# not a control for any of them.
m = BaselineModel()
m.observe("opus|a|s|tools", 100)
m.observe("sonnet|b|m|notools", 300)
mean, _, n = m.lookup("gpt|totally|xl|tools")
assert m.lookup("gpt|totally|xl|tools") == (0.0, 0.0, 0)
def test_the_global_mean_remains_available_on_request(self):
m = BaselineModel()
m.observe("opus|a|s|tools", 100)
m.observe("sonnet|b|m|notools", 300)
mean, _, n = m.lookup("gpt|totally|xl|tools", fall_back_to_global=True)
assert mean == 200.0 # global mean of 100 and 300
assert n == 2
def test_prefix_backoff_merges_every_neighbour_not_the_first_one_hashed(self):
# Taking the first matching stratum in dict order made the answer
# depend on insertion order, so a round-tripped ledger could score the
# same request differently.
m = BaselineModel()
m.observe("opus|ask|l|tools", 1000)
m.observe("opus|ask|l|notools", 100)
mean, _, n = m.lookup("opus|ask|l|other")
assert (mean, n) == (550.0, 2)
reversed_order = BaselineModel()
reversed_order.observe("opus|ask|l|notools", 100)
reversed_order.observe("opus|ask|l|tools", 1000)
assert reversed_order.lookup("opus|ask|l|other") == m.lookup("opus|ask|l|other")
def test_roundtrip_serialization(self):
m = BaselineModel()
for v in (10, 20, 30):
@@ -288,6 +312,28 @@ class TestEstimateFromBaseline:
# ---------------------------------------------------------------------------
class TestEstimateExcludesUnobservedStrata:
def test_requests_without_baseline_evidence_are_left_out(self):
ledger = SavingsLedger()
for _ in range(10):
ledger.baseline.observe("opus|ask|l|tools", 1000)
ledger.record("treatment", "opus|ask|l|tools", 800)
# A family the baseline never saw, with far shorter replies. Scoring it
# against the global mean would credit ~950 saved tokens per request.
for _ in range(40):
ledger.record("treatment", "sonnet|new_user_ask|m|notools", 50)
est = ledger.estimate_from_baseline()
assert est.n_requests == 10, "only the observed stratum is scored"
assert abs(est.tokens_saved - 2000) < 1e-6 # 10 * (1000 - 800)
assert abs(est.pct - 20.0) < 1e-6
def test_per_request_savings_are_zero_without_evidence(self):
ledger = SavingsLedger()
ledger.baseline.observe("opus|ask|l|tools", 1000)
assert ledger.baseline.lookup("sonnet|new_user_ask|m|notools") == (0.0, 0.0, 0)
class TestEstimateFromHoldout:
def test_none_without_control_data(self):
ledger = SavingsLedger()