mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(savings): stop scoring unobserved strata against the global mean (#3128)
## Description
`BaselineModel.lookup` fell back to the all-requests mean whenever
prefix back-off found nothing. That fallback is not a control for
anything.
The baseline is seeded once, by `learn --verbosity` reading session
history that predates the shaper. It therefore only ever covers the
model families the user ran *before* installing. It also cannot be
relearned: every transcript written since is already shaped, so a
re-learn would collapse the baseline onto the treatment mean. Each
family a user adopts later resolves to a single number derived from a
different population, permanently.
Measured on a real ledger (15,658 baseline samples over 16 strata, all
`opus`; 74,243 treatment requests over 83 strata):
| | |
|---|---|
| Requests scored against the global mean | 48% (every fable, sonnet,
haiku, gpt turn) |
| Global mean applied to them | 1,083 output tokens |
| `sonnet\|new_user_ask\|m\|notools` | 4,666 requests, replies average
**73** tokens, each credited ~1,010 saved |
| Share of total reported savings from the fallback | **74%** |
| Reported reduction | 42.6% |
| Reduction over strata the baseline actually observed | 32.1% |
Scoring a short no-tool ask against a long tool-calling turn is not a
synthetic control, it is a unit conversion.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
**Reported savings will drop** for any user whose traffic has moved
beyond the model family their baseline was seeded from. That is the
point of the change, but it is a visible number moving, so flagging it
explicitly.
## Changes Made
- `lookup` returns `(0.0, 0.0, 0)` when neither the exact stratum nor
prefix back-off finds evidence. Both estimators already treat `n == 0`
as "no evidence" and skip the request, so what changes is which requests
qualify, not how they are scored.
- The global fallback stays reachable via `lookup(key,
fall_back_to_global=True)` for callers that want it; nothing in the tree
passes it today.
- `glob` is still observed and still backs `total_samples`, so the
ledger format and the "has a baseline been seeded" check are unaffected.
- Also in this commit: prefix back-off merges every matching neighbour
instead of returning the first in dict order. The old loop made the
answer depend on insertion order, so a round-tripped ledger could score
the same request differently. Happy to split this out if you would
rather review it separately.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_output_savings.py tests/test_verbosity_learn.py tests/test_output_savings_cli.py -q
tests/test_output_savings.py ........................................... [ 66%]
tests/test_verbosity_learn.py ................... [ 95%]
tests/test_output_savings_cli.py ... [100%]
65 passed in 1.12s
$ uvx ruff check headroom/proxy/output_savings.py tests/test_output_savings.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/proxy/output_savings.py
Success: no issues found in 1 source file
```
`test_lookup_falls_back_to_global` is rewritten into two tests: one
pinning the new default, one pinning that the fallback still works when
asked for.
## Real Behavior Proof
- Environment: macOS 24.6.0, Python 3.10.18, uv, upstream/main at
17522fb0. Ledger from a shipped desktop build,
`~/.headroom/output_savings.json`, 7.4KB.
- Exact command / steps: reimplemented both back-off policies against
that ledger and recomputed `estimate_from_baseline` under each.
Confirmed the current-policy recomputation reproduces what the running
proxy reports on `/stats` before comparing.
- Observed result: current policy reproduces the live `/stats` figures
exactly (`tokens_saved` 31,493,556, `baseline_tokens` 73,986,108, 42.6%,
74,297 requests), which validates the recomputation. Under the new
policy the same ledger gives 11,540,868 / 35,961,142 / 32.1% over 38,729
requests. Splitting the current-policy total by how each stratum
resolved: 8.29M from exact stratum hits, 2.76M from prefix back-off,
20.43M from the global mean -- 74% of the reported savings resting on
the fallback this PR removes.
- Not tested: one user's ledger. A baseline seeded from a corpus
spanning several model families would lose less; one seeded before a
model switch would lose more.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. Baseline lookup is not behind a
rollout gate.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes, and visibly. Reported savings
drop for any user whose traffic has moved past the model family their
baseline was seeded from: 42.6% -> 32.1% on the real ledger measured
above. Requests with no baseline evidence are now skipped rather than
scored against an unrelated mean, so `n_requests` in the estimate drops
with it. That is the fix, but it is a number users can see moving.
- Kill switch / disable path: No runtime flag. `lookup(key,
fall_back_to_global=True)` restores the previous behavior at the call
site, and `glob` is still accumulated, so the old number remains
computable from the same ledger.
- Unsafe override required: No
- Qualification impact: `test_lookup_falls_back_to_global` is split into
two tests, one pinning the new default and one pinning the opt-in
fallback. No assertion was relaxed.
- Rollback path: Revert the commit. The ledger format is unchanged and
every estimate is recomputed from it on read, so the previous figures
return immediately with no migration or backfill.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
76ef2c3bc9
commit
f8645257c0
@@ -229,12 +229,23 @@ class BaselineModel:
|
||||
self.strata.setdefault(key, _Accum()).merge(acc)
|
||||
self.glob.merge(other.glob)
|
||||
|
||||
def lookup(self, key: str) -> tuple[float, float, int]:
|
||||
def lookup(self, key: str, *, fall_back_to_global: bool = False) -> tuple[float, float, int]:
|
||||
"""Return ``(mean, var, n)`` for *key* with hierarchical back-off.
|
||||
|
||||
Falls back by trimming trailing (least-specific) stratum fields, then
|
||||
to the global mean. Back-off keeps the estimate defined for strata the
|
||||
baseline never saw, at the cost of specificity.
|
||||
Falls back by trimming trailing (least-specific) stratum fields, so a
|
||||
stratum the baseline never saw is still scored against its nearest
|
||||
observed neighbours. Returns ``(0.0, 0.0, 0)`` when even that finds
|
||||
nothing -- callers already treat ``n == 0`` as "no evidence".
|
||||
|
||||
``fall_back_to_global`` restores the old last resort of the
|
||||
all-requests mean. It is off by default because that mean is not a
|
||||
control for anything: the baseline is seeded once, from whatever the
|
||||
user ran *before* installing, so every model family they adopt later
|
||||
resolves to it. On a real ledger that meant 48% of requests -- sonnet
|
||||
and fable turns whose replies average 43-770 tokens -- being scored
|
||||
against one opus-derived mean of 1,083, which alone produced 74% of the
|
||||
reported savings. Scoring a short no-tool ask against a long
|
||||
tool-calling turn is not a synthetic control, it is a unit conversion.
|
||||
"""
|
||||
acc = self.strata.get(key)
|
||||
if acc is not None and acc.n > 0:
|
||||
@@ -242,11 +253,16 @@ class BaselineModel:
|
||||
parts = key.split("|")
|
||||
while len(parts) > 1:
|
||||
parts = parts[:-1]
|
||||
prefix = "|".join(parts)
|
||||
prefix = "|".join(parts) + "|"
|
||||
neighbours = _Accum()
|
||||
for k, a in self.strata.items():
|
||||
if k.startswith(prefix + "|") and a.n > 0:
|
||||
return a.mean, a.var, a.n
|
||||
return self.glob.mean, self.glob.var, self.glob.n
|
||||
if a.n > 0 and k.startswith(prefix):
|
||||
neighbours.merge(a)
|
||||
if neighbours.n > 0:
|
||||
return neighbours.mean, neighbours.var, neighbours.n
|
||||
if fall_back_to_global:
|
||||
return self.glob.mean, self.glob.var, self.glob.n
|
||||
return 0.0, 0.0, 0
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
|
||||
@@ -189,14 +189,38 @@ class TestBaselineModel:
|
||||
assert mean == 500.0
|
||||
assert n == 1
|
||||
|
||||
def test_lookup_falls_back_to_global(self):
|
||||
def test_an_unobserved_family_is_not_scored_against_the_global_mean(self):
|
||||
# The baseline is seeded once, from what the user ran before installing.
|
||||
# Every family adopted later lands here, and the all-requests mean is
|
||||
# not a control for any of them.
|
||||
m = BaselineModel()
|
||||
m.observe("opus|a|s|tools", 100)
|
||||
m.observe("sonnet|b|m|notools", 300)
|
||||
mean, _, n = m.lookup("gpt|totally|xl|tools")
|
||||
assert m.lookup("gpt|totally|xl|tools") == (0.0, 0.0, 0)
|
||||
|
||||
def test_the_global_mean_remains_available_on_request(self):
|
||||
m = BaselineModel()
|
||||
m.observe("opus|a|s|tools", 100)
|
||||
m.observe("sonnet|b|m|notools", 300)
|
||||
mean, _, n = m.lookup("gpt|totally|xl|tools", fall_back_to_global=True)
|
||||
assert mean == 200.0 # global mean of 100 and 300
|
||||
assert n == 2
|
||||
|
||||
def test_prefix_backoff_merges_every_neighbour_not_the_first_one_hashed(self):
|
||||
# Taking the first matching stratum in dict order made the answer
|
||||
# depend on insertion order, so a round-tripped ledger could score the
|
||||
# same request differently.
|
||||
m = BaselineModel()
|
||||
m.observe("opus|ask|l|tools", 1000)
|
||||
m.observe("opus|ask|l|notools", 100)
|
||||
mean, _, n = m.lookup("opus|ask|l|other")
|
||||
assert (mean, n) == (550.0, 2)
|
||||
|
||||
reversed_order = BaselineModel()
|
||||
reversed_order.observe("opus|ask|l|notools", 100)
|
||||
reversed_order.observe("opus|ask|l|tools", 1000)
|
||||
assert reversed_order.lookup("opus|ask|l|other") == m.lookup("opus|ask|l|other")
|
||||
|
||||
def test_roundtrip_serialization(self):
|
||||
m = BaselineModel()
|
||||
for v in (10, 20, 30):
|
||||
@@ -288,6 +312,28 @@ class TestEstimateFromBaseline:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestEstimateExcludesUnobservedStrata:
|
||||
def test_requests_without_baseline_evidence_are_left_out(self):
|
||||
ledger = SavingsLedger()
|
||||
for _ in range(10):
|
||||
ledger.baseline.observe("opus|ask|l|tools", 1000)
|
||||
ledger.record("treatment", "opus|ask|l|tools", 800)
|
||||
# A family the baseline never saw, with far shorter replies. Scoring it
|
||||
# against the global mean would credit ~950 saved tokens per request.
|
||||
for _ in range(40):
|
||||
ledger.record("treatment", "sonnet|new_user_ask|m|notools", 50)
|
||||
|
||||
est = ledger.estimate_from_baseline()
|
||||
assert est.n_requests == 10, "only the observed stratum is scored"
|
||||
assert abs(est.tokens_saved - 2000) < 1e-6 # 10 * (1000 - 800)
|
||||
assert abs(est.pct - 20.0) < 1e-6
|
||||
|
||||
def test_per_request_savings_are_zero_without_evidence(self):
|
||||
ledger = SavingsLedger()
|
||||
ledger.baseline.observe("opus|ask|l|tools", 1000)
|
||||
assert ledger.baseline.lookup("sonnet|new_user_ask|m|notools") == (0.0, 0.0, 0)
|
||||
|
||||
|
||||
class TestEstimateFromHoldout:
|
||||
def test_none_without_control_data(self):
|
||||
ledger = SavingsLedger()
|
||||
|
||||
Reference in New Issue
Block a user