Files
headroom/tests/test_cache/test_ttl_estimator.py
Yuchuan He 93df7ae4f7 fix(cache): exclude cold starts from TTL hit evidence (#4728)
## Description

`PrefixCacheTracker.classify_cache_miss()` deliberately returns
`is_miss=False, reason="cold_start"` when no prefix was cached on the
previous turn. The TTL estimator treated every falsey `is_miss` value as
a hit. Three cold starts and three later expiry misses could therefore
produce a learned TTL with zero actual hit observations. A long cold
start could also suppress a valid estimate by raising its supposed life
bound.

Count life evidence only for explicit `reason="hit", is_miss=False`
rows, and accept expiry evidence only with a boolean miss outcome. The
evidence floors, per-model grouping and conservative expiry rounding are
unchanged. This follows the recording and upper-bound fixes in #4711 and
#4710 by checking what the recorded outcomes actually prove.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Distinguish cold starts from positive cache hits when collecting TTL
bounds.
- Ignore malformed/ambiguous hit outcomes instead of letting them
satisfy the sample floor.
- Cover actual tracker cold starts, mixed sample floors, long cold
starts alongside genuine hits, and non-hit/malformed controls.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Owner test file with new regressions on original production code: 9 failed, 28 passed
pytest tests/test_cache tests/test_cold_prefix.py: 369 passed, 3 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed (1924 files)
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: built, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The three cache skips require optional spaCy/embedding models.
Critical-suite skips require optional relevance models; the existing
token-count approximation warning remains in that log.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development venv;
clean baseline `1da8930ab47a2fc5339ded97e5a41bfe6fc419eb`, patch
`be03ab5a8ff74f18c5b90232f6d5874e1af8d9ec`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH=.` for the selected checkout. Real local files, public APIs
and controlled idle inputs; no provider credentials.
- Exact command / steps: Run `python proof.py` from either checkout with
the following identical script. Both before and after runs were outside
pytest.

```python
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile

from headroom.cache.prefix_tracker import PrefixCacheTracker
from headroom.cache.ttl_observations import record_cache_observation

with tempfile.TemporaryDirectory() as directory:
    obs = Path(directory) / "observations.jsonl"
    out = Path(directory) / "learned.json"
    os.environ["HEADROOM_CACHE_TTL_LEARN"] = "1"
    os.environ["HEADROOM_STATELESS"] = "0"
    os.environ["HEADROOM_CACHE_TTL_OBS_PATH"] = str(obs)
    for idle in (120, 200, 240):
        tracker = PrefixCacheTracker("openai")
        attr = tracker.classify_cache_miss(0, [], idle_seconds=idle)
        record_cache_observation(provider="openai", model="fixture", attribution=attr)
    tracker = PrefixCacheTracker("openai")
    tracker.update_from_response(cache_read_tokens=0, cache_write_tokens=1000, messages=[])
    for idle in (600, 700, 800):
        attr = tracker.classify_cache_miss(0, [], idle_seconds=idle)
        record_cache_observation(provider="openai", model="fixture", attribution=attr)
    result = subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs", str(obs), "--out", str(out)], capture_output=True, text=True, check=True)
    rows = [json.loads(line) for line in obs.read_text(encoding="utf-8").splitlines()]
    print(json.dumps({"reasons": [r["reason"] for r in rows],
        "actual_hit_rows": sum(r["reason"] == "hit" for r in rows),
        "learned_file_written": out.exists(),
        "learned_table": json.loads(out.read_text()) if out.exists() else None,
        "cli_output": result.stdout.strip()}))
```

- Observed result: The real tracker and recorder produce three
`cold_start` and three `ttl_expiry` rows. Before the fix the CLI wrote a
600-second estimate claiming three hits; after it reports no estimable
keys and creates no learned file. This uses controlled idle inputs to
verify evidence classification.

```text
Before: {"reasons": ["cold_start", "cold_start", "cold_start", "ttl_expiry", "ttl_expiry", "ttl_expiry"], "actual_hit_rows": 0, "learned_file_written": true, "learned_table": {"openai/fixture": {"hits": 3, "max_hit_idle": 240, "ttl_expiry_misses": 3, "ttl_seconds": 600, "updated_at": 1794297956.945}}, "cli_output": "headroom-cache-ttl: wrote 1 estimate(s) (1 total) to C:\\Users\\ASUS\\AppData\\Local\\Temp\\tmp5ol13684\\learned.json"}
After:  {"reasons": ["cold_start", "cold_start", "cold_start", "ttl_expiry", "ttl_expiry", "ttl_expiry"], "actual_hit_rows": 0, "learned_file_written": false, "learned_table": null, "cli_output": "headroom-cache-ttl: no estimable keys in C:\\Users\\ASUS\\AppData\\Local\\Temp\\tmpnnetshza\\observations.jsonl (6 rows); nothing written"}
```

- Not tested: Live proxy/provider traffic, measured provider TTLs,
billing or model quality, and the complete Python suite. The runtime
proof verifies the local classification/file/CLI/decision behavior
above; hosted CI is tracked separately.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing optional TTL-learning/cold-prefix
behavior; no new flag.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Cold starts no longer create or
inflate a learned life bound. Valid explicit hit/expiry observations
retain their estimates; existing learned-table merge behavior is
unchanged.
- Kill switch / disable path: Do not run the optional estimator, or
remove the learned table to use the existing provider/config fallback.
- Unsafe override required: No.
- Qualification impact: Owner cache/cold-prefix tests and original local
gates passed; hosted checks provide remote qualification.
- Rollback path: Revert this commit. No dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Implementation and local validation were completed with Codex
assistance. The author reviewed the code and validation record.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4728"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
2026-11-10 09:09:55 -06:00

353 lines
16 KiB
Python

"""Tests for the offline cache-TTL estimator (`headroom-cache-ttl`)."""
from __future__ import annotations
import json
import pytest
from headroom.cache.ttl_estimator import (
estimate_ttls,
main,
write_learned,
)
from headroom.cache.ttl_observations import resolve_learned_ttl
def _row(
provider: str = "openai",
model: str = "gpt-5.5",
*,
idle: float,
hit: bool,
reason: str | None = None,
) -> dict:
return {
"ts": 1000.0,
"provider": provider,
"model": model,
"reason": reason if reason is not None else ("hit" if hit else "ttl_expiry"),
"idle_seconds": idle,
"ttl_assumed": 300,
"is_miss": not hit,
"cache_read": 1000 if hit else 0,
"expected_cached": 1000,
}
def _corpus(hit_idles: list[float], expiry_idles: list[float]) -> list[dict]:
return [_row(idle=i, hit=True) for i in hit_idles] + [
_row(idle=i, hit=False) for i in expiry_idles
]
def _legacy_aggregate(ttl: int) -> dict:
# The exact record shape the earlier per-provider estimator wrote under a
# bare provider key; `max_hit_idle` is what marks it as estimator output.
return {
"ttl_seconds": ttl,
"max_hit_idle": ttl // 2,
"hits": 5,
"ttl_expiry_misses": 4,
"updated_at": 1000.0,
}
class TestEstimateTtls:
def test_cold_starts_do_not_supply_hit_evidence(self):
# The real tracker reports cold_start with is_miss=False: nothing was
# cached last turn, so this is neither a miss nor proof of a live cache.
from headroom.cache.prefix_tracker import PrefixCacheTracker
tracker = PrefixCacheTracker("openai")
rows = []
for idle in (120, 200, 240):
attribution = tracker.classify_cache_miss(0, [], idle_seconds=idle)
assert attribution.reason == "cold_start" and not attribution.is_miss
rows.append(_row(idle=idle, hit=not attribution.is_miss, reason=attribution.reason))
rows += _corpus([], [600, 700, 800])
assert estimate_ttls(rows) == {}
def test_cold_starts_cannot_top_up_the_hit_sample_floor(self):
rows = _corpus([120, 240], [600, 700, 800])
rows.append(_row(idle=180, hit=True, reason="cold_start"))
assert estimate_ttls(rows) == {}
def test_long_cold_start_does_not_invalidate_real_hit_bounds(self):
rows = _corpus([120, 200, 240], [600, 700, 800])
rows.append(_row(idle=3600, hit=True, reason="cold_start"))
estimate = estimate_ttls(rows)["openai/gpt-5.5"]
assert estimate["ttl_seconds"] == 600
assert estimate["hits"] == 3
assert estimate["max_hit_idle"] == 240
@pytest.mark.parametrize("reason", [None, "unknown", "prefix_change", "ttl_expiry"])
def test_non_hit_reason_is_not_life_evidence(self, reason):
rows = _corpus([120, 240], [600, 700, 800])
row = _row(idle=180, hit=True)
row["reason"] = reason
rows.append(row)
assert estimate_ttls(rows) == {}
@pytest.mark.parametrize("is_miss", [None, 0, "false"])
def test_malformed_outcome_is_not_life_evidence(self, is_miss):
rows = _corpus([120, 240], [600, 700, 800])
row = _row(idle=180, hit=True)
row["is_miss"] = is_miss
rows.append(row)
assert estimate_ttls(rows) == {}
@pytest.mark.parametrize(
("hit_idles", "expiry_idles", "expected"),
[
([300.1, 300.2, 300.4], [300.8, 301.4, 302.5], 301),
([0.1, 0.2, 0.4], [0.8, 1.4, 2.5], 1),
([120, 300, 480], [600, 900, 1200], 600),
],
)
def test_whole_second_ttl_keeps_the_observed_upper_bound(
self, hit_idles, expiry_idles, expected
):
estimate = estimate_ttls(_corpus(hit_idles, expiry_idles))["openai/gpt-5.5"]
assert estimate["ttl_seconds"] >= min(expiry_idles)
assert estimate["ttl_seconds"] > max(hit_idles)
assert estimate["ttl_seconds"] == expected
def test_upper_bound_of_interval_is_emitted(self):
# Alive at up to 480s, first observed death beyond that at 600s
# -> TTL estimate is the safe upper end: 600.
table = estimate_ttls(_corpus([120, 300, 480], [600, 900, 1200]))
assert table["openai/gpt-5.5"]["ttl_seconds"] == 600
assert table["openai/gpt-5.5"]["max_hit_idle"] == 480
assert "openai" not in table # provider aggregates are never emitted
def test_death_at_or_below_max_hit_idle_is_not_an_upper_bound(self):
# Deaths at 400/450 sit below a hit at 480 (variable eviction);
# only the 700s death is beyond every observed life.
table = estimate_ttls(_corpus([120, 300, 480], [400, 450, 700]))
assert table["openai/gpt-5.5"]["ttl_seconds"] == 700
def test_no_death_beyond_life_skips_key(self):
# Every observed death overlaps the life range: no safe upper end.
table = estimate_ttls(_corpus([120, 480, 900], [400, 450, 600]))
assert table == {}
def test_insufficient_samples_skip_key(self):
assert estimate_ttls(_corpus([480], [600, 700, 800])) == {} # 1 hit < 3
assert estimate_ttls(_corpus([100, 200, 480], [600])) == {} # 1 expiry < 3
# Thresholds are tunable.
table = estimate_ttls(_corpus([480], [600]), min_hits=1, min_expiry_misses=1)
assert table["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_sub_one_floors_do_not_disable_the_evidence_requirement(self):
# An interval needs one life and one death sample to exist; floors below
# 1 must not let a key through to max()/min() on an empty list.
rows = [_row(idle=i, hit=False) for i in (600, 700, 800)] # deaths only
assert estimate_ttls(rows, min_hits=0, min_expiry_misses=0) == {}
assert estimate_ttls(rows, min_hits=-5, min_expiry_misses=-5) == {}
def test_non_ttl_expiry_misses_are_ignored(self):
rows = _corpus([100, 200, 480], [600, 700]) + [
_row(idle=550, hit=False, reason="prefix_change"),
_row(idle=560, hit=False, reason="cold_start"),
]
# prefix_change/cold_start misses say nothing about TTL: still only
# 2 ttl_expiry rows -> below the min_expiry_misses=3 floor.
assert estimate_ttls(rows) == {}
def test_zero_idle_and_malformed_rows_are_ignored(self):
rows = _corpus([100, 200, 480], [600, 700, 800]) + [
_row(idle=0.0, hit=True),
{"provider": "", "idle_seconds": 50, "is_miss": False},
{"provider": "openai", "model": "gpt-5.5", "idle_seconds": "nan-ish"},
{"provider": "openai", "idle_seconds": 50, "is_miss": False}, # no model
"not a dict", # type: ignore[list-item]
]
assert estimate_ttls(rows)["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_non_finite_idles_are_ignored(self):
# NaN/±Inf pass a bare `idle <= 0` (NaN comparisons are all False), then
# poison max()/min() and raise ValueError/OverflowError in int() — one
# such row must not take the whole batch down with it.
rows = _corpus([100, 200, 480], [600, 700, 800]) + [
_row(idle=float(v), hit=h) for v in ("nan", "inf", "-inf") for h in (True, False)
]
assert estimate_ttls(rows)["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_one_models_death_cannot_close_anothers_life_interval(self):
# gpt-5.5 is alive at up to 1000s and never observed dead; mini dies at
# 1200s+. Pooled per-provider these would emit ttl=1200, which the
# consumer would then apply to gpt-5.5 (real TTL far higher) and to any
# unseen model, recompacting a still-warm prefix. Only mini is estimable.
rows = (
[_row(idle=i, hit=True) for i in (900, 950, 1000)]
+ [_row(model="gpt-5.5-mini", idle=i, hit=True) for i in (100, 200, 300)]
+ [_row(model="gpt-5.5-mini", idle=i, hit=False) for i in (1200, 1300, 1400)]
)
table = estimate_ttls(rows)
assert table["openai/gpt-5.5-mini"]["ttl_seconds"] == 1200
assert "openai/gpt-5.5" not in table # no death evidence of its own
assert "openai" not in table
class TestWriteLearned:
def test_merge_preserves_existing_keys(self, tmp_path):
out = tmp_path / "learned.json"
out.write_text(json.dumps({"kimi/k2": {"ttl_seconds": 900}}))
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
data = json.loads(out.read_text())
assert data["kimi/k2"]["ttl_seconds"] == 900
assert data["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_legacy_bare_provider_keys_are_purged_on_write(self, tmp_path):
# Files written by the earlier per-provider version of this tool carry
# exactly the unsound cross-model aggregates the estimator no longer
# emits; a merge that preserves them would keep feeding the consumer's
# provider fallback forever. They are recognized by the estimator's own
# metadata shape — a bare-provider key withOUT it is an operator-written
# fallback and must survive the purge.
out = tmp_path / "learned.json"
out.write_text(
json.dumps({"openai": _legacy_aggregate(1200), "kimi/k2": {"ttl_seconds": 900}})
)
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
data = json.loads(out.read_text())
assert "openai" not in data
assert data["kimi/k2"]["ttl_seconds"] == 900
assert data["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_manual_bare_provider_fallbacks_survive_the_purge(self, tmp_path):
# Both hand-written forms resolve_learned_ttl accepts for a bare
# provider key: a {"ttl_seconds": N} dict and a plain number.
out = tmp_path / "learned.json"
out.write_text(
json.dumps(
{"openai": {"ttl_seconds": 3600}, "kimi": 900, "codex": _legacy_aggregate(1200)}
)
)
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
data = json.loads(out.read_text())
assert data["openai"] == {"ttl_seconds": 3600}
assert data["kimi"] == 900
assert "codex" not in data
def test_corrupt_existing_file_is_replaced_not_fatal(self, tmp_path):
out = tmp_path / "learned.json"
out.write_text("{not json")
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_non_dict_existing_file_is_replaced_not_merged(self, tmp_path):
out = tmp_path / "learned.json"
out.write_text("[1, 2]")
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
assert json.loads(out.read_text()) == {"openai/gpt-5.5": {"ttl_seconds": 600}}
class TestMain:
def test_fractional_upper_bound_survives_cli_and_resolver(self, paths):
obs, out = paths
rows = _corpus([300.1, 300.2, 300.4], [300.8, 301.4, 302.5])
obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8")
assert main([]) == 0
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 301
assert resolve_learned_ttl("openai", "gpt-5.5") == 301
@pytest.fixture()
def paths(self, tmp_path, monkeypatch):
obs = tmp_path / "obs.jsonl"
out = tmp_path / "learned.json"
monkeypatch.setenv("HEADROOM_CACHE_TTL_OBS_PATH", str(obs))
monkeypatch.setenv("HEADROOM_CACHE_TTL_LEARNED_PATH", str(out))
return obs, out
def test_end_to_end_resolve_learned_ttl_reads_output(self, paths):
obs, out = paths
rows = _corpus([120, 300, 480], [600, 900])
rows += [_row(idle=650, hit=False)]
# Trailing junk the parser must skip: invalid JSON, a blank line,
# and valid JSON that is not an object.
obs.write_text("\n".join(json.dumps(r) for r in rows) + '\nnot-json\n\n[1, 2]\n"scalar"\n')
assert main([]) == 0
assert resolve_learned_ttl("openai", "gpt-5.5") == 600
# An unseen model gets nothing rather than gpt-5.5's bound: the consumer
# falls back to its own static default instead of a TTL learned from a
# cache population it has no evidence about.
assert resolve_learned_ttl("openai", "other-model") is None
def test_upgrade_purges_legacy_provider_aggregate(self, paths):
# Upgrade regression: a learned file from the earlier per-provider
# version must not keep serving its aggregate to unseen models after
# a run of the model-scoped estimator — while an operator-written
# provider fallback in the same file keeps resolving.
obs, out = paths
out.write_text(
json.dumps({"openai": _legacy_aggregate(1200), "kimi": {"ttl_seconds": 3600}})
)
obs.write_text("\n".join(json.dumps(r) for r in _corpus([120, 300, 480], [600, 900, 1200])))
assert main([]) == 0
data = json.loads(out.read_text())
assert "openai" not in data
assert data["openai/gpt-5.5"]["ttl_seconds"] == 600
assert resolve_learned_ttl("openai", "unseen-model") is None
assert resolve_learned_ttl("kimi", "unseen-model") == 3600
def test_upgrade_purge_runs_even_with_no_estimable_data(self, paths, capsys):
# The purge must not depend on today's window producing an estimate.
obs, out = paths
out.write_text(json.dumps({"openai": _legacy_aggregate(1200)}))
obs.write_text(json.dumps(_row(idle=120, hit=True)) + "\n")
assert main([]) == 0
assert json.loads(out.read_text()) == {}
assert resolve_learned_ttl("openai", "unseen-model") is None
def test_non_finite_jsonl_row_does_not_abort_the_run(self, paths):
obs, out = paths
# json.dumps emits bare NaN/Infinity and json.loads reads them back, so
# the recorder can put one in the log without any hand-editing.
# First row on purpose: max() keeps a leading NaN (every later `x > nan`
# is False), so this is the ordering that takes the run down.
rows = [_row(idle=float("nan"), hit=True)] + _corpus([120, 300, 480], [600, 900, 1200])
obs.write_text("\n".join(json.dumps(r) for r in rows) + "\n")
assert "NaN" in obs.read_text()
assert main([]) == 0
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_sub_one_sample_floors_are_rejected(self, paths):
with pytest.raises(SystemExit):
main(["--min-hits", "0"])
with pytest.raises(SystemExit):
main(["--min-expiry-misses", "-1"])
def test_no_estimable_data_writes_nothing(self, paths, capsys):
obs, out = paths
obs.write_text(json.dumps(_row(idle=120, hit=True)) + "\n")
assert main([]) == 0
assert not out.exists()
assert "nothing written" in capsys.readouterr().out
def test_missing_obs_file_is_not_an_error(self, paths):
assert main([]) == 0
def test_dry_run_prints_without_writing(self, paths, capsys):
obs, out = paths
obs.write_text("\n".join(json.dumps(r) for r in _corpus([120, 300, 480], [600, 900, 1200])))
assert main(["--dry-run"]) == 0
assert not out.exists()
assert json.loads(capsys.readouterr().out)["openai/gpt-5.5"]["ttl_seconds"] == 600
def test_explicit_paths_override_env(self, paths, tmp_path):
obs2 = tmp_path / "other.jsonl"
out2 = tmp_path / "other.json"
obs2.write_text(
"\n".join(json.dumps(r) for r in _corpus([120, 300, 480], [600, 900, 1200]))
)
assert main(["--obs", str(obs2), "--out", str(out2)]) == 0
assert json.loads(out2.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 600