mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description `PrefixCacheTracker.classify_cache_miss()` deliberately returns `is_miss=False, reason="cold_start"` when no prefix was cached on the previous turn. The TTL estimator treated every falsey `is_miss` value as a hit. Three cold starts and three later expiry misses could therefore produce a learned TTL with zero actual hit observations. A long cold start could also suppress a valid estimate by raising its supposed life bound. Count life evidence only for explicit `reason="hit", is_miss=False` rows, and accept expiry evidence only with a boolean miss outcome. The evidence floors, per-model grouping and conservative expiry rounding are unchanged. This follows the recording and upper-bound fixes in #4711 and #4710 by checking what the recorded outcomes actually prove. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Distinguish cold starts from positive cache hits when collecting TTL bounds. - Ignore malformed/ambiguous hit outcomes instead of letting them satisfy the sample floor. - Cover actual tracker cold starts, mixed sample floors, long cold starts alongside genuine hits, and non-hit/malformed controls. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Owner test file with new regressions on original production code: 9 failed, 28 passed pytest tests/test_cache tests/test_cold_prefix.py: 369 passed, 3 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed (1924 files) Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: built, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The three cache skips require optional spaCy/embedding models. Critical-suite skips require optional relevance models; the existing token-count approximation warning remains in that log. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development venv; clean baseline `1da8930ab47a2fc5339ded97e5a41bfe6fc419eb`, patch `be03ab5a8ff74f18c5b90232f6d5874e1af8d9ec`. `HEADROOM_OFFLINE=1`, `PYTHONPATH=.` for the selected checkout. Real local files, public APIs and controlled idle inputs; no provider credentials. - Exact command / steps: Run `python proof.py` from either checkout with the following identical script. Both before and after runs were outside pytest. ```python import json import os from pathlib import Path import subprocess import sys import tempfile from headroom.cache.prefix_tracker import PrefixCacheTracker from headroom.cache.ttl_observations import record_cache_observation with tempfile.TemporaryDirectory() as directory: obs = Path(directory) / "observations.jsonl" out = Path(directory) / "learned.json" os.environ["HEADROOM_CACHE_TTL_LEARN"] = "1" os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_CACHE_TTL_OBS_PATH"] = str(obs) for idle in (120, 200, 240): tracker = PrefixCacheTracker("openai") attr = tracker.classify_cache_miss(0, [], idle_seconds=idle) record_cache_observation(provider="openai", model="fixture", attribution=attr) tracker = PrefixCacheTracker("openai") tracker.update_from_response(cache_read_tokens=0, cache_write_tokens=1000, messages=[]) for idle in (600, 700, 800): attr = tracker.classify_cache_miss(0, [], idle_seconds=idle) record_cache_observation(provider="openai", model="fixture", attribution=attr) result = subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs", str(obs), "--out", str(out)], capture_output=True, text=True, check=True) rows = [json.loads(line) for line in obs.read_text(encoding="utf-8").splitlines()] print(json.dumps({"reasons": [r["reason"] for r in rows], "actual_hit_rows": sum(r["reason"] == "hit" for r in rows), "learned_file_written": out.exists(), "learned_table": json.loads(out.read_text()) if out.exists() else None, "cli_output": result.stdout.strip()})) ``` - Observed result: The real tracker and recorder produce three `cold_start` and three `ttl_expiry` rows. Before the fix the CLI wrote a 600-second estimate claiming three hits; after it reports no estimable keys and creates no learned file. This uses controlled idle inputs to verify evidence classification. ```text Before: {"reasons": ["cold_start", "cold_start", "cold_start", "ttl_expiry", "ttl_expiry", "ttl_expiry"], "actual_hit_rows": 0, "learned_file_written": true, "learned_table": {"openai/fixture": {"hits": 3, "max_hit_idle": 240, "ttl_expiry_misses": 3, "ttl_seconds": 600, "updated_at": 1794297956.945}}, "cli_output": "headroom-cache-ttl: wrote 1 estimate(s) (1 total) to C:\\Users\\ASUS\\AppData\\Local\\Temp\\tmp5ol13684\\learned.json"} After: {"reasons": ["cold_start", "cold_start", "cold_start", "ttl_expiry", "ttl_expiry", "ttl_expiry"], "actual_hit_rows": 0, "learned_file_written": false, "learned_table": null, "cli_output": "headroom-cache-ttl: no estimable keys in C:\\Users\\ASUS\\AppData\\Local\\Temp\\tmpnnetshza\\observations.jsonl (6 rows); nothing written"} ``` - Not tested: Live proxy/provider traffic, measured provider TTLs, billing or model quality, and the complete Python suite. The runtime proof verifies the local classification/file/CLI/decision behavior above; hosted CI is tracked separately. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing optional TTL-learning/cold-prefix behavior; no new flag. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Cold starts no longer create or inflate a learned life bound. Valid explicit hit/expiry observations retain their estimates; existing learned-table merge behavior is unchanged. - Kill switch / disable path: Do not run the optional estimator, or remove the learned table to use the existing provider/config fallback. - Unsafe override required: No. - Qualification impact: Owner cache/cold-prefix tests and original local gates passed; hosted checks provide remote qualification. - Rollback path: Revert this commit. No dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Implementation and local validation were completed with Codex assistance. The author reviewed the code and validation record. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4728" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end -->
353 lines
16 KiB
Python
353 lines
16 KiB
Python
"""Tests for the offline cache-TTL estimator (`headroom-cache-ttl`)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.ttl_estimator import (
|
|
estimate_ttls,
|
|
main,
|
|
write_learned,
|
|
)
|
|
from headroom.cache.ttl_observations import resolve_learned_ttl
|
|
|
|
|
|
def _row(
|
|
provider: str = "openai",
|
|
model: str = "gpt-5.5",
|
|
*,
|
|
idle: float,
|
|
hit: bool,
|
|
reason: str | None = None,
|
|
) -> dict:
|
|
return {
|
|
"ts": 1000.0,
|
|
"provider": provider,
|
|
"model": model,
|
|
"reason": reason if reason is not None else ("hit" if hit else "ttl_expiry"),
|
|
"idle_seconds": idle,
|
|
"ttl_assumed": 300,
|
|
"is_miss": not hit,
|
|
"cache_read": 1000 if hit else 0,
|
|
"expected_cached": 1000,
|
|
}
|
|
|
|
|
|
def _corpus(hit_idles: list[float], expiry_idles: list[float]) -> list[dict]:
|
|
return [_row(idle=i, hit=True) for i in hit_idles] + [
|
|
_row(idle=i, hit=False) for i in expiry_idles
|
|
]
|
|
|
|
|
|
def _legacy_aggregate(ttl: int) -> dict:
|
|
# The exact record shape the earlier per-provider estimator wrote under a
|
|
# bare provider key; `max_hit_idle` is what marks it as estimator output.
|
|
return {
|
|
"ttl_seconds": ttl,
|
|
"max_hit_idle": ttl // 2,
|
|
"hits": 5,
|
|
"ttl_expiry_misses": 4,
|
|
"updated_at": 1000.0,
|
|
}
|
|
|
|
|
|
class TestEstimateTtls:
|
|
def test_cold_starts_do_not_supply_hit_evidence(self):
|
|
# The real tracker reports cold_start with is_miss=False: nothing was
|
|
# cached last turn, so this is neither a miss nor proof of a live cache.
|
|
from headroom.cache.prefix_tracker import PrefixCacheTracker
|
|
|
|
tracker = PrefixCacheTracker("openai")
|
|
rows = []
|
|
for idle in (120, 200, 240):
|
|
attribution = tracker.classify_cache_miss(0, [], idle_seconds=idle)
|
|
assert attribution.reason == "cold_start" and not attribution.is_miss
|
|
rows.append(_row(idle=idle, hit=not attribution.is_miss, reason=attribution.reason))
|
|
rows += _corpus([], [600, 700, 800])
|
|
assert estimate_ttls(rows) == {}
|
|
|
|
def test_cold_starts_cannot_top_up_the_hit_sample_floor(self):
|
|
rows = _corpus([120, 240], [600, 700, 800])
|
|
rows.append(_row(idle=180, hit=True, reason="cold_start"))
|
|
assert estimate_ttls(rows) == {}
|
|
|
|
def test_long_cold_start_does_not_invalidate_real_hit_bounds(self):
|
|
rows = _corpus([120, 200, 240], [600, 700, 800])
|
|
rows.append(_row(idle=3600, hit=True, reason="cold_start"))
|
|
estimate = estimate_ttls(rows)["openai/gpt-5.5"]
|
|
assert estimate["ttl_seconds"] == 600
|
|
assert estimate["hits"] == 3
|
|
assert estimate["max_hit_idle"] == 240
|
|
|
|
@pytest.mark.parametrize("reason", [None, "unknown", "prefix_change", "ttl_expiry"])
|
|
def test_non_hit_reason_is_not_life_evidence(self, reason):
|
|
rows = _corpus([120, 240], [600, 700, 800])
|
|
row = _row(idle=180, hit=True)
|
|
row["reason"] = reason
|
|
rows.append(row)
|
|
assert estimate_ttls(rows) == {}
|
|
|
|
@pytest.mark.parametrize("is_miss", [None, 0, "false"])
|
|
def test_malformed_outcome_is_not_life_evidence(self, is_miss):
|
|
rows = _corpus([120, 240], [600, 700, 800])
|
|
row = _row(idle=180, hit=True)
|
|
row["is_miss"] = is_miss
|
|
rows.append(row)
|
|
assert estimate_ttls(rows) == {}
|
|
|
|
@pytest.mark.parametrize(
|
|
("hit_idles", "expiry_idles", "expected"),
|
|
[
|
|
([300.1, 300.2, 300.4], [300.8, 301.4, 302.5], 301),
|
|
([0.1, 0.2, 0.4], [0.8, 1.4, 2.5], 1),
|
|
([120, 300, 480], [600, 900, 1200], 600),
|
|
],
|
|
)
|
|
def test_whole_second_ttl_keeps_the_observed_upper_bound(
|
|
self, hit_idles, expiry_idles, expected
|
|
):
|
|
estimate = estimate_ttls(_corpus(hit_idles, expiry_idles))["openai/gpt-5.5"]
|
|
assert estimate["ttl_seconds"] >= min(expiry_idles)
|
|
assert estimate["ttl_seconds"] > max(hit_idles)
|
|
assert estimate["ttl_seconds"] == expected
|
|
|
|
def test_upper_bound_of_interval_is_emitted(self):
|
|
# Alive at up to 480s, first observed death beyond that at 600s
|
|
# -> TTL estimate is the safe upper end: 600.
|
|
table = estimate_ttls(_corpus([120, 300, 480], [600, 900, 1200]))
|
|
assert table["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
assert table["openai/gpt-5.5"]["max_hit_idle"] == 480
|
|
assert "openai" not in table # provider aggregates are never emitted
|
|
|
|
def test_death_at_or_below_max_hit_idle_is_not_an_upper_bound(self):
|
|
# Deaths at 400/450 sit below a hit at 480 (variable eviction);
|
|
# only the 700s death is beyond every observed life.
|
|
table = estimate_ttls(_corpus([120, 300, 480], [400, 450, 700]))
|
|
assert table["openai/gpt-5.5"]["ttl_seconds"] == 700
|
|
|
|
def test_no_death_beyond_life_skips_key(self):
|
|
# Every observed death overlaps the life range: no safe upper end.
|
|
table = estimate_ttls(_corpus([120, 480, 900], [400, 450, 600]))
|
|
assert table == {}
|
|
|
|
def test_insufficient_samples_skip_key(self):
|
|
assert estimate_ttls(_corpus([480], [600, 700, 800])) == {} # 1 hit < 3
|
|
assert estimate_ttls(_corpus([100, 200, 480], [600])) == {} # 1 expiry < 3
|
|
# Thresholds are tunable.
|
|
table = estimate_ttls(_corpus([480], [600]), min_hits=1, min_expiry_misses=1)
|
|
assert table["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_sub_one_floors_do_not_disable_the_evidence_requirement(self):
|
|
# An interval needs one life and one death sample to exist; floors below
|
|
# 1 must not let a key through to max()/min() on an empty list.
|
|
rows = [_row(idle=i, hit=False) for i in (600, 700, 800)] # deaths only
|
|
assert estimate_ttls(rows, min_hits=0, min_expiry_misses=0) == {}
|
|
assert estimate_ttls(rows, min_hits=-5, min_expiry_misses=-5) == {}
|
|
|
|
def test_non_ttl_expiry_misses_are_ignored(self):
|
|
rows = _corpus([100, 200, 480], [600, 700]) + [
|
|
_row(idle=550, hit=False, reason="prefix_change"),
|
|
_row(idle=560, hit=False, reason="cold_start"),
|
|
]
|
|
# prefix_change/cold_start misses say nothing about TTL: still only
|
|
# 2 ttl_expiry rows -> below the min_expiry_misses=3 floor.
|
|
assert estimate_ttls(rows) == {}
|
|
|
|
def test_zero_idle_and_malformed_rows_are_ignored(self):
|
|
rows = _corpus([100, 200, 480], [600, 700, 800]) + [
|
|
_row(idle=0.0, hit=True),
|
|
{"provider": "", "idle_seconds": 50, "is_miss": False},
|
|
{"provider": "openai", "model": "gpt-5.5", "idle_seconds": "nan-ish"},
|
|
{"provider": "openai", "idle_seconds": 50, "is_miss": False}, # no model
|
|
"not a dict", # type: ignore[list-item]
|
|
]
|
|
assert estimate_ttls(rows)["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_non_finite_idles_are_ignored(self):
|
|
# NaN/±Inf pass a bare `idle <= 0` (NaN comparisons are all False), then
|
|
# poison max()/min() and raise ValueError/OverflowError in int() — one
|
|
# such row must not take the whole batch down with it.
|
|
rows = _corpus([100, 200, 480], [600, 700, 800]) + [
|
|
_row(idle=float(v), hit=h) for v in ("nan", "inf", "-inf") for h in (True, False)
|
|
]
|
|
assert estimate_ttls(rows)["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_one_models_death_cannot_close_anothers_life_interval(self):
|
|
# gpt-5.5 is alive at up to 1000s and never observed dead; mini dies at
|
|
# 1200s+. Pooled per-provider these would emit ttl=1200, which the
|
|
# consumer would then apply to gpt-5.5 (real TTL far higher) and to any
|
|
# unseen model, recompacting a still-warm prefix. Only mini is estimable.
|
|
rows = (
|
|
[_row(idle=i, hit=True) for i in (900, 950, 1000)]
|
|
+ [_row(model="gpt-5.5-mini", idle=i, hit=True) for i in (100, 200, 300)]
|
|
+ [_row(model="gpt-5.5-mini", idle=i, hit=False) for i in (1200, 1300, 1400)]
|
|
)
|
|
table = estimate_ttls(rows)
|
|
assert table["openai/gpt-5.5-mini"]["ttl_seconds"] == 1200
|
|
assert "openai/gpt-5.5" not in table # no death evidence of its own
|
|
assert "openai" not in table
|
|
|
|
|
|
class TestWriteLearned:
|
|
def test_merge_preserves_existing_keys(self, tmp_path):
|
|
out = tmp_path / "learned.json"
|
|
out.write_text(json.dumps({"kimi/k2": {"ttl_seconds": 900}}))
|
|
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
|
|
data = json.loads(out.read_text())
|
|
assert data["kimi/k2"]["ttl_seconds"] == 900
|
|
assert data["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_legacy_bare_provider_keys_are_purged_on_write(self, tmp_path):
|
|
# Files written by the earlier per-provider version of this tool carry
|
|
# exactly the unsound cross-model aggregates the estimator no longer
|
|
# emits; a merge that preserves them would keep feeding the consumer's
|
|
# provider fallback forever. They are recognized by the estimator's own
|
|
# metadata shape — a bare-provider key withOUT it is an operator-written
|
|
# fallback and must survive the purge.
|
|
out = tmp_path / "learned.json"
|
|
out.write_text(
|
|
json.dumps({"openai": _legacy_aggregate(1200), "kimi/k2": {"ttl_seconds": 900}})
|
|
)
|
|
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
|
|
data = json.loads(out.read_text())
|
|
assert "openai" not in data
|
|
assert data["kimi/k2"]["ttl_seconds"] == 900
|
|
assert data["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_manual_bare_provider_fallbacks_survive_the_purge(self, tmp_path):
|
|
# Both hand-written forms resolve_learned_ttl accepts for a bare
|
|
# provider key: a {"ttl_seconds": N} dict and a plain number.
|
|
out = tmp_path / "learned.json"
|
|
out.write_text(
|
|
json.dumps(
|
|
{"openai": {"ttl_seconds": 3600}, "kimi": 900, "codex": _legacy_aggregate(1200)}
|
|
)
|
|
)
|
|
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
|
|
data = json.loads(out.read_text())
|
|
assert data["openai"] == {"ttl_seconds": 3600}
|
|
assert data["kimi"] == 900
|
|
assert "codex" not in data
|
|
|
|
def test_corrupt_existing_file_is_replaced_not_fatal(self, tmp_path):
|
|
out = tmp_path / "learned.json"
|
|
out.write_text("{not json")
|
|
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
|
|
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_non_dict_existing_file_is_replaced_not_merged(self, tmp_path):
|
|
out = tmp_path / "learned.json"
|
|
out.write_text("[1, 2]")
|
|
write_learned({"openai/gpt-5.5": {"ttl_seconds": 600}}, str(out))
|
|
assert json.loads(out.read_text()) == {"openai/gpt-5.5": {"ttl_seconds": 600}}
|
|
|
|
|
|
class TestMain:
|
|
def test_fractional_upper_bound_survives_cli_and_resolver(self, paths):
|
|
obs, out = paths
|
|
rows = _corpus([300.1, 300.2, 300.4], [300.8, 301.4, 302.5])
|
|
obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8")
|
|
|
|
assert main([]) == 0
|
|
|
|
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 301
|
|
assert resolve_learned_ttl("openai", "gpt-5.5") == 301
|
|
|
|
@pytest.fixture()
|
|
def paths(self, tmp_path, monkeypatch):
|
|
obs = tmp_path / "obs.jsonl"
|
|
out = tmp_path / "learned.json"
|
|
monkeypatch.setenv("HEADROOM_CACHE_TTL_OBS_PATH", str(obs))
|
|
monkeypatch.setenv("HEADROOM_CACHE_TTL_LEARNED_PATH", str(out))
|
|
return obs, out
|
|
|
|
def test_end_to_end_resolve_learned_ttl_reads_output(self, paths):
|
|
obs, out = paths
|
|
rows = _corpus([120, 300, 480], [600, 900])
|
|
rows += [_row(idle=650, hit=False)]
|
|
# Trailing junk the parser must skip: invalid JSON, a blank line,
|
|
# and valid JSON that is not an object.
|
|
obs.write_text("\n".join(json.dumps(r) for r in rows) + '\nnot-json\n\n[1, 2]\n"scalar"\n')
|
|
|
|
assert main([]) == 0
|
|
assert resolve_learned_ttl("openai", "gpt-5.5") == 600
|
|
# An unseen model gets nothing rather than gpt-5.5's bound: the consumer
|
|
# falls back to its own static default instead of a TTL learned from a
|
|
# cache population it has no evidence about.
|
|
assert resolve_learned_ttl("openai", "other-model") is None
|
|
|
|
def test_upgrade_purges_legacy_provider_aggregate(self, paths):
|
|
# Upgrade regression: a learned file from the earlier per-provider
|
|
# version must not keep serving its aggregate to unseen models after
|
|
# a run of the model-scoped estimator — while an operator-written
|
|
# provider fallback in the same file keeps resolving.
|
|
obs, out = paths
|
|
out.write_text(
|
|
json.dumps({"openai": _legacy_aggregate(1200), "kimi": {"ttl_seconds": 3600}})
|
|
)
|
|
obs.write_text("\n".join(json.dumps(r) for r in _corpus([120, 300, 480], [600, 900, 1200])))
|
|
|
|
assert main([]) == 0
|
|
data = json.loads(out.read_text())
|
|
assert "openai" not in data
|
|
assert data["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
assert resolve_learned_ttl("openai", "unseen-model") is None
|
|
assert resolve_learned_ttl("kimi", "unseen-model") == 3600
|
|
|
|
def test_upgrade_purge_runs_even_with_no_estimable_data(self, paths, capsys):
|
|
# The purge must not depend on today's window producing an estimate.
|
|
obs, out = paths
|
|
out.write_text(json.dumps({"openai": _legacy_aggregate(1200)}))
|
|
obs.write_text(json.dumps(_row(idle=120, hit=True)) + "\n")
|
|
|
|
assert main([]) == 0
|
|
assert json.loads(out.read_text()) == {}
|
|
assert resolve_learned_ttl("openai", "unseen-model") is None
|
|
|
|
def test_non_finite_jsonl_row_does_not_abort_the_run(self, paths):
|
|
obs, out = paths
|
|
# json.dumps emits bare NaN/Infinity and json.loads reads them back, so
|
|
# the recorder can put one in the log without any hand-editing.
|
|
# First row on purpose: max() keeps a leading NaN (every later `x > nan`
|
|
# is False), so this is the ordering that takes the run down.
|
|
rows = [_row(idle=float("nan"), hit=True)] + _corpus([120, 300, 480], [600, 900, 1200])
|
|
obs.write_text("\n".join(json.dumps(r) for r in rows) + "\n")
|
|
assert "NaN" in obs.read_text()
|
|
|
|
assert main([]) == 0
|
|
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_sub_one_sample_floors_are_rejected(self, paths):
|
|
with pytest.raises(SystemExit):
|
|
main(["--min-hits", "0"])
|
|
with pytest.raises(SystemExit):
|
|
main(["--min-expiry-misses", "-1"])
|
|
|
|
def test_no_estimable_data_writes_nothing(self, paths, capsys):
|
|
obs, out = paths
|
|
obs.write_text(json.dumps(_row(idle=120, hit=True)) + "\n")
|
|
assert main([]) == 0
|
|
assert not out.exists()
|
|
assert "nothing written" in capsys.readouterr().out
|
|
|
|
def test_missing_obs_file_is_not_an_error(self, paths):
|
|
assert main([]) == 0
|
|
|
|
def test_dry_run_prints_without_writing(self, paths, capsys):
|
|
obs, out = paths
|
|
obs.write_text("\n".join(json.dumps(r) for r in _corpus([120, 300, 480], [600, 900, 1200])))
|
|
assert main(["--dry-run"]) == 0
|
|
assert not out.exists()
|
|
assert json.loads(capsys.readouterr().out)["openai/gpt-5.5"]["ttl_seconds"] == 600
|
|
|
|
def test_explicit_paths_override_env(self, paths, tmp_path):
|
|
obs2 = tmp_path / "other.jsonl"
|
|
out2 = tmp_path / "other.json"
|
|
obs2.write_text(
|
|
"\n".join(json.dumps(r) for r in _corpus([120, 300, 480], [600, 900, 1200]))
|
|
)
|
|
assert main(["--obs", str(obs2), "--out", str(out2)]) == 0
|
|
assert json.loads(out2.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 600
|