fix(cache): round learned TTL upper bounds up to whole seconds (#4710)

## Description

The TTL estimator selects the first observed expiry after the largest
observed hit as its conservative upper bound, then truncated it with
`int()`. Hits through 300.4 seconds and an expiry at 300.8 seconds
therefore produced a 300-second TTL, below evidence that the cache was
still alive. Round the selected upper bound up with `math.ceil()` when
writing whole seconds. Integer observations, evidence floors, model
scoping and the existing cold-prefix margin keep their behavior.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Round the selected expiry upper bound toward the next whole second.
- Cover fractional and subsecond intervals, an unchanged integral
control, and CLI output read back by the production resolver.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed

### Test Output

```text
New regressions on unmodified main production code: 3 failed, 1 passed
pytest tests/test_cache tests/test_cold_prefix.py: 352 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
  Rust workspace: 1739 passed, 3 ignored
  Native extension: built, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`8c2338800fa7cbc5357e6de866916fe94b03c45a`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs, real temporary files and the actual estimator subprocess.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.

```python
import json
import os
from pathlib import Path
import subprocess
import sys
from tempfile import TemporaryDirectory
from headroom.cache.prefix_tracker import PrefixCacheTracker
from headroom.cache.ttl_observations import resolve_learned_ttl
from headroom.transforms.cold_prefix import is_cold_prefix

rows = [
    {"provider": "openai", "model": "fixture", "idle_seconds": idle,
     "is_miss": miss, "reason": "ttl_expiry" if miss else "hit"}
    for miss, values in [(False, [300.1, 300.2, 300.4]), (True, [300.8, 301.4, 302.5])]
    for idle in values
]
with TemporaryDirectory() as directory:
    obs, out = Path(directory) / "obs.jsonl", Path(directory) / "learned.json"
    obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8", newline="\n")
    subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs", str(obs),
                    "--out", str(out)], check=True, capture_output=True)
    os.environ["HEADROOM_CACHE_TTL_LEARNED_PATH"] = str(out)
    ttl = resolve_learned_ttl("openai", "fixture")
    tracker = PrefixCacheTracker("openai")
    tracker._idle_seconds_at_fetch = 360.4
    print(json.dumps({"file_ttl": json.loads(out.read_text())["openai/fixture"]["ttl_seconds"],
                      "resolved_ttl": ttl,
                      "cold_at_controlled_idle_360_4_default_margin": is_cold_prefix(tracker, ttl_seconds=ttl)}))
```

- Observed result: The genuine estimator CLI writes 301, the real-file
resolver returns 301, and the controlled 360.4-second idle is no longer
above the learned TTL plus the unchanged 60-second margin. This is a
decision-boundary check, not a provider TTL measurement.

```text
Before: {"file_ttl": 300, "resolved_ttl": 300, "cold_at_controlled_idle_360_4_default_margin": true}
After:  {"file_ttl": 301, "resolved_ttl": 301, "cold_at_controlled_idle_360_4_default_margin": false}
```

- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.

## Runtime Rollout Safety

- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Newly learned fractional TTL
boundaries round upward by less than one second instead of downward.
Integral TTLs and existing learned files are unchanged.
- Kill switch / disable path: Do not run the optional estimator;
removing the learned table restores the existing fallback.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the code and
validation record.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4710"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
This commit is contained in:
Yuchuan He
2026-11-09 12:32:33 -06:00
committed by GitHub
parent 5438f0035d
commit 13347e339c
2 changed files with 29 additions and 1 deletions
+3 -1
View File
@@ -130,7 +130,9 @@ def estimate_ttls(
if not deaths_after_life:
continue # bounds overlap: no idle at which the cache is provably dead
table[key] = {
"ttl_seconds": int(min(deaths_after_life)),
# Keep the upper bound when converting fractional idle gaps to
# whole seconds; truncation can put the TTL below an observed HIT.
"ttl_seconds": math.ceil(min(deaths_after_life)),
"max_hit_idle": int(max_hit_idle),
"hits": len(key_hits),
"ttl_expiry_misses": len(key_expiries),
+26
View File
@@ -54,6 +54,22 @@ def _legacy_aggregate(ttl: int) -> dict:
class TestEstimateTtls:
@pytest.mark.parametrize(
("hit_idles", "expiry_idles", "expected"),
[
([300.1, 300.2, 300.4], [300.8, 301.4, 302.5], 301),
([0.1, 0.2, 0.4], [0.8, 1.4, 2.5], 1),
([120, 300, 480], [600, 900, 1200], 600),
],
)
def test_whole_second_ttl_keeps_the_observed_upper_bound(
self, hit_idles, expiry_idles, expected
):
estimate = estimate_ttls(_corpus(hit_idles, expiry_idles))["openai/gpt-5.5"]
assert estimate["ttl_seconds"] >= min(expiry_idles)
assert estimate["ttl_seconds"] > max(hit_idles)
assert estimate["ttl_seconds"] == expected
def test_upper_bound_of_interval_is_emitted(self):
# Alive at up to 480s, first observed death beyond that at 600s
# -> TTL estimate is the safe upper end: 600.
@@ -186,6 +202,16 @@ class TestWriteLearned:
class TestMain:
def test_fractional_upper_bound_survives_cli_and_resolver(self, paths):
obs, out = paths
rows = _corpus([300.1, 300.2, 300.4], [300.8, 301.4, 302.5])
obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8")
assert main([]) == 0
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 301
assert resolve_learned_ttl("openai", "gpt-5.5") == 301
@pytest.fixture()
def paths(self, tmp_path, monkeypatch):
obs = tmp_path / "obs.jsonl"