mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
fix(cache): round learned TTL upper bounds up to whole seconds (#4710)
## Description
The TTL estimator selects the first observed expiry after the largest
observed hit as its conservative upper bound, then truncated it with
`int()`. Hits through 300.4 seconds and an expiry at 300.8 seconds
therefore produced a 300-second TTL, below evidence that the cache was
still alive. Round the selected upper bound up with `math.ceil()` when
writing whole seconds. Integer observations, evidence floors, model
scoping and the existing cold-prefix margin keep their behavior.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Round the selected expiry upper bound toward the next whole second.
- Cover fractional and subsecond intervals, an unchanged integral
control, and CLI output read back by the production resolver.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed
### Test Output
```text
New regressions on unmodified main production code: 3 failed, 1 passed
pytest tests/test_cache tests/test_cold_prefix.py: 352 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
Rust workspace: 1739 passed, 3 ignored
Native extension: built, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`8c2338800fa7cbc5357e6de866916fe94b03c45a`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs, real temporary files and the actual estimator subprocess.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.
```python
import json
import os
from pathlib import Path
import subprocess
import sys
from tempfile import TemporaryDirectory
from headroom.cache.prefix_tracker import PrefixCacheTracker
from headroom.cache.ttl_observations import resolve_learned_ttl
from headroom.transforms.cold_prefix import is_cold_prefix
rows = [
{"provider": "openai", "model": "fixture", "idle_seconds": idle,
"is_miss": miss, "reason": "ttl_expiry" if miss else "hit"}
for miss, values in [(False, [300.1, 300.2, 300.4]), (True, [300.8, 301.4, 302.5])]
for idle in values
]
with TemporaryDirectory() as directory:
obs, out = Path(directory) / "obs.jsonl", Path(directory) / "learned.json"
obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8", newline="\n")
subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs", str(obs),
"--out", str(out)], check=True, capture_output=True)
os.environ["HEADROOM_CACHE_TTL_LEARNED_PATH"] = str(out)
ttl = resolve_learned_ttl("openai", "fixture")
tracker = PrefixCacheTracker("openai")
tracker._idle_seconds_at_fetch = 360.4
print(json.dumps({"file_ttl": json.loads(out.read_text())["openai/fixture"]["ttl_seconds"],
"resolved_ttl": ttl,
"cold_at_controlled_idle_360_4_default_margin": is_cold_prefix(tracker, ttl_seconds=ttl)}))
```
- Observed result: The genuine estimator CLI writes 301, the real-file
resolver returns 301, and the controlled 360.4-second idle is no longer
above the learned TTL plus the unchanged 60-second margin. This is a
decision-boundary check, not a provider TTL measurement.
```text
Before: {"file_ttl": 300, "resolved_ttl": 300, "cold_at_controlled_idle_360_4_default_margin": true}
After: {"file_ttl": 301, "resolved_ttl": 301, "cold_at_controlled_idle_360_4_default_margin": false}
```
- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.
## Runtime Rollout Safety
- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Newly learned fractional TTL
boundaries round upward by less than one second instead of downward.
Integral TTLs and existing learned files are unchanged.
- Kill switch / disable path: Do not run the optional estimator;
removing the learned table restores the existing fallback.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the code and
validation record.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4710"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
This commit is contained in:
Vendored
+3
-1
@@ -130,7 +130,9 @@ def estimate_ttls(
|
||||
if not deaths_after_life:
|
||||
continue # bounds overlap: no idle at which the cache is provably dead
|
||||
table[key] = {
|
||||
"ttl_seconds": int(min(deaths_after_life)),
|
||||
# Keep the upper bound when converting fractional idle gaps to
|
||||
# whole seconds; truncation can put the TTL below an observed HIT.
|
||||
"ttl_seconds": math.ceil(min(deaths_after_life)),
|
||||
"max_hit_idle": int(max_hit_idle),
|
||||
"hits": len(key_hits),
|
||||
"ttl_expiry_misses": len(key_expiries),
|
||||
|
||||
@@ -54,6 +54,22 @@ def _legacy_aggregate(ttl: int) -> dict:
|
||||
|
||||
|
||||
class TestEstimateTtls:
|
||||
@pytest.mark.parametrize(
|
||||
("hit_idles", "expiry_idles", "expected"),
|
||||
[
|
||||
([300.1, 300.2, 300.4], [300.8, 301.4, 302.5], 301),
|
||||
([0.1, 0.2, 0.4], [0.8, 1.4, 2.5], 1),
|
||||
([120, 300, 480], [600, 900, 1200], 600),
|
||||
],
|
||||
)
|
||||
def test_whole_second_ttl_keeps_the_observed_upper_bound(
|
||||
self, hit_idles, expiry_idles, expected
|
||||
):
|
||||
estimate = estimate_ttls(_corpus(hit_idles, expiry_idles))["openai/gpt-5.5"]
|
||||
assert estimate["ttl_seconds"] >= min(expiry_idles)
|
||||
assert estimate["ttl_seconds"] > max(hit_idles)
|
||||
assert estimate["ttl_seconds"] == expected
|
||||
|
||||
def test_upper_bound_of_interval_is_emitted(self):
|
||||
# Alive at up to 480s, first observed death beyond that at 600s
|
||||
# -> TTL estimate is the safe upper end: 600.
|
||||
@@ -186,6 +202,16 @@ class TestWriteLearned:
|
||||
|
||||
|
||||
class TestMain:
|
||||
def test_fractional_upper_bound_survives_cli_and_resolver(self, paths):
|
||||
obs, out = paths
|
||||
rows = _corpus([300.1, 300.2, 300.4], [300.8, 301.4, 302.5])
|
||||
obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8")
|
||||
|
||||
assert main([]) == 0
|
||||
|
||||
assert json.loads(out.read_text())["openai/gpt-5.5"]["ttl_seconds"] == 301
|
||||
assert resolve_learned_ttl("openai", "gpt-5.5") == 301
|
||||
|
||||
@pytest.fixture()
|
||||
def paths(self, tmp_path, monkeypatch):
|
||||
obs = tmp_path / "obs.jsonl"
|
||||
|
||||
Reference in New Issue
Block a user