mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
fix(cache): record TTL observations with relative filenames (#4711)
## Description
With `HEADROOM_CACHE_TTL_LEARN=1` and
`HEADROOM_CACHE_TTL_OBS_PATH=observations.jsonl`, the recorder passed an
empty dirname to `os.makedirs()`. Its best-effort exception handler then
discarded every observation, leaving the estimator with no input. Use
`.` for a bare relative filename, matching the estimator's existing
output-path handling. Pin the JSONL append to LF on Windows as required
by the repository's state-file convention.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Resolve the missing parent of a relative filename to `.` and write
JSONL with `newline="\n"`.
- Cover bare/nested/absolute paths, disabled and stateless modes,
unwritable parents, and the Windows newline contract.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed
### Test Output
```text
New regressions on unmodified main production code: 2 failed, 5 passed
pytest tests/test_cache tests/test_cold_prefix.py: 355 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
Rust workspace: 1739 passed, 3 ignored
Native extension: built, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`8bf51914e16c769ccd5154c54ca061b1c93c64d6`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs, real temporary files and the actual estimator subprocess.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.
```python
import json
import os
from pathlib import Path
import subprocess
import sys
from tempfile import TemporaryDirectory
from headroom.cache.prefix_tracker import CacheMissAttribution
from headroom.cache.ttl_observations import record_cache_observation
os.environ["HEADROOM_CACHE_TTL_LEARN"] = "1"
os.environ.pop("HEADROOM_STATELESS", None)
os.environ["HEADROOM_CACHE_TTL_OBS_PATH"] = "observations.jsonl"
previous = Path.cwd()
with TemporaryDirectory() as directory:
try:
os.chdir(directory)
for miss, idles in [(False, [120, 300, 480]), (True, [600, 900, 1200])]:
for idle in idles:
record_cache_observation(provider="openai", model="fixture", attribution=
CacheMissAttribution(is_miss=miss, reason="ttl_expiry" if miss else "hit",
idle_seconds=idle, cache_ttl_seconds=300))
raw = Path("observations.jsonl").read_bytes() if Path("observations.jsonl").exists() else b""
subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs",
"observations.jsonl", "--out", "learned.json"], check=True, capture_output=True)
learned = json.loads(Path("learned.json").read_text()) if Path("learned.json").exists() else {}
print(json.dumps({"rows": len(raw.splitlines()), "contains_cr": b"\r" in raw,
"learned_ttl": learned.get("openai/fixture", {}).get("ttl_seconds")}))
finally:
os.chdir(previous)
```
- Observed result: The real recorder now appends six observations to the
relative file (three hits and three expiry misses), with LF bytes. The
genuine estimator CLI reads them and writes a 600-second TTL. Before the
patch, there was no observation file or learned output.
```text
Before: {"rows": 0, "contains_cr": false, "learned_ttl": null}
After: {"rows": 6, "contains_cr": false, "learned_ttl": 600}
```
- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.
## Runtime Rollout Safety
- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Enabled observation recording works
for bare relative filenames; JSONL appends use LF on every platform.
Existing opt-in and stateless gates are preserved.
- Kill switch / disable path: Unset `HEADROOM_CACHE_TTL_LEARN`, or use
the existing stateless mode.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the code and
validation record.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4711"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
This commit is contained in:
Vendored
+2
-2
@@ -67,7 +67,7 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->
|
||||
return
|
||||
try:
|
||||
path = _obs_path()
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
|
||||
try:
|
||||
if os.path.getsize(path) > _OBS_MAX_BYTES:
|
||||
os.replace(path, path + ".1") # bounded: single-backup rotation
|
||||
@@ -84,7 +84,7 @@ def record_cache_observation(*, provider: str, model: str, attribution: Any) ->
|
||||
"cache_read": int(getattr(attribution, "cache_read_tokens", 0) or 0),
|
||||
"expected_cached": int(getattr(attribution, "expected_cached_tokens", 0) or 0),
|
||||
}
|
||||
with open(path, "a", encoding="utf-8") as f:
|
||||
with open(path, "a", encoding="utf-8", newline="\n") as f:
|
||||
f.write(json.dumps(row) + "\n")
|
||||
except Exception:
|
||||
pass # observability must never break a request
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import json
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -10,6 +11,78 @@ from headroom.cache.prefix_tracker import PrefixCacheTracker
|
||||
from headroom.transforms.cold_prefix import is_cold_prefix
|
||||
|
||||
|
||||
@pytest.mark.parametrize("layout", ["filename", "nested", "absolute"])
|
||||
def test_observation_path_records_each_turn(tmp_path, monkeypatch, layout):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
path = {
|
||||
"filename": "observations.jsonl",
|
||||
"nested": "nested/observations.jsonl",
|
||||
"absolute": str(tmp_path / "absolute" / "observations.jsonl"),
|
||||
}[layout]
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_LEARN", "1")
|
||||
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_OBS_PATH", path)
|
||||
for miss in (False, True):
|
||||
attribution = SimpleNamespace(reason="ttl_expiry" if miss else "hit", is_miss=miss)
|
||||
ttl_observations.record_cache_observation(
|
||||
provider="openai", model="fixture", attribution=attribution
|
||||
)
|
||||
|
||||
with open(path, encoding="utf-8") as stream:
|
||||
rows = [json.loads(line) for line in stream]
|
||||
assert [row["is_miss"] for row in rows] == [False, True]
|
||||
assert all(row["model"] == "fixture" for row in rows)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stateless", [False, True])
|
||||
def test_relative_observation_path_respects_write_gates(tmp_path, monkeypatch, stateless):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_OBS_PATH", "observations.jsonl")
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_LEARN", "1" if stateless else "0")
|
||||
monkeypatch.setenv("HEADROOM_STATELESS", "1" if stateless else "0")
|
||||
ttl_observations.record_cache_observation(
|
||||
provider="openai", model="fixture", attribution=SimpleNamespace()
|
||||
)
|
||||
assert not (tmp_path / "observations.jsonl").exists()
|
||||
|
||||
|
||||
def test_observation_path_failure_still_does_not_raise(tmp_path, monkeypatch):
|
||||
parent = tmp_path / "not-a-directory"
|
||||
parent.write_bytes(b"keep")
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_OBS_PATH", str(parent / "observations.jsonl"))
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_LEARN", "1")
|
||||
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
||||
ttl_observations.record_cache_observation(
|
||||
provider="openai", model="fixture", attribution=SimpleNamespace()
|
||||
)
|
||||
assert parent.read_bytes() == b"keep"
|
||||
|
||||
|
||||
@pytest.mark.windows_newline
|
||||
def test_observation_writer_pins_lf(tmp_path, monkeypatch):
|
||||
import builtins
|
||||
|
||||
path = tmp_path / "observations.jsonl"
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_OBS_PATH", str(path))
|
||||
monkeypatch.setenv("HEADROOM_CACHE_TTL_LEARN", "1")
|
||||
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
|
||||
real_open = builtins.open
|
||||
writes = []
|
||||
|
||||
def recording_open(*args, **kwargs):
|
||||
if args[0] == str(path) and len(args) > 1 and args[1] == "a":
|
||||
writes.append(kwargs.get("newline"))
|
||||
return real_open(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(builtins, "open", recording_open)
|
||||
ttl_observations.record_cache_observation(
|
||||
provider="openai", model="fixture", attribution=SimpleNamespace()
|
||||
)
|
||||
assert writes == ["\n"]
|
||||
assert path.read_bytes().endswith(b"\n")
|
||||
assert b"\r" not in path.read_bytes()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def learned_file(tmp_path, monkeypatch):
|
||||
path = tmp_path / "learned.json"
|
||||
|
||||
Reference in New Issue
Block a user