mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
fix/openclaw-python-launcher-path
36
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
4eb742abd8 |
fix(cache): refresh semantic cache recency on updates (#4767)
## Description Overwriting a SemanticCache entry refreshed its response and timestamps but left its old position in the OrderedDict. With capacity two, put(A), put(B), update(A), put(C) evicted the freshly updated A and retained the older B. Move a successfully stored entry to the end of the LRU. Cache state operations now share an RLock, including semantic scans and touches, while embedding callbacks execute outside it. This happens after embedding generation and entry construction, so a failed update still preserves the original eviction order. It complements the query-only retrieval fix in #4709: a successfully refreshed response should remain retrievable under subsequent capacity pressure. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Refresh LRU position after successful insertion/replacement. - Cover query-only and context-hash entries, with and without namespaces. - Keep failed embedding updates from refreshing recency, and retain the no-eviction-on-update contract from #2094. - Synchronize cache/index/statistics operations and the complete similarity-scan/touch sequence. Embedding callbacks run outside the state lock; add two-thread scan/update and nonblocking exact-hit regressions. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Initial new regressions against original production code: 4 failed, 1 passed pytest tests/test_cache tests/test_cold_prefix.py: 398 passed, 3 skipped Review follow-up on the initial PR head: 1 failed Original pre-commit and commit-msg hooks (including follow-up): passed Whole-tree Ruff check / format --check: passed (1932 files) Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: built, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The three cache skips require optional spaCy/embedding models. Critical-suite skips require optional relevance models. The existing token-count approximation warning remains visible in the gate log. Initial broader runs failed because the new worktrees lacked the native extension; these were rerun successfully after building it. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development venv; clean baseline `200c838c7393d20ae56a0c338f832ed27133c398`, patch `ffa5198d3e4a7ab634e49efb8b6c6b59e4315acf`. `HEADROOM_OFFLINE=1`, `PYTHONPATH=.` for the selected checkout. No paid/provider credentials. - Exact command / steps: Save the following identical script as `proof.py` and run `python proof.py` from either checkout. Before and after runs were outside pytest. ```python """A real SemanticCache overwrite followed by a capacity-triggered eviction.""" import json from headroom.cache.semantic import SemanticCache, SemanticCacheConfig reports = [] for context in (False, True): for namespace in ("", "provider/model"): cache = SemanticCache(SemanticCacheConfig(max_entries=2)) hashes = ["context-a", "context-b", "context-c"] if context else [None] * 3 cache.put("first", {"answer": "old"}, hashes[0], namespace) cache.put("second", {"answer": "older"}, hashes[1], namespace) cache.put("first", {"answer": "updated"}, hashes[0], namespace) before = cache.get_stats() cache.put("third", {"answer": "new"}, hashes[2], namespace) first = cache.get("first", hashes[0], namespace) second = cache.get("second", hashes[1], namespace) reports.append({"context": context, "namespace": namespace, "update_evictions": before["evictions"], "updated_response": first.response if first else None, "older_response": second.response if second else None, "entries": cache.get_stats()["entries"], "evictions": cache.get_stats()["evictions"]}) print(json.dumps(reports, indent=2)) ``` Review follow-up: run `python concurrency_proof.py` with the script below on the initial PR head and the revised head. It coordinates two real cache callers at a similarity-scan boundary. ```python """A real two-thread scan/update interleaving, without mocked cache results.""" import json import threading from headroom.cache.semantic import SemanticCache scanning, release, writer_progress = threading.Event(), threading.Event(), threading.Event() class PausedCache(SemanticCache): def _cosine_similarity(self, a, b): if not scanning.is_set(): scanning.set() if not release.wait(10): raise RuntimeError("Scan release missing") return super()._cosine_similarity(a, b) cache = PausedCache(embedding_fn=lambda query: [1.0, 0.0]) cache.put("first", "old", "first") cache.put("second", "two", "second") real = threading.RLock() class SignalsWriterLock: def __enter__(self): if threading.current_thread().name == "cache-update": writer_progress.set() return real.__enter__() def __exit__(self, *exc): return real.__exit__(*exc) cache._lock = SignalsWriterLock() errors, results = [], [] def read(): try: entry = cache.get("third", "third") results.append(entry.response if entry else None) except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc)) def update(): try: cache.put("first", "updated", "first") except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc)) finally: writer_progress.set() reader = threading.Thread(target=read) writer = threading.Thread(target=update, name="cache-update") reader.start() try: assert scanning.wait(10) writer.start() assert writer_progress.wait(10) finally: release.set() reader.join(10) if writer.ident is not None: writer.join(10) assert not reader.is_alive() and not writer.is_alive() print(json.dumps({"errors": errors, "lookup_responses": results, "updated_response": cache.get("first", "first").response})) # Existing tie behavior: an ordinary exact hit already changes scan order. # No put/update occurs between the two semantic lookups in this control. ties = SemanticCache(embedding_fn=lambda query: [1.0, 0.0]) ties.put("first", "A", "first") ties.put("second", "B", "second") first = ties.get("one unseen query", "unseen-1").response ties.get("first", "first") second = ties.get("another unseen query", "unseen-2").response print(json.dumps({"equal_score_control": [first, second], "policy": "first match in current LRU order; no most-recent tie guarantee"})) ``` - Observed result: The real SemanticCache API reproduces the sequence for four context/namespace combinations. Before the patch, the updated response is missing and the older response survives. After it, updated_response is updated and older_response is None; each case still has two entries and exactly one eviction. No get() is made between update(A) and put(C), which would mask the defect by touching A. The concurrent proof raises OrderedDict mutated during iteration on the initial PR head and completes without errors on the revised head. The equal-score control returns A then B on both original main and revised code: ties still follow current LRU scan order, with no added freshness guarantee. - Not tested: Paid/live embedding services, model quality, production hit-rate/latency improvements and the complete Python suite. The runtime proof uses exact matching without an embedding dependency. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing cache behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Successful replacement counts as recent use. Capacity, exact/context isolation, TTL and failed-store behavior remain unchanged. - Kill switch / disable path: Disable the existing optional semantic cache in its caller, or revert this commit. - Unsafe override required: No. - Qualification impact: Related cache tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit. No dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Implementation and local validation were completed with Codex assistance. The author reviewed the initial code and validation record; the review follow-up was checked with Codex. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4767" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> |
||
|
|
a82088847b |
fix(cache): restore query-only semantic cache lookups (#4709)
## Description
`SemanticCache.put()` stores a query-derived entry when `messages_hash`
is omitted, but `get()` only tried exact lookup when that hash was
supplied. With the default `embedding_fn=None`, `put("What failed?",
response)` followed by `get("What failed?")` always missed. Look up the
same query-derived key before embedding search. Require a nonblank
query, the same namespace and query text, and an entry stored without a
context hash, so this fallback cannot consume a context-keyed response.
`SemanticCacheLayer` also skips both cache reads and writes if
complete-message serialization fails. An unknown context identity must
not become an intentional query-only request.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Add exact query-only retrieval before semantic search, with context
and namespace guards.
- Reject unhashable Layer contexts (objects, sets and cycles) for both
lookup and storage.
- Cover round trips, disabled matching, expiry, LRU refresh,
embedder-outage exact hits, blank queries and a query-hash/context-key
collision control.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed
### Test Output
```text
Original regressions on unmodified main production code: 7 failed, 3 passed
Review follow-up regressions on the initial PR head: 6 failed
pytest tests/test_cache tests/test_cold_prefix.py: 364 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
Rust workspace: 1739 passed, 3 ignored
Native extension: built, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`eff3172c2483861e60f07f97179b8d4139a7aa7f`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.
```python
import json
from headroom.cache import SemanticCache
cache = SemanticCache()
cache.put("What failed?", {"answer": "disk full"}, namespace="project-a")
hit = cache.get("What failed?", namespace="project-a")
other = cache.get("What failed?", namespace="project-b")
scoped = cache.get("What failed?", messages_hash="context", namespace="project-a")
cache.put("What failed?", "context answer", messages_hash="context", namespace="project-a")
exact = cache.get("What failed?", messages_hash="context", namespace="project-a")
print(json.dumps({
"query_only_response": hit.response if hit else None,
"other_namespace_hit": other is not None,
"context_request_consumed_query_only_entry": scoped is not None,
"explicit_context_response": exact.response if exact else None,
}))
```
- Observed result: The query-only request now returns the stored
response; a different namespace and an explicit context request still
miss that entry. Explicit context-hash lookup still returns its own
response.
```text
Before: {"query_only_response": null, "other_namespace_hit": false, "context_request_consumed_query_only_entry": false, "explicit_context_response": "context answer"}
After: {"query_only_response": {"answer": "disk full"}, "other_namespace_hit": false, "context_request_consumed_query_only_entry": false, "explicit_context_response": "context answer"}
```
Review follow-up: the same standalone `python review-proof.py` was run
before and after the Layer guard:
```python
import json
from headroom.cache import AnthropicCacheOptimizer, OptimizationContext, SemanticCacheLayer
layer = SemanticCacheLayer(AnthropicCacheOptimizer())
context = OptimizationContext(provider="anthropic", model="fixture")
first = [{"role": "user", "content": "Hello!", "opaque": object()}]
second = [{"role": "user", "content": "Hello!", "opaque": object()}]
key = layer.store_response(first, "first conversation's answer", context)
result = layer.process(second, context)
print(json.dumps({"unhashable_store_accepted": bool(key),
"other_conversation_cache_hit": result.semantic_cache_hit,
"cached_response": result.cached_response}))
```
```text
Before follow-up: {"unhashable_store_accepted": true, "other_conversation_cache_hit": true, "cached_response": "first conversation's answer"}
After follow-up: {"unhashable_store_accepted": false, "other_conversation_cache_hit": false, "cached_response": null}
```
- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.
## Runtime Rollout Safety
- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Exact query-only entries become
retrievable; blank queries still require an exact context hash. Existing
context-hash and embedding matching remain available for serializable
requests; the Layer bypasses caching for unhashable messages.
- Kill switch / disable path: Set
`SemanticCacheConfig.use_exact_matching=False`, or do not use the
optional cache.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the initial code
and validation record; the review follow-up is documented above.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4709"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
acb58a0c00 |
fix(cache): scope learned TTL cache to its source file (#4727)
## Description `resolve_learned_ttl()` re-read `HEADROOM_CACHE_TTL_LEARNED_PATH` on each call but cached the loaded table using only mtime. Two different files with the same mtime therefore shared one cached table. Switching from a 300-second file to an 1800-second file still returned 300; a controlled 600-second idle was then classified as cold. The same collision occurs when a relative path resolves under a different working directory. Include the absolute source path in the cache identity alongside mtime. A source change reloads the selected file; repeated reads of the same unchanged file remain cached. This completes the file-to-decision part of the TTL flow exercised by #4711 and #4710. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Cache the absolute source path only after a successful load, and reload when that path changes. - Keep missing files best-effort and ignore invalid table shapes on the new source. - Keep fixture initialization compatible with unpatched production code; the final test module replays its four behavioral failures on main. - Cover equal-mtime source switches and switch-back, relative paths across directories, a warm-prefix decision, and unchanged-file read-once behavior. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Owner test file with new regressions on original production code: 4 failed, 32 passed pytest tests/test_cache tests/test_cold_prefix.py: 365 passed, 3 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed (1924 files) Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: built, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The three cache skips require optional spaCy/embedding models. Critical-suite skips require optional relevance models; the existing token-count approximation warning remains in that log. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development venv; clean baseline `1da8930ab47a2fc5339ded97e5a41bfe6fc419eb`, patch `4b863c460828d095f4e00020a90d5dd3c00cf5c0`. `HEADROOM_OFFLINE=1`, `PYTHONPATH=.` for the selected checkout. Real local files, public APIs and controlled idle inputs; no provider credentials. - Exact command / steps: Run `python proof.py` from either checkout with the following identical script. Both before and after runs were outside pytest. ```python import json import os from pathlib import Path import tempfile from headroom.cache.ttl_estimator import write_learned from headroom.cache.ttl_observations import resolve_learned_ttl from headroom.cache.prefix_tracker import PrefixCacheTracker from headroom.transforms.cold_prefix import is_cold_prefix with tempfile.TemporaryDirectory() as directory: root = Path(directory) first, second = root / "first.json", root / "second.json" for path, ttl in ((first, 300), (second, 1800)): write_learned({"openai/fixture": {"ttl_seconds": ttl}}, str(path)) os.utime(path, ns=(1_700_000_000_000_000_000,) * 2) os.environ["HEADROOM_CACHE_TTL_LEARNED_PATH"] = str(first) initial = resolve_learned_ttl("openai", "fixture") os.environ["HEADROOM_CACHE_TTL_LEARNED_PATH"] = str(second) switched = resolve_learned_ttl("openai", "fixture") tracker = PrefixCacheTracker("openai") tracker._idle_seconds_at_fetch = 600 print(json.dumps({"same_mtime": first.stat().st_mtime == second.stat().st_mtime, "first_ttl": initial, "second_file_ttl": 1800, "resolved_after_switch": switched, "cold_at_600_seconds": is_cold_prefix(tracker, ttl_seconds=switched)})) ``` - Observed result: The genuine learned-table writer creates two files, whose mtimes are set equal. After selecting the second path, the resolver now returns 1800 instead of 300; the actual cold-prefix helper keeps a controlled 600-second idle warm. ```text Before: {"same_mtime": true, "first_ttl": 300, "second_file_ttl": 1800, "resolved_after_switch": 300, "cold_at_600_seconds": true} After: {"same_mtime": true, "first_ttl": 300, "second_file_ttl": 1800, "resolved_after_switch": 1800, "cold_at_600_seconds": false} ``` - Not tested: Live proxy/provider traffic, measured provider TTLs, billing or model quality, and the complete Python suite. The runtime proof verifies the local classification/file/CLI/decision behavior above; hosted CI is tracked separately. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing optional TTL-learning/cold-prefix behavior; no new flag. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: The learned table follows its selected source file even when another source shares its timestamp. The existing mtime refresh, value validation and provider/model precedence remain intact. - Kill switch / disable path: Remove the optional learned file to use existing fallback TTLs; existing cold-recompaction flags control their callers. - Unsafe override required: No. - Qualification impact: Owner cache/cold-prefix tests and original local gates passed; hosted checks provide remote qualification. - Rollback path: Revert this commit. No dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Implementation and local validation were completed with Codex assistance. The author reviewed the initial code and validation record; the fixture-only compatibility follow-up was checked with Codex. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4727" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> |
||
|
|
93df7ae4f7 |
fix(cache): exclude cold starts from TTL hit evidence (#4728)
## Description `PrefixCacheTracker.classify_cache_miss()` deliberately returns `is_miss=False, reason="cold_start"` when no prefix was cached on the previous turn. The TTL estimator treated every falsey `is_miss` value as a hit. Three cold starts and three later expiry misses could therefore produce a learned TTL with zero actual hit observations. A long cold start could also suppress a valid estimate by raising its supposed life bound. Count life evidence only for explicit `reason="hit", is_miss=False` rows, and accept expiry evidence only with a boolean miss outcome. The evidence floors, per-model grouping and conservative expiry rounding are unchanged. This follows the recording and upper-bound fixes in #4711 and #4710 by checking what the recorded outcomes actually prove. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Distinguish cold starts from positive cache hits when collecting TTL bounds. - Ignore malformed/ambiguous hit outcomes instead of letting them satisfy the sample floor. - Cover actual tracker cold starts, mixed sample floors, long cold starts alongside genuine hits, and non-hit/malformed controls. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Owner test file with new regressions on original production code: 9 failed, 28 passed pytest tests/test_cache tests/test_cold_prefix.py: 369 passed, 3 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed (1924 files) Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: built, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The three cache skips require optional spaCy/embedding models. Critical-suite skips require optional relevance models; the existing token-count approximation warning remains in that log. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development venv; clean baseline `1da8930ab47a2fc5339ded97e5a41bfe6fc419eb`, patch `be03ab5a8ff74f18c5b90232f6d5874e1af8d9ec`. `HEADROOM_OFFLINE=1`, `PYTHONPATH=.` for the selected checkout. Real local files, public APIs and controlled idle inputs; no provider credentials. - Exact command / steps: Run `python proof.py` from either checkout with the following identical script. Both before and after runs were outside pytest. ```python import json import os from pathlib import Path import subprocess import sys import tempfile from headroom.cache.prefix_tracker import PrefixCacheTracker from headroom.cache.ttl_observations import record_cache_observation with tempfile.TemporaryDirectory() as directory: obs = Path(directory) / "observations.jsonl" out = Path(directory) / "learned.json" os.environ["HEADROOM_CACHE_TTL_LEARN"] = "1" os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_CACHE_TTL_OBS_PATH"] = str(obs) for idle in (120, 200, 240): tracker = PrefixCacheTracker("openai") attr = tracker.classify_cache_miss(0, [], idle_seconds=idle) record_cache_observation(provider="openai", model="fixture", attribution=attr) tracker = PrefixCacheTracker("openai") tracker.update_from_response(cache_read_tokens=0, cache_write_tokens=1000, messages=[]) for idle in (600, 700, 800): attr = tracker.classify_cache_miss(0, [], idle_seconds=idle) record_cache_observation(provider="openai", model="fixture", attribution=attr) result = subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs", str(obs), "--out", str(out)], capture_output=True, text=True, check=True) rows = [json.loads(line) for line in obs.read_text(encoding="utf-8").splitlines()] print(json.dumps({"reasons": [r["reason"] for r in rows], "actual_hit_rows": sum(r["reason"] == "hit" for r in rows), "learned_file_written": out.exists(), "learned_table": json.loads(out.read_text()) if out.exists() else None, "cli_output": result.stdout.strip()})) ``` - Observed result: The real tracker and recorder produce three `cold_start` and three `ttl_expiry` rows. Before the fix the CLI wrote a 600-second estimate claiming three hits; after it reports no estimable keys and creates no learned file. This uses controlled idle inputs to verify evidence classification. ```text Before: {"reasons": ["cold_start", "cold_start", "cold_start", "ttl_expiry", "ttl_expiry", "ttl_expiry"], "actual_hit_rows": 0, "learned_file_written": true, "learned_table": {"openai/fixture": {"hits": 3, "max_hit_idle": 240, "ttl_expiry_misses": 3, "ttl_seconds": 600, "updated_at": 1794297956.945}}, "cli_output": "headroom-cache-ttl: wrote 1 estimate(s) (1 total) to C:\\Users\\ASUS\\AppData\\Local\\Temp\\tmp5ol13684\\learned.json"} After: {"reasons": ["cold_start", "cold_start", "cold_start", "ttl_expiry", "ttl_expiry", "ttl_expiry"], "actual_hit_rows": 0, "learned_file_written": false, "learned_table": null, "cli_output": "headroom-cache-ttl: no estimable keys in C:\\Users\\ASUS\\AppData\\Local\\Temp\\tmpnnetshza\\observations.jsonl (6 rows); nothing written"} ``` - Not tested: Live proxy/provider traffic, measured provider TTLs, billing or model quality, and the complete Python suite. The runtime proof verifies the local classification/file/CLI/decision behavior above; hosted CI is tracked separately. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing optional TTL-learning/cold-prefix behavior; no new flag. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Cold starts no longer create or inflate a learned life bound. Valid explicit hit/expiry observations retain their estimates; existing learned-table merge behavior is unchanged. - Kill switch / disable path: Do not run the optional estimator, or remove the learned table to use the existing provider/config fallback. - Unsafe override required: No. - Qualification impact: Owner cache/cold-prefix tests and original local gates passed; hosted checks provide remote qualification. - Rollback path: Revert this commit. No dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Implementation and local validation were completed with Codex assistance. The author reviewed the code and validation record. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4728" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> |
||
|
|
51d9dc59b3 |
fix(cache): record TTL observations with relative filenames (#4711)
## Description
With `HEADROOM_CACHE_TTL_LEARN=1` and
`HEADROOM_CACHE_TTL_OBS_PATH=observations.jsonl`, the recorder passed an
empty dirname to `os.makedirs()`. Its best-effort exception handler then
discarded every observation, leaving the estimator with no input. Use
`.` for a bare relative filename, matching the estimator's existing
output-path handling. Pin the JSONL append to LF on Windows as required
by the repository's state-file convention.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Resolve the missing parent of a relative filename to `.` and write
JSONL with `newline="\n"`.
- Cover bare/nested/absolute paths, disabled and stateless modes,
unwritable parents, and the Windows newline contract.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed
### Test Output
```text
New regressions on unmodified main production code: 2 failed, 5 passed
pytest tests/test_cache tests/test_cold_prefix.py: 355 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
Rust workspace: 1739 passed, 3 ignored
Native extension: built, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`8bf51914e16c769ccd5154c54ca061b1c93c64d6`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs, real temporary files and the actual estimator subprocess.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.
```python
import json
import os
from pathlib import Path
import subprocess
import sys
from tempfile import TemporaryDirectory
from headroom.cache.prefix_tracker import CacheMissAttribution
from headroom.cache.ttl_observations import record_cache_observation
os.environ["HEADROOM_CACHE_TTL_LEARN"] = "1"
os.environ.pop("HEADROOM_STATELESS", None)
os.environ["HEADROOM_CACHE_TTL_OBS_PATH"] = "observations.jsonl"
previous = Path.cwd()
with TemporaryDirectory() as directory:
try:
os.chdir(directory)
for miss, idles in [(False, [120, 300, 480]), (True, [600, 900, 1200])]:
for idle in idles:
record_cache_observation(provider="openai", model="fixture", attribution=
CacheMissAttribution(is_miss=miss, reason="ttl_expiry" if miss else "hit",
idle_seconds=idle, cache_ttl_seconds=300))
raw = Path("observations.jsonl").read_bytes() if Path("observations.jsonl").exists() else b""
subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs",
"observations.jsonl", "--out", "learned.json"], check=True, capture_output=True)
learned = json.loads(Path("learned.json").read_text()) if Path("learned.json").exists() else {}
print(json.dumps({"rows": len(raw.splitlines()), "contains_cr": b"\r" in raw,
"learned_ttl": learned.get("openai/fixture", {}).get("ttl_seconds")}))
finally:
os.chdir(previous)
```
- Observed result: The real recorder now appends six observations to the
relative file (three hits and three expiry misses), with LF bytes. The
genuine estimator CLI reads them and writes a 600-second TTL. Before the
patch, there was no observation file or learned output.
```text
Before: {"rows": 0, "contains_cr": false, "learned_ttl": null}
After: {"rows": 6, "contains_cr": false, "learned_ttl": 600}
```
- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.
## Runtime Rollout Safety
- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Enabled observation recording works
for bare relative filenames; JSONL appends use LF on every platform.
Existing opt-in and stateless gates are preserved.
- Kill switch / disable path: Unset `HEADROOM_CACHE_TTL_LEARN`, or use
the existing stateless mode.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the code and
validation record.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4711"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
|
||
|
|
13347e339c |
fix(cache): round learned TTL upper bounds up to whole seconds (#4710)
## Description
The TTL estimator selects the first observed expiry after the largest
observed hit as its conservative upper bound, then truncated it with
`int()`. Hits through 300.4 seconds and an expiry at 300.8 seconds
therefore produced a 300-second TTL, below evidence that the cache was
still alive. Round the selected upper bound up with `math.ceil()` when
writing whole seconds. Integer observations, evidence floors, model
scoping and the existing cold-prefix margin keep their behavior.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Round the selected expiry upper bound toward the next whole second.
- Cover fractional and subsecond intervals, an unchanged integral
control, and CLI output read back by the production resolver.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed
### Test Output
```text
New regressions on unmodified main production code: 3 failed, 1 passed
pytest tests/test_cache tests/test_cold_prefix.py: 352 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
Rust workspace: 1739 passed, 3 ignored
Native extension: built, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`8c2338800fa7cbc5357e6de866916fe94b03c45a`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs, real temporary files and the actual estimator subprocess.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.
```python
import json
import os
from pathlib import Path
import subprocess
import sys
from tempfile import TemporaryDirectory
from headroom.cache.prefix_tracker import PrefixCacheTracker
from headroom.cache.ttl_observations import resolve_learned_ttl
from headroom.transforms.cold_prefix import is_cold_prefix
rows = [
{"provider": "openai", "model": "fixture", "idle_seconds": idle,
"is_miss": miss, "reason": "ttl_expiry" if miss else "hit"}
for miss, values in [(False, [300.1, 300.2, 300.4]), (True, [300.8, 301.4, 302.5])]
for idle in values
]
with TemporaryDirectory() as directory:
obs, out = Path(directory) / "obs.jsonl", Path(directory) / "learned.json"
obs.write_text("\n".join(json.dumps(row) for row in rows), encoding="utf-8", newline="\n")
subprocess.run([sys.executable, "-m", "headroom.cache.ttl_estimator", "--obs", str(obs),
"--out", str(out)], check=True, capture_output=True)
os.environ["HEADROOM_CACHE_TTL_LEARNED_PATH"] = str(out)
ttl = resolve_learned_ttl("openai", "fixture")
tracker = PrefixCacheTracker("openai")
tracker._idle_seconds_at_fetch = 360.4
print(json.dumps({"file_ttl": json.loads(out.read_text())["openai/fixture"]["ttl_seconds"],
"resolved_ttl": ttl,
"cold_at_controlled_idle_360_4_default_margin": is_cold_prefix(tracker, ttl_seconds=ttl)}))
```
- Observed result: The genuine estimator CLI writes 301, the real-file
resolver returns 301, and the controlled 360.4-second idle is no longer
above the learned TTL plus the unchanged 60-second margin. This is a
decision-boundary check, not a provider TTL measurement.
```text
Before: {"file_ttl": 300, "resolved_ttl": 300, "cold_at_controlled_idle_360_4_default_margin": true}
After: {"file_ttl": 301, "resolved_ttl": 301, "cold_at_controlled_idle_360_4_default_margin": false}
```
- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.
## Runtime Rollout Safety
- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Newly learned fractional TTL
boundaries round upward by less than one second instead of downward.
Integral TTLs and existing learned files are unchanged.
- Kill switch / disable path: Do not run the optional estimator;
removing the learned table restores the existing fallback.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the code and
validation record.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4710"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
|
||
|
|
911bb74074 |
fix(cache): reject nonpositive semantic cache capacities (#4301)
## Description Closes #4282. Reject `max_entries <= 0` at the start of `SemanticCache.put()`, before expiration cleanup, eviction, or overwriting a response. Otherwise the eviction loop cannot terminate once the cache is empty. The check is at write time because the public configuration can change after construction. Positive capacities retain their existing behavior; zero is not reinterpreted as a silent no-cache mode. Thanks to @oxura for the diagnosis, reproduction, and published fork fix linked in the issue. This PR independently implements the guard and adds regression coverage. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change ## Changes Made - Add the capacity guard and document the `ValueError`. - Cover both invalid initial capacities and runtime changes to zero/negative capacity. - Verify rejected inserts and overwrites preserve entries and statistics, and that restoring positive capacity allows writes again. ## Testing - [x] Focused cache unit and client-integration tests - [x] Ruff lint and formatting - [x] Mypy and commitlint through the required commit hooks - [x] Rust formatting and Clippy - [x] Native extension rebuild/import verification and the standard critical Python pre-push suite - [x] Six new regression cases fail before the fix and pass afterward - [x] Separate public-API smoke test without mocks or provider requests - [x] Full normal pre-push gate, including 1,658 Rust tests ```text uv run --no-sync python -m pytest tests/test_cache -q 303 passed, 2 skipped uv run --no-sync ruff check headroom/cache/semantic.py tests/test_cache/test_semantic.py All checks passed! uv run --no-sync ruff format --check headroom/cache/semantic.py tests/test_cache/test_semantic.py 2 files already formatted uv run --no-sync make ci-precheck-python ci-precheck-commitlint 177 passed, 4 skipped, 1 warning ORT_DYLIB_PATH="$(uv run --no-sync python -c 'from headroom._ort import ensure_ort_dylib_pinned; print(ensure_ort_dylib_pinned())')" \ uv run --no-sync git push --set-upstream fork fix/semantic-cache-positive-capacity 1658 Rust tests passed 177 critical Python tests passed, 4 skipped ci-precheck PASSED — safe to push. ``` **Local environment:** the optional ONNX Runtime dependency was initially absent. Installing and pinning it resolved the Kompress model-load stall. The latest runtime (1.31) then reproduced the already-tracked macOS thread-count test issue #4191; using the repository's locked version (1.26.0) allowed the entire normal pre-push gate to pass. No hooks were bypassed, and no dependencies, test skips, or Rust files were changed in this PR. ## Real Behavior Proof - Environment: macOS arm64, Python 3.12.0, current source installed with `uv sync --extra dev`; ONNX Runtime 1.26.0 for the native verification gate. The public cache smoke uses default exact matching, without an embedding function, credentials, or provider/model calls. - Before: four independent subprocesses using the public `SemanticCacheLayer` API hung and were stopped after three seconds each: initial capacity 0/-1 and capacity changed to 0/-1 after storing a response. - Observed result: all four cases raise `ValueError` promptly, preserve the previous cache state/response, and resume successful stores and cache hits once capacity is restored to 1. - Not tested: Windows/Linux runtime, remote providers, or semantic-cache matching against a real embedding service. The full Python suite is not claimed; the cache/client-integration and standard critical suites were run. - Exact command / steps: run the standalone public-API smoke below from the checkout. Pass the same context when storing and looking up, as required by current main cache namespacing. Reproducible post-fix smoke: ```bash uv run --no-sync python - <<'PY' from headroom.cache import ( AnthropicCacheOptimizer, OptimizationContext, SemanticCacheLayer, ) context = OptimizationContext(provider="anthropic", model="claude-3-opus") old = [{"role": "user", "content": "original request"}] new = [{"role": "user", "content": "replacement request"}] for changed in (False, True): for capacity in (0, -1): layer = SemanticCacheLayer( AnthropicCacheOptimizer(), max_entries=1 if changed else capacity, ) if changed: layer.store_response(old, {"text": "original answer"}, context) layer.cache.config.max_entries = capacity before = layer.get_stats() try: layer.store_response(new, {"text": "replacement answer"}, context) except ValueError as error: assert str(error) == "max_entries must be positive" else: raise AssertionError("invalid capacity accepted") assert layer.get_stats() == before if changed: result = layer.process(old, context) assert result.semantic_cache_hit assert result.cached_response == {"text": "original answer"} layer.cache.config.max_entries = 1 layer.store_response(new, {"text": "replacement answer"}, context) result = layer.process(new, context) assert result.semantic_cache_hit assert result.cached_response == {"text": "replacement answer"} state = "changed" if changed else "initial" print(f"{state} capacity={capacity}: rejected; state preserved; recovery OK") PY ``` Observed output: ```text initial capacity=0: rejected; state preserved; recovery OK initial capacity=-1: rejected; state preserved; recovery OK changed capacity=0: rejected; state preserved; recovery OK changed capacity=-1: rejected; state preserved; recovery OK ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing cache correctness fix. - Minimum rollout channel: normal CI-qualified release. - Stable/default behavior changed: invalid nonpositive capacities raise instead of hanging or overwriting an existing key; positive capacities and defaults are unchanged. - Kill switch / disable path: none needed; configure a positive capacity. - Unsafe override required: no. - Qualification impact: cache tests and normal repository hooks. - Rollback path: revert this commit, which would restore the invalid-capacity bug. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] The change follows the existing code style. - [x] Public behavior and errors are documented in the configuration comment and method docstring. - [x] Regression tests prove the fix and preserve valid-capacity behavior. - [x] No dependencies, feature flags, or unrelated changes were added. - [x] `CHANGELOG.md` was not edited. - [x] No co-author trailer or history rewrite. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4301" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Maintainer requalification on Windows/Python 3.12 against current main: all 347 cache tests passed, two optional tests skipped; changed-file lint, formatting, and whitespace checks passed. The public-API smoke above was rerun with current-main context namespacing and produced all four stated successful rejection/state-preservation/recovery lines. The separate proxy cache noted by Devin is outside this library-cache fix; no claim of fixing it is made. No live provider or fresh native build is claimed. Hosted qualification for the published sync remains pending. Co-authored-by: Romin Parekh <roparekh@adobe.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
0d4c450d77 |
fix(cache): build semantic entry before evicting so a failed embedding keeps live entries (#4581)
## Description `SemanticCache.put()` evicted the oldest entry before calling the embedding function. When the embedding raised (e.g. an OpenAI 503), the new entry was never stored but a live one was already gone, so a cache at capacity lost a valid response on a transient failure. Fixes #4283 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/cache/semantic.py`: move the capacity eviction loop in `put()` to after the embedding is computed and the `CacheEntry` is built, right before insertion. The embedding exception still propagates; successful stores evict exactly as before. - `tests/test_cache/test_semantic.py`: regression `test_put_failure_after_eviction_keeps_live_entry` (max_entries=1, embedding fails on the second put; old entry survives, size stays 1; a later successful put evicts normally). ## Testing - [x] Unit tests pass - [x] Regression test fails on `main` and passes with the fix ``` $ python -m pytest tests/test_cache/test_semantic.py -q 37 passed in 1.94s # on upstream/main source: FAILED tests/test_cache/test_semantic.py::TestSemanticCache::test_put_failure_after_eviction_keeps_live_entry 1 failed, 36 passed in 1.72s $ ruff check / ruff format --check (0.16.8) All checks passed! 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, uv tool Python env, openai 2.54.0, headroom upstream/main vs this branch; local HTTP embeddings endpoint (first call 200, then 503) - Exact command / steps: SemanticCache(max_entries=1) with embedding_fn = real OpenAI client (base_url=local endpoint, max_retries=0); put("old query"), put("new query") -> 503, then get("old query") - Observed result: on upstream/main the failed put leaves 0 entries and the old lookup misses (falls through to the failing embedding); with this fix the 503 still raises InternalServerError, the cache keeps 1 entry and the old exact lookup returns "old response" - Not tested: a hosted OpenAI endpoint under real overload ``` == with fix after first put: entries = 1 second put raised: InternalServerError after failed put: entries = 1 old exact lookup: old response == upstream/main after first put: entries = 1 second put raised: InternalServerError after failed put: entries = 0 openai.InternalServerError: Error code: 503 (old entry gone; get() re-embeds and fails) ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only on embedding failure, where the existing entry is now kept instead of evicted - Kill switch / disable path: n/a (semantic cache can be disabled as before) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
99cdaaf1c7 |
fix(cache): namespace SemanticCacheLayer by provider, model and metadata (#4473)
## Description `SemanticCacheLayer` keyed exact matches on the messages hash alone and its embedding search scanned every stored entry, so `OptimizationContext.provider`, `.model` and `.metadata` never fenced a lookup. Identical (or semantically similar) messages sent to a different model, provider, or with different metadata (e.g. temperature) were served the first producer's cached response. Fixes #4281 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `CacheEntry` gains a `namespace` field. - `SemanticCache.get` / `put` take an optional `namespace` (default `""`). The cache key and exact-match index key are `sha256(namespace + "\0" + base)`, where base is the messages hash or the query-derived fallback. Exact hits also verify `entry.namespace`, and `_find_similar` skips entries from other namespaces. One global LRU / `max_entries` limit is kept. - `SemanticCacheLayer._compute_namespace` builds the namespace as sorted JSON of `provider`, `model`, `metadata`. `request_id`, `timestamp` and `query` are excluded, so they do not affect matching. If the context is not JSON-serializable, the layer skips the cache (lookup delegates straight to the provider optimizer, store returns `""`) instead of sharing entries under an incomplete identity. - Regression tests in `tests/test_cache/test_semantic.py`, including the context-free `store_response` policy: it stores under the unscoped namespace and only matches lookups whose context also has empty provider/model/metadata. ## Testing - [x] Unit tests added / updated - [x] Existing test suite passes locally ``` $ python -m pytest tests/test_cache -q ======================= 313 passed, 2 skipped ======================= $ ruff check headroom/cache tests/test_cache All checks passed! $ ruff format --check headroom/cache tests/test_cache 31 files already formatted ``` `mypy` on `headroom/cache/semantic.py` reports no errors in the changed file (only the pre-existing Windows-only `os.fchmod` / `O_NOFOLLOW` notes elsewhere). ## Real Behavior Proof - Environment: Windows 11, Python 3.13, branch rebased on upstream/main; standalone script importing the public `headroom.cache` API (not the test suite) - Exact command / steps: `python proof.py`. Stores one response under `anthropic/claude-3-opus, temperature=0` via `SemanticCacheLayer.store_response`, then calls `process` with varied contexts on both the exact path and the embedding path (a deterministic `embedding_fn`, and a different query with an identical embedding). Ran the same script against `upstream/main`'s `semantic.py` for comparison. - Observed result: fix branch: same context hit=True, new request_id hit=True, other model / other provider / other temperature hit=False, semantic same context hit=True, semantic other model hit=False, unserializable metadata store key=''. On upstream/main: other model hit=True, other temperature hit=True, semantic other model hit=True (the #4281 bug). - Client-level check: `HeadroomClient(OpenAIProvider(), enable_semantic_cache=True, default_mode="optimize")` over a mocked upstream SDK. One `chat.completions.create(model="gpt-4o")` call (miss, goes upstream), then `client._store_response_in_semantic_cache(...)` populates with the client's own provider+model context, then `create` for `gpt-4o` and `gpt-4o-mini`. Fix branch: gpt-4o served from cache, gpt-4o-mini went upstream (2 upstream calls). upstream/main: gpt-4o-mini was served the gpt-4o cached response (1 upstream call). - Not tested: the proxy (it does not use `SemanticCacheLayer`). Note `HeadroomClient._store_response_in_semantic_cache` has no call site on main, so the client never auto-populates the cache today; the client check above invokes it directly. Wiring it in is out of scope for this fix. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: not applicable (library bug fix) - Stable/default behavior changed: yes, only in that cached responses no longer cross provider/model/metadata boundaries; same-context hits are unchanged - Kill switch / disable path: not applicable; callers that don't use SemanticCacheLayer are unaffected - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
929a40a80e |
fix(cache): validate malformed learned TTL values (#4364)
## Description
The learned-TTL resolver promises best-effort, never-raising behavior,
but after JSON loading it assumed an object and converted numbers
outside its error guard. A nonempty JSON array/scalar raises
AttributeError, and NaN/Infinity raise ValueError/OverflowError. Boolean
and nonpositive TTLs are accepted as short durations, so they can
incorrectly classify a still-warm prefix as cold.
The OpenAI HEADROOM_COLD_RECOMPACT path calls this resolver before its
cold-prefix decision. Ignore malformed learned data instead of letting
optional state interrupt that path or authorize an early rewrite. Valid
model-specific values still take precedence over provider fallbacks.
Found through source inspection. Existing PR2670 implements the
estimator and handles invalid observation rows, not learned-table
validation. No matching fix/issue found in function-name and learned-TTL
searches. Base
|
||
|
|
1bcdca2d9d |
fix(cache): invalidate configured registry instances on replacement (#4363)
## Description
CacheOptimizerRegistry caches explicit configurations under
registration-plus-config-ID keys, but register(override=True) and
unregister only removed the unconfigured registration-name key. Reusing
a CacheConfig after replacing a plugin therefore returned the previous
optimizer implementation; unregister/re-register also resurrected an old
instance.
This fixes that lifecycle invariant: every cached configuration belongs
to the current registration. Existing references already held by callers
remain unchanged; subsequent registry lookups resolve the replacement.
Found through source inspection; no matching existing issue or fix found
in class-name issue/PR searches or the broader optimizer/override PR
search. Base
|
||
|
|
2480100915 |
fix(cache): keep indented lines intact in OpenAI whitespace normalization (#4343)
## Description `OpenAICacheOptimizer._normalize_whitespace` measured a line's indentation, collapsed the line with `" ".join(stripped.split())` (which already drops the indentation), and then sliced `content_part[leading:]`. So every indented line in a system prompt lost its first `leading` characters: `" def f():"` became `" f():"`, `" return x"` became `" "`, and tab-indented lines lost characters and had their tabs turned into spaces. Unindented lines were fine, which is why the existing test didn't catch it. The fix keeps the line's original leading whitespace and appends the collapsed content: `line[:leading] + content_part`. Inner runs of spaces are still collapsed, trailing whitespace is still trimmed, and tabs in the indentation are now kept as tabs instead of being rewritten as spaces. Fixes #4336 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/openai.py`: in `_normalize_whitespace`, emit `line[:leading] + content_part` instead of `" " * leading + content_part[leading:]`. - `tests/test_cache/test_openai.py`: `test_whitespace_normalization_keeps_indented_lines_intact` checks that a space-indented code block round-trips unchanged, that a tab-indented block keeps its tabs while inner spaces are collapsed, and that the same system prompt comes out of `optimize()` unchanged. It fails on `main` and passes with the fix. - Follow-up from review: trim only surrounding newlines at the end of `_normalize_whitespace`, so a prompt that starts with an indented line keeps its indentation (`test_whitespace_normalization_keeps_first_line_indentation`). ## Testing - [x] Unit tests pass (`pytest`) - `tests/test_cache/`: 298 passed, 2 skipped - [x] Linting passes (`ruff check .`) - `ruff check` and `ruff format --check` on the two changed files - [x] Type checking passes (`mypy headroom`) - `mypy headroom/cache/openai.py`, no issues - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New test on `main` (fix stashed): ```text $ pytest tests/test_cache/test_openai.py -q -k indented E AssertionError: assert 'Example:\n f():' == 'Example:\n ... return x' E Example: E - def f(): E ? ---- - E + f(): E - return x ======================= 1 failed, 13 deselected in 0.26s ======================= ``` With the fix: ```text $ pytest tests/test_cache/test_openai.py -q -k indented ======================= 1 passed, 13 deselected in 0.24s ======================= $ pytest tests/test_cache/ -q ======================= 298 passed, 2 skipped in 15.08s ======================== ``` ## Real Behavior Proof - Environment: macOS 26.2, Python 3.14.7, repo deps via `uv sync --extra dev`, optimizer resolved the same way `HeadroomClient` does it (`CacheOptimizerRegistry.get("openai")`), default `CacheConfig`. - Exact command / steps: a small script that gets the `"openai"` optimizer from the registry and runs `optimize()` on a system prompt with a space-indented code block, a nested bullet and a tab-indented block: ```python from headroom.cache import OptimizationContext from headroom.cache.registry import CacheOptimizerRegistry opt = CacheOptimizerRegistry.get("openai") prompt = "You are a code reviewer. Follow this style:\n def handler(event):\n return process(event)\n - nested bullet\n\tif ok:\n\t\treturn x" res = opt.optimize([{"role": "system", "content": prompt}, {"role": "user", "content": "hi"}], OptimizationContext(provider="openai", model="gpt-4o")) print(repr(res.messages[0]["content"])) ``` - Observed result: indentation and text are preserved after the fix. Before and after output: Before (`main`): ```text 'You are a code reviewer. Follow this style:\n handler(event):\n rocess(event)\n nested bullet\n f ok:\n turn x' ``` After: ```text 'You are a code reviewer. Follow this style:\n def handler(event):\n return process(event)\n - nested bullet\n\tif ok:\n\t\treturn x' ``` Indentation and text are preserved; the double space in `nested bullet` and `return x` is still collapsed. - Not tested: a live request through `HeadroomClient` to the OpenAI API (no key used); the optimizer output above is exactly what that path sends upstream. The Anthropic and Google optimizers were not touched. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, only in that indented system-prompt lines are no longer truncated. Tab indentation is now kept as tabs instead of being converted to spaces. - Kill switch / disable path: `CacheConfig(normalize_whitespace=False)` (unchanged) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Comments and docs: the existing comments already describe the intended behavior ("Collapse multiple spaces in content (not indentation)"), so no new comment or doc change was needed. The cache prefix hash for prompts with indented lines will change once, since the normalized text is now correct. --------- Co-authored-by: Abelo9996 <Abelo9996@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
1cf4d053d2 |
fix(cache): stop Claude Code forks from cold-starting the conversation's lineage (#4043)
### Maintainer qualification refresh (2026-10-09) Current head: `a91a1ec24ab20b6dcea0f6e7f8d41feb7d0beea0`. The newly supplied live Claude Code/Anthropic before-after evidence resolves the previous runtime-proof hold. Clean synchronization with current main `01d9194` preserves contributor logic. Independently reran all 88 fork-lineage/prefix tests with the existing native extension; current-head hosted checks are green. Approved for human merge review. The live capture below remains correctly attributed to the preceding implementation head, not represented as a new capture of this synchronization commit. No merge performed. ## Description Claude Code sends side requests alongside an agent's turns. The subagent progress summary and the parent session's side query both re-send the history plus one prompt, with the breakpoint one message earlier, so they read the prompt cache without writing it. resolve_tracker matched such a fork as a continuation and replaced the lineage snapshot with it. The next real turn extends the history before the fork, not the fork, so it matched no lineage. It started a cold tracker (frozen=0), and the token-mode pipeline rewrote history the provider had already cached. Production, 2026-10-08: a review subagent busted Anthropic's cache on 35 of 81 turns. cache_read_input_tokens fell to the 6,065-token system prompt while 100-150k tokens were re-written. Its 600k-token parent did the same after its own side queries. In the proxy log, every bust is the first main request after a fork and shows no frozen prefix. Fork-free sessions on the same proxy were 96/96 and 130/130 full hits. A lineage now keeps its last three request snapshots. A request that extends an earlier one resumes that turn instead of starting cold. Each response is also recorded against the turn that sent it, through a request-scoped ContextVar (the same pattern as the tenant key and CCR store). A fork's response can no longer replace the main line's frozen prefix or overlay source, whether it lands before or after the in-flight main turn's response. Templated fan-out siblings that extend a shared earlier snapshot now stay on one tracker and each resumes its own turn's state. Splitting them would mint a new lineage per fork and exhaust the per-session cap. The fan-out test asserts the per-line state instead of object identity. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: each lineage keeps a `deque(maxlen=_LINEAGE_HISTORY)` (3) of request snapshots instead of one. `resolve_tracker` resumes the turn a request extends, and the turn it resolved is held in a request-scoped `ContextVar` (`_resolved_turn`) so the response is recorded against that turn. - `tests/test_anthropic_fork_lineage_cache.py` (new): handler-level regression. A main turn after a fork (progress summary or side query) must replay its cached prefix byte for byte, and a fork must not replace the lineage it branched from, whichever response lands first. - `tests/test_cache/test_prefix_tracker.py`: the fan-out test asserts per-line state instead of tracker object identity. ## Testing - [x] Unit tests pass (`pytest tests/test_anthropic_fork_lineage_cache.py tests/test_cache/ tests/test_proxy_anthropic_cache_stability.py`: 272 passed, with `test_cache/test_dynamic_detector.py` run on its own: 57 passed, 2 skipped) - [x] Linting passes (`ruff check` on the changed files) - [x] Type checking passes for the changed module (`mypy headroom/cache/prefix_tracker.py`: no issues) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # new test file, without the fix (parent commit) FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[summary] FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[side-query] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[fork-first] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[main-first] AssertionError: main turn 2 re-sent turn 1's history with different bytes ============================== 4 failed ============================== # with the fix ============================== 4 passed ============================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.16, real Claude Code 2.1.295 sessions against the Anthropic API through `headroom proxy --mode token` (details below). - Exact command / steps: the scripted Claude Code session described below, run once on `17abee27` (before) and once on `9b27664a` (after). - Observed result: main turns right after a fork arrived with a cold tracker 5/5 before, 0/5 after; cached tokens lost 3,363 before, 0 after. - Not tested: the full collapse to the system prompt seen in production fork-heavy sessions (did not reproduce in this short session); a deployed proxy running this fix. Live before/after capture on 2026-10-09. Real Claude Code 2.1.295 sessions against the Anthropic API (subscription login, `claude-sonnet-5-5`), through `headroom proxy --mode token` on Windows 11 / Python 3.13.16, with the native extension built in each tree. - **Before:** `17abee27` (merge base with `main`) - **After:** `9b27664a` (this PR) Both ran the same scripted session: a stream-json SDK client with `agentProgressSummaries` and `promptSuggestions` on, one general-purpose subagent that `cat`s ten ~40 KB JSON files one tool call at a time, then three parent follow-up turns. Each request's kind comes from Claude Code's own debug log (`[API REQUEST] source=…`). `sdk` and `agent:*` are main turns; `agent_summary` and `prompt_suggestion` are forks. Token counts are the upstream `usage` values from the proxy's PERF lines. | Main turns right after a fork | Before | After | |---|---|---| | Count | 5 | 5 | | Arrived with a cold tracker (no `Pipeline: freezing` line) | **5/5** | **0/5** | | Read the full prior prefix | 4/5 | **5/5** | | Cached tokens lost | 3,363 | 0 | - **Before**, subagent turn #15 (after its 23-message `agent_summary` fork): cold, `cache_read` 22,137 against the 25,500 cached by the previous turn, `cache_creation` 4,902. - **After**, the matching subagent turn #17 (after a 21-message `agent_summary` fork): frozen 20/23, `cache_read` 25,311 = 24,192 + 1,119, exactly the previous turn's read + write. This short session did not reproduce the full collapse to the system prompt (~6k read) seen in production fork-heavy sessions. In most cold turns the pipeline happened to regenerate identical bytes. The capture does show the mechanism deterministically: before, every fork leaves the next main turn of that conversation starting cold, and one of those cold turns rewrote 3.4k cached tokens. After, none start cold and every post-fork turn reads its whole prior prefix. <details><summary>Per-request tables (before / after)</summary> **Before (`17abee27`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 0 | 15,072 | | 2 | main (subagent) | 2 | - | 5,406 | 7,859 | | 3 | main (subagent) | 5 | 3 | 13,265 | 660 | | 4 | main (subagent) | 7 | 3 | 13,925 | 1,176 | | 5 | main (subagent) | 9 | 3 | 15,101 | 1,319 | | 6 | main (subagent) | 11 | 3 | 16,420 | 1,138 | | 7 | fork (agent_summary) | 11 | 3 | 16,476 | 0 | | 8 | main (subagent) <- after fork | 13 | - | 17,558 | 1,127 | | 9 | main (subagent) | 15 | 3 | 18,685 | 1,134 | | 10 | main (subagent) | 17 | 3 | 19,819 | 1,126 | | 11 | main (subagent) | 19 | 3 | 20,945 | 1,136 | | 12 | main (subagent) | 21 | 3 | 22,081 | 1,133 | | 13 | main (subagent) | 23 | 3 | 23,214 | 2,286 | | 14 | fork (agent_summary) | 23 | 22 | 23,321 | 0 | | 15 | main (subagent) <- after fork | 25 | - | 22,137 | 4,902 | | 16 | main (subagent) | 27 | 24 | 27,039 | 1,260 | | 17 | main (subagent) | 29 | 24 | 28,299 | 7,523 | | 18 | main (parent) | 4 | 3 | 15,072 | 2,204 | | 19 | fork (prompt_suggestion) | 6 | 5 | 17,276 | 335 | | 20 | main (parent) <- after fork | 6 | - | 17,611 | 31 | | 21 | fork (prompt_suggestion) | 8 | 7 | 17,642 | 362 | | 22 | main (parent) <- after fork | 8 | - | 18,004 | 29 | | 23 | main (parent) | 10 | 9 | 18,033 | 242 | | 24 | fork (prompt_suggestion) | 12 | 9 | 18,275 | 128 | | 25 | main (parent) <- after fork | 12 | - | 18,403 | 18 | | 26 | fork (prompt_suggestion) | 14 | 9 | 18,421 | 142 | **After (`9b27664a`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 5,406 | 9,666 | | 2 | main (subagent) | 2 | - | 10,523 | 2,742 | | 3 | main (parent) | 4 | 3 | 15,072 | 692 | | 4 | main (subagent) | 5 | 3 | 13,265 | 1,681 | | 5 | fork (prompt_suggestion) | 6 | 3 | 15,764 | 25 | | 6 | main (subagent) | 7 | 3 | 14,946 | 1,246 | | 7 | main (subagent) | 9 | 3 | 16,192 | 1,138 | | 8 | main (subagent) | 11 | 3 | 17,330 | 2,303 | | 9 | main (subagent) | 13 | 3 | 19,633 | 1,167 | | 10 | main (subagent) | 15 | 3 | 20,800 | 1,137 | | 11 | main (subagent) | 17 | 3 | 21,937 | 1,129 | | 12 | main (subagent) | 19 | 3 | 23,066 | 1,126 | | 13 | main (subagent) | 21 | 20 | 24,192 | 1,119 | | 14 | main (parent) <- after fork | 6 | 3 | 15,789 | 31 | | 15 | fork (prompt_suggestion) | 8 | 3 | 15,820 | 108 | | 16 | fork (agent_summary) | 21 | 20 | 24,248 | 0 | | 17 | main (subagent) <- after fork | 23 | 20 | 25,311 | 710 | | 18 | main (parent) <- after fork | 8 | 3 | 15,928 | 851 | | 19 | fork (prompt_suggestion) | 10 | 3 | 16,779 | 688 | | 20 | main (parent) <- after fork | 10 | 3 | 17,467 | 29 | | 21 | main (parent) | 12 | 3 | 17,496 | 130 | | 22 | main (parent) | 14 | 3 | 17,626 | 249 | | 23 | fork (prompt_suggestion) | 16 | 3 | 17,875 | 131 | | 24 | main (parent) <- after fork | 16 | 3 | 18,006 | 18 | | 25 | fork (prompt_suggestion) | 18 | 3 | 18,024 | 136 | </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, prefix-tracker lineage matching for requests that extend an earlier snapshot of the same lineage - Kill switch / disable path: none - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Not covered here: a few small partial misses in production that forks don't explain, and a synthetic-only risk that an older tool result gets recompressed once it falls past a lagging frozen boundary. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4043" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
9553d4292f |
fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description Session cleanup currently drops prefix trackers after the default 600-second session lifetime even when Anthropic's request selects a one-hour cache TTL. Losing the tracker also loses the frozen forwarded prefix and causes the cache-preserving overlay to replay different bytes. Closes #4017. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Resolve the request's existing Anthropic cache TTL before selecting the prefix lineage and pass that TTL through tracker selection/creation. - Retain each lineage for the greater of the configured session floor and its observed/current provider TTL plus the existing 60-second safety margin. - Keep the lifetime per lineage rather than changing global configuration: a five-minute session does not inherit another session's one-hour lifetime. - Preserve normal expiry beyond the provider lifetime and margin. - Honor configured cache TTL before a confirmed cache write; discard expired state before accepting a later request's longer TTL. - Re-register the live lineage family after lookup/cleanup so recreated sessions cannot share a tracker's frozen state across sibling conversations or tool profiles. - Update existing fake callbacks to the changed tracker contract, preserve all consumer byte-stability assertions, and remove one handler-to-store wiring-only test rather than re-pinning its internal identity assertions. - Document local tracker retention separately from provider cache availability. ## Testing - [x] Unit tests pass (`pytest`), including frozen-prefix byte stability and independent TTL expiry. - [x] Changed-file lint, configured pre-commit mypy and manual production timestamp/overlay checks pass. ### Test Output ```text Review correction: 273 passed, 2 dependency warnings in 83.79s {"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true} ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate explicitly used the installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 12.87s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_cache/test_prefix_tracker.py python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py ``` **74 passed** in tracker tests; **54 passed** in request/cache tests (one existing Starlette TestClient deprecation warning). Two new regressions fail on baseline because the request TTL is not accepted; a separate production runtime comparison demonstrates the actual pre-fix loss, not just the missing parameter. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together with `tests/test_prefix_tracker_client_rewrote_reply.py`, `tests/test_cache_mode_delta_marker.py`, and `tests/test_content_router_mcp_small_result.py`: **150 passed**. The actual production timestamp/byte-retention smoke was repeated successfully on the rebased code, covering compatibility with main's newly merged client-reply normalization. Review regression and affected HTTP-consumer suite: **273 passed**, including tracker expiry, Anthropic/OpenAI cache stability, byte-faithful forwarding, header isolation, buffered timeouts, CCR injection, and immutable system prompts. All affected two-argument test callbacks now accept the TTL contract. The obsolete walker-call-order test was removed; wire-byte assertions remain. The new configured-TTL/expiry cases fail against the published baseline; the post-cleanup regression also verifies that an unrelated conversation gets a different tracker after recreation. Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644). A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit hook passed. No unrelated source/permissions changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real `SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token estimator and `overlay_cached_prefix`, no mocked store or mocked clock. - Exact command / steps: parse `cache_control: {type: ephemeral, ttl: 1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed frozen message with different original/forwarded prefix bytes; age `_last_activity` to `time.time()-660` and make periodic cleanup due; resume the same lineage and compare the tracker, frozen count, and forwarded-prefix bytes. Also age beyond 3901 seconds and compare independent 300/3600-second sessions. - Before: resolved TTL 3600; after 660 seconds, tracker identity changes, frozen count drops **1→0**, and forwarded-prefix bytes differ. - Observed result: after the fix, after 660 seconds the same tracker remains, frozen count stays **1**, and forwarded-prefix bytes are identical. The one-hour session survives independently, the five-minute session expires, and the one-hour lineage still expires beyond TTL plus margin. - Review runtime proof against published head `e1a107df`: actual stores and prefix overlays confirm configured TTL retention before the first cache write, no resurrection of expired state by a later one-hour request, eventual expiry, and separate trackers after the maintainer's 3901-second cleanup/recreation sequence. On that exact baseline the unrelated lookup reuses the wrong tracker, although the overlay guard preserves the particular unrelated input bytes; after the fix both identity isolation and byte preservation hold. Identical messages under different tool profiles also receive a cold, separate tracker after the expired bare-key transition. - Not tested: Anthropic cloud cache availability, billed savings, or an actual one-hour wait. The local production state and byte invariants were exercised by aging actual timestamps, not by a paid upstream request. Preserving a tracker cannot guarantee that the provider retains its cache. ## Runtime Rollout Safety - Rollout-managed feature(s): none; repair of existing TTL-aware cache-prefix behavior. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: eligible local prefix trackers survive for their provider cache lifetime. Short-TTL/default sessions remain bounded by their own session floor. - Kill switch / disable path: the existing prefix-freeze disable path remains available; no new flag. - Unsafe override required: no. - Qualification impact: local Anthropic prefix lineage selection/retention, not cache pricing or subscription quota policy. - Rollback path: revert the commit to restore previous retention. No persistent schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing cache TTL parser and safety margin reused; cache docs updated. - [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes, and existing request byte-stability behavior. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. Remote CI results are not inferred from local verification. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
16eaec1e95 |
fix(cache): keep prefix lineage across a replaced system tail (#3934)
## Description Claude Code can end a `/v1/messages` request with a temporary `role:"system"` reminder and replace it on the next turn. `SessionTrackerStore.resolve_tracker` records the full history, so the stored chain ends in the old reminder and is never a prefix of the next request: `_classify_history_canonical` returns `diverged` and the request gets a fresh, empty tracker. The confirmed frozen prefix and the previously forwarded messages are gone, the #3380 replay has nothing to replay, and any recompressed history re-writes the provider prompt cache. The reminder moves every turn, so this repeats for the whole session. This change lets lineage resolution set aside that one replaced trailing system message, only when nothing matched as recorded. It changes tracker selection only: forwarded bytes, the replay body, and breakpoint placement are untouched, and the existing replay already stops at the first changed message, so the new reminder goes out as the client sent it. Closes #3933 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: new `_extends_past_system_tail(current, previous)`. It is true when `previous` ends in a `role:"system"` message, everything before that message includes user/assistant history and is unchanged in `current`, and `current` is longer than `previous`. Because `current` must be longer, siblings that only swap their final instruction stay separate. - `SessionTrackerStore.resolve_tracker`: after the existing exact/append, block-append and block-rewrite passes find nothing (and there are no rewrite candidates, so an ambiguous rewrite tie is not overridden), Anthropic requests consider chains with the same cache affinity that pass that check. Only a unique longest candidate is taken; two equal-length candidates start a fresh lineage, as before. The full current snapshot is stored as usual. - `extract_cache_stable_delta` (cache mode's prefix/delta split): a `diverged` request that replaced the previous trailing system reminder is split too. Every recorded message before the reminder, plus the model reply the handlers record after it, must match the current request position for position (canonical compare, same rule via `_extends_past_system_tail`). The previously forwarded bytes are replayed without the old reminder, and everything after it is the delta to compress. Before this, cache mode tagged these turns `cache_mode_prefix_mismatch` and forwarded the new messages raw. - `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns off both the lineage fallback and the delta split. It uses the same parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`. - `tests/test_cache/test_prefix_tracker.py`: - Tracker retention: the tracker and its confirmed frozen count are kept, and the existing overlay restores the unchanged prefix. - A reminder that moves and changes over several turns: full snapshots are stored. - Isolation: different cache affinity, OpenAI provider, changed earlier history, changed leading system message, changed historical system message, same-length siblings `[user: task, system: A]` vs `[user: task, system: B]`, ambiguous equal-length siblings, exact-match precedence, and the kill switch. - The delta split: with and without the recorded reply, a system message followed by a client turn is not treated as a reminder, a mismatched reply is rejected, and the kill switch disables it. - `tests/test_cache_mode_cold_start.py`: drives `handle_anthropic_messages` in `mode="cache"` with the real `SessionTrackerStore` over three turns. The first turn compresses a tool result and later turns replace the reminder. The test asserts that the third request replays the second request's compressed prefix byte-for-byte (ignoring `cache_control`), compresses the new tool result, and ends in the current reminder. ## Testing - [x] Unit tests pass (`pytest`) - targeted files below, not the whole suite - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - on `headroom/cache/prefix_tracker.py` - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New tests with the source at `main` ( |
||
|
|
7ce2580ff9 |
fix(cache/google): tolerate timezone-aware cache expiry timestamps (#3210)
## Description
`GoogleCacheOptimizer` tracks Google `CachedContent` lifecycles via
`CachedContentInfo`, whose `is_expired` / `ttl_remaining_seconds` /
`age_seconds` compared a stored `expires_at` / `created_at` against a
**naive** `datetime.now()`:
```python
@property
def is_expired(self) -> bool:
return datetime.now() >= self.expires_at
```
The documented integration registers the timestamp straight off Google's
SDK:
```python
cached_content = genai.caching.CachedContent.create(model=..., contents=..., ttl=timedelta(hours=1))
optimizer.register_cache(cache_id=cached_content.name, ..., expires_at=cached_content.expire_time)
```
`CachedContent.expire_time` is a timezone-**aware** RFC3339 value.
Comparing or subtracting it against a naive `datetime.now()` raises
`TypeError: can't compare offset-naive and offset-aware datetimes`. This
crashes `register_cache` itself (it logs `ttl_remaining_seconds`), and
every consumer of `is_expired`: `get_reusable_cache`,
`cleanup_expired_caches`, and the `list_*` helpers. A single aware entry
also poisons `cleanup_expired_caches`, which iterates `is_expired` over
the whole registry. `CachedContentInfo.from_dict` reconstructs the same
aware value from a persisted `expires_at`, so the crash survives a
save/restore round-trip.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/google.py`: added a `_now_matching(reference)` helper
that returns `datetime.now(reference.tzinfo)`, and use it in
`CachedContentInfo.is_expired`, `ttl_remaining_seconds`, and
`age_seconds`. "Now" is anchored to the same awareness as the value it
is measured against, so an aware `expires_at` compares against an aware
now, and a naive one keeps the exact prior behavior
(`datetime.now(None)` is naive local time).
- `tests/test_cache/test_google.py`: added
`test_cache_registration_accepts_aware_expiry`,
`test_cache_lookup_expired_aware`, and
`test_from_dict_roundtrip_preserves_aware_expiry`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_cache/test_google.py -> 21 passed in 0.72s
uvx ruff@0.16.2 check headroom/cache/google.py tests/test_cache/test_google.py -> All checks passed!
uvx mypy@1.20.2 headroom/cache/google.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: before the fix, `optimizer.register_cache(...,
expires_at=datetime.now(timezone.utc) + timedelta(hours=1))` raised
`TypeError: can't subtract offset-naive and offset-aware datetimes` at
`google.py:108` (`ttl_remaining_seconds`, called from `register_cache`'s
log line). After the fix, the same call registers cleanly,
`get_reusable_cache` returns the entry, `cleanup_expired_caches` returns
`[]`, an aware past-expiry entry reports `is_expired == True`, and naive
registrations behave exactly as before (`is_expired == False`,
`ttl_remaining_seconds > 0`).
- Observed result: aware and naive expiry timestamps both work; no
`TypeError`; naive-path values unchanged.
- Not tested: no live `genai.caching.CachedContent.create` call (the
optimizer intentionally makes no API calls); the aware timestamp Google
returns is reproduced with `datetime.now(timezone.utc)`, which has the
same awareness.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. `GoogleCacheOptimizer` is an
SDK-side cache lifecycle utility (`headroom.cache`), not a
rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Naive timestamps take
`datetime.now(None)`, identical to the previous `datetime.now()`; only
the previously-crashing aware case changes, from `TypeError` to a
correct comparison.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none; behavior for existing naive callers is
byte-for-byte unchanged.
- Rollback path: revert this PR; the properties return to a naive
`datetime.now()`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, unchanged semantics)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
`_now_matching` is deliberately minimal: it does not rewrite stored
timestamps or change the serialization format, so a mixed registry of
naive and aware entries each compare correctly on their own terms. This
matches the module's existing convention of leaving caller-supplied
timestamps as-is.
|
||
|
|
52c7aa2e6f |
feat(cache): add the headroom-cache-ttl offline TTL estimator the docs reference (#2670)
## Description
Adds the missing offline estimator half of the cache-TTL learning seam:
a `headroom-cache-ttl` console script that reads
`cache_ttl_observations.jsonl` (written by the proxy under
`HEADROOM_CACHE_TTL_LEARN=1`) and writes the `cache_ttl_learned.json`
table that `resolve_learned_ttl()` already consults.
The docs and `headroom/cache/ttl_observations.py`'s docstring both
reference this estimator as the thing that turns observations into
learned TTLs — but the tool does not ship:
```
$ git grep -l "headroom-cache-ttl" v0.33.0
docs/content/docs/configuration.mdx
headroom/cache/ttl_observations.py # docstring reference only
$ headroom-cache-ttl
command not found
```
So today `HEADROOM_CACHE_TTL_LEARN=1` collects observations that nothing
ever reads, and `HEADROOM_COLD_RECOMPACT` cold detection for providers
that don't expose a TTL (OpenAI/Codex, Kimi) stays on the static 300s
guess forever — exactly the aggressive-in-the-wrong-direction value the
docs warn about (a resume in the 6–60 min band can recompact a
still-warm OpenAI cache).
Filed as an implementation of the tool the docs already describe rather
than a new design; happy to move/rename if you'd prefer a `headroom`
subcommand over a second console script.
Closes #
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `headroom/cache/ttl_estimator.py` with the `headroom-cache-ttl`
CLI (`[project.scripts]` entry in `pyproject.toml`, named exactly as the
docs reference it). Stdlib only, no new dependencies.
- **Interval estimation, per the contract documented in
`ttl_observations.py`:** hit idles bound the TTL from below (cache
proven alive), `ttl_expiry` miss idles bound it from above (cache proven
dead). The estimator emits the interval's **upper** end — smallest
observed death-idle beyond the largest observed life-idle — because the
consumer treats a turn as cold only when `idle > ttl`: overestimating
merely skips a recompaction, underestimating would bust a warm cache.
- Keys with no death-evidence beyond their last observed life are
**skipped** (no safe upper bound → static default stays). Sample floors:
≥3 hits and ≥3 `ttl_expiry` misses per key (`--min-hits` /
`--min-expiry-misses`).
- Emits `provider/model` keys **only** — no bare-`provider` aggregate. A
`ttl_expiry` miss for model B says nothing about model A's TTL, so a
pooled per-provider interval could hand the consumer a TTL below a
model's real one (the costly direction). Unseen models fall back to the
consumer's static default; extra metadata fields (`max_hit_idle`, sample
counts, `updated_at`) ride along and are ignored by the reader.
- Atomic tmp+rename write, **merged** with the existing table so
observation-window rotation doesn't un-learn old keys. On every write,
bare-provider aggregates left behind by earlier revisions of this tool
(recognized by their generated metadata shape, e.g. `max_hit_idle`) are
purged; operator-written provider fallbacks (`{"ttl_seconds": N}` or a
plain number, which `resolve_learned_ttl()` still supports) are
preserved.
- Offline/out-of-process by design (zero live-feedback risk on the hot
path); the proxy picks up a rewritten table by mtime, no restart needed.
- Docs: "Running the estimator" section added under the cold-prefix hook
in `configuration.mdx`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
New `tests/test_cache/test_ttl_estimator.py` — 23 tests: upper-bound
selection, overlapping bounds (variable eviction) skipping, sample
floors, non-`ttl_expiry` misses ignored, malformed/non-finite rows and
lines skipped, one model's death evidence cannot close another's
interval (no provider pooling), merge-preserves-existing-keys,
legacy-aggregate purge with manual-fallback preservation (upgrade
regression, both directions), corrupt-existing-file recovery, and CLI
end-to-end through `resolve_learned_ttl()` (env paths, explicit flags,
`--dry-run`).
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_cache/
======================= 277 passed, 2 skipped in 16.54s ========================
$ uvx ruff check headroom/cache/ttl_estimator.py tests/test_cache/test_ttl_estimator.py
All checks passed!
$ uvx ruff format --check headroom/cache/ttl_estimator.py tests/test_cache/test_ttl_estimator.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/cache/ttl_estimator.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15 (Darwin 24.6.0), Python 3.12 via `uv run
--frozen`, branch `feat/cache-ttl-estimator` on top of `main`.
- Exact command / steps: wrote a synthetic observation log in the exact
row shape `record_cache_observation` emits (3 hits at idle 120/300/480s,
3 `ttl_expiry` misses at 600/900/1200s for `openai/gpt-5.5`), plus a
pre-existing `learned.json` containing a legacy `openai` estimator
aggregate and a hand-written `kimi` fallback, then (at
|
||
|
|
455f4f263c |
fix(cache/semantic): don't semantic-match an empty query across contexts (#3226)
## Description
`SemanticCache.get()` matches on the **embedding of the last user
message** whenever an `embedding_fn` is wired. That query is empty
(`""`) for the overwhelming majority of agent/tool turns — a
`tool_result` continuation carries no text block, so
`SemanticCacheLayer._extract_query` returns `""`. A real sentence
embedder maps `""` to a fixed **non-zero** vector, so every empty-query
turn is ~identical to every other in embedding space. The exact
`messages_hash` guard (correctly chosen so `"continue"`/`"yes"` turns in
different contexts don't collide) is then bypassed by the semantic path:
an empty-query request misses on its unique hash, falls through to
embedding matching, and hits a **different conversation's** stored
response.
Reproduction (realistic embedder, non-zero for `""`):
```python
c = SemanticCache(embedding_fn=embed)
c.put(query="", response={"answer": "A"}, messages_hash="ctxA") # conversation A
c.get(query="", messages_hash="ctxB") # conversation B, different context
# -> returned A's response (cross-context false hit)
```
Measured on 330 real Claude Code transcripts (28,441 requests): **95.7%
have an empty extracted query**, so this is the dominant case, not a
corner case. The exact-hash path is unaffected; only the
embedding-similarity path is.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/semantic.py`:
- `get()`: gate the semantic-similarity branch on `query.strip()` — an
empty/blank query can only ever hit via its exact `messages_hash`
(context-complete), never via embedding similarity.
- `put()`: store no embedding for an empty/blank query, so such an entry
is skipped by `_find_similar` (which ignores entries with no embedding)
and can never be a match target.
- `tests/test_cache/test_semantic.py`: added
`test_empty_query_never_semantic_matches` (cross-context empty-query
miss, exact-hash still hits, whitespace treated as empty).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_cache/test_semantic.py -> 22 passed in 2.39s
uvx ruff@0.16.2 check headroom/cache/semantic.py tests/test_cache/test_semantic.py -> All checks passed!
uvx mypy@1.20.2 headroom/cache/semantic.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: before the fix, two different-context
empty-query requests (`ctxA` then `ctxB`) returned `ctxA`'s response via
the embedding path. After the fix, the second returns `None`, while
`ctxA`'s own exact-hash lookup still returns its response, and a
legitimate non-empty semantic hit (`"What is the weather today?"` ->
`"How is the weather?"`) still works.
- Observed result: empty/blank queries no longer semantic-match across
contexts; exact-hash and non-empty semantic matching are unchanged.
- Not tested: no live embedder model wired (the current client wires
none — the embedding path is exercised with an injected `embedding_fn`,
which is the documented usage).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. `SemanticCache` is an SDK-side cache
(`headroom.cache`), not a rollout-channel-gated runtime feature;
semantic matching only runs when a caller injects an `embedding_fn`.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Exact-hash matching and non-empty
semantic matching are unchanged; only empty/blank-query semantic
matching (a false-hit source) is removed.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none; correctness-only.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
|
||
|
|
7bfb1d7f38 |
fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)
## Description Running headroom as the proxy for Claude Code destroys Anthropic prompt-cache reuse (#2085: ~4.4x cache-creation inflation, 2.5–3x net cost). Tracing live Claude Code traffic through the proxy shows **two independent session-identity defects**, both of which orphan or thrash the frozen-prefix state; this PR fixes both. ### Defect 1: `<system-reminder>` turns rotate the fallback session id mid-conversation Claude Code interleaves reminder turns into the history as actual `role:"system"` messages (hook output, skills lists, file-truncation notices). `compute_session_id` hashed **every** system message, so the id rotated each time a reminder landed. Live trace (subagent reading two 80KB files; sid changes exactly when the truncation reminder appears, and the tracker restarts at turn 0): ``` REQ#2 sid=68d4ee666990 nmsg=3 [0]SYSTEM<<top-level system>> [1]user [2]SYSTEM<<skills reminder>> REQ#3 sid=6944948c9fb2 nmsg=6 ... [5]SYSTEM<<Truncated: PARTIAL view ...>> <- id rotated ``` Everything keyed on the session id is orphaned at that moment: the prefix tracker (freeze never survives past a reminder-bearing turn), beta-header stickiness, the CCR and memory-tool registries, and the compression cache. **Fix:** hash only the **leading run** of system messages (everything before the first non-system turn) — the top-level system prompt on the Anthropic path (folded in as the synthetic first message), the conventional leading system message(s) on the OpenAI path. Stable for the life of a conversation; mid-history system turns are content, not identity. ### Defect 2: conversations sharing a (now stable) id thrash one tracker With ids stable, the fallback tuple `model + system prompt` is identical across every same-type parallel subagent (and any sessions reusing one system prompt) — all of them collapse onto one `PrefixCacheTracker`, and their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider cache is re-written instead of read. Reproduced against the real code paths (script below): ``` 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` **Fix:** `SessionTrackerStore.resolve_tracker` — within a session id, reuse the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one); a diverging or rewritten history (client-side compaction) starts a fresh lineage. Matching uses the repo's existing canonical cross-turn equivalence (`_canonicalize_for_prefix_compare`, the same one the cache-stable delta path uses) on the **original client bytes**, so moved cache breakpoints, string<->block sugar, transport annotations, or a tail-mutating `pre_compress` hook never read as a rewrite. Byte-identical histories (templated fan-outs before they diverge) intentionally share a tracker — their provider cache line is identical too. ### Both fixes together, on live Claude Code traffic (sonnet, 2 parallel Explore agents) ``` main conversation: sid=5b7e245a... one tracker, turns 0->4, id stable across reminders agents (collide): sid=2bdffc9e... -> lineage bare (alpha) turns 0->1->2 -> lineage "~1" (beta) turns 0->1->2 ``` Before: the agents' ids rotated per reminder (every tracker stuck at turn 0), and whenever they did share an id they thrashed one tracker (`0/9` stable prefixes in the repro). ### Why not key the session id on conversation content? Draft #1912 folds the first user turn into the fallback id; this change composes with it, but identity-level keying alone can't close #2085: identical first turns (templated fan-outs) still collide, and everything keyed on the session id rotates with it when the client rewrites history. The "session" (client/workspace grouping) and the "conversation" (positional cache lineage) are different identities; only the tracker holds positional per-turn state that thrashes under collision — beta stickiness is a monotone union and the compression cache is content-addressed — so lineage resolution lives one level below the session id and leaves the id semantics (and every other consumer) untouched. ## Changes Made - `headroom/cache/prefix_tracker.py`: - `compute_session_id`: harvest only the leading system run (defect 1). - `SessionTrackerStore.resolve_tracker`: conversation-lineage resolution (defect 2). First lineage lives under the bare session id — single-conversation sessions behave byte-identically to before; degrades to `get_or_create` when messages are absent or prefix freeze is disabled. - Lineages are capped per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32). **Over-cap conversations share one overflow tracker instead of evicting an established lineage** — any eviction policy degrades every conversation once the working set exceeds the cap (under round-robin the victim is always the conversation about to arrive), while overflow sharing degrades only the over-cap tail, to exactly the pre-lineage shared behavior; `0` disables lineage splitting. Chains are stored as structural snapshots that normalize `NaN` (`json.loads` accepts bare NaN, and `NaN != NaN` would read a byte-identical resend as a rewrite). Synthetic lineage keys use a `\x00` separator, which cannot appear in an HTTP header value, so they can never collide with a client-supplied `x-headroom-session-id`. - `headroom/proxy/handlers/anthropic.py`, `openai.py`: the session id and the lineage both derive from the **same original client bytes** (a turn-dependent hook rewrite can no longer rotate one without the other); anthropic folds in its synthetic system message so explicit-header clients with different system prompts stay separate. Plus a docstring correction in `streaming.py` that falsely claimed its coarse mid-turn key "mirrors" `compute_session_id`. - `tests/test_cache/test_prefix_tracker.py`: 24 new test cases — reminder-rotation regression; interleaved isolation + per-conversation turn state; identical-first-turn share-then-split; cache_control movement (3 cases); representation churn (string<->block sugar / streaming `index` / Bedrock cachePoint); rewritten history → fresh lineage (compacted / middle-edited / truncated); legacy no-messages / freeze-disabled / empty-canonical fallbacks; NaN-in-tool-payload stability; overflow sharing, established-lineages-survive-cap, and a cap+1 round-robin no-cliff guard; TTL cleanup; session-id-not-rotated-by-lineage guard. One existing test renamed (`uses_all_system_messages` → `distinguishes_leading_system_run`) to match the new contract. - Three SimpleNamespace stub stores in existing tests gained a `resolve_tracker` field (handlers call it unconditionally — a silent `hasattr` fallback would degrade to the pre-fix behavior with no signal). One of them is the cold-start fast-pass suite (#2073), which landed while this branch was in review. - `CHANGELOG.md` entry. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Testing - [x] Unit tests pass (`pytest`) — 11 failed, 8652 passed, 528 skipped in 4:37 (the 11 are pre-existing on unmodified `main` — verified by rerunning the same node ids on a clean checkout: gh-CLI/onnx/PID-reuse/deadline flakes and order-dependent cases, none touching session/cache/proxy paths) - [x] Linting passes (`ruff check .`) — All checks passed (ruff 0.15.17, CI-pinned; `ruff format --check .` clean) - [x] Type checking passes (`mypy headroom`) — Success: no issues found in 471 source files - [x] New tests added for new functionality — 24 test cases; the rotation/isolation/no-cliff ones fail on `main` - [x] Manual testing performed — live Claude Code end-to-end, below ### Test Output ```text $ python -m pytest tests/ -q 11 failed, 8652 passed, 528 skipped, 5857 warnings in 276.68s (0:04:36) # same 11 fail on unmodified main (env/order-dependent: test_wrap_claude_base_url pid-reuse, # copilot_auth gh-cli fallback, image_compression onnx, content_router deadline, rtk/output-shaper/dedup order flakes) $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 63 passed $ uvx ruff@0.15.17 check . && uvx ruff@0.15.17 format --check . All checks passed! / 1208 files already formatted $ mypy headroom Success: no issues found in 471 source files $ python repro_2085.py 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, `uv sync --extra dev --extra proxy`; real Claude Code CLI pointed at the proxy via `ANTHROPIC_BASE_URL=http://127.0.0.1:8790`, real Anthropic backend. - Exact command / steps: ran Claude Code sessions that launch 2–3 parallel Explore subagents (each reading multi-KB JSON files, several tool-loop turns each), with an observability wrapper printing each request's resolved session id, tracker identity, and turn counter inside the proxy. - Observed result: on `main`, subagent session ids rotate on reminder-bearing turns (trackers permanently stuck at turn 0); when conversations do share an id they share one tracker whose turn counter interleaves all of them. On this branch: ids stable for the life of each conversation; colliding subagents resolve to separate lineages (`bare`, `~1`) with clean per-conversation turn progressions (trace above). Unit-level repro shows forwarded-prefix stability going 0/9 → 8/8 for the interleaved shape. - Not tested: reporter-scale cache-economics (his 4.4x needs his long-session workload against a paid backend); happy to coordinate with @RomanAlexanderW on a before/after — the number to watch is the cache-read ratio in Claude Code transcripts recovering toward ~96%. <details> <summary>repro_2085.py</summary> ```python """Repro for #2085: concurrent conversations sharing a fallback session id (same model + system prompt — e.g. a Claude Code session and its parallel subagents) collapse onto one PrefixCacheTracker and thrash its frozen-prefix state -> byte-unstable forwarded prefixes -> the provider prompt cache is re-written on nearly every call. Uses headroom's real code paths. Run from the repo root: python ../repro_2085.py """ from headroom.cache.prefix_tracker import PrefixFreezeConfig, SessionTrackerStore MODEL = "claude-sonnet-5" # Claude Code system prompt: long, static, identical across the main session # and every parallel subagent of the same type. SYSTEM = ("You are Claude Code, Anthropic's official CLI for Claude. " * 40)[:2000] def convo(name: str, turns: int) -> list[dict]: msgs = [{"role": "system", "content": SYSTEM}] for t in range(turns): msgs.append({"role": "user", "content": f"[{name}] user turn {t}: " + ("x" * 800)}) msgs.append( {"role": "assistant", "content": f"[{name}] tool_result {t}: " + ('{"data": 1}' * 200)} ) return msgs class _Req: # request stub: no x-headroom-session-id header headers: dict = {} # --- Part 1: identity collision (real derivation) ---------------------------- store = SessionTrackerStore(PrefixFreezeConfig()) id_a = store.compute_session_id(_Req(), MODEL, convo("A", 3)) id_b = store.compute_session_id(_Req(), MODEL, convo("B", 5)) print(f"1) fallback session ids: A={id_a} B={id_b} -> COLLIDE={id_a == id_b}") # --- Part 2: interleaved conversations thrash the freeze state --------------- def run(interleave: bool, lineage_resolution: bool) -> tuple[int, int, int]: store = SessionTrackerStore(PrefixFreezeConfig()) stable_turns = 0 later_turns = 0 seq = [] for t in range(1, 6): seq.append(("A", convo("A", t))) if interleave: seq.append(("B", convo("B", t))) for _name, msgs in seq: sid = store.compute_session_id(_Req(), MODEL, msgs) if lineage_resolution: tracker = store.resolve_tracker(sid, "anthropic", messages=msgs) else: tracker = store.get_or_create(sid, "anthropic") if tracker._turn_number > 0: later_turns += 1 if tracker._forwarded_prefix_stable(msgs): stable_turns += 1 tracker.update_from_response( cache_read_tokens=5000 * len(msgs), cache_write_tokens=2000, messages=msgs, ) return stable_turns, later_turns, store.active_sessions for label, interleave, fixed in ( ("single conversation, legacy ", False, False), ("interleaved (subagents), legacy ", True, False), ("interleaved, lineage resolution ", True, True), ): stable, later, sessions = run(interleave, fixed) print(f"2) {label}: stable prefix on {stable}/{later} later turns, trackers={sessions}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (CHANGELOG only — no docs describe the tracker store) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Addresses the session-identity mechanisms of #2085; intentionally does not `Closes` it — the reporter should confirm the cache-read ratio recovers on live traffic first. - Composes with draft #1912 (first-user-turn fallback id). - Known bounded tradeoffs (all strictly milder than the per-turn thrash this fixes): a fork-style branch that resends a parent's full history adopts the parent's lineage, costing the parent one cold restart at its next turn; a request that aborts before the response and is retried with different bytes starts a fresh lineage; history truncation/tail-edit starts a fresh lineage even though the shorter provider prefix may still be warm. - Hot-path cost, measured on a 199-message/2.1MB agentic history: canonical projection 0.21ms + structural snapshot 0.92ms + match loop 0.06ms with 32 candidate lineages (2.27ms absolute worst case) ≈ **1.3ms per request** — same order as the handler's existing request deepcopy (0.80ms) and below one `json.dumps` of the body (2.9ms). Chain memory is structure-only (~180-330KB per lineage; message strings are shared with state the tracker already retains). - Known semantic shift to flag: hashing only the leading system run means conversations distinguished ONLY by mid-list system messages (e.g. clients injecting a per-conversation system context late in the list) now share a fallback id. The tracker is protected by lineage resolution; the residual sharing concentrates in the CCR sticky-tool registry and the monotone beta union — the same pre-existing class as same-system-prompt conversations today. Happy to file the CCR-stickiness scoping as a follow-up. - Out of scope, observed while tracing: `SessionCcrTracker.has_done_ccr` mildly cross-contaminates conversations sharing an id (monotone, no thrash) — can file separately if useful. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
f8eaaeb26a |
fix(cache): normalize embeddings before the semantic similarity check (#2122)
## Description
The semantic tier of the dynamic-content detector compares an
unnormalized dot product against a cosine threshold, so it flags almost
everything as dynamic and strips the static content it is supposed to
protect.
`SemanticDetector` pre-computes exemplar embeddings and, per sentence,
scores similarity with `np.dot` and compares to `semantic_threshold`:
```python
self._exemplar_embeddings = self._model.encode(self.DYNAMIC_EXEMPLARS, convert_to_numpy=True)
...
sentence_embeddings = self._model.encode(sentence_texts, convert_to_numpy=True)
similarities = np.dot(sentence_embeddings, self._exemplar_embeddings.T)
...
if max_sim < self.config.semantic_threshold: # semantic_threshold defaults to 0.7
continue
```
`sentence_transformers.encode(..., convert_to_numpy=True)` does **not**
normalize by default. So `np.dot` here is an inner product whose
magnitude scales with the embedding norms (typically ~5-15 for MiniLM),
not a cosine similarity in [0, 1]. Comparing that against
`semantic_threshold=0.7` (documented and configured as a 0-1 similarity)
is a scale mismatch: nearly every sentence clears the threshold, so the
semantic tier classifies almost all text as dynamic, moves it into
`dynamic_content`, and empties `static_content` — busting the very cache
the detector exists to protect.
A standalone repro: an unrelated sentence with a true cosine of ~0.1 to
an exemplar produces a raw dot of ~9.1 (well over 0.7); normalized, it
correctly scores ~0.09 and stays static.
The correct behavior is used by the in-repo siblings:
`prediction/feature_extractor.py` passes `normalize_embeddings=True`,
and `memory/adapters/embedders.py` L2-normalizes before dot-product
similarity. This detector did neither.
## Fix
Pass `normalize_embeddings=True` to both `encode` calls (exemplars in
`__init__` and sentences in `detect`). Both sides of the dot product are
then unit vectors, so `np.dot` is a true cosine similarity in [-1, 1],
comparable to `semantic_threshold`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/dynamic_detector.py`: add `normalize_embeddings=True`
to the exemplar encode (`__init__`) and the sentence encode (`detect`),
with comments explaining the cosine requirement.
- `tests/test_cache/test_dynamic_detector.py`: add
`TestSemanticDetectorNormalization` — a recording fake model asserts
both encode calls pass `normalize_embeddings=True` (via `object.__new__`
for `detect`, and a monkeypatched registry for `__init__`). No model
download needed.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorNormalization
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/cache/dynamic_detector.py
tests/test_cache/test_dynamic_detector.py headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ python -m py_compile headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`, numpy.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the scale mismatch
with a dependency-free numpy script (no sentence-transformers), and left
the full pytest to CI.
- Exact command / steps: built a MiniLM-dimension exemplar direction and
a sentence direction with a true cosine of ~0.1 (genuinely not dynamic),
gave them realistic un-normalized magnitudes (~9 and ~11), and computed
the old `np.dot` of the raw vectors versus the new `np.dot` of the
normalized vectors, against the 0.7 threshold.
- Observed result: old raw dot ~9.1 (far above 0.7 -> the unrelated
sentence is wrongly flagged dynamic); new cosine ~0.09 (below 0.7 ->
correctly kept static), and always within [-1, 1]. The new tests assert
both encode calls pass `normalize_embeddings=True`.
- Not tested: a real sentence-transformers model end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
adds one keyword argument to two `encode` calls, verified by the numpy
proof and the new fake-model tests for CI. The fix brings this detector
in line with the two sibling call sites that already normalize.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
908a9a1bb1 |
fix(cache): stop DynamicContentDetector false positives corrupting cached prompts (#2110) (#2119)
## Description `DynamicContentDetector` / `RegexDetector` in `headroom/cache/dynamic_detector.py` (used by the `cache_aligner` transform) misclassified ordinary English words and code identifiers (e.g. `in_pr`) as "dynamic content," extracting them from the system prompt and re-appending a `[Dynamic Context]` tail that grows unboundedly and corrupts the cached prompt over a session. Fix tightens detection to require genuinely-dynamic shapes (timestamps, UUIDs, hashes, numbers-with-units, ISO dates) rather than bare tokens — no hardcoded wordlist — and bounds the tail. `cache_aligner` is off by default, so blast radius is limited, but the detector logic is now correct. Closes #2110 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/cache/dynamic_detector.py`: raise the evidence bar so ordinary words/identifiers aren't extracted; bound the dynamic tail. - `tests/test_cache/test_dynamic_detector.py`: assert false positives (ordinary words/identifiers) are NOT extracted while real dynamic values still are. ## Testing - [x] Unit tests pass (`pytest tests/test_cache/test_dynamic_detector.py`) - [x] Linting passes (`ruff check`) ### Test Output ```text 55 passed, 2 skipped ruff: All checks passed! ``` ## Real Behavior Proof - Before: identifiers like `in_pr` extracted into a growing `[Dynamic Context]` tail, corrupting cached prompts. - After: ordinary tokens stay in place; only genuinely-dynamic values are detected. |
||
|
|
cf6367add4 |
fix(cache/semantic): don't evict an unrelated entry on an update at capacity (#2094)
## Description
`SemanticCache.put` can evict a perfectly good, unrelated entry when it
merely updates a key that is already cached.
The method runs its at-capacity eviction loop *before* it computes the
entry's key:
```python
self._cleanup_expired()
# Evict if at capacity
while len(self._cache) >= self.config.max_entries:
self._evict_oldest()
...
key = messages_hash or self._generate_key(query)
...
self._cache[key] = entry
```
So when the same key is stored again while the cache is full (a
duplicate store, or a retried request that produces the same
`messages_hash`), the loop fires because `len == max_entries`, evicts
the LRU-oldest *distinct* entry, and only then overwrites the existing
key in place. Writing to an already-present key does not grow the map,
so nothing needed to be evicted — but an unrelated live entry is now
gone, and the next `get` for it is a false miss.
Concretely, with `max_entries=2` and keys `[h1, h2]`, re-storing `h2`
evicts `h1`, leaving `[h2]` even though only two distinct keys were ever
stored.
The sibling `CompressionCache.store_compressed` gets this right: it
deletes the existing key first, inserts, and only then trims — so
re-storing a present key never drops an unrelated entry.
## Fix
Compute the key first, then run the eviction loop only while the key is
genuinely new:
```python
key = messages_hash or self._generate_key(query)
while key not in self._cache and len(self._cache) >= self.config.max_entries:
self._evict_oldest()
```
An in-place update of an existing key no longer evicts anything; adding
a new key still trims to make room exactly as before.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/semantic.py`: move the cache-key computation above the
eviction loop and gate the loop on `key not in self._cache` so an
in-place update never evicts.
- `tests/test_cache/test_semantic.py`: add
`test_update_at_capacity_does_not_evict_unrelated_entry`.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
$ python -m py_compile headroom/cache/semantic.py tests/test_cache/test_semantic.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the eviction logic with a
dependency-free script that replicates the `OrderedDict` +
`_evict_oldest` (popitem last=False) behavior for the old vs new loop,
and left the full pytest to CI.
- Exact command / steps: with `max_entries=2`, store `h1` then `h2`,
then re-store the already-present `h2`, under both the old loop (evict
before key dedup) and the new loop (evict only when key is new).
- Observed result: old loop leaves `['h2']` and `get(h1)` returns `None`
(h1 wrongly evicted); new loop leaves `['h1', 'h2']` with `get(h1)`
intact and `h2` updated. The regression test asserts h1 survives and h2
reflects the update.
- Not tested: a live embedding-backed cache round-trip; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized reordering of two existing
statements plus a loop guard, verified by the standalone proof and the
new regression test for CI. This is a different defect from the earlier
messages-hash keying fix — that one was about which slot a request maps
to; this one is about eviction dropping a live entry on an in-place
update.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
ec6e60ea3e |
fix(proxy/anthropic): scope session id by top-level system prompt (#2070)
## Description
`SessionTrackerStore.compute_session_id`
(`headroom/cache/prefix_tracker.py`) computes a fallback
session id (when no `x-headroom-session-id` header is present) from
`model` + system-prompt text.
But it harvests system text **only** from `messages` entries with `role
== "system"`:
```python
for msg in messages:
if msg.get("role") == "system":
... # collect system text
system_content = json.dumps(system_parts, ...)
key = f"{model}:{system_content}"
```
Anthropic's `/v1/messages` carries the system prompt as a **top-level**
`body["system"]` field —
it never sends `role:"system"` entries inside `messages`. And
`x-headroom-session-id` is a
Headroom-internal header no client sends. So for every genuine Anthropic
request `system_parts`
is empty and the id collapses to `md5(f"{model}:[]")` — **every
conversation on the same model
shares one session id**, and therefore one `PrefixCacheTracker` and all
session-sticky state.
The colliding state cross-contaminates across conversations
(`anthropic.py:1052`):
- sticky `headroom_retrieve` / memory tools keyed purely on `session_id`
(no content guard) get
injected into another conversation's tool list — busting its tools cache
and adding tools its
client never requested;
- sticky `anthropic-beta` header tokens leak across conversations;
- `frozen_message_count` and the per-session compression cache
cross-contaminate.
(The sibling `StreamingMixin._get_session_key` already reads
`body.get("system")` and its docstring
claims to mirror `compute_session_id` — which it did not.)
Closes: no issue filed — found while auditing the session/prefix
tracker.
## Fix
Add an optional `system` parameter to `compute_session_id` and fold its
text (a plain string or a
list of `{"type":"text"}` blocks) into the hash. The Anthropic handler
passes `body.get("system")`.
OpenAI callers don't pass it (defaults to `None`), so their behavior is
unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/prefix_tracker.py`: `compute_session_id` accepts an
optional `system` and folds it into the id.
- `headroom/proxy/handlers/anthropic.py`: pass
`system=body.get("system")` when computing the session id.
- `tests/test_cache/test_prefix_tracker.py`: add
`test_compute_session_id_distinguishes_top_level_system` (distinct
systems → distinct ids; list-form == string-form; `system=None`
unchanged).
## Testing
- [x] New regression test added
(`tests/test_cache/test_prefix_tracker.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/prefix_tracker.py headroom/proxy/handlers/anthropic.py tests/test_cache/test_prefix_tracker.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the hash logic with
a dependency-free script and left the full pytest to CI.
- Exact command / steps: computed ids for two conversations with the
same model and messages but different top-level `system` prompts,
through the old (never-folds-system) and new logic.
- Observed result: the old logic collapses both to one id (the leak);
the new logic separates them, folds list-form system the same as
string-form, and leaves the `system=None` (OpenAI) path unchanged:
```text
OLD: A=97d8857ba27010bb B=97d8857ba27010bb same=True
NEW: A=1e838c0f6e3980a6 B=18ec49bfa8240852 same=False
SESSION-ID SYSTEM FIX VERIFIED (old collapses Anthropic convos; new separates them)
```
- Not tested: a full two-conversation proxy run asserting no sticky-tool
leakage (needs the heavy stack). The fix is confined to
`compute_session_id` + the one handler call site, and the new test
drives the method directly. Full local `pytest` deferred to CI (OOM, per
above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Backward-compatible: the new `system` parameter defaults to `None`, so
the OpenAI call sites (`openai.py`) need no change and their session ids
are identical.
- @JerrettDavis tagging you — this one lets one Anthropic conversation's
sticky tools/headers leak into another on the same model, so it seemed
worth surfacing. Thanks!
|
||
|
|
d8783ab89b |
fix(cache/semantic): key entries by context hash, not query text (#2022)
## Description
`SemanticCache` (`headroom/cache/semantic.py`) derives each entry's key
from the **query text
only** — where `query` is just the trailing user message — and its
exact-match lookup returns
the slot without checking the stored entry's `messages_hash`:
```python
# put()
key = self._generate_key(query) # sha256(query)[:16]
self._cache[key] = entry
if messages_hash:
self._hash_index[messages_hash] = key
# get() — exact-match branch
key = self._hash_index.get(messages_hash)
if key and key in self._cache:
entry = self._cache[key]
...
return entry # never checks entry.messages_hash
```
So two requests that share a trailing user message but differ in earlier
context map to the
**same** key. The second `put` overwrites the first, and the first
request's `messages_hash`
still points at that (now overwritten) slot — so it is served the
**other conversation's**
cached response.
Trailing messages like `"continue"`, `"yes"`, `"fix it"`, `"run the
tests"` are extremely
common in agentic/coding sessions, so this collides constantly. It's
independent of the
proxy-level `_compute_key` fix (that's about what goes *into*
`messages_hash`; here the entry
is stored under a query-only key regardless of how good the hash is).
This `SemanticCache` is
the one used by the SDK client's `enable_semantic_cache` path.
Concretely:
1. `put("run the tests", A, messages_hash=HA)` → key `K = sha256("run
the tests")`; `_cache[K]=A`.
2. `put("run the tests", B, messages_hash=HB)` → same `K`; `_cache[K]`
overwritten with `B`.
3. `get("run the tests", HA)` → `_hash_index[HA]=K`, `K in _cache` →
returns **B**.
Closes: no issue filed — found while auditing the cache key derivation.
## Fix
1. Key entries by the full-context `messages_hash` when present, falling
back to the query hash
only when no hash is supplied:
```python
key = messages_hash or self._generate_key(query)
```
2. Defensively verify `entry.messages_hash == messages_hash` in the
exact-match branch of `get`,
so any residual stale mapping becomes a miss rather than wrong data.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/semantic.py`: key `put` entries by `messages_hash`
when present; verify `entry.messages_hash` in the `get` exact-match
branch.
- `tests/test_cache/test_semantic.py`: add
`test_same_query_different_context_does_not_collide` and
`test_exact_match_verifies_messages_hash`.
## Testing
- [x] New regression tests added (`tests/test_cache/test_semantic.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the `put`/`get`
logic with a dependency-free script and left the full pytest to CI.
- Exact command / steps: stored responses A and B under the same query
`"run the tests"` with different `messages_hash`, then read each hash
back — through both the old (query-keyed) and new (hash-keyed) logic.
- Observed result: the old logic serves B's response to request A; the
new logic isolates them:
```text
OLD: A->RESPONSE_B B->RESPONSE_B
NEW: A->RESPONSE_A B->RESPONSE_B
SEMANTIC CACHE COLLISION FIX VERIFIED (OLD served B to A; NEW isolates)
```
- Not tested: the full SDK `HeadroomClient` round-trip with
`enable_semantic_cache=True` (needs the heavy stack). The fix is
confined to `SemanticCache.put`/`get` and the new tests drive them
directly. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Small, contained fix — the key derivation plus a verification guard,
no new dependencies.
- @JerrettDavis tagging you — this one can serve one conversation's
cached response to another when the last message matches, so it seemed
worth surfacing. Thanks!
|
||
|
|
0f606b6281 |
fix(cache): avoid fallback session collisions (#1827)
## Description Cache-mode session tracking currently collapses unrelated conversations when they share a large static first system prompt. The fallback session-id hash ignores later system messages entirely, so dynamic per-conversation context can get cut out of the key and two different sessions reuse the same `PrefixCacheTracker`. This hashes the full ordered system-text payload instead, while leaving explicit `x-headroom-session-id` overrides untouched. Refs #1808. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Collected all system-text content when building the fallback cache session id. - Stopped truncating fallback session-id input to the first 500 characters of the first system message. - Added a regression that proves two conversations with different later system context no longer collide. - Added a preservation test that appending only non-system turns keeps the same fallback session id. - Applied the pinned Ruff formatter to three pre-existing files on the current base so the repo-wide lint job passes unchanged semantics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cache/test_prefix_tracker.py -q`) - [x] Linting passes (`uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_cache/test_prefix_tracker.py -q 40 passed, 1 warning in 0.15s uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py All checks passed! uv run ruff check . All checks passed! uv run ruff format --check . 1046 files already formatted ``` ## Real Behavior Proof - Environment: Windows, project `uv` environment, focused cache-tracker regression. - Exact command / steps: run `tests/test_cache/test_prefix_tracker.py` on `origin/main` with the new collision regression present, then rerun the same file on this branch. - Observed result: base returns the same session id for two conversations that differ only in a later system message and fails `assert id_a != id_b`; head passes the focused file and keeps the fallback session id stable when only non-system turns are appended. - Not tested: live proxy traffic through a real agentic client. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This is only the session-collision half of #1808. The duplicate-response-header fix stays separate so this PR can reference the issue without claiming the whole bug report is resolved. The extra formatting-only diff comes from the current base failing the pinned full-repo Ruff format check. |
||
|
|
4658721ea0 |
feat(cache): attribute prompt-cache misses to TTL lapse vs prefix change (#1313) (#1343)
## Description A low prompt-cache hit rate is hard to act on without knowing *why* turns miss. Two very different causes need very different responses: - **TTL lapse** — the session went idle longer than the provider's cache lifetime, so the entry expired. The fix is a longer TTL (e.g. Anthropic's 1h breakpoint instead of the 5m default). - **Prefix change** — the cacheable message prefix shifted, so the new request couldn't match the cached key. A longer TTL won't help here at all. Right now those look identical from the dashboard (just "cache_read was 0"). This adds the attribution so a user can actually decide 5m vs 1h. Closes #1313 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `PrefixCacheTracker` already kept the previous turn's forwarded messages and a per-turn activity timestamp, so the signal was already there — it just wasn't being read. - **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn expected a cached prefix (non-zero cached tokens last turn) but read 0 this turn, returns `ttl_expiry` if the idle gap exceeded the provider cache TTL, else `prefix_change` if the forwarded prefix differs from last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed, a coincident content change is moot, and the 5m-vs-1h decision is exactly what the TTL signal answers. A 1h-breakpoint session can widen the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and hits return `is_miss=False`. - **Anthropic handlers (streaming + non-streaming)** — classify BEFORE `update_from_response` overwrites the last-turn state the classifier reads, then record the reason. - **`prometheus_metrics.py`** — a per-provider/per-reason counter, `record_cache_miss_attribution()`, reset handling, and a `headroom_cache_miss_attribution_total{provider,reason}` export series. - **`cost.py`** — `build_prefix_cache_stats()` aggregates a `miss_attribution` block (per-provider + totals, with the ttl/prefix split as a % of *attributed* misses, so `unknown` doesn't dilute the headline). - **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix change" headline. Scoped to Anthropic for this first cut (where the tracker is fully wired); OpenAI/Gemini can follow once the shape is proven. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 38 passed # 29 existing + 9 new classifier tests (TestClassifyCacheMiss). $ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q 5 passed, 8 deselected # new: counter bucketing, stats aggregation, empty case, /metrics export, reset. ``` The full `test_proxy_cache_ttl_metrics.py` / `test_proxy_dashboard_stats_cache.py` files have some failures in this sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) — those spin up the proxy server / Rust `_core` extension, which isn't built here. I confirmed via `git stash` that they fail identically on `main` without my changes, so they're pre-existing and unrelated. My additions to the stats dict are purely additive and don't break any passing assertion. ## Real Behavior Proof - Environment: Windows 11, Python 3.10. The Rust `_core` extension and a live proxy aren't available in this checkout. - Exact command / steps: drove `classify_cache_miss()` through every branch with a faithful warm-then-miss sequence; drove `record_cache_miss_attribution()` → `build_prefix_cache_stats()` → `export()` end to end. - Observed result: classifier returns `cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL wins the tie when both signals fire, a growing (append-only) prefix is treated as stable, and the 1h override widens the window. The stats builder produces `miss_attribution.totals` (`ttl_expiry`/`prefix_change`/`unknown`/`total` + `ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and `by_provider`; `/metrics` emits `headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`. - Not tested: a live Anthropic session through the running proxy with a real idle-then-resume to confirm the handler wiring fires end-to-end. I verified the handler integration by reading scope/order (classify before `update_from_response`, `provider_name`/`self.metrics` in scope) and unit-tested every layer it calls, but didn't exercise the actual server loop. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The classifier is intentionally pure (takes the cache-read result + current forwarded messages + an optional idle override) so it's order-independent and unit-testable without a live tracker clock. - No README/docs change yet — this surfaces in the dashboard and `/metrics`, which are self-describing; happy to add a docs page if you'd like one. - CHANGELOG.md isn't touched — release-please generates it from the `feat(cache):` commit subject. - Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a per-provider breakdown row in the dashboard panel (the stats already carry `by_provider`). |
||
|
|
3b0bceecf4 |
fix(cache): name the missing piece in semantic detector guard (#1018)
## Description
The `test` and `test-extras` CI jobs are currently red on `main`:
`tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorGuards::test_none_exemplars_early_return`
fails. This PR fixes the underlying regression. It is independent of any
feature branch (it only touches `headroom/cache/dynamic_detector.py` and
its test).
#950 folded the exemplar-embeddings None-check into the model
None-guard:
```python
if self._model is None or self._exemplar_embeddings is None:
return [], self._load_error or "semantic detector is not initialized"
```
So a `SemanticDetector` with a loaded model but unset exemplar
embeddings now returns the generic *"semantic detector is not
initialized"* message, shadowing the specific *"exemplar embeddings not
initialized"* message and leaving the later guard as unreachable dead
code. That directly contradicts #950's own
`test_none_exemplars_early_return`, which asserts the specific message —
hence the red main.
## Type of Change
- [ ] New feature
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change
- [ ] Documentation
## Changes Made
- `SemanticDetector.detect`: split the combined guard into two checks —
`_model` then `_exemplar_embeddings` — both *before* `encode()`. The
model-missing case keeps the generic message; the exemplar-missing case
reports the specific *"exemplar embeddings not initialized"*. Checking
before `encode()` avoids a wasted encode in the error path and preserves
the mypy narrowing for `np.dot(..., self._exemplar_embeddings.T)`. The
previously-shadowed duplicate guard is removed.
- `tests/test_cache/test_dynamic_detector.py`:
`test_missing_exemplar_embeddings_returns_warning` sets the same
model-present / exemplar-None state, so its assertion is aligned to the
specific message to match `test_none_exemplars_early_return`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New and existing unit tests pass locally with my changes
### Test Output
```text
$ pytest tests/test_cache/test_dynamic_detector.py -q
38 passed, 2 skipped in 9.94s
$ pytest tests/test_cache/ -q
198 passed, 2 skipped in 11.58s
$ ruff check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ ruff format --check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
2 files already formatted
$ mypy headroom/cache/dynamic_detector.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: local macOS, repo .venv, Python 3.11.9, numpy installed
- Exact command / steps: construct a `SemanticDetector` via
`object.__new__` with `_model` set (mock) and `_exemplar_embeddings =
None`, then call `.detect(...)`. Before fix: on `upstream/main`
(
|
||
|
|
1ec9320888 |
fix(cache): guard None exemplar embeddings in dynamic detector (#950)
## Description `mypy headroom --ignore-missing-imports` fails on `main` at `headroom/cache/dynamic_detector.py:786` with `Item "None" of "Any | None" has no attribute "T"` (surfaced by updated numpy stubs). This breaks the `lint` job for every open PR that merges current main. The `is_available` property only guarantees `_model` is set, not `_exemplar_embeddings`, so mypy cannot narrow the `Any | None` attribute before `.T` — and if it were ever None this is a real runtime crash, not just a type nit. Closes # <!-- broken-main lint failure; no tracked issue --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/dynamic_detector.py`: add an explicit `self._exemplar_embeddings is None` guard before the `np.dot(..., .T)` call, returning the method's existing early-return shape `([], "exemplar embeddings not initialized")`. Narrows the type for mypy and prevents a latent `None.T` crash. - `tests/test_cache/test_dynamic_detector.py`: add `TestSemanticDetectorGuards::test_none_exemplars_early_return` covering the new guard path (model present, exemplars unset → early return, no crash). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom --ignore-missing-imports --no-incremental (0 errors — was: "Found 1 error in 1 file" at dynamic_detector.py:786) $ ruff check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py All checks passed! $ pytest tests/test_cache/test_dynamic_detector.py -q 37 passed, 2 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, branch `fix/dynamic-detector-mypy` from current `origin/main`. - Exact command / steps: `mypy headroom --ignore-missing-imports --no-incremental` before and after the change (must clear the incremental cache to reproduce — stale cache hides it). - Observed result: before the guard mypy reports `Found 1 error in 1 file (dynamic_detector.py:786)`; after, 0 errors. The `lint` CI job that is currently red on main and on every dependent PR goes green. - Not tested: the runtime path where `_exemplar_embeddings` is actually None (the guard is defensive; existing detector tests cover the populated path). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — type/CI fix with no UI surface. See **Test Output** above. ## Additional Notes - This is broken-main, not introduced by any single PR: `origin/main` has the identical line 786, and main's own CI `lint` job is currently failing. Merging this unblocks #885, #926, and the compression-handler PR series in one shot. - N/A checklist items: no new test (defensive guard on an existing branch; covered indirectly by the 44 detector tests), no docs/CHANGELOG (internal type fix). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
b51cda10d7 |
docs(evals): add session probes section to evals README (#888)
## Description Follow-up to #862. That PR's body described a **Session Probes** section in `headroom/evals/README.md`, but the file edit missed the commit (edited in the wrong checkout). This adds the missing 22-line docs-only section: the record-then-score workflow for `HEADROOM_PROBE_RECORD_DIR` + `headroom evals probes`, including the plaintext-recording privacy note. Refs #861 (session-probe eval harness — this README section was part of that feature's spec). ## Type of Change - [x] Documentation update ## Changes Made - Add a **Session Probes (real recorded sessions)** section to `headroom/evals/README.md` (+22 lines, no code change): the two-step record (`HEADROOM_PROBE_RECORD_DIR=… headroom proxy start`) then score (`headroom evals probes --recordings …`) workflow, the three probe dimensions (exact numerics, artifact trail, error evidence), the retained/recoverable/lost classification, retention bucketing by ratio + per-transform grouping, and the `--json-output` flag. - Includes the opt-in privacy note: recordings contain full conversation content in plaintext and stay on the local machine. ## Testing - [x] Documentation builds/renders correctly - [x] Linting passes (`ruff check .`) - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ git diff --stat upstream/main..HEAD headroom/evals/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) Docs-only change — no code paths touched. The commands and flags documented (HEADROOM_PROBE_RECORD_DIR, `headroom evals probes`, --recordings, --json-output) are the surface shipped and tested in #862. ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rendered the edited `headroom/evals/README.md` and cross-checked every documented flag/command against the implemented CLI from #862 (`headroom evals probes`, `HEADROOM_PROBE_RECORD_DIR`, `--recordings`, `--json-output`) - Observed result: the new section renders correctly and every command/flag it names exists in the shipped probe harness; no code paths are changed by this PR, so behavior is unchanged - Not tested: nothing additional — docs-only change with no executable surface of its own ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Pure documentation backfill for #862; the feature itself (recorder + retention probes) already merged. PR body updated to satisfy the PR-governance template gate. |
||
|
|
22dad133e2 |
fix: eliminate prefix cache busts from frozen count underestimation
Root cause: CompressionCache.compute_frozen_count() stopped at the first tool_result not in its cache, capping frozen_message_count at 2. Tool results excluded by content_router (Read/Glob) or skipped (ratio too high) never entered the cache, so every subsequent message was eligible for recompression — causing 192 cache busts per session. Four fixes: 1. Add _stable_hashes set to CompressionCache so excluded/skipped tool_results don't block the frozen count walk 2. Fix _estimate_message_tokens to count tool_result content and tool_use input fields (were counted as 0 tokens in Anthropic format) 3. Fix streaming handler to include assistant response and original_messages in prefix tracker updates (parity with non-streaming) 4. TTL-aware batch recompression: defer first-time compressions within the 5-min cache TTL window, batching them at the boundary to trade many small busts for one |
||
|
|
8f438b0674 |
Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude - It will also install rtk-ai locally - rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code |
||
|
|
3ffe68618a |
Add pluggable storage backend abstraction for CompressionStore
- Add CompressionStoreBackend protocol for duck-typed backends - Add InMemoryBackend as default thread-safe implementation - Refactor CompressionStore to accept optional backend parameter - Add comprehensive backend contract tests (28 tests) |
||
|
|
313fe0158a | Fix ruff formatting | ||
|
|
dd832fee0c |
Add TOIN field-level learning and comprehensive integration tests
Features: - Add field-level learning to TOIN from retrieved items - CompressionStore now passes retrieved_items to TOIN for learning - Add FieldSemantics class for tracking field usage patterns Test improvements: - Add TestCacheOptimizerInvocation to verify optimizer is actually invoked - Add TestSemanticCacheIntegration to verify cache hit returns without API call - Add TestSessionStatsTracking to verify session stats are tracked - Add TestEndToEndTOINIntegration for full CCR cycle with TOIN - Add critical field_semantics assertions to catch feedback loop bugs Fixes: - Remove unused imports and variables (ruff linting) Bump version to 0.2.11 |
||
|
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
|
7a05808e0f |
Add cache optimization module with scalable dynamic content detection
Implements a comprehensive cache optimization layer for LLM providers: - Provider-specific optimizers (Anthropic, OpenAI, Google) with distinct caching strategies: explicit breakpoints, prefix stabilization, and CachedContent API respectively - Scalable dynamic content detector using three strategies: 1. Structural detection: "Label: value" patterns (language-agnostic) 2. Entropy-based detection: high-entropy strings (IDs, tokens, hashes) 3. Universal patterns: ISO 8601, UUIDs, JWTs, hex hashes - NO hardcoded locale-specific patterns (no month names, etc.) - Semantic caching layer with LRU eviction and TTL support - Plugin registry for provider selection and custom optimizers - 131 tests, real-world benchmarks showing 20-55% compression at <0.3ms |