mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Overwriting a SemanticCache entry refreshed its response and timestamps but left its old position in the OrderedDict. With capacity two, put(A), put(B), update(A), put(C) evicted the freshly updated A and retained the older B. Move a successfully stored entry to the end of the LRU. Cache state operations now share an RLock, including semantic scans and touches, while embedding callbacks execute outside it. This happens after embedding generation and entry construction, so a failed update still preserves the original eviction order. It complements the query-only retrieval fix in #4709: a successfully refreshed response should remain retrievable under subsequent capacity pressure. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Refresh LRU position after successful insertion/replacement. - Cover query-only and context-hash entries, with and without namespaces. - Keep failed embedding updates from refreshing recency, and retain the no-eviction-on-update contract from #2094. - Synchronize cache/index/statistics operations and the complete similarity-scan/touch sequence. Embedding callbacks run outside the state lock; add two-thread scan/update and nonblocking exact-hit regressions. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Initial new regressions against original production code: 4 failed, 1 passed pytest tests/test_cache tests/test_cold_prefix.py: 398 passed, 3 skipped Review follow-up on the initial PR head: 1 failed Original pre-commit and commit-msg hooks (including follow-up): passed Whole-tree Ruff check / format --check: passed (1932 files) Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: built, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The three cache skips require optional spaCy/embedding models. Critical-suite skips require optional relevance models. The existing token-count approximation warning remains visible in the gate log. Initial broader runs failed because the new worktrees lacked the native extension; these were rerun successfully after building it. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development venv; clean baseline `200c838c7393d20ae56a0c338f832ed27133c398`, patch `ffa5198d3e4a7ab634e49efb8b6c6b59e4315acf`. `HEADROOM_OFFLINE=1`, `PYTHONPATH=.` for the selected checkout. No paid/provider credentials. - Exact command / steps: Save the following identical script as `proof.py` and run `python proof.py` from either checkout. Before and after runs were outside pytest. ```python """A real SemanticCache overwrite followed by a capacity-triggered eviction.""" import json from headroom.cache.semantic import SemanticCache, SemanticCacheConfig reports = [] for context in (False, True): for namespace in ("", "provider/model"): cache = SemanticCache(SemanticCacheConfig(max_entries=2)) hashes = ["context-a", "context-b", "context-c"] if context else [None] * 3 cache.put("first", {"answer": "old"}, hashes[0], namespace) cache.put("second", {"answer": "older"}, hashes[1], namespace) cache.put("first", {"answer": "updated"}, hashes[0], namespace) before = cache.get_stats() cache.put("third", {"answer": "new"}, hashes[2], namespace) first = cache.get("first", hashes[0], namespace) second = cache.get("second", hashes[1], namespace) reports.append({"context": context, "namespace": namespace, "update_evictions": before["evictions"], "updated_response": first.response if first else None, "older_response": second.response if second else None, "entries": cache.get_stats()["entries"], "evictions": cache.get_stats()["evictions"]}) print(json.dumps(reports, indent=2)) ``` Review follow-up: run `python concurrency_proof.py` with the script below on the initial PR head and the revised head. It coordinates two real cache callers at a similarity-scan boundary. ```python """A real two-thread scan/update interleaving, without mocked cache results.""" import json import threading from headroom.cache.semantic import SemanticCache scanning, release, writer_progress = threading.Event(), threading.Event(), threading.Event() class PausedCache(SemanticCache): def _cosine_similarity(self, a, b): if not scanning.is_set(): scanning.set() if not release.wait(10): raise RuntimeError("Scan release missing") return super()._cosine_similarity(a, b) cache = PausedCache(embedding_fn=lambda query: [1.0, 0.0]) cache.put("first", "old", "first") cache.put("second", "two", "second") real = threading.RLock() class SignalsWriterLock: def __enter__(self): if threading.current_thread().name == "cache-update": writer_progress.set() return real.__enter__() def __exit__(self, *exc): return real.__exit__(*exc) cache._lock = SignalsWriterLock() errors, results = [], [] def read(): try: entry = cache.get("third", "third") results.append(entry.response if entry else None) except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc)) def update(): try: cache.put("first", "updated", "first") except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc)) finally: writer_progress.set() reader = threading.Thread(target=read) writer = threading.Thread(target=update, name="cache-update") reader.start() try: assert scanning.wait(10) writer.start() assert writer_progress.wait(10) finally: release.set() reader.join(10) if writer.ident is not None: writer.join(10) assert not reader.is_alive() and not writer.is_alive() print(json.dumps({"errors": errors, "lookup_responses": results, "updated_response": cache.get("first", "first").response})) # Existing tie behavior: an ordinary exact hit already changes scan order. # No put/update occurs between the two semantic lookups in this control. ties = SemanticCache(embedding_fn=lambda query: [1.0, 0.0]) ties.put("first", "A", "first") ties.put("second", "B", "second") first = ties.get("one unseen query", "unseen-1").response ties.get("first", "first") second = ties.get("another unseen query", "unseen-2").response print(json.dumps({"equal_score_control": [first, second], "policy": "first match in current LRU order; no most-recent tie guarantee"})) ``` - Observed result: The real SemanticCache API reproduces the sequence for four context/namespace combinations. Before the patch, the updated response is missing and the older response survives. After it, updated_response is updated and older_response is None; each case still has two entries and exactly one eviction. No get() is made between update(A) and put(C), which would mask the defect by touching A. The concurrent proof raises OrderedDict mutated during iteration on the initial PR head and completes without errors on the revised head. The equal-score control returns A then B on both original main and revised code: ties still follow current LRU scan order, with no added freshness guarantee. - Not tested: Paid/live embedding services, model quality, production hit-rate/latency improvements and the complete Python suite. The runtime proof uses exact matching without an embedding dependency. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing cache behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Successful replacement counts as recent use. Capacity, exact/context isolation, TTL and failed-store behavior remain unchanged. - Kill switch / disable path: Disable the existing optional semantic cache in its caller, or revert this commit. - Unsafe override required: No. - Qualification impact: Related cache tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit. No dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Implementation and local validation were completed with Codex assistance. The author reviewed the initial code and validation record; the review follow-up was checked with Codex. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4767" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end -->