Files
Yuchuan He 4eb742abd8 fix(cache): refresh semantic cache recency on updates (#4767)
## Description

Overwriting a SemanticCache entry refreshed its response and timestamps
but left its old position in the OrderedDict. With capacity two, put(A),
put(B), update(A), put(C) evicted the freshly updated A and retained the
older B.

Move a successfully stored entry to the end of the LRU. Cache state
operations now share an RLock, including semantic scans and touches,
while embedding callbacks execute outside it. This happens after
embedding generation and entry construction, so a failed update still
preserves the original eviction order. It complements the query-only
retrieval fix in #4709: a successfully refreshed response should remain
retrievable under subsequent capacity pressure.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Refresh LRU position after successful insertion/replacement.
- Cover query-only and context-hash entries, with and without
namespaces.
- Keep failed embedding updates from refreshing recency, and retain the
no-eviction-on-update contract from #2094.

- Synchronize cache/index/statistics operations and the complete
similarity-scan/touch sequence. Embedding callbacks run outside the
state lock; add two-thread scan/update and nonblocking exact-hit
regressions.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Initial new regressions against original production code: 4 failed, 1 passed
pytest tests/test_cache tests/test_cold_prefix.py:
  398 passed, 3 skipped
Review follow-up on the initial PR head: 1 failed
Original pre-commit and commit-msg hooks (including follow-up): passed
Whole-tree Ruff check / format --check: passed (1932 files)
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: built, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The three cache skips require optional spaCy/embedding models.
Critical-suite skips require optional relevance models. The existing
token-count approximation warning remains visible in the gate log.
Initial broader runs failed because the new worktrees lacked the native
extension; these were rerun successfully after building it.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development venv;
clean baseline `200c838c7393d20ae56a0c338f832ed27133c398`, patch
`ffa5198d3e4a7ab634e49efb8b6c6b59e4315acf`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH=.` for the selected checkout. No paid/provider credentials.
- Exact command / steps: Save the following identical script as
`proof.py` and run `python proof.py` from either checkout. Before and
after runs were outside pytest.

```python
"""A real SemanticCache overwrite followed by a capacity-triggered eviction."""
import json
from headroom.cache.semantic import SemanticCache, SemanticCacheConfig

reports = []
for context in (False, True):
    for namespace in ("", "provider/model"):
        cache = SemanticCache(SemanticCacheConfig(max_entries=2))
        hashes = ["context-a", "context-b", "context-c"] if context else [None] * 3
        cache.put("first", {"answer": "old"}, hashes[0], namespace)
        cache.put("second", {"answer": "older"}, hashes[1], namespace)
        cache.put("first", {"answer": "updated"}, hashes[0], namespace)
        before = cache.get_stats()
        cache.put("third", {"answer": "new"}, hashes[2], namespace)
        first = cache.get("first", hashes[0], namespace)
        second = cache.get("second", hashes[1], namespace)
        reports.append({"context": context, "namespace": namespace,
            "update_evictions": before["evictions"], "updated_response": first.response if first else None,
            "older_response": second.response if second else None,
            "entries": cache.get_stats()["entries"], "evictions": cache.get_stats()["evictions"]})
print(json.dumps(reports, indent=2))
```


Review follow-up: run `python concurrency_proof.py` with the script
below on the initial PR head and the revised head. It coordinates two
real cache callers at a similarity-scan boundary.

```python
"""A real two-thread scan/update interleaving, without mocked cache results."""
import json
import threading
from headroom.cache.semantic import SemanticCache

scanning, release, writer_progress = threading.Event(), threading.Event(), threading.Event()

class PausedCache(SemanticCache):
    def _cosine_similarity(self, a, b):
        if not scanning.is_set():
            scanning.set()
            if not release.wait(10): raise RuntimeError("Scan release missing")
        return super()._cosine_similarity(a, b)

cache = PausedCache(embedding_fn=lambda query: [1.0, 0.0])
cache.put("first", "old", "first")
cache.put("second", "two", "second")
real = threading.RLock()

class SignalsWriterLock:
    def __enter__(self):
        if threading.current_thread().name == "cache-update": writer_progress.set()
        return real.__enter__()

    def __exit__(self, *exc): return real.__exit__(*exc)

cache._lock = SignalsWriterLock()
errors, results = [], []

def read():
    try:
        entry = cache.get("third", "third")
        results.append(entry.response if entry else None)
    except Exception as exc:
        errors.append(type(exc).__name__ + ": " + str(exc))

def update():
    try: cache.put("first", "updated", "first")
    except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc))
    finally: writer_progress.set()

reader = threading.Thread(target=read)
writer = threading.Thread(target=update, name="cache-update")
reader.start()
try:
    assert scanning.wait(10)
    writer.start()
    assert writer_progress.wait(10)
finally:
    release.set()
    reader.join(10)
    if writer.ident is not None: writer.join(10)
assert not reader.is_alive() and not writer.is_alive()
print(json.dumps({"errors": errors, "lookup_responses": results,
    "updated_response": cache.get("first", "first").response}))

# Existing tie behavior: an ordinary exact hit already changes scan order.
# No put/update occurs between the two semantic lookups in this control.
ties = SemanticCache(embedding_fn=lambda query: [1.0, 0.0])
ties.put("first", "A", "first")
ties.put("second", "B", "second")
first = ties.get("one unseen query", "unseen-1").response
ties.get("first", "first")
second = ties.get("another unseen query", "unseen-2").response
print(json.dumps({"equal_score_control": [first, second],
    "policy": "first match in current LRU order; no most-recent tie guarantee"}))
```

- Observed result: The real SemanticCache API reproduces the sequence
for four context/namespace combinations. Before the patch, the updated
response is missing and the older response survives. After it,
updated_response is updated and older_response is None; each case still
has two entries and exactly one eviction. No get() is made between
update(A) and put(C), which would mask the defect by touching A. The
concurrent proof raises OrderedDict mutated during iteration on the
initial PR head and completes without errors on the revised head. The
equal-score control returns A then B on both original main and revised
code: ties still follow current LRU scan order, with no added freshness
guarantee.
- Not tested: Paid/live embedding services, model quality, production
hit-rate/latency improvements and the complete Python suite. The runtime
proof uses exact matching without an embedding dependency.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Successful replacement counts as
recent use. Capacity, exact/context isolation, TTL and failed-store
behavior remain unchanged.
- Kill switch / disable path: Disable the existing optional semantic
cache in its caller, or revert this commit.
- Unsafe override required: No.
- Qualification impact: Related cache tests and original local gates
passed; hosted CI is tracked separately.
- Rollback path: Revert this commit. No dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Implementation and local validation were completed with Codex
assistance. The author reviewed the initial code and validation record;
the review follow-up was checked with Codex.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4767"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
2026-11-11 10:00:26 -06:00
..
2026-01-17 17:14:12 -08:00