mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(cache): build semantic entry before evicting so a failed embedding keeps live entries (#4581)
## Description `SemanticCache.put()` evicted the oldest entry before calling the embedding function. When the embedding raised (e.g. an OpenAI 503), the new entry was never stored but a live one was already gone, so a cache at capacity lost a valid response on a transient failure. Fixes #4283 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/cache/semantic.py`: move the capacity eviction loop in `put()` to after the embedding is computed and the `CacheEntry` is built, right before insertion. The embedding exception still propagates; successful stores evict exactly as before. - `tests/test_cache/test_semantic.py`: regression `test_put_failure_after_eviction_keeps_live_entry` (max_entries=1, embedding fails on the second put; old entry survives, size stays 1; a later successful put evicts normally). ## Testing - [x] Unit tests pass - [x] Regression test fails on `main` and passes with the fix ``` $ python -m pytest tests/test_cache/test_semantic.py -q 37 passed in 1.94s # on upstream/main source: FAILED tests/test_cache/test_semantic.py::TestSemanticCache::test_put_failure_after_eviction_keeps_live_entry 1 failed, 36 passed in 1.72s $ ruff check / ruff format --check (0.16.8) All checks passed! 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, uv tool Python env, openai 2.54.0, headroom upstream/main vs this branch; local HTTP embeddings endpoint (first call 200, then 503) - Exact command / steps: SemanticCache(max_entries=1) with embedding_fn = real OpenAI client (base_url=local endpoint, max_retries=0); put("old query"), put("new query") -> 503, then get("old query") - Observed result: on upstream/main the failed put leaves 0 entries and the old lookup misses (falls through to the failing embedding); with this fix the 503 still raises InternalServerError, the cache keeps 1 entry and the old exact lookup returns "old response" - Not tested: a hosted OpenAI endpoint under real overload ``` == with fix after first put: entries = 1 second put raised: InternalServerError after failed put: entries = 1 old exact lookup: old response == upstream/main after first put: entries = 1 second put raised: InternalServerError after failed put: entries = 0 openai.InternalServerError: Error code: 503 (old entry gone; get() re-embeds and fails) ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only on embedding failure, where the existing entry is now kept instead of evicted - Kill switch / disable path: n/a (semantic cache can be disabled as before) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
5dcb1e4ade
commit
0d4c450d77
Vendored
+12
-9
@@ -225,15 +225,6 @@ class SemanticCache:
|
||||
base = messages_hash or self._generate_key(query)
|
||||
key = self._combined_key(namespace, base)
|
||||
|
||||
# Evict if adding a NEW key would exceed capacity. Overwriting a key that
|
||||
# is already present is an in-place update that does not grow the map, so
|
||||
# it must NOT evict — the old code ran the eviction loop before computing
|
||||
# the key, so re-storing an existing entry at capacity dropped an
|
||||
# unrelated live entry and turned a later lookup for it into a false miss.
|
||||
# (Mirrors CompressionCache.store_compressed, which deletes-then-inserts.)
|
||||
while key not in self._cache and len(self._cache) >= self.config.max_entries:
|
||||
self._evict_oldest()
|
||||
|
||||
# Generate embedding if available — but never for an empty/blank query.
|
||||
# A stored empty-query entry with an embedding would be a false-match
|
||||
# target for the semantic get() path; leaving its embedding empty makes
|
||||
@@ -254,6 +245,18 @@ class SemanticCache:
|
||||
namespace=namespace,
|
||||
)
|
||||
|
||||
# Evict if adding a NEW key would exceed capacity. Overwriting a key that
|
||||
# is already present is an in-place update that does not grow the map, so
|
||||
# it must NOT evict — the old code ran the eviction loop before computing
|
||||
# the key, so re-storing an existing entry at capacity dropped an
|
||||
# unrelated live entry and turned a later lookup for it into a false miss.
|
||||
# (Mirrors CompressionCache.store_compressed, which deletes-then-inserts.)
|
||||
# The embedding call above may raise, so evict only once the new entry is
|
||||
# built — never before, or a failed store would drop a live entry without
|
||||
# replacing it.
|
||||
while key not in self._cache and len(self._cache) >= self.config.max_entries:
|
||||
self._evict_oldest()
|
||||
|
||||
self._cache[key] = entry
|
||||
|
||||
# Index by hash for fast exact matching
|
||||
|
||||
@@ -123,6 +123,52 @@ class TestSemanticCache:
|
||||
got2 = cache.get("query2", messages_hash="h2")
|
||||
assert got2 is not None and got2.response == "response2b"
|
||||
|
||||
def test_put_failure_after_eviction_keeps_live_entry(self):
|
||||
"""A failed embedding must not silently drop an already-evicted entry.
|
||||
|
||||
The eviction loop used to run before the embedding call, so if the
|
||||
embedding function raised, an old live entry was already evicted while
|
||||
the new entry was never stored — a permanent data loss on a transient
|
||||
error. The exception must still propagate.
|
||||
"""
|
||||
config = SemanticCacheConfig(max_entries=1, similarity_threshold=0.9)
|
||||
cache = SemanticCache(config)
|
||||
|
||||
call_count = {"n": 0}
|
||||
|
||||
def flaky_embedding(text: str) -> list[float]:
|
||||
call_count["n"] += 1
|
||||
if call_count["n"] == 1:
|
||||
return [1.0, 0.0]
|
||||
raise RuntimeError("embedding backend unavailable")
|
||||
|
||||
cache._embedding_fn = flaky_embedding
|
||||
|
||||
# First put succeeds and fills the cache to capacity.
|
||||
cache.put("a", "response-a", messages_hash="ha")
|
||||
assert cache.get("a", messages_hash="ha") is not None
|
||||
assert len(cache._cache) == 1
|
||||
|
||||
# Second put raises, but the live entry must survive untouched.
|
||||
with pytest.raises(RuntimeError, match="embedding backend unavailable"):
|
||||
cache.put("b", "response-b", messages_hash="hb")
|
||||
|
||||
assert cache.get("a", messages_hash="ha") is not None
|
||||
assert len(cache._cache) == 1
|
||||
|
||||
# With the embedding working again, a new put evicts "a" normally.
|
||||
def stable_embedding(text: str) -> list[float]:
|
||||
# Distinct vectors per query so "a" and "c" never false-hit.
|
||||
if text == "a":
|
||||
return [1.0, 0.0]
|
||||
return [0.0, 1.0]
|
||||
|
||||
cache._embedding_fn = stable_embedding
|
||||
cache.put("c", "response-c", messages_hash="hc")
|
||||
assert cache.get("a", messages_hash="ha") is None
|
||||
assert cache.get("c", messages_hash="hc") is not None
|
||||
assert len(cache._cache) == 1
|
||||
|
||||
def test_ttl_expiration(self):
|
||||
"""Test TTL expiration."""
|
||||
config = SemanticCacheConfig(ttl_seconds=1)
|
||||
|
||||
Reference in New Issue
Block a user