fix(cache): build semantic entry before evicting so a failed embedding keeps live entries (#4581)

## Description

`SemanticCache.put()` evicted the oldest entry before calling the
embedding function. When the embedding raised (e.g. an OpenAI 503), the
new entry was never stored but a live one was already gone, so a cache
at capacity lost a valid response on a transient failure.

Fixes #4283

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cache/semantic.py`: move the capacity eviction loop in
`put()` to after the embedding is computed and the `CacheEntry` is
built, right before insertion. The embedding exception still propagates;
successful stores evict exactly as before.
- `tests/test_cache/test_semantic.py`: regression
`test_put_failure_after_eviction_keeps_live_entry` (max_entries=1,
embedding fails on the second put; old entry survives, size stays 1; a
later successful put evicts normally).

## Testing

- [x] Unit tests pass
- [x] Regression test fails on `main` and passes with the fix

```
$ python -m pytest tests/test_cache/test_semantic.py -q
37 passed in 1.94s
# on upstream/main source:
FAILED tests/test_cache/test_semantic.py::TestSemanticCache::test_put_failure_after_eviction_keeps_live_entry
1 failed, 36 passed in 1.72s
$ ruff check / ruff format --check (0.16.8)
All checks passed! 2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, uv tool Python env, openai 2.54.0, headroom
upstream/main vs this branch; local HTTP embeddings endpoint (first call
200, then 503)
- Exact command / steps: SemanticCache(max_entries=1) with embedding_fn
= real OpenAI client (base_url=local endpoint, max_retries=0); put("old
query"), put("new query") -> 503, then get("old query")
- Observed result: on upstream/main the failed put leaves 0 entries and
the old lookup misses (falls through to the failing embedding); with
this fix the 503 still raises InternalServerError, the cache keeps 1
entry and the old exact lookup returns "old response"
- Not tested: a hosted OpenAI endpoint under real overload

```
== with fix
after first put: entries = 1
second put raised: InternalServerError
after failed put: entries = 1
old exact lookup: old response
== upstream/main
after first put: entries = 1
second put raised: InternalServerError
after failed put: entries = 0
openai.InternalServerError: Error code: 503 (old entry gone; get() re-embeds and fails)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on embedding failure, where the
existing entry is now kept instead of evicted
- Kill switch / disable path: n/a (semantic cache can be disabled as
before)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Parideboy
2026-11-04 08:36:09 -06:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 5dcb1e4ade
commit 0d4c450d77
2 changed files with 58 additions and 9 deletions
+12 -9
View File
@@ -225,15 +225,6 @@ class SemanticCache:
base = messages_hash or self._generate_key(query)
key = self._combined_key(namespace, base)
# Evict if adding a NEW key would exceed capacity. Overwriting a key that
# is already present is an in-place update that does not grow the map, so
# it must NOT evict — the old code ran the eviction loop before computing
# the key, so re-storing an existing entry at capacity dropped an
# unrelated live entry and turned a later lookup for it into a false miss.
# (Mirrors CompressionCache.store_compressed, which deletes-then-inserts.)
while key not in self._cache and len(self._cache) >= self.config.max_entries:
self._evict_oldest()
# Generate embedding if available — but never for an empty/blank query.
# A stored empty-query entry with an embedding would be a false-match
# target for the semantic get() path; leaving its embedding empty makes
@@ -254,6 +245,18 @@ class SemanticCache:
namespace=namespace,
)
# Evict if adding a NEW key would exceed capacity. Overwriting a key that
# is already present is an in-place update that does not grow the map, so
# it must NOT evict — the old code ran the eviction loop before computing
# the key, so re-storing an existing entry at capacity dropped an
# unrelated live entry and turned a later lookup for it into a false miss.
# (Mirrors CompressionCache.store_compressed, which deletes-then-inserts.)
# The embedding call above may raise, so evict only once the new entry is
# built — never before, or a failed store would drop a live entry without
# replacing it.
while key not in self._cache and len(self._cache) >= self.config.max_entries:
self._evict_oldest()
self._cache[key] = entry
# Index by hash for fast exact matching
+46
View File
@@ -123,6 +123,52 @@ class TestSemanticCache:
got2 = cache.get("query2", messages_hash="h2")
assert got2 is not None and got2.response == "response2b"
def test_put_failure_after_eviction_keeps_live_entry(self):
"""A failed embedding must not silently drop an already-evicted entry.
The eviction loop used to run before the embedding call, so if the
embedding function raised, an old live entry was already evicted while
the new entry was never stored — a permanent data loss on a transient
error. The exception must still propagate.
"""
config = SemanticCacheConfig(max_entries=1, similarity_threshold=0.9)
cache = SemanticCache(config)
call_count = {"n": 0}
def flaky_embedding(text: str) -> list[float]:
call_count["n"] += 1
if call_count["n"] == 1:
return [1.0, 0.0]
raise RuntimeError("embedding backend unavailable")
cache._embedding_fn = flaky_embedding
# First put succeeds and fills the cache to capacity.
cache.put("a", "response-a", messages_hash="ha")
assert cache.get("a", messages_hash="ha") is not None
assert len(cache._cache) == 1
# Second put raises, but the live entry must survive untouched.
with pytest.raises(RuntimeError, match="embedding backend unavailable"):
cache.put("b", "response-b", messages_hash="hb")
assert cache.get("a", messages_hash="ha") is not None
assert len(cache._cache) == 1
# With the embedding working again, a new put evicts "a" normally.
def stable_embedding(text: str) -> list[float]:
# Distinct vectors per query so "a" and "c" never false-hit.
if text == "a":
return [1.0, 0.0]
return [0.0, 1.0]
cache._embedding_fn = stable_embedding
cache.put("c", "response-c", messages_hash="hc")
assert cache.get("a", messages_hash="ha") is None
assert cache.get("c", messages_hash="hc") is not None
assert len(cache._cache) == 1
def test_ttl_expiration(self):
"""Test TTL expiration."""
config = SemanticCacheConfig(ttl_seconds=1)