Commit Graph
10 Commits
Author SHA1 Message Date
Yuchuan He 4eb742abd8 fix(cache): refresh semantic cache recency on updates (#4767)
## Description

Overwriting a SemanticCache entry refreshed its response and timestamps
but left its old position in the OrderedDict. With capacity two, put(A),
put(B), update(A), put(C) evicted the freshly updated A and retained the
older B.

Move a successfully stored entry to the end of the LRU. Cache state
operations now share an RLock, including semantic scans and touches,
while embedding callbacks execute outside it. This happens after
embedding generation and entry construction, so a failed update still
preserves the original eviction order. It complements the query-only
retrieval fix in #4709: a successfully refreshed response should remain
retrievable under subsequent capacity pressure.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Refresh LRU position after successful insertion/replacement.
- Cover query-only and context-hash entries, with and without
namespaces.
- Keep failed embedding updates from refreshing recency, and retain the
no-eviction-on-update contract from #2094.

- Synchronize cache/index/statistics operations and the complete
similarity-scan/touch sequence. Embedding callbacks run outside the
state lock; add two-thread scan/update and nonblocking exact-hit
regressions.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Initial new regressions against original production code: 4 failed, 1 passed
pytest tests/test_cache tests/test_cold_prefix.py:
  398 passed, 3 skipped
Review follow-up on the initial PR head: 1 failed
Original pre-commit and commit-msg hooks (including follow-up): passed
Whole-tree Ruff check / format --check: passed (1932 files)
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: built, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The three cache skips require optional spaCy/embedding models.
Critical-suite skips require optional relevance models. The existing
token-count approximation warning remains visible in the gate log.
Initial broader runs failed because the new worktrees lacked the native
extension; these were rerun successfully after building it.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development venv;
clean baseline `200c838c7393d20ae56a0c338f832ed27133c398`, patch
`ffa5198d3e4a7ab634e49efb8b6c6b59e4315acf`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH=.` for the selected checkout. No paid/provider credentials.
- Exact command / steps: Save the following identical script as
`proof.py` and run `python proof.py` from either checkout. Before and
after runs were outside pytest.

```python
"""A real SemanticCache overwrite followed by a capacity-triggered eviction."""
import json
from headroom.cache.semantic import SemanticCache, SemanticCacheConfig

reports = []
for context in (False, True):
    for namespace in ("", "provider/model"):
        cache = SemanticCache(SemanticCacheConfig(max_entries=2))
        hashes = ["context-a", "context-b", "context-c"] if context else [None] * 3
        cache.put("first", {"answer": "old"}, hashes[0], namespace)
        cache.put("second", {"answer": "older"}, hashes[1], namespace)
        cache.put("first", {"answer": "updated"}, hashes[0], namespace)
        before = cache.get_stats()
        cache.put("third", {"answer": "new"}, hashes[2], namespace)
        first = cache.get("first", hashes[0], namespace)
        second = cache.get("second", hashes[1], namespace)
        reports.append({"context": context, "namespace": namespace,
            "update_evictions": before["evictions"], "updated_response": first.response if first else None,
            "older_response": second.response if second else None,
            "entries": cache.get_stats()["entries"], "evictions": cache.get_stats()["evictions"]})
print(json.dumps(reports, indent=2))
```


Review follow-up: run `python concurrency_proof.py` with the script
below on the initial PR head and the revised head. It coordinates two
real cache callers at a similarity-scan boundary.

```python
"""A real two-thread scan/update interleaving, without mocked cache results."""
import json
import threading
from headroom.cache.semantic import SemanticCache

scanning, release, writer_progress = threading.Event(), threading.Event(), threading.Event()

class PausedCache(SemanticCache):
    def _cosine_similarity(self, a, b):
        if not scanning.is_set():
            scanning.set()
            if not release.wait(10): raise RuntimeError("Scan release missing")
        return super()._cosine_similarity(a, b)

cache = PausedCache(embedding_fn=lambda query: [1.0, 0.0])
cache.put("first", "old", "first")
cache.put("second", "two", "second")
real = threading.RLock()

class SignalsWriterLock:
    def __enter__(self):
        if threading.current_thread().name == "cache-update": writer_progress.set()
        return real.__enter__()

    def __exit__(self, *exc): return real.__exit__(*exc)

cache._lock = SignalsWriterLock()
errors, results = [], []

def read():
    try:
        entry = cache.get("third", "third")
        results.append(entry.response if entry else None)
    except Exception as exc:
        errors.append(type(exc).__name__ + ": " + str(exc))

def update():
    try: cache.put("first", "updated", "first")
    except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc))
    finally: writer_progress.set()

reader = threading.Thread(target=read)
writer = threading.Thread(target=update, name="cache-update")
reader.start()
try:
    assert scanning.wait(10)
    writer.start()
    assert writer_progress.wait(10)
finally:
    release.set()
    reader.join(10)
    if writer.ident is not None: writer.join(10)
assert not reader.is_alive() and not writer.is_alive()
print(json.dumps({"errors": errors, "lookup_responses": results,
    "updated_response": cache.get("first", "first").response}))

# Existing tie behavior: an ordinary exact hit already changes scan order.
# No put/update occurs between the two semantic lookups in this control.
ties = SemanticCache(embedding_fn=lambda query: [1.0, 0.0])
ties.put("first", "A", "first")
ties.put("second", "B", "second")
first = ties.get("one unseen query", "unseen-1").response
ties.get("first", "first")
second = ties.get("another unseen query", "unseen-2").response
print(json.dumps({"equal_score_control": [first, second],
    "policy": "first match in current LRU order; no most-recent tie guarantee"}))
```

- Observed result: The real SemanticCache API reproduces the sequence
for four context/namespace combinations. Before the patch, the updated
response is missing and the older response survives. After it,
updated_response is updated and older_response is None; each case still
has two entries and exactly one eviction. No get() is made between
update(A) and put(C), which would mask the defect by touching A. The
concurrent proof raises OrderedDict mutated during iteration on the
initial PR head and completes without errors on the revised head. The
equal-score control returns A then B on both original main and revised
code: ties still follow current LRU scan order, with no added freshness
guarantee.
- Not tested: Paid/live embedding services, model quality, production
hit-rate/latency improvements and the complete Python suite. The runtime
proof uses exact matching without an embedding dependency.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Successful replacement counts as
recent use. Capacity, exact/context isolation, TTL and failed-store
behavior remain unchanged.
- Kill switch / disable path: Disable the existing optional semantic
cache in its caller, or revert this commit.
- Unsafe override required: No.
- Qualification impact: Related cache tests and original local gates
passed; hosted CI is tracked separately.
- Rollback path: Revert this commit. No dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Implementation and local validation were completed with Codex
assistance. The author reviewed the initial code and validation record;
the review follow-up was checked with Codex.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4767"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
2026-11-11 10:00:26 -06:00
Yuchuan HeandJD Davis a82088847b fix(cache): restore query-only semantic cache lookups (#4709)
## Description

`SemanticCache.put()` stores a query-derived entry when `messages_hash`
is omitted, but `get()` only tried exact lookup when that hash was
supplied. With the default `embedding_fn=None`, `put("What failed?",
response)` followed by `get("What failed?")` always missed. Look up the
same query-derived key before embedding search. Require a nonblank
query, the same namespace and query text, and an entry stored without a
context hash, so this fallback cannot consume a context-keyed response.



`SemanticCacheLayer` also skips both cache reads and writes if
complete-message serialization fails. An unknown context identity must
not become an intentional query-only request.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add exact query-only retrieval before semantic search, with context
and namespace guards.
- Reject unhashable Layer contexts (objects, sets and cycles) for both
lookup and storage.
- Cover round trips, disabled matching, expiry, LRU refresh,
embedder-outage exact hits, blank queries and a query-hash/context-key
collision control.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed

### Test Output

```text
Original regressions on unmodified main production code: 7 failed, 3 passed
Review follow-up regressions on the initial PR head: 6 failed
pytest tests/test_cache tests/test_cold_prefix.py: 364 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
  Rust workspace: 1739 passed, 3 ignored
  Native extension: built, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`eff3172c2483861e60f07f97179b8d4139a7aa7f`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.

```python
import json
from headroom.cache import SemanticCache

cache = SemanticCache()
cache.put("What failed?", {"answer": "disk full"}, namespace="project-a")
hit = cache.get("What failed?", namespace="project-a")
other = cache.get("What failed?", namespace="project-b")
scoped = cache.get("What failed?", messages_hash="context", namespace="project-a")
cache.put("What failed?", "context answer", messages_hash="context", namespace="project-a")
exact = cache.get("What failed?", messages_hash="context", namespace="project-a")
print(json.dumps({
    "query_only_response": hit.response if hit else None,
    "other_namespace_hit": other is not None,
    "context_request_consumed_query_only_entry": scoped is not None,
    "explicit_context_response": exact.response if exact else None,
}))
```

- Observed result: The query-only request now returns the stored
response; a different namespace and an explicit context request still
miss that entry. Explicit context-hash lookup still returns its own
response.

```text
Before: {"query_only_response": null, "other_namespace_hit": false, "context_request_consumed_query_only_entry": false, "explicit_context_response": "context answer"}
After:  {"query_only_response": {"answer": "disk full"}, "other_namespace_hit": false, "context_request_consumed_query_only_entry": false, "explicit_context_response": "context answer"}
```

Review follow-up: the same standalone `python review-proof.py` was run
before and after the Layer guard:

```python
import json
from headroom.cache import AnthropicCacheOptimizer, OptimizationContext, SemanticCacheLayer

layer = SemanticCacheLayer(AnthropicCacheOptimizer())
context = OptimizationContext(provider="anthropic", model="fixture")
first = [{"role": "user", "content": "Hello!", "opaque": object()}]
second = [{"role": "user", "content": "Hello!", "opaque": object()}]
key = layer.store_response(first, "first conversation's answer", context)
result = layer.process(second, context)
print(json.dumps({"unhashable_store_accepted": bool(key),
                  "other_conversation_cache_hit": result.semantic_cache_hit,
                  "cached_response": result.cached_response}))
```

```text
Before follow-up: {"unhashable_store_accepted": true, "other_conversation_cache_hit": true, "cached_response": "first conversation's answer"}
After follow-up:  {"unhashable_store_accepted": false, "other_conversation_cache_hit": false, "cached_response": null}
```

- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.

## Runtime Rollout Safety

- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Exact query-only entries become
retrievable; blank queries still require an exact context hash. Existing
context-hash and embedding matching remain available for serializable
requests; the Layer bypasses caching for unhashable messages.
- Kill switch / disable path: Set
`SemanticCacheConfig.use_exact_matching=False`, or do not use the
optional cache.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the initial code
and validation record; the review follow-up is documented above.


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4709"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-10 22:26:27 -06:00
911bb74074 fix(cache): reject nonpositive semantic cache capacities (#4301)
## Description

Closes #4282.

Reject `max_entries <= 0` at the start of `SemanticCache.put()`, before
expiration cleanup, eviction, or overwriting a response. Otherwise the
eviction loop cannot terminate once the cache is empty.

The check is at write time because the public configuration can change
after construction. Positive capacities retain their existing behavior;
zero is not reinterpreted as a silent no-cache mode.

Thanks to @oxura for the diagnosis, reproduction, and published fork fix
linked in the issue. This PR independently implements the guard and adds
regression coverage.

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change

## Changes Made

- Add the capacity guard and document the `ValueError`.
- Cover both invalid initial capacities and runtime changes to
zero/negative capacity.
- Verify rejected inserts and overwrites preserve entries and
statistics, and that restoring positive capacity allows writes again.

## Testing

- [x] Focused cache unit and client-integration tests
- [x] Ruff lint and formatting
- [x] Mypy and commitlint through the required commit hooks
- [x] Rust formatting and Clippy
- [x] Native extension rebuild/import verification and the standard
critical Python pre-push suite
- [x] Six new regression cases fail before the fix and pass afterward
- [x] Separate public-API smoke test without mocks or provider requests
- [x] Full normal pre-push gate, including 1,658 Rust tests

```text
uv run --no-sync python -m pytest tests/test_cache -q
303 passed, 2 skipped

uv run --no-sync ruff check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!

uv run --no-sync ruff format --check headroom/cache/semantic.py tests/test_cache/test_semantic.py
2 files already formatted

uv run --no-sync make ci-precheck-python ci-precheck-commitlint
177 passed, 4 skipped, 1 warning

ORT_DYLIB_PATH="$(uv run --no-sync python -c 'from headroom._ort import ensure_ort_dylib_pinned; print(ensure_ort_dylib_pinned())')" \
  uv run --no-sync git push --set-upstream fork fix/semantic-cache-positive-capacity
1658 Rust tests passed
177 critical Python tests passed, 4 skipped
ci-precheck PASSED — safe to push.
```

**Local environment:** the optional ONNX Runtime dependency was
initially absent. Installing and pinning it resolved the Kompress
model-load stall. The latest runtime (1.31) then reproduced the
already-tracked macOS thread-count test issue #4191; using the
repository's locked version (1.26.0) allowed the entire normal pre-push
gate to pass. No hooks were bypassed, and no dependencies, test skips,
or Rust files were changed in this PR.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.12.0, current source installed with
`uv sync --extra dev`; ONNX Runtime 1.26.0 for the native verification
gate. The public cache smoke uses default exact matching, without an
embedding function, credentials, or provider/model calls.
- Before: four independent subprocesses using the public
`SemanticCacheLayer` API hung and were stopped after three seconds each:
initial capacity 0/-1 and capacity changed to 0/-1 after storing a
response.
- Observed result: all four cases raise `ValueError` promptly, preserve
the previous cache state/response, and resume successful stores and
cache hits once capacity is restored to 1.
- Not tested: Windows/Linux runtime, remote providers, or semantic-cache
matching against a real embedding service. The full Python suite is not
claimed; the cache/client-integration and standard critical suites were
run.

- Exact command / steps: run the standalone public-API smoke below from
the checkout. Pass the same context when storing and looking up, as
required by current main cache namespacing.

Reproducible post-fix smoke:

```bash
uv run --no-sync python - <<'PY'
from headroom.cache import (
    AnthropicCacheOptimizer, OptimizationContext, SemanticCacheLayer,
)

context = OptimizationContext(provider="anthropic", model="claude-3-opus")
old = [{"role": "user", "content": "original request"}]
new = [{"role": "user", "content": "replacement request"}]
for changed in (False, True):
    for capacity in (0, -1):
        layer = SemanticCacheLayer(
            AnthropicCacheOptimizer(), max_entries=1 if changed else capacity,
        )
        if changed:
            layer.store_response(old, {"text": "original answer"}, context)
            layer.cache.config.max_entries = capacity
        before = layer.get_stats()
        try:
            layer.store_response(new, {"text": "replacement answer"}, context)
        except ValueError as error:
            assert str(error) == "max_entries must be positive"
        else:
            raise AssertionError("invalid capacity accepted")
        assert layer.get_stats() == before
        if changed:
            result = layer.process(old, context)
            assert result.semantic_cache_hit
            assert result.cached_response == {"text": "original answer"}
        layer.cache.config.max_entries = 1
        layer.store_response(new, {"text": "replacement answer"}, context)
        result = layer.process(new, context)
        assert result.semantic_cache_hit
        assert result.cached_response == {"text": "replacement answer"}
        state = "changed" if changed else "initial"
        print(f"{state} capacity={capacity}: rejected; state preserved; recovery OK")
PY
```

Observed output:

```text
initial capacity=0: rejected; state preserved; recovery OK
initial capacity=-1: rejected; state preserved; recovery OK
changed capacity=0: rejected; state preserved; recovery OK
changed capacity=-1: rejected; state preserved; recovery OK
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing cache correctness fix.
- Minimum rollout channel: normal CI-qualified release.
- Stable/default behavior changed: invalid nonpositive capacities raise
instead of hanging or overwriting an existing key; positive capacities
and defaults are unchanged.
- Kill switch / disable path: none needed; configure a positive
capacity.
- Unsafe override required: no.
- Qualification impact: cache tests and normal repository hooks.
- Rollback path: revert this commit, which would restore the
invalid-capacity bug.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] The change follows the existing code style.
- [x] Public behavior and errors are documented in the configuration
comment and method docstring.
- [x] Regression tests prove the fix and preserve valid-capacity
behavior.
- [x] No dependencies, feature flags, or unrelated changes were added.
- [x] `CHANGELOG.md` was not edited.
- [x] No co-author trailer or history rewrite.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4301"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Maintainer requalification on Windows/Python 3.12 against current main:
all 347 cache tests passed, two optional tests skipped; changed-file
lint, formatting, and whitespace checks passed. The public-API smoke
above was rerun with current-main context namespacing and produced all
four stated successful rejection/state-preservation/recovery lines. The
separate proxy cache noted by Devin is outside this library-cache fix;
no claim of fixing it is made. No live provider or fresh native build is
claimed. Hosted qualification for the published sync remains pending.

Co-authored-by: Romin Parekh <roparekh@adobe.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:45:17 -06:00
0d4c450d77 fix(cache): build semantic entry before evicting so a failed embedding keeps live entries (#4581)
## Description

`SemanticCache.put()` evicted the oldest entry before calling the
embedding function. When the embedding raised (e.g. an OpenAI 503), the
new entry was never stored but a live one was already gone, so a cache
at capacity lost a valid response on a transient failure.

Fixes #4283

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cache/semantic.py`: move the capacity eviction loop in
`put()` to after the embedding is computed and the `CacheEntry` is
built, right before insertion. The embedding exception still propagates;
successful stores evict exactly as before.
- `tests/test_cache/test_semantic.py`: regression
`test_put_failure_after_eviction_keeps_live_entry` (max_entries=1,
embedding fails on the second put; old entry survives, size stays 1; a
later successful put evicts normally).

## Testing

- [x] Unit tests pass
- [x] Regression test fails on `main` and passes with the fix

```
$ python -m pytest tests/test_cache/test_semantic.py -q
37 passed in 1.94s
# on upstream/main source:
FAILED tests/test_cache/test_semantic.py::TestSemanticCache::test_put_failure_after_eviction_keeps_live_entry
1 failed, 36 passed in 1.72s
$ ruff check / ruff format --check (0.16.8)
All checks passed! 2 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, uv tool Python env, openai 2.54.0, headroom
upstream/main vs this branch; local HTTP embeddings endpoint (first call
200, then 503)
- Exact command / steps: SemanticCache(max_entries=1) with embedding_fn
= real OpenAI client (base_url=local endpoint, max_retries=0); put("old
query"), put("new query") -> 503, then get("old query")
- Observed result: on upstream/main the failed put leaves 0 entries and
the old lookup misses (falls through to the failing embedding); with
this fix the 503 still raises InternalServerError, the cache keeps 1
entry and the old exact lookup returns "old response"
- Not tested: a hosted OpenAI endpoint under real overload

```
== with fix
after first put: entries = 1
second put raised: InternalServerError
after failed put: entries = 1
old exact lookup: old response
== upstream/main
after first put: entries = 1
second put raised: InternalServerError
after failed put: entries = 0
openai.InternalServerError: Error code: 503 (old entry gone; get() re-embeds and fails)
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on embedding failure, where the
existing entry is now kept instead of evicted
- Kill switch / disable path: n/a (semantic cache can be disabled as
before)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:36:09 -06:00
99cdaaf1c7 fix(cache): namespace SemanticCacheLayer by provider, model and metadata (#4473)
## Description

`SemanticCacheLayer` keyed exact matches on the messages hash alone and
its embedding search scanned every stored entry, so
`OptimizationContext.provider`, `.model` and `.metadata` never fenced a
lookup. Identical (or semantically similar) messages sent to a different
model, provider, or with different metadata (e.g. temperature) were
served the first producer's cached response.

Fixes #4281

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `CacheEntry` gains a `namespace` field.
- `SemanticCache.get` / `put` take an optional `namespace` (default
`""`). The cache key and exact-match index key are `sha256(namespace +
"\0" + base)`, where base is the messages hash or the query-derived
fallback. Exact hits also verify `entry.namespace`, and `_find_similar`
skips entries from other namespaces. One global LRU / `max_entries`
limit is kept.
- `SemanticCacheLayer._compute_namespace` builds the namespace as sorted
JSON of `provider`, `model`, `metadata`. `request_id`, `timestamp` and
`query` are excluded, so they do not affect matching. If the context is
not JSON-serializable, the layer skips the cache (lookup delegates
straight to the provider optimizer, store returns `""`) instead of
sharing entries under an incomplete identity.
- Regression tests in `tests/test_cache/test_semantic.py`, including the
context-free `store_response` policy: it stores under the unscoped
namespace and only matches lookups whose context also has empty
provider/model/metadata.

## Testing

- [x] Unit tests added / updated
- [x] Existing test suite passes locally

```
$ python -m pytest tests/test_cache -q
======================= 313 passed, 2 skipped =======================
$ ruff check headroom/cache tests/test_cache
All checks passed!
$ ruff format --check headroom/cache tests/test_cache
31 files already formatted
```

`mypy` on `headroom/cache/semantic.py` reports no errors in the changed
file (only the pre-existing Windows-only `os.fchmod` / `O_NOFOLLOW`
notes elsewhere).

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, branch rebased on upstream/main;
standalone script importing the public `headroom.cache` API (not the
test suite)
- Exact command / steps: `python proof.py`. Stores one response under
`anthropic/claude-3-opus, temperature=0` via
`SemanticCacheLayer.store_response`, then calls `process` with varied
contexts on both the exact path and the embedding path (a deterministic
`embedding_fn`, and a different query with an identical embedding). Ran
the same script against `upstream/main`'s `semantic.py` for comparison.
- Observed result: fix branch: same context hit=True, new request_id
hit=True, other model / other provider / other temperature hit=False,
semantic same context hit=True, semantic other model hit=False,
unserializable metadata store key=''. On upstream/main: other model
hit=True, other temperature hit=True, semantic other model hit=True (the
#4281 bug).
- Client-level check: `HeadroomClient(OpenAIProvider(),
enable_semantic_cache=True, default_mode="optimize")` over a mocked
upstream SDK. One `chat.completions.create(model="gpt-4o")` call (miss,
goes upstream), then `client._store_response_in_semantic_cache(...)`
populates with the client's own provider+model context, then `create`
for `gpt-4o` and `gpt-4o-mini`. Fix branch: gpt-4o served from cache,
gpt-4o-mini went upstream (2 upstream calls). upstream/main: gpt-4o-mini
was served the gpt-4o cached response (1 upstream call).
- Not tested: the proxy (it does not use `SemanticCacheLayer`). Note
`HeadroomClient._store_response_in_semantic_cache` has no call site on
main, so the client never auto-populates the cache today; the client
check above invokes it directly. Wiring it in is out of scope for this
fix.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: not applicable (library bug fix)
- Stable/default behavior changed: yes, only in that cached responses no
longer cross provider/model/metadata boundaries; same-context hits are
unchanged
- Kill switch / disable path: not applicable; callers that don't use
SemanticCacheLayer are unaffected
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 23:36:34 -05:00
Abhay Singh 455f4f263c fix(cache/semantic): don't semantic-match an empty query across contexts (#3226)
## Description

`SemanticCache.get()` matches on the **embedding of the last user
message** whenever an `embedding_fn` is wired. That query is empty
(`""`) for the overwhelming majority of agent/tool turns — a
`tool_result` continuation carries no text block, so
`SemanticCacheLayer._extract_query` returns `""`. A real sentence
embedder maps `""` to a fixed **non-zero** vector, so every empty-query
turn is ~identical to every other in embedding space. The exact
`messages_hash` guard (correctly chosen so `"continue"`/`"yes"` turns in
different contexts don't collide) is then bypassed by the semantic path:
an empty-query request misses on its unique hash, falls through to
embedding matching, and hits a **different conversation's** stored
response.

Reproduction (realistic embedder, non-zero for `""`):

```python
c = SemanticCache(embedding_fn=embed)
c.put(query="", response={"answer": "A"}, messages_hash="ctxA")   # conversation A
c.get(query="", messages_hash="ctxB")   # conversation B, different context
# -> returned A's response (cross-context false hit)
```

Measured on 330 real Claude Code transcripts (28,441 requests): **95.7%
have an empty extracted query**, so this is the dominant case, not a
corner case. The exact-hash path is unaffected; only the
embedding-similarity path is.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/semantic.py`:
- `get()`: gate the semantic-similarity branch on `query.strip()` — an
empty/blank query can only ever hit via its exact `messages_hash`
(context-complete), never via embedding similarity.
- `put()`: store no embedding for an empty/blank query, so such an entry
is skipped by `_find_similar` (which ignores entries with no embedding)
and can never be a match target.
- `tests/test_cache/test_semantic.py`: added
`test_empty_query_never_semantic_matches` (cross-context empty-query
miss, exact-hash still hits, whitespace treated as empty).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_cache/test_semantic.py  ->  22 passed in 2.39s
uvx ruff@0.16.2 check headroom/cache/semantic.py tests/test_cache/test_semantic.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/cache/semantic.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: before the fix, two different-context
empty-query requests (`ctxA` then `ctxB`) returned `ctxA`'s response via
the embedding path. After the fix, the second returns `None`, while
`ctxA`'s own exact-hash lookup still returns its response, and a
legitimate non-empty semantic hit (`"What is the weather today?"` ->
`"How is the weather?"`) still works.
- Observed result: empty/blank queries no longer semantic-match across
contexts; exact-hash and non-empty semantic matching are unchanged.
- Not tested: no live embedder model wired (the current client wires
none — the embedding path is exercised with an injected `embedding_fn`,
which is the documented usage).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. `SemanticCache` is an SDK-side cache
(`headroom.cache`), not a rollout-channel-gated runtime feature;
semantic matching only runs when a caller injects an `embedding_fn`.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Exact-hash matching and non-empty
semantic matching are unchanged; only empty/blank-query semantic
matching (a false-hit source) is removed.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none; correctness-only.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
2026-08-23 11:51:33 -07:00
Abhay SinghandTejas Chopra cf6367add4 fix(cache/semantic): don't evict an unrelated entry on an update at capacity (#2094)
## Description

`SemanticCache.put` can evict a perfectly good, unrelated entry when it
merely updates a key that is already cached.

The method runs its at-capacity eviction loop *before* it computes the
entry's key:

```python
self._cleanup_expired()

# Evict if at capacity
while len(self._cache) >= self.config.max_entries:
    self._evict_oldest()
...
key = messages_hash or self._generate_key(query)
...
self._cache[key] = entry
```

So when the same key is stored again while the cache is full (a
duplicate store, or a retried request that produces the same
`messages_hash`), the loop fires because `len == max_entries`, evicts
the LRU-oldest *distinct* entry, and only then overwrites the existing
key in place. Writing to an already-present key does not grow the map,
so nothing needed to be evicted — but an unrelated live entry is now
gone, and the next `get` for it is a false miss.

Concretely, with `max_entries=2` and keys `[h1, h2]`, re-storing `h2`
evicts `h1`, leaving `[h2]` even though only two distinct keys were ever
stored.

The sibling `CompressionCache.store_compressed` gets this right: it
deletes the existing key first, inserts, and only then trims — so
re-storing a present key never drops an unrelated entry.

## Fix

Compute the key first, then run the eviction loop only while the key is
genuinely new:

```python
key = messages_hash or self._generate_key(query)

while key not in self._cache and len(self._cache) >= self.config.max_entries:
    self._evict_oldest()
```

An in-place update of an existing key no longer evicts anything; adding
a new key still trims to make room exactly as before.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/semantic.py`: move the cache-key computation above the
eviction loop and gate the loop on `key not in self._cache` so an
in-place update never evicts.
- `tests/test_cache/test_semantic.py`: add
`test_update_at_capacity_does_not_evict_unrelated_entry`.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
$ python -m py_compile headroom/cache/semantic.py tests/test_cache/test_semantic.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the eviction logic with a
dependency-free script that replicates the `OrderedDict` +
`_evict_oldest` (popitem last=False) behavior for the old vs new loop,
and left the full pytest to CI.
- Exact command / steps: with `max_entries=2`, store `h1` then `h2`,
then re-store the already-present `h2`, under both the old loop (evict
before key dedup) and the new loop (evict only when key is new).
- Observed result: old loop leaves `['h2']` and `get(h1)` returns `None`
(h1 wrongly evicted); new loop leaves `['h1', 'h2']` with `get(h1)`
intact and `h2` updated. The regression test asserts h1 survives and h2
reflects the update.
- Not tested: a live embedding-backed cache round-trip; full local
`pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized reordering of two existing
statements plus a loop guard, verified by the standalone proof and the
new regression test for CI. This is a different defect from the earlier
messages-hash keying fix — that one was about which slot a request maps
to; this one is about eviction dropping a live entry on an in-place
update.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-13 10:50:23 -04:00
Abhay Singh d8783ab89b fix(cache/semantic): key entries by context hash, not query text (#2022)
## Description

`SemanticCache` (`headroom/cache/semantic.py`) derives each entry's key
from the **query text
only** — where `query` is just the trailing user message — and its
exact-match lookup returns
the slot without checking the stored entry's `messages_hash`:

```python
# put()
key = self._generate_key(query)          # sha256(query)[:16]
self._cache[key] = entry
if messages_hash:
    self._hash_index[messages_hash] = key

# get() — exact-match branch
key = self._hash_index.get(messages_hash)
if key and key in self._cache:
    entry = self._cache[key]
    ...
    return entry                         # never checks entry.messages_hash
```

So two requests that share a trailing user message but differ in earlier
context map to the
**same** key. The second `put` overwrites the first, and the first
request's `messages_hash`
still points at that (now overwritten) slot — so it is served the
**other conversation's**
cached response.

Trailing messages like `"continue"`, `"yes"`, `"fix it"`, `"run the
tests"` are extremely
common in agentic/coding sessions, so this collides constantly. It's
independent of the
proxy-level `_compute_key` fix (that's about what goes *into*
`messages_hash`; here the entry
is stored under a query-only key regardless of how good the hash is).
This `SemanticCache` is
the one used by the SDK client's `enable_semantic_cache` path.

Concretely:
1. `put("run the tests", A, messages_hash=HA)` → key `K = sha256("run
the tests")`; `_cache[K]=A`.
2. `put("run the tests", B, messages_hash=HB)` → same `K`; `_cache[K]`
overwritten with `B`.
3. `get("run the tests", HA)` → `_hash_index[HA]=K`, `K in _cache` →
returns **B**.

Closes: no issue filed — found while auditing the cache key derivation.

## Fix

1. Key entries by the full-context `messages_hash` when present, falling
back to the query hash
   only when no hash is supplied:
   ```python
   key = messages_hash or self._generate_key(query)
   ```
2. Defensively verify `entry.messages_hash == messages_hash` in the
exact-match branch of `get`,
   so any residual stale mapping becomes a miss rather than wrong data.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cache/semantic.py`: key `put` entries by `messages_hash`
when present; verify `entry.messages_hash` in the `get` exact-match
branch.
- `tests/test_cache/test_semantic.py`: add
`test_same_query_different_context_does_not_collide` and
`test_exact_match_verifies_messages_hash`.

## Testing

- [x] New regression tests added (`tests/test_cache/test_semantic.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).

```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the `put`/`get`
logic with a dependency-free script and left the full pytest to CI.
- Exact command / steps: stored responses A and B under the same query
`"run the tests"` with different `messages_hash`, then read each hash
back — through both the old (query-keyed) and new (hash-keyed) logic.
- Observed result: the old logic serves B's response to request A; the
new logic isolates them:

```text
OLD: A->RESPONSE_B  B->RESPONSE_B
NEW: A->RESPONSE_A  B->RESPONSE_B
SEMANTIC CACHE COLLISION FIX VERIFIED (OLD served B to A; NEW isolates)
```

- Not tested: the full SDK `HeadroomClient` round-trip with
`enable_semantic_cache=True` (needs the heavy stack). The fix is
confined to `SemanticCache.put`/`get` and the new tests drive them
directly. Full local `pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Small, contained fix — the key derivation plus a verification guard,
no new dependencies.
- @JerrettDavis tagging you — this one can serve one conversation's
cached response to another when the last message matches, so it seemed
worth surfacing. Thanks!
2026-07-11 10:11:09 -05:00
chopratejas e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas 7a05808e0f Add cache optimization module with scalable dynamic content detection
Implements a comprehensive cache optimization layer for LLM providers:

- Provider-specific optimizers (Anthropic, OpenAI, Google) with distinct
  caching strategies: explicit breakpoints, prefix stabilization, and
  CachedContent API respectively

- Scalable dynamic content detector using three strategies:
  1. Structural detection: "Label: value" patterns (language-agnostic)
  2. Entropy-based detection: high-entropy strings (IDs, tokens, hashes)
  3. Universal patterns: ISO 8601, UUIDs, JWTs, hex hashes

- NO hardcoded locale-specific patterns (no month names, etc.)

- Semantic caching layer with LRU eviction and TTL support

- Plugin registry for provider selection and custom optimizers

- 131 tests, real-world benchmarks showing 20-55% compression at <0.3ms
2026-01-07 14:07:49 -08:00