mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
4eb742abd8 |
fix(cache): refresh semantic cache recency on updates (#4767)
## Description Overwriting a SemanticCache entry refreshed its response and timestamps but left its old position in the OrderedDict. With capacity two, put(A), put(B), update(A), put(C) evicted the freshly updated A and retained the older B. Move a successfully stored entry to the end of the LRU. Cache state operations now share an RLock, including semantic scans and touches, while embedding callbacks execute outside it. This happens after embedding generation and entry construction, so a failed update still preserves the original eviction order. It complements the query-only retrieval fix in #4709: a successfully refreshed response should remain retrievable under subsequent capacity pressure. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Refresh LRU position after successful insertion/replacement. - Cover query-only and context-hash entries, with and without namespaces. - Keep failed embedding updates from refreshing recency, and retain the no-eviction-on-update contract from #2094. - Synchronize cache/index/statistics operations and the complete similarity-scan/touch sequence. Embedding callbacks run outside the state lock; add two-thread scan/update and nonblocking exact-hit regressions. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Initial new regressions against original production code: 4 failed, 1 passed pytest tests/test_cache tests/test_cold_prefix.py: 398 passed, 3 skipped Review follow-up on the initial PR head: 1 failed Original pre-commit and commit-msg hooks (including follow-up): passed Whole-tree Ruff check / format --check: passed (1932 files) Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: built, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The three cache skips require optional spaCy/embedding models. Critical-suite skips require optional relevance models. The existing token-count approximation warning remains visible in the gate log. Initial broader runs failed because the new worktrees lacked the native extension; these were rerun successfully after building it. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development venv; clean baseline `200c838c7393d20ae56a0c338f832ed27133c398`, patch `ffa5198d3e4a7ab634e49efb8b6c6b59e4315acf`. `HEADROOM_OFFLINE=1`, `PYTHONPATH=.` for the selected checkout. No paid/provider credentials. - Exact command / steps: Save the following identical script as `proof.py` and run `python proof.py` from either checkout. Before and after runs were outside pytest. ```python """A real SemanticCache overwrite followed by a capacity-triggered eviction.""" import json from headroom.cache.semantic import SemanticCache, SemanticCacheConfig reports = [] for context in (False, True): for namespace in ("", "provider/model"): cache = SemanticCache(SemanticCacheConfig(max_entries=2)) hashes = ["context-a", "context-b", "context-c"] if context else [None] * 3 cache.put("first", {"answer": "old"}, hashes[0], namespace) cache.put("second", {"answer": "older"}, hashes[1], namespace) cache.put("first", {"answer": "updated"}, hashes[0], namespace) before = cache.get_stats() cache.put("third", {"answer": "new"}, hashes[2], namespace) first = cache.get("first", hashes[0], namespace) second = cache.get("second", hashes[1], namespace) reports.append({"context": context, "namespace": namespace, "update_evictions": before["evictions"], "updated_response": first.response if first else None, "older_response": second.response if second else None, "entries": cache.get_stats()["entries"], "evictions": cache.get_stats()["evictions"]}) print(json.dumps(reports, indent=2)) ``` Review follow-up: run `python concurrency_proof.py` with the script below on the initial PR head and the revised head. It coordinates two real cache callers at a similarity-scan boundary. ```python """A real two-thread scan/update interleaving, without mocked cache results.""" import json import threading from headroom.cache.semantic import SemanticCache scanning, release, writer_progress = threading.Event(), threading.Event(), threading.Event() class PausedCache(SemanticCache): def _cosine_similarity(self, a, b): if not scanning.is_set(): scanning.set() if not release.wait(10): raise RuntimeError("Scan release missing") return super()._cosine_similarity(a, b) cache = PausedCache(embedding_fn=lambda query: [1.0, 0.0]) cache.put("first", "old", "first") cache.put("second", "two", "second") real = threading.RLock() class SignalsWriterLock: def __enter__(self): if threading.current_thread().name == "cache-update": writer_progress.set() return real.__enter__() def __exit__(self, *exc): return real.__exit__(*exc) cache._lock = SignalsWriterLock() errors, results = [], [] def read(): try: entry = cache.get("third", "third") results.append(entry.response if entry else None) except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc)) def update(): try: cache.put("first", "updated", "first") except Exception as exc: errors.append(type(exc).__name__ + ": " + str(exc)) finally: writer_progress.set() reader = threading.Thread(target=read) writer = threading.Thread(target=update, name="cache-update") reader.start() try: assert scanning.wait(10) writer.start() assert writer_progress.wait(10) finally: release.set() reader.join(10) if writer.ident is not None: writer.join(10) assert not reader.is_alive() and not writer.is_alive() print(json.dumps({"errors": errors, "lookup_responses": results, "updated_response": cache.get("first", "first").response})) # Existing tie behavior: an ordinary exact hit already changes scan order. # No put/update occurs between the two semantic lookups in this control. ties = SemanticCache(embedding_fn=lambda query: [1.0, 0.0]) ties.put("first", "A", "first") ties.put("second", "B", "second") first = ties.get("one unseen query", "unseen-1").response ties.get("first", "first") second = ties.get("another unseen query", "unseen-2").response print(json.dumps({"equal_score_control": [first, second], "policy": "first match in current LRU order; no most-recent tie guarantee"})) ``` - Observed result: The real SemanticCache API reproduces the sequence for four context/namespace combinations. Before the patch, the updated response is missing and the older response survives. After it, updated_response is updated and older_response is None; each case still has two entries and exactly one eviction. No get() is made between update(A) and put(C), which would mask the defect by touching A. The concurrent proof raises OrderedDict mutated during iteration on the initial PR head and completes without errors on the revised head. The equal-score control returns A then B on both original main and revised code: ties still follow current LRU scan order, with no added freshness guarantee. - Not tested: Paid/live embedding services, model quality, production hit-rate/latency improvements and the complete Python suite. The runtime proof uses exact matching without an embedding dependency. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing cache behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Successful replacement counts as recent use. Capacity, exact/context isolation, TTL and failed-store behavior remain unchanged. - Kill switch / disable path: Disable the existing optional semantic cache in its caller, or revert this commit. - Unsafe override required: No. - Qualification impact: Related cache tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit. No dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Implementation and local validation were completed with Codex assistance. The author reviewed the initial code and validation record; the review follow-up was checked with Codex. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4767" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> |
||
|
|
a82088847b |
fix(cache): restore query-only semantic cache lookups (#4709)
## Description
`SemanticCache.put()` stores a query-derived entry when `messages_hash`
is omitted, but `get()` only tried exact lookup when that hash was
supplied. With the default `embedding_fn=None`, `put("What failed?",
response)` followed by `get("What failed?")` always missed. Look up the
same query-derived key before embedding search. Require a nonblank
query, the same namespace and query text, and an entry stored without a
context hash, so this fallback cannot consume a context-keyed response.
`SemanticCacheLayer` also skips both cache reads and writes if
complete-message serialization fails. An unknown context identity must
not become an intentional query-only request.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Add exact query-only retrieval before semantic search, with context
and namespace guards.
- Reject unhashable Layer contexts (objects, sets and cycles) for both
lookup and storage.
- Cover round trips, disabled matching, expiry, LRU refresh,
embedder-outage exact hits, blank queries and a query-hash/context-key
collision control.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (`mypy headroom`, original pre-commit hook)
- [x] New regression tests added
- [x] Standalone runtime proof completed
### Test Output
```text
Original regressions on unmodified main production code: 7 failed, 3 passed
Review follow-up regressions on the initial PR head: 6 failed
pytest tests/test_cache tests/test_cold_prefix.py: 364 passed, 3 skipped
Original pre-commit + commit-msg checks: passed
ruff 0.17.3 check . / format --check .: passed (1921 files formatted)
Original pre-push make ci-precheck: passed
Rust workspace: 1739 passed, 3 ignored
Native extension: built, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
The three cache-suite skips require optional spaCy/an embedding model.
The critical-suite skips require optional relevance models. Its existing
token-count approximation warning remains visible in the gate log.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing development venv;
base `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b`, patch
`eff3172c2483861e60f07f97179b8d4139a7aa7f`. `HEADROOM_OFFLINE=1`,
`PYTHONPATH` set to the selected checkout. The script uses production
APIs.
- Exact command / steps: Run `python proof.py` from the selected
checkout, using the following standalone script. The same file was run
against clean main and this branch, outside pytest.
```python
import json
from headroom.cache import SemanticCache
cache = SemanticCache()
cache.put("What failed?", {"answer": "disk full"}, namespace="project-a")
hit = cache.get("What failed?", namespace="project-a")
other = cache.get("What failed?", namespace="project-b")
scoped = cache.get("What failed?", messages_hash="context", namespace="project-a")
cache.put("What failed?", "context answer", messages_hash="context", namespace="project-a")
exact = cache.get("What failed?", messages_hash="context", namespace="project-a")
print(json.dumps({
"query_only_response": hit.response if hit else None,
"other_namespace_hit": other is not None,
"context_request_consumed_query_only_entry": scoped is not None,
"explicit_context_response": exact.response if exact else None,
}))
```
- Observed result: The query-only request now returns the stored
response; a different namespace and an explicit context request still
miss that entry. Explicit context-hash lookup still returns its own
response.
```text
Before: {"query_only_response": null, "other_namespace_hit": false, "context_request_consumed_query_only_entry": false, "explicit_context_response": "context answer"}
After: {"query_only_response": {"answer": "disk full"}, "other_namespace_hit": false, "context_request_consumed_query_only_entry": false, "explicit_context_response": "context answer"}
```
Review follow-up: the same standalone `python review-proof.py` was run
before and after the Layer guard:
```python
import json
from headroom.cache import AnthropicCacheOptimizer, OptimizationContext, SemanticCacheLayer
layer = SemanticCacheLayer(AnthropicCacheOptimizer())
context = OptimizationContext(provider="anthropic", model="fixture")
first = [{"role": "user", "content": "Hello!", "opaque": object()}]
second = [{"role": "user", "content": "Hello!", "opaque": object()}]
key = layer.store_response(first, "first conversation's answer", context)
result = layer.process(second, context)
print(json.dumps({"unhashable_store_accepted": bool(key),
"other_conversation_cache_hit": result.semantic_cache_hit,
"cached_response": result.cached_response}))
```
```text
Before follow-up: {"unhashable_store_accepted": true, "other_conversation_cache_hit": true, "cached_response": "first conversation's answer"}
After follow-up: {"unhashable_store_accepted": false, "other_conversation_cache_hit": false, "cached_response": null}
```
- Not tested: Live model/provider traffic, embedding-model quality,
hosted CI and the complete Python test suite. Runtime validation above
is local public-API/filesystem/CLI behavior; no paid request was made.
## Runtime Rollout Safety
- Rollout-managed feature(s): No new rollout flags; existing optional
cache/TTL-learning surfaces only.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Exact query-only entries become
retrievable; blank queries still require an exact context hash. Existing
context-hash and embedding matching remain available for serializable
requests; the Layer bypasses caching for unhashable messages.
- Kill switch / disable path: Set
`SemanticCacheConfig.use_exact_matching=False`, or do not use the
optional cache.
- Unsafe override required: No.
- Qualification impact: Relevant cache tests and the original local
gates pass; hosted qualification is left to this PR's checks.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Implementation, regression tests and standalone validation were
completed with Codex assistance. The author reviewed the initial code
and validation record; the review follow-up is documented above.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4709"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
911bb74074 |
fix(cache): reject nonpositive semantic cache capacities (#4301)
## Description Closes #4282. Reject `max_entries <= 0` at the start of `SemanticCache.put()`, before expiration cleanup, eviction, or overwriting a response. Otherwise the eviction loop cannot terminate once the cache is empty. The check is at write time because the public configuration can change after construction. Positive capacities retain their existing behavior; zero is not reinterpreted as a silent no-cache mode. Thanks to @oxura for the diagnosis, reproduction, and published fork fix linked in the issue. This PR independently implements the guard and adds regression coverage. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change ## Changes Made - Add the capacity guard and document the `ValueError`. - Cover both invalid initial capacities and runtime changes to zero/negative capacity. - Verify rejected inserts and overwrites preserve entries and statistics, and that restoring positive capacity allows writes again. ## Testing - [x] Focused cache unit and client-integration tests - [x] Ruff lint and formatting - [x] Mypy and commitlint through the required commit hooks - [x] Rust formatting and Clippy - [x] Native extension rebuild/import verification and the standard critical Python pre-push suite - [x] Six new regression cases fail before the fix and pass afterward - [x] Separate public-API smoke test without mocks or provider requests - [x] Full normal pre-push gate, including 1,658 Rust tests ```text uv run --no-sync python -m pytest tests/test_cache -q 303 passed, 2 skipped uv run --no-sync ruff check headroom/cache/semantic.py tests/test_cache/test_semantic.py All checks passed! uv run --no-sync ruff format --check headroom/cache/semantic.py tests/test_cache/test_semantic.py 2 files already formatted uv run --no-sync make ci-precheck-python ci-precheck-commitlint 177 passed, 4 skipped, 1 warning ORT_DYLIB_PATH="$(uv run --no-sync python -c 'from headroom._ort import ensure_ort_dylib_pinned; print(ensure_ort_dylib_pinned())')" \ uv run --no-sync git push --set-upstream fork fix/semantic-cache-positive-capacity 1658 Rust tests passed 177 critical Python tests passed, 4 skipped ci-precheck PASSED — safe to push. ``` **Local environment:** the optional ONNX Runtime dependency was initially absent. Installing and pinning it resolved the Kompress model-load stall. The latest runtime (1.31) then reproduced the already-tracked macOS thread-count test issue #4191; using the repository's locked version (1.26.0) allowed the entire normal pre-push gate to pass. No hooks were bypassed, and no dependencies, test skips, or Rust files were changed in this PR. ## Real Behavior Proof - Environment: macOS arm64, Python 3.12.0, current source installed with `uv sync --extra dev`; ONNX Runtime 1.26.0 for the native verification gate. The public cache smoke uses default exact matching, without an embedding function, credentials, or provider/model calls. - Before: four independent subprocesses using the public `SemanticCacheLayer` API hung and were stopped after three seconds each: initial capacity 0/-1 and capacity changed to 0/-1 after storing a response. - Observed result: all four cases raise `ValueError` promptly, preserve the previous cache state/response, and resume successful stores and cache hits once capacity is restored to 1. - Not tested: Windows/Linux runtime, remote providers, or semantic-cache matching against a real embedding service. The full Python suite is not claimed; the cache/client-integration and standard critical suites were run. - Exact command / steps: run the standalone public-API smoke below from the checkout. Pass the same context when storing and looking up, as required by current main cache namespacing. Reproducible post-fix smoke: ```bash uv run --no-sync python - <<'PY' from headroom.cache import ( AnthropicCacheOptimizer, OptimizationContext, SemanticCacheLayer, ) context = OptimizationContext(provider="anthropic", model="claude-3-opus") old = [{"role": "user", "content": "original request"}] new = [{"role": "user", "content": "replacement request"}] for changed in (False, True): for capacity in (0, -1): layer = SemanticCacheLayer( AnthropicCacheOptimizer(), max_entries=1 if changed else capacity, ) if changed: layer.store_response(old, {"text": "original answer"}, context) layer.cache.config.max_entries = capacity before = layer.get_stats() try: layer.store_response(new, {"text": "replacement answer"}, context) except ValueError as error: assert str(error) == "max_entries must be positive" else: raise AssertionError("invalid capacity accepted") assert layer.get_stats() == before if changed: result = layer.process(old, context) assert result.semantic_cache_hit assert result.cached_response == {"text": "original answer"} layer.cache.config.max_entries = 1 layer.store_response(new, {"text": "replacement answer"}, context) result = layer.process(new, context) assert result.semantic_cache_hit assert result.cached_response == {"text": "replacement answer"} state = "changed" if changed else "initial" print(f"{state} capacity={capacity}: rejected; state preserved; recovery OK") PY ``` Observed output: ```text initial capacity=0: rejected; state preserved; recovery OK initial capacity=-1: rejected; state preserved; recovery OK changed capacity=0: rejected; state preserved; recovery OK changed capacity=-1: rejected; state preserved; recovery OK ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing cache correctness fix. - Minimum rollout channel: normal CI-qualified release. - Stable/default behavior changed: invalid nonpositive capacities raise instead of hanging or overwriting an existing key; positive capacities and defaults are unchanged. - Kill switch / disable path: none needed; configure a positive capacity. - Unsafe override required: no. - Qualification impact: cache tests and normal repository hooks. - Rollback path: revert this commit, which would restore the invalid-capacity bug. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] The change follows the existing code style. - [x] Public behavior and errors are documented in the configuration comment and method docstring. - [x] Regression tests prove the fix and preserve valid-capacity behavior. - [x] No dependencies, feature flags, or unrelated changes were added. - [x] `CHANGELOG.md` was not edited. - [x] No co-author trailer or history rewrite. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4301" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Maintainer requalification on Windows/Python 3.12 against current main: all 347 cache tests passed, two optional tests skipped; changed-file lint, formatting, and whitespace checks passed. The public-API smoke above was rerun with current-main context namespacing and produced all four stated successful rejection/state-preservation/recovery lines. The separate proxy cache noted by Devin is outside this library-cache fix; no claim of fixing it is made. No live provider or fresh native build is claimed. Hosted qualification for the published sync remains pending. Co-authored-by: Romin Parekh <roparekh@adobe.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
0d4c450d77 |
fix(cache): build semantic entry before evicting so a failed embedding keeps live entries (#4581)
## Description `SemanticCache.put()` evicted the oldest entry before calling the embedding function. When the embedding raised (e.g. an OpenAI 503), the new entry was never stored but a live one was already gone, so a cache at capacity lost a valid response on a transient failure. Fixes #4283 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/cache/semantic.py`: move the capacity eviction loop in `put()` to after the embedding is computed and the `CacheEntry` is built, right before insertion. The embedding exception still propagates; successful stores evict exactly as before. - `tests/test_cache/test_semantic.py`: regression `test_put_failure_after_eviction_keeps_live_entry` (max_entries=1, embedding fails on the second put; old entry survives, size stays 1; a later successful put evicts normally). ## Testing - [x] Unit tests pass - [x] Regression test fails on `main` and passes with the fix ``` $ python -m pytest tests/test_cache/test_semantic.py -q 37 passed in 1.94s # on upstream/main source: FAILED tests/test_cache/test_semantic.py::TestSemanticCache::test_put_failure_after_eviction_keeps_live_entry 1 failed, 36 passed in 1.72s $ ruff check / ruff format --check (0.16.8) All checks passed! 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, uv tool Python env, openai 2.54.0, headroom upstream/main vs this branch; local HTTP embeddings endpoint (first call 200, then 503) - Exact command / steps: SemanticCache(max_entries=1) with embedding_fn = real OpenAI client (base_url=local endpoint, max_retries=0); put("old query"), put("new query") -> 503, then get("old query") - Observed result: on upstream/main the failed put leaves 0 entries and the old lookup misses (falls through to the failing embedding); with this fix the 503 still raises InternalServerError, the cache keeps 1 entry and the old exact lookup returns "old response" - Not tested: a hosted OpenAI endpoint under real overload ``` == with fix after first put: entries = 1 second put raised: InternalServerError after failed put: entries = 1 old exact lookup: old response == upstream/main after first put: entries = 1 second put raised: InternalServerError after failed put: entries = 0 openai.InternalServerError: Error code: 503 (old entry gone; get() re-embeds and fails) ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only on embedding failure, where the existing entry is now kept instead of evicted - Kill switch / disable path: n/a (semantic cache can be disabled as before) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
99cdaaf1c7 |
fix(cache): namespace SemanticCacheLayer by provider, model and metadata (#4473)
## Description `SemanticCacheLayer` keyed exact matches on the messages hash alone and its embedding search scanned every stored entry, so `OptimizationContext.provider`, `.model` and `.metadata` never fenced a lookup. Identical (or semantically similar) messages sent to a different model, provider, or with different metadata (e.g. temperature) were served the first producer's cached response. Fixes #4281 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `CacheEntry` gains a `namespace` field. - `SemanticCache.get` / `put` take an optional `namespace` (default `""`). The cache key and exact-match index key are `sha256(namespace + "\0" + base)`, where base is the messages hash or the query-derived fallback. Exact hits also verify `entry.namespace`, and `_find_similar` skips entries from other namespaces. One global LRU / `max_entries` limit is kept. - `SemanticCacheLayer._compute_namespace` builds the namespace as sorted JSON of `provider`, `model`, `metadata`. `request_id`, `timestamp` and `query` are excluded, so they do not affect matching. If the context is not JSON-serializable, the layer skips the cache (lookup delegates straight to the provider optimizer, store returns `""`) instead of sharing entries under an incomplete identity. - Regression tests in `tests/test_cache/test_semantic.py`, including the context-free `store_response` policy: it stores under the unscoped namespace and only matches lookups whose context also has empty provider/model/metadata. ## Testing - [x] Unit tests added / updated - [x] Existing test suite passes locally ``` $ python -m pytest tests/test_cache -q ======================= 313 passed, 2 skipped ======================= $ ruff check headroom/cache tests/test_cache All checks passed! $ ruff format --check headroom/cache tests/test_cache 31 files already formatted ``` `mypy` on `headroom/cache/semantic.py` reports no errors in the changed file (only the pre-existing Windows-only `os.fchmod` / `O_NOFOLLOW` notes elsewhere). ## Real Behavior Proof - Environment: Windows 11, Python 3.13, branch rebased on upstream/main; standalone script importing the public `headroom.cache` API (not the test suite) - Exact command / steps: `python proof.py`. Stores one response under `anthropic/claude-3-opus, temperature=0` via `SemanticCacheLayer.store_response`, then calls `process` with varied contexts on both the exact path and the embedding path (a deterministic `embedding_fn`, and a different query with an identical embedding). Ran the same script against `upstream/main`'s `semantic.py` for comparison. - Observed result: fix branch: same context hit=True, new request_id hit=True, other model / other provider / other temperature hit=False, semantic same context hit=True, semantic other model hit=False, unserializable metadata store key=''. On upstream/main: other model hit=True, other temperature hit=True, semantic other model hit=True (the #4281 bug). - Client-level check: `HeadroomClient(OpenAIProvider(), enable_semantic_cache=True, default_mode="optimize")` over a mocked upstream SDK. One `chat.completions.create(model="gpt-4o")` call (miss, goes upstream), then `client._store_response_in_semantic_cache(...)` populates with the client's own provider+model context, then `create` for `gpt-4o` and `gpt-4o-mini`. Fix branch: gpt-4o served from cache, gpt-4o-mini went upstream (2 upstream calls). upstream/main: gpt-4o-mini was served the gpt-4o cached response (1 upstream call). - Not tested: the proxy (it does not use `SemanticCacheLayer`). Note `HeadroomClient._store_response_in_semantic_cache` has no call site on main, so the client never auto-populates the cache today; the client check above invokes it directly. Wiring it in is out of scope for this fix. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: not applicable (library bug fix) - Stable/default behavior changed: yes, only in that cached responses no longer cross provider/model/metadata boundaries; same-context hits are unchanged - Kill switch / disable path: not applicable; callers that don't use SemanticCacheLayer are unaffected - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
455f4f263c |
fix(cache/semantic): don't semantic-match an empty query across contexts (#3226)
## Description
`SemanticCache.get()` matches on the **embedding of the last user
message** whenever an `embedding_fn` is wired. That query is empty
(`""`) for the overwhelming majority of agent/tool turns — a
`tool_result` continuation carries no text block, so
`SemanticCacheLayer._extract_query` returns `""`. A real sentence
embedder maps `""` to a fixed **non-zero** vector, so every empty-query
turn is ~identical to every other in embedding space. The exact
`messages_hash` guard (correctly chosen so `"continue"`/`"yes"` turns in
different contexts don't collide) is then bypassed by the semantic path:
an empty-query request misses on its unique hash, falls through to
embedding matching, and hits a **different conversation's** stored
response.
Reproduction (realistic embedder, non-zero for `""`):
```python
c = SemanticCache(embedding_fn=embed)
c.put(query="", response={"answer": "A"}, messages_hash="ctxA") # conversation A
c.get(query="", messages_hash="ctxB") # conversation B, different context
# -> returned A's response (cross-context false hit)
```
Measured on 330 real Claude Code transcripts (28,441 requests): **95.7%
have an empty extracted query**, so this is the dominant case, not a
corner case. The exact-hash path is unaffected; only the
embedding-similarity path is.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/semantic.py`:
- `get()`: gate the semantic-similarity branch on `query.strip()` — an
empty/blank query can only ever hit via its exact `messages_hash`
(context-complete), never via embedding similarity.
- `put()`: store no embedding for an empty/blank query, so such an entry
is skipped by `_find_similar` (which ignores entries with no embedding)
and can never be a match target.
- `tests/test_cache/test_semantic.py`: added
`test_empty_query_never_semantic_matches` (cross-context empty-query
miss, exact-hash still hits, whitespace treated as empty).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_cache/test_semantic.py -> 22 passed in 2.39s
uvx ruff@0.16.2 check headroom/cache/semantic.py tests/test_cache/test_semantic.py -> All checks passed!
uvx mypy@1.20.2 headroom/cache/semantic.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: before the fix, two different-context
empty-query requests (`ctxA` then `ctxB`) returned `ctxA`'s response via
the embedding path. After the fix, the second returns `None`, while
`ctxA`'s own exact-hash lookup still returns its response, and a
legitimate non-empty semantic hit (`"What is the weather today?"` ->
`"How is the weather?"`) still works.
- Observed result: empty/blank queries no longer semantic-match across
contexts; exact-hash and non-empty semantic matching are unchanged.
- Not tested: no live embedder model wired (the current client wires
none — the embedding path is exercised with an injected `embedding_fn`,
which is the documented usage).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. `SemanticCache` is an SDK-side cache
(`headroom.cache`), not a rollout-channel-gated runtime feature;
semantic matching only runs when a caller injects an `embedding_fn`.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Exact-hash matching and non-empty
semantic matching are unchanged; only empty/blank-query semantic
matching (a false-hit source) is removed.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: none; correctness-only.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
|
||
|
|
cf6367add4 |
fix(cache/semantic): don't evict an unrelated entry on an update at capacity (#2094)
## Description
`SemanticCache.put` can evict a perfectly good, unrelated entry when it
merely updates a key that is already cached.
The method runs its at-capacity eviction loop *before* it computes the
entry's key:
```python
self._cleanup_expired()
# Evict if at capacity
while len(self._cache) >= self.config.max_entries:
self._evict_oldest()
...
key = messages_hash or self._generate_key(query)
...
self._cache[key] = entry
```
So when the same key is stored again while the cache is full (a
duplicate store, or a retried request that produces the same
`messages_hash`), the loop fires because `len == max_entries`, evicts
the LRU-oldest *distinct* entry, and only then overwrites the existing
key in place. Writing to an already-present key does not grow the map,
so nothing needed to be evicted — but an unrelated live entry is now
gone, and the next `get` for it is a false miss.
Concretely, with `max_entries=2` and keys `[h1, h2]`, re-storing `h2`
evicts `h1`, leaving `[h2]` even though only two distinct keys were ever
stored.
The sibling `CompressionCache.store_compressed` gets this right: it
deletes the existing key first, inserts, and only then trims — so
re-storing a present key never drops an unrelated entry.
## Fix
Compute the key first, then run the eviction loop only while the key is
genuinely new:
```python
key = messages_hash or self._generate_key(query)
while key not in self._cache and len(self._cache) >= self.config.max_entries:
self._evict_oldest()
```
An in-place update of an existing key no longer evicts anything; adding
a new key still trims to make room exactly as before.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/semantic.py`: move the cache-key computation above the
eviction loop and gate the loop on `key not in self._cache` so an
in-place update never evicts.
- `tests/test_cache/test_semantic.py`: add
`test_update_at_capacity_does_not_evict_unrelated_entry`.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
$ python -m py_compile headroom/cache/semantic.py tests/test_cache/test_semantic.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the eviction logic with a
dependency-free script that replicates the `OrderedDict` +
`_evict_oldest` (popitem last=False) behavior for the old vs new loop,
and left the full pytest to CI.
- Exact command / steps: with `max_entries=2`, store `h1` then `h2`,
then re-store the already-present `h2`, under both the old loop (evict
before key dedup) and the new loop (evict only when key is new).
- Observed result: old loop leaves `['h2']` and `get(h1)` returns `None`
(h1 wrongly evicted); new loop leaves `['h1', 'h2']` with `get(h1)`
intact and `h2` updated. The regression test asserts h1 survives and h2
reflects the update.
- Not tested: a live embedding-backed cache round-trip; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized reordering of two existing
statements plus a loop guard, verified by the standalone proof and the
new regression test for CI. This is a different defect from the earlier
messages-hash keying fix — that one was about which slot a request maps
to; this one is about eviction dropping a live entry on an in-place
update.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
d8783ab89b |
fix(cache/semantic): key entries by context hash, not query text (#2022)
## Description
`SemanticCache` (`headroom/cache/semantic.py`) derives each entry's key
from the **query text
only** — where `query` is just the trailing user message — and its
exact-match lookup returns
the slot without checking the stored entry's `messages_hash`:
```python
# put()
key = self._generate_key(query) # sha256(query)[:16]
self._cache[key] = entry
if messages_hash:
self._hash_index[messages_hash] = key
# get() — exact-match branch
key = self._hash_index.get(messages_hash)
if key and key in self._cache:
entry = self._cache[key]
...
return entry # never checks entry.messages_hash
```
So two requests that share a trailing user message but differ in earlier
context map to the
**same** key. The second `put` overwrites the first, and the first
request's `messages_hash`
still points at that (now overwritten) slot — so it is served the
**other conversation's**
cached response.
Trailing messages like `"continue"`, `"yes"`, `"fix it"`, `"run the
tests"` are extremely
common in agentic/coding sessions, so this collides constantly. It's
independent of the
proxy-level `_compute_key` fix (that's about what goes *into*
`messages_hash`; here the entry
is stored under a query-only key regardless of how good the hash is).
This `SemanticCache` is
the one used by the SDK client's `enable_semantic_cache` path.
Concretely:
1. `put("run the tests", A, messages_hash=HA)` → key `K = sha256("run
the tests")`; `_cache[K]=A`.
2. `put("run the tests", B, messages_hash=HB)` → same `K`; `_cache[K]`
overwritten with `B`.
3. `get("run the tests", HA)` → `_hash_index[HA]=K`, `K in _cache` →
returns **B**.
Closes: no issue filed — found while auditing the cache key derivation.
## Fix
1. Key entries by the full-context `messages_hash` when present, falling
back to the query hash
only when no hash is supplied:
```python
key = messages_hash or self._generate_key(query)
```
2. Defensively verify `entry.messages_hash == messages_hash` in the
exact-match branch of `get`,
so any residual stale mapping becomes a miss rather than wrong data.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/semantic.py`: key `put` entries by `messages_hash`
when present; verify `entry.messages_hash` in the `get` exact-match
branch.
- `tests/test_cache/test_semantic.py`: add
`test_same_query_different_context_does_not_collide` and
`test_exact_match_verifies_messages_hash`.
## Testing
- [x] New regression tests added (`tests/test_cache/test_semantic.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the `put`/`get`
logic with a dependency-free script and left the full pytest to CI.
- Exact command / steps: stored responses A and B under the same query
`"run the tests"` with different `messages_hash`, then read each hash
back — through both the old (query-keyed) and new (hash-keyed) logic.
- Observed result: the old logic serves B's response to request A; the
new logic isolates them:
```text
OLD: A->RESPONSE_B B->RESPONSE_B
NEW: A->RESPONSE_A B->RESPONSE_B
SEMANTIC CACHE COLLISION FIX VERIFIED (OLD served B to A; NEW isolates)
```
- Not tested: the full SDK `HeadroomClient` round-trip with
`enable_semantic_cache=True` (needs the heavy stack). The fix is
confined to `SemanticCache.put`/`get` and the new tests drive them
directly. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Small, contained fix — the key derivation plus a verification guard,
no new dependencies.
- @JerrettDavis tagging you — this one can serve one conversation's
cached response to another when the last message matches, so it seemed
worth surfacing. Thanks!
|
||
|
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
|
7a05808e0f |
Add cache optimization module with scalable dynamic content detection
Implements a comprehensive cache optimization layer for LLM providers: - Provider-specific optimizers (Anthropic, OpenAI, Google) with distinct caching strategies: explicit breakpoints, prefix stabilization, and CachedContent API respectively - Scalable dynamic content detector using three strategies: 1. Structural detection: "Label: value" patterns (language-agnostic) 2. Entropy-based detection: high-entropy strings (IDs, tokens, hashes) 3. Universal patterns: ISO 8601, UUIDs, JWTs, hex hashes - NO hardcoded locale-specific patterns (no month names, etc.) - Semantic caching layer with LRU eviction and TTL support - Plugin registry for provider selection and custom optimizers - 131 tests, real-world benchmarks showing 20-55% compression at <0.3ms |