mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 16:45:34 +08:00
feat/https-compose-caddy
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
327d92414f |
fix(cache): keep literal cache_control fields in the response-cache key (#4799)
## Description The proxy response-cache key policy strips `cache_control` recursively at every dict depth, with a single carve-out for JSON Schema `properties` names (#327). But `cache_control` is overloaded: on a content block, a `system` entry or a tool definition it is a transport prompt-cache directive, while inside tool arguments and JSON Schema literals it is ordinary application data the model actually sees. So two requests that differ only in that data get the same key. With `cache_enabled` on by default, the second caller is replayed the first caller's response for the full TTL. Reproduced on `main` (`e995eb9f`) against the real `SemanticCache` API, no mock: ```python msgs = [{"role": "assistant", "content": [{"type": "tool_use", "id": "c1", "name": "configure", "input": {"cache_control": "first"}}]}] changed = copy.deepcopy(msgs); changed[0]["content"][0]["input"]["cache_control"] = "second" c = SemanticCache() await c.set(msgs, "m", b'{"id":"first"}', {}, partition="caller") print("replayed:", (await c.get(changed, "m", partition="caller")) is not None) ``` Before (this branch's parent, `e995eb9f`): ```text changed tool payload replayed: True changed schema replayed: True ``` Closes #4272 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/semantic_cache_key_policy.py`: `strip_cache_control` no longer recurses into fields whose value is opaque payload — `input`, `arguments`, `input_schema`, `parameters`, `schema`, `properties`, `const`, `default`, `enum`, `example`, `examples`, `text` — so a `cache_control`-named field there stays in the key. Everywhere else (content blocks, `system`, tool definitions) it is still dropped, so a moved transport breakpoint still does not fragment the key. The narrower `properties`-only carve-out falls out of the general rule. - `tests/test_proxy_semantic_cache_key_policy.py`: covers the mandatory-partition precondition (`ValueError` when no partition is passed), which was the one uncovered line Codecov flagged in this change. - `tests/test_semantic_cache_literal_payloads.py`: new regression suite covering tool `input`, tool `input_schema` `default`, and `response_format` schema `const`, each through the real `SemanticCache` (changed request misses, original still hits), plus a guard for the decoded-dict `function.arguments` shape Headroom accepts (`handlers/openai.py` parses either), a guard that OpenAI's JSON-*string* `arguments` is passed through verbatim, and guards that transport `cache_control` is still ignored and nested transport annotations are still dropped. One pure module changed; no provider, no native rebuild. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `mypy` is unchecked because the pre-commit hook fails on 11 pre-existing Windows-only `attr-defined` errors in files this PR does not touch (`headroom/fileperms.py`, `headroom/ccr/mcp_server.py`, `headroom/proxy/interceptors/astgrep.py`: `os.fchmod`, `os.O_DIRECTORY`, `fcntl`); neither changed file produces a mypy error. ### Test Output Regression tests fail before the fix. With `headroom/proxy/semantic_cache_key_policy.py` reverted and the new tests kept: ```text $ git show HEAD~1:headroom/proxy/semantic_cache_key_policy.py > headroom/proxy/semantic_cache_key_policy.py $ uv run pytest tests/test_semantic_cache_literal_payloads.py -q FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_does_not_alias_cached_responses[tool_input] FAILED ... [tool_arguments_dict] FAILED ... [tool_schema] FAILED ... [response_schema] FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_changes_the_key[tool_input] FAILED ... [tool_arguments_dict] FAILED ... [tool_schema] FAILED ... [response_schema] FAILED tests/test_semantic_cache_literal_payloads.py::test_strip_cache_control_still_drops_nested_transport_annotations 9 failed, 3 passed in 0.64s ``` The three that pass against the old policy are the guards: the OpenAI string-`arguments` case (a string is never rewritten), the schema-property-name case (#327), and the moved-transport-breakpoint case. After the fix: ```text $ uv run pytest tests/test_semantic_cache_literal_payloads.py tests/test_proxy_semantic_cache_key_policy.py \ tests/test_semantic_cache_key_policy.py tests/test_proxy_response_cache_partition.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_openai_response_cache_key.py tests/test_proxy_openai_cache_key_integration.py -q 94 passed, 1 warning in 28.39s $ uv run pytest <the four policy/partition files> -q \ --cov=headroom.proxy.semantic_cache_key_policy --cov-report=term-missing --cov-branch Name Stmts Miss Branch BrPart Cover headroom\proxy\semantic_cache_key_policy.py 54 0 16 0 100% ``` $ uv run pytest tests/test_cache/test_semantic.py tests/test_openai_response_cache_key.py \ tests/test_proxy_openai_cache_key_integration.py tests/test_proxy_response_cache_store_guard.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_shared_context.py -q 133 passed, 1 warning in 34.11s $ uv run pytest tests/test_cache_control_move_bust.py tests/test_anthropic_fork_lineage_cache.py \ tests/test_cross_turn_cache_safety.py tests/test_cache/test_prefix_tracker.py -q 113 passed, 1 warning in 6.21s $ uv run ruff check headroom/proxy/semantic_cache_key_policy.py tests/test_semantic_cache_literal_payloads.py All checks passed! $ uv run ruff format --check <both files> 1 file already formatted ``` ## Real Behavior Proof - Environment: Windows 11, `uv run` against the project venv (Python 3.13.11), branch `fix/4272-literal-payload-cache-keys` cut from upstream `main` `e995eb9f`. No mocked cache and no provider: the reproducer calls the real `SemanticCache.set`/`get` in `headroom/proxy/semantic_cache.py`. - Exact command / steps: run the reproducer above as a script under `uv run python` — store a response under a tool input containing `{"cache_control": "first"}`, deep-copy the messages, change that value to `"second"`, then `get` with the changed messages; repeat for a tool `input_schema` `default` and a `response_format` schema `const`. - Observed result: before the fix the changed request hit the cache in both cases (`changed tool payload replayed: True`, `changed schema replayed: True`). After the fix both print `False`, and the original request still hits (asserted in each regression test), so the key was not merely widened. - Not tested: the real OpenAI HTTP path with a decoded-dict `arguments` field, which is a nonstandard client encoding (`arguments` is normally a JSON string, covered by `test_openai_arguments_string_is_not_rewritten`); the full `make ci-precheck` (Rust/native rebuild) and the end-to-end proxy HTTP path; the change is confined to the pure key policy, which the tests above exercise directly through the real `SemanticCache`. `headroom/cache/prefix_tracker.py` has the same blunt recursive strip, but it serves prefix-content equality rather than response replay and is left as a separate follow-up. ## Runtime Rollout Safety - Rollout-managed feature(s): none; the proxy response cache (`cache_enabled`) is not a rollout-gated feature. - Minimum rollout channel: n/a - Stable/default behavior changed: yes, deliberately — for requests whose tool data or JSON Schema values contain a `cache_control`-named field, two distinct requests that used to share one key now get distinct keys. Requests without such a field produce the same key as before, so hit rate there is unchanged. - Kill switch / disable path: `cache_enabled = false`, or the existing proxy config for the response cache; no new switch. - Unsafe override required: no - Qualification impact: none; no rollout-managed surface touched. - Rollback path: revert this commit. Cache keys are process-local (`SemanticCache` is in-memory), so a revert needs no cache migration or invalidation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — not needed: the key policy's docstrings describe the new rule, and no user-facing docs claim `cache_control` is stripped from payloads. - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes `headroom/cache/prefix_tracker.py:165` has the same blunt recursive strip, used for prefix-content equality rather than response replay. Worth a follow-up, deliberately out of scope here so the response-replay fix stays a one-module change. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
ee6cfcccf6 |
fix(security): partition the response cache by credential and principal (#3862)
## Why On a proxy shared by several callers, the semantic response cache was keyed only by request content and upstream (#3349). A second caller who sent the identical request with a different provider key was served the completion generated for the first caller, under the first caller's key, and never reached the upstream. That discloses completions across tenants (finding 01-F15). ## What changes - `compute_cache_partition()` in `headroom/proxy/semantic_cache_key_policy.py`: an HMAC-SHA256 under a per-process random key over the caller's credential and account-selector headers plus the authenticated principal. Headers counted: every header the shared credential-header rule (`internal_header_policy.is_credential_header`) matches, which covers `authorization`, `proxy-authorization`, `cookie` and `x-api-key`, and any header whose name ends in `api-key`, `key`, `token`, `secret`, `account`, `account-id`, `organization` or `project` (so `x-api-key`, `api-key`, `x-goog-api-key`, `chatgpt-account-id`, `openai-organization`, `openai-project` and so on). Matching by name shape keeps new providers partitioned by default. Cookie is used as a whole value; it is only ever HMAC input. `x-headroom-*` headers (including the proxy's own token, which the security gate has already removed) and idempotency keys are excluded. A request with no credential and no principal gets the shared `anon` partition. - The partition is an HMAC, not a plain hash, so it cannot be used to confirm a guessed credential. - `identity.resolve_authenticated_principal()` returns a principal only when an identity resolver is installed (`set_identity_resolver`). The OSS default identity is the same for every network caller and can be chosen by header on loopback, so it is not treated as a principal. - If an installed identity resolver raises or returns no principal, the request fails closed (`resolve_authenticated_principal()` raises `UnresolvedPrincipalError` for an empty result): `compute_request_cache_partition()` logs a warning and returns `None`, and both handlers skip the response cache for that request (no lookup, no store). It never falls back to the credential-only partition, which tenants on one operator key would share. With no resolver installed, the credential-only partition is unchanged. - Both handlers compute the partition only when the response cache can be used (cache enabled and non-streaming), so streaming and cache-disabled requests do not call the identity resolver. - `partition` is a required keyword on `SemanticCache.get`, `set` and `_compute_key`, and on `compute_semantic_cache_key`. A future call site that forgets it fails immediately instead of silently sharing. - The `/v1/messages` and `/v1/chat/completions` handlers compute the partition once, next to the existing cache-key snapshot, and reuse it at store time so lookup and store agree. Other caches in the request path are not changed. The OpenAI Responses unit cache and the CCR retrieval store are content-addressed, so a hit requires already holding the content. CCR entries are not bound to a caller; that is a separate issue. The `/v1/compress` pipeline cache holds no user content, and the SDK `SemanticCache` is in-process for one caller. ## Behaviour change The proxy response cache no longer shares entries between callers that present different provider credentials or, with an identity resolver installed, different principals. - One developer with one key: same hit rate. - Deployments where callers send no provider credential and the proxy supplies the operator's key: same hit rate; they share the `anon` partition. - Shared proxy with per-caller keys: entries are no longer shared between keys. That is the fix. - Compatible upstream gateways that authenticate by `Cookie` or `Proxy-Authorization`: entries are no longer shared between sessions. - Callers whose only credential is a proxy-token `Authorization` header, or deployments using oauth2, share one partition unless an identity resolver is installed, because core cannot tell them apart. ## Tests `tests/test_proxy_response_cache_partition.py`: - The partition function: differs per credential and is stable for one credential; every credential and account header shape partitions; other headers do not fragment the cache; no credential and no principal gives the shared partition; it is keyed, not a plain hash; a principal separates callers sharing one credential. - `SemanticCache` refuses a call without a partition and keeps partitions apart. - End to end through the real Anthropic and OpenAI handlers: a second caller is never served the first caller's cached response, whether the credential is `x-api-key`/`Authorization`, `Cookie` or `Proxy-Authorization`. The `Cookie` and `Proxy-Authorization` cases fail without the latest change. With an identity resolver installed, callers on one operator key are partitioned. These end-to-end tests fail on `main`. - A resolver that raises, or returns an empty principal, yields no partition, and end to end (both handlers) callers on one operator key never share a cached response and nothing is stored. These fail without the fail-closed change. With no resolver installed, the partition is the credential-only one. - With the response cache disabled, neither handler calls the identity resolver. Existing cache tests now pass a fixed partition. The buffered-stream CCR test derives the same partition the handler computes. Manual run: two callers send the identical `/v1/messages` request with different `x-api-key` values through a proxy in front of a capture upstream. On `main` the second caller gets the first caller's answer and only the first key reaches the upstream. On this branch each caller gets its own answer, and a repeat by the first caller is still a cache hit. Not tested: multi-worker deployments (the cache and its key are per process, as before). ## Merge order / conflicts Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 → #3863. Textual conflict: #3860 adds an import next to the one this PR adds in `handlers/anthropic.py` and `handlers/openai.py`. Whichever merges second keeps both imports. Revised 2026-10-04 after review: forwarded `Cookie` and `Proxy-Authorization` headers are now partition input (via the shared credential-header rule), so cookie-authenticated callers no longer share the anonymous partition. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3862" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
b84c4c9f71 |
fix(cache): preserve cache_control schema properties (#3348)
## Description
Preserve legitimate JSON Schema properties named `cache_control` when
building semantic response-cache keys. The normalizer now strips only
Anthropic prompt-cache annotations (`{"type": "ephemeral"}` with
optional `ttl`) instead of every key with that name.
Closes #3345
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Narrow `cache_control` removal to the documented prompt-cache
annotation shape.
- Keep both semantic-cache policy modules synchronized.
- Add regressions proving user-defined schema properties remain in cache
identity.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/pytest -q tests/test_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key_integration.py
42 passed, 1 warning in 1.74s
$ .venv/bin/ruff check <changed files>
All checks passed!
$ .venv/bin/ruff format --check <changed files>
4 files already formatted
$ .venv/bin/mypy headroom/proxy/semantic_cache_key.py headroom/proxy/semantic_cache_key_policy.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Linux, Python 3.14, current `main` plus this patch,
in-memory `SemanticCache`.
- Exact command / steps: instantiated production `SemanticCache`,
computed keys for otherwise-identical tool schemas with and without a
boolean property named `cache_control`, and compared the keys.
- Observed result: the two production cache keys are distinct, so the
schemas no longer collide.
```text
collision=False
keys_distinct=True
```
- Not tested: live Anthropic API traffic; full repository test suite.
## Runtime Rollout Safety
- Rollout-managed feature(s): Semantic response cache key normalization.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes, only for tool/schema payloads
containing a non-annotation `cache_control` key.
- Kill switch / disable path: disable semantic caching with
`cache_enabled=False`.
- Unsafe override required: No.
- Qualification impact: cache entries for affected schemas are correctly
partitioned after deployment.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A.
## Additional Notes
Documentation is unchanged because this corrects internal cache-key
normalization without changing the public API. The warning above is the
existing Starlette `TestClient` deprecation warning.
## Maintainer refresh (2026-09-16)
Current main removed the unused duplicate `semantic_cache_key.py`; this
refresh preserves that deletion and applies the schema-property fix only
in `semantic_cache_key_policy.py`, which the proxy imports. Both
policy-level and SemanticCache integration regressions remain. All 35
focused cache-key tests pass, and focused Ruff checks pass. Reported
governance checks pass; the local test results are recorded above.
---------
Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
|
||
|
|
e9c6276e24 |
refactor(proxy): delete duplicate semantic_cache_key module (#3472)
## Description `headroom/proxy/semantic_cache_key.py` (added in #1953) and `headroom/proxy/semantic_cache_key_policy.py` (added in #1964) were the same 42-line module: identical ASTs, differing only in two docstrings. Production `SemanticCache` already imported the `_policy` copy; the other copy's only consumer was its own test file. This deletes the copy nothing runs and points its test at the copy that remains. The retained module is byte-for-byte the one the proxy was already executing, so no cache key changes. The module path was never exported from `headroom.proxy` and has no reference anywhere outside that one test (checked across `headroom/`, `tests/`, `plugins/`, `sdk/`, `scripts/`, `benchmarks/`, docs, wiki and CI config), so no compatibility re-export is kept. Closes nothing; this is the consolidation flagged in the repository efficiency audit. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/semantic_cache_key.py`: deleted (42 lines). - `tests/test_proxy_semantic_cache_key_policy.py`: import repointed from the deleted module to `headroom.proxy.semantic_cache_key_policy`; the tests themselves are untouched. Nothing else. In particular `headroom/proxy/semantic_cache.py` is not touched because it already imported the retained module. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed No new tests: the change deletes a duplicate and repoints one import. The existing suites for both module paths, the `SemanticCache` key regression tests and the Anthropic/OpenAI cache-key integration tests all pass unchanged before and after. ### Test Output ```text $ pytest tests/test_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key_policy.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_proxy_openai_cache_key_integration.py tests/test_openai_response_cache_key.py \ tests/test_cache/test_semantic.py -q 67 passed, 1 warning in 4.20s # identical count on the unchanged base: 67 passed $ pytest --collect-only -q tests 12549 tests collected in 16.14s # no module fails to import after the deletion $ pre-commit run --files tests/test_proxy_semantic_cache_key_policy.py headroom/proxy/semantic_cache.py ruff (legacy alias)......................................................Passed ruff format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: macOS, Python 3.12 venv, worktree based on `main` @ `e67b3c8a`. - Exact command / steps: `git diff --no-index` of the two modules on `main` (only the two docstring lines differ); `grep -rn "semantic_cache_key" .` before and after the deletion to enumerate every reference; the pytest runs above; `compute_semantic_cache_key` evaluated for representative requests (plain messages, `cache_control` on content blocks, `system`/`tools`/`temperature` kwargs, a tool schema with a property named `cache_control`) through both the retained module and the deleted copy loaded from `main`, with identical hashes. - Observed result: every semantic-cache key is unchanged; the only import of the deleted path was the one test file, now repointed; full test collection succeeds. - Not tested: no live upstream provider call; no production traffic. Neither exercises anything this change touches, since the executed code is identical. ## Runtime Rollout Safety - Rollout-managed feature(s): None. The semantic response cache key policy is not rollout-gated. - Minimum rollout channel: N/A. No rollout-managed feature is touched. - Stable/default behavior changed: No. The proxy executes the same module it did before; cache keys, `cache_control` stripping and the `SemanticCache._compute_key` wrapper are unchanged. - Kill switch / disable path: N/A. No flag. Reverting the commit restores the duplicate file. - Unsafe override required: No. - Qualification impact: None. No compression decision, cache hit/miss behavior, CCR, TTL or persistence path is affected. - Rollback path: `git revert` of this single commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. No user-visible surface changes. ## Additional Notes - Maintainer context on #3348 (preserve `cache_control` schema properties; open, approved by a collaborator, blocked on required checks): that PR currently patches both copies. A `git merge-tree` of the two heads shows exactly one modify/delete conflict on the deleted file and a clean auto-merge of the test file. Whichever lands first, the other rebases trivially. If this lands first, #3348 drops its hunk for the deleted file and its `_policy` and test hunks apply as-is. If #3348 lands first, this PR re-deletes the file and nothing else changes. This PR deliberately does not absorb #3348's fix, to keep the two reviews independent. - Two test files still cover the one remaining module (`tests/test_semantic_cache_key_policy.py` and `tests/test_proxy_semantic_cache_key_policy.py`, with three near-duplicate cases between them). They are left as-is here so #3348's new test applies cleanly; merging them is a candidate follow-up once #3348 settles. - No documentation references the deleted module path, so there is nothing to update. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
2cae0f8eaf |
fix(proxy/cache): strip cache_control from messages in the semantic cache key (#3086)
## Description
The proxy semantic response-cache key (`compute_semantic_cache_key`)
strips `cache_control` from the response-shaping fields (`system`,
`tools`, ...) so that a moved prompt-cache breakpoint does not fragment
the key:
```python
{
"model": model,
"messages": messages, # hashed verbatim
**{k: strip_cache_control(v) for k, v in key_fields.items()}, # stripped
}
```
But `messages` was hashed **verbatim**. Messages are the primary key
component, and on the Anthropic path they are the most common place a
client (e.g. Claude Code) places and *moves* a `cache_control`
breakpoint between turns (on the last user turn / a `tool_result`
block). So two otherwise-identical requests that differed only in a
message-level breakpoint produced different keys and missed the semantic
cache — the exact fragmentation the `strip_cache_control` helper exists
to prevent, applied to everything except the field that matters most.
The existing tests pin the strip for `system`
(`test_cache_control_breakpoint_move_same_key`) and `tools`
(`test_tools_cache_control_ignored`), but never covered a message-level
breakpoint, so the gap went unnoticed.
## Fix
Apply `strip_cache_control` to `messages` as well. `cache_control` is a
prompt-caching directive for the upstream provider that never changes
the generated completion, so removing the annotation before hashing is
sound: message *content* still differentiates the key, and two requests
that differ only in a `cache_control` breakpoint now share the cache
entry (whose stored response body is identical either way).
The proxy carries two in-sync copies of this pure policy
(`semantic_cache_key_policy.py`, imported by the runtime
`SemanticCache`, and `semantic_cache_key.py`, imported by the policy
test); both are updated identically so they do not diverge.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/semantic_cache_key_policy.py` and
`headroom/proxy/semantic_cache_key.py`: hash
`strip_cache_control(messages)` instead of `messages`, with a docstring
explaining why message-level breakpoints must not fragment the key.
- `tests/test_proxy_semantic_cache_key.py`: added
`test_message_cache_control_breakpoint_move_same_key` (behavioral,
through `SemanticCache._compute_key`) and
`test_message_content_change_still_distinct_key` (guards that stripping
does not collapse genuinely different messages).
- `tests/test_proxy_semantic_cache_key_policy.py`: added
`test_semantic_cache_key_ignores_moved_message_cache_control` at the
pure-policy level.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
### Test Output
```text
tests/test_proxy_semantic_cache_key.py + tests/test_proxy_semantic_cache_key_policy.py 33 passed
# uvx ruff@0.15.22 check -> All checks passed!
# uvx mypy@1.20.2 (both policy modules) -> Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted the two policy modules and ran the new
tests to capture the bug (`python -m pytest
tests/test_proxy_semantic_cache_key.py::test_message_cache_control_breakpoint_move_same_key
tests/test_proxy_semantic_cache_key_policy.py::test_semantic_cache_key_ignores_moved_message_cache_control`
-> both failed with two distinct SHA-256 keys for messages that differ
only in a `cache_control` breakpoint); restored the fix; re-ran both key
suites (`python -m pytest tests/test_proxy_semantic_cache_key.py
tests/test_proxy_semantic_cache_key_policy.py` -> 33 passed); ran the
wider `tests/test_cache/` suite and confirmed the only failures
(`test_client_integration.py`) reproduce identically on clean `main` and
are unrelated to this change; then `uvx ruff@0.15.22 format`, `uvx
ruff@0.15.22 check`, and `uvx mypy@1.20.2` on both modules.
- Observed result: before the fix, a request whose last message carries
`cache_control: {type: ephemeral}` hashes to a different key than the
same request without it; after the fix they hash identically (a cache
hit), while messages with different text still hash differently.
- Not tested: a live multi-turn proxy session measuring the hit-rate
improvement (the key contract is verified directly through
`SemanticCache._compute_key` and the pure policy, which is what the
runtime calls).
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the pure semantic-cache key
policy behind `SemanticCache`, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: yes, as a bug fix. Requests that
differ only in a message-level `cache_control` breakpoint now share a
semantic-cache key (a hit) instead of missing. No request that differs
in message content, model, or any shaping field changes key. Because the
cache key changes shape, any entries stored under the old (un-stripped)
keys are simply not reused and age out under the existing TTL/LRU — a
one-time cold start for the affected entries, never a wrong response.
- Kill switch / disable path: the semantic cache itself is already gated
by the existing cache-enable configuration; disabling it bypasses this
path entirely.
- Unsafe override required: no.
- Qualification impact: higher semantic-cache hit rate on the Anthropic
path where clients move `cache_control` breakpoints between turns; no
change to which distinct requests are considered equal beyond ignoring
the caching directive.
- Rollback path: revert this PR; the key returns to hashing messages
verbatim.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title
## Additional Notes
Same class as the `system`/`tools` breakpoint handling already in place
(issue #327 kept the strip from fragmenting the key on a hit); this
extends it to messages, the primary key component. The two in-sync
policy copies are updated together to avoid divergence; consolidating
them into one module is left out of scope for this bug fix.
|
||
|
|
740fb9bc16 |
refactor(cache): isolate semantic key policy (#1953)
## Description Extracts proxy semantic-cache key normalization and hashing into a pure policy module while preserving `SemanticCache._compute_key` for existing callers and tests. This separates deterministic cache-key construction from the async cache adapter and LRU storage concerns. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.semantic_cache_key` for pure cache-control stripping and semantic cache key construction. - Updated `SemanticCache._compute_key` to delegate to the extracted policy while preserving the local `_strip_cache_control` compatibility alias. - Added direct tests for the extracted semantic-cache key policy. - Kept the LiteLLM callback compatibility shim required for repo-wide type checking on fresh branches. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_proxy_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py tests/test_litellm_callback.py tests/test_compress_api.py::TestLiteLLMCallback -q 46 passed in 11.83s python -m ruff check . All checks passed! python -m ruff format --check . 1095 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 409 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, local worktree based on `headroomlabs/main`. - Exact command / steps: ran focused semantic cache key tests, handler cache-key integration tests, LiteLLM callback tests, Ruff lint/format checks, mypy over `headroom`, and staged gitleaks scan. - Observed result: all local checks passed; staged secret scan found no leaks. - Not tested: full CI matrix and deployment flows; those are covered by GitHub Actions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. GitHub reported existing Dependabot alerts on the default branch during push; this PR does not change dependencies, and the staged secret scan is clean. |