Commit Graph
6 Commits
Author SHA1 Message Date
ParideboyandJerrettDavis 327d92414f fix(cache): keep literal cache_control fields in the response-cache key (#4799)
## Description

The proxy response-cache key policy strips `cache_control` recursively
at every dict depth, with a single carve-out for JSON Schema
`properties` names (#327). But `cache_control` is overloaded: on a
content block, a `system` entry or a tool definition it is a transport
prompt-cache directive, while inside tool arguments and JSON Schema
literals it is ordinary application data the model actually sees.

So two requests that differ only in that data get the same key. With
`cache_enabled` on by default, the second caller is replayed the first
caller's response for the full TTL.

Reproduced on `main` (`e995eb9f`) against the real `SemanticCache` API,
no mock:

```python
msgs = [{"role": "assistant", "content": [{"type": "tool_use", "id": "c1",
         "name": "configure", "input": {"cache_control": "first"}}]}]
changed = copy.deepcopy(msgs); changed[0]["content"][0]["input"]["cache_control"] = "second"
c = SemanticCache()
await c.set(msgs, "m", b'{"id":"first"}', {}, partition="caller")
print("replayed:", (await c.get(changed, "m", partition="caller")) is not None)
```

Before (this branch's parent, `e995eb9f`):

```text
changed tool payload replayed: True
changed schema replayed: True
```

Closes #4272

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache_key_policy.py`: `strip_cache_control`
no longer recurses into fields whose value is opaque payload — `input`,
`arguments`, `input_schema`, `parameters`, `schema`, `properties`,
`const`, `default`, `enum`, `example`, `examples`, `text` — so a
`cache_control`-named field there stays in the key. Everywhere else
(content blocks, `system`, tool definitions) it is still dropped, so a
moved transport breakpoint still does not fragment the key. The narrower
`properties`-only carve-out falls out of the general rule.
- `tests/test_proxy_semantic_cache_key_policy.py`: covers the
mandatory-partition precondition (`ValueError` when no partition is
passed), which was the one uncovered line Codecov flagged in this
change.
- `tests/test_semantic_cache_literal_payloads.py`: new regression suite
covering tool `input`, tool `input_schema` `default`, and
`response_format` schema `const`, each through the real `SemanticCache`
(changed request misses, original still hits), plus a guard for the
decoded-dict `function.arguments` shape Headroom accepts
(`handlers/openai.py` parses either), a guard that OpenAI's
JSON-*string* `arguments` is passed through verbatim, and guards that
transport `cache_control` is still ignored and nested transport
annotations are still dropped.

One pure module changed; no provider, no native rebuild.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy` is unchecked because the pre-commit hook fails on 11 pre-existing
Windows-only `attr-defined` errors in files this PR does not touch
(`headroom/fileperms.py`, `headroom/ccr/mcp_server.py`,
`headroom/proxy/interceptors/astgrep.py`: `os.fchmod`, `os.O_DIRECTORY`,
`fcntl`); neither changed file produces a mypy error.

### Test Output

Regression tests fail before the fix. With
`headroom/proxy/semantic_cache_key_policy.py` reverted and the new tests
kept:

```text
$ git show HEAD~1:headroom/proxy/semantic_cache_key_policy.py > headroom/proxy/semantic_cache_key_policy.py
$ uv run pytest tests/test_semantic_cache_literal_payloads.py -q
FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_does_not_alias_cached_responses[tool_input]
FAILED ... [tool_arguments_dict]
FAILED ... [tool_schema]
FAILED ... [response_schema]
FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_changes_the_key[tool_input]
FAILED ... [tool_arguments_dict]
FAILED ... [tool_schema]
FAILED ... [response_schema]
FAILED tests/test_semantic_cache_literal_payloads.py::test_strip_cache_control_still_drops_nested_transport_annotations
9 failed, 3 passed in 0.64s
```

The three that pass against the old policy are the guards: the OpenAI
string-`arguments` case (a string is never rewritten), the
schema-property-name
case (#327), and the moved-transport-breakpoint case.

After the fix:

```text
$ uv run pytest tests/test_semantic_cache_literal_payloads.py tests/test_proxy_semantic_cache_key_policy.py \
    tests/test_semantic_cache_key_policy.py tests/test_proxy_response_cache_partition.py \
    tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \
    tests/test_openai_response_cache_key.py tests/test_proxy_openai_cache_key_integration.py -q
94 passed, 1 warning in 28.39s

$ uv run pytest <the four policy/partition files> -q \
    --cov=headroom.proxy.semantic_cache_key_policy --cov-report=term-missing --cov-branch
Name                                          Stmts   Miss Branch BrPart  Cover
headroom\proxy\semantic_cache_key_policy.py      54      0     16      0   100%
```

$ uv run pytest tests/test_cache/test_semantic.py
tests/test_openai_response_cache_key.py \
tests/test_proxy_openai_cache_key_integration.py
tests/test_proxy_response_cache_store_guard.py \
tests/test_proxy_semantic_cache_key.py
tests/test_proxy_semantic_cache_key_integration.py \
    tests/test_shared_context.py -q
133 passed, 1 warning in 34.11s

$ uv run pytest tests/test_cache_control_move_bust.py
tests/test_anthropic_fork_lineage_cache.py \
tests/test_cross_turn_cache_safety.py
tests/test_cache/test_prefix_tracker.py -q
113 passed, 1 warning in 6.21s

$ uv run ruff check headroom/proxy/semantic_cache_key_policy.py
tests/test_semantic_cache_literal_payloads.py
All checks passed!
$ uv run ruff format --check <both files>
1 file already formatted
```

## Real Behavior Proof

- Environment: Windows 11, `uv run` against the project venv (Python 3.13.11), branch `fix/4272-literal-payload-cache-keys` cut from upstream `main` `e995eb9f`. No mocked cache and no provider: the reproducer calls the real `SemanticCache.set`/`get` in `headroom/proxy/semantic_cache.py`.
- Exact command / steps: run the reproducer above as a script under `uv run python` — store a response under a tool input containing `{"cache_control": "first"}`, deep-copy the messages, change that value to `"second"`, then `get` with the changed messages; repeat for a tool `input_schema` `default` and a `response_format` schema `const`.
- Observed result: before the fix the changed request hit the cache in both cases (`changed tool payload replayed: True`, `changed schema replayed: True`). After the fix both print `False`, and the original request still hits (asserted in each regression test), so the key was not merely widened.
- Not tested: the real OpenAI HTTP path with a decoded-dict `arguments` field, which is a nonstandard client encoding (`arguments` is normally a JSON string, covered by `test_openai_arguments_string_is_not_rewritten`); the full `make ci-precheck` (Rust/native rebuild) and the end-to-end proxy HTTP path; the change is confined to the pure key policy, which the tests above exercise directly through the real `SemanticCache`. `headroom/cache/prefix_tracker.py` has the same blunt recursive strip, but it serves prefix-content equality rather than response replay and is left as a separate follow-up.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the proxy response cache (`cache_enabled`) is not a rollout-gated feature.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, deliberately — for requests whose tool data or JSON Schema values contain a `cache_control`-named field, two distinct requests that used to share one key now get distinct keys. Requests without such a field produce the same key as before, so hit rate there is unchanged.
- Kill switch / disable path: `cache_enabled = false`, or the existing proxy config for the response cache; no new switch.
- Unsafe override required: no
- Qualification impact: none; no rollout-managed surface touched.
- Rollback path: revert this commit. Cache keys are process-local (`SemanticCache` is in-memory), so a revert needs no cache migration or invalidation.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — not needed: the key policy's docstrings describe the new rule, and no user-facing docs claim `cache_control` is stripped from payloads.
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

`headroom/cache/prefix_tracker.py:165` has the same blunt recursive strip, used for prefix-content equality rather than response replay. Worth a follow-up, deliberately out of scope here so the response-replay fix stays a one-module change.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:01:23 -06:00
ee6cfcccf6 fix(security): partition the response cache by credential and principal (#3862)
## Why

On a proxy shared by several callers, the semantic response cache was
keyed only by request content and upstream (#3349). A second caller who
sent the identical request with a different provider key was served the
completion generated for the first caller, under the first caller's key,
and never reached the upstream. That discloses completions across
tenants (finding 01-F15).

## What changes

- `compute_cache_partition()` in
`headroom/proxy/semantic_cache_key_policy.py`: an HMAC-SHA256 under a
per-process random key over the caller's credential and account-selector
headers plus the authenticated principal. Headers counted: every header
the shared credential-header rule
(`internal_header_policy.is_credential_header`) matches, which covers
`authorization`, `proxy-authorization`, `cookie` and `x-api-key`, and
any header whose name ends in `api-key`, `key`, `token`, `secret`,
`account`, `account-id`, `organization` or `project` (so `x-api-key`,
`api-key`, `x-goog-api-key`, `chatgpt-account-id`,
`openai-organization`, `openai-project` and so on). Matching by name
shape keeps new providers partitioned by default. Cookie is used as a
whole value; it is only ever HMAC input. `x-headroom-*` headers
(including the proxy's own token, which the security gate has already
removed) and idempotency keys are excluded. A request with no credential
and no principal gets the shared `anon` partition.
- The partition is an HMAC, not a plain hash, so it cannot be used to
confirm a guessed credential.
- `identity.resolve_authenticated_principal()` returns a principal only
when an identity resolver is installed (`set_identity_resolver`). The
OSS default identity is the same for every network caller and can be
chosen by header on loopback, so it is not treated as a principal.
- If an installed identity resolver raises or returns no principal, the
request fails closed (`resolve_authenticated_principal()` raises
`UnresolvedPrincipalError` for an empty result):
`compute_request_cache_partition()` logs a warning and returns `None`,
and both handlers skip the response cache for that request (no lookup,
no store). It never falls back to the credential-only partition, which
tenants on one operator key would share. With no resolver installed, the
credential-only partition is unchanged.
- Both handlers compute the partition only when the response cache can
be used (cache enabled and non-streaming), so streaming and
cache-disabled requests do not call the identity resolver.
- `partition` is a required keyword on `SemanticCache.get`, `set` and
`_compute_key`, and on `compute_semantic_cache_key`. A future call site
that forgets it fails immediately instead of silently sharing.
- The `/v1/messages` and `/v1/chat/completions` handlers compute the
partition once, next to the existing cache-key snapshot, and reuse it at
store time so lookup and store agree.

Other caches in the request path are not changed. The OpenAI Responses
unit cache and the CCR retrieval store are content-addressed, so a hit
requires already holding the content. CCR entries are not bound to a
caller; that is a separate issue. The `/v1/compress` pipeline cache
holds no user content, and the SDK `SemanticCache` is in-process for one
caller.

## Behaviour change

The proxy response cache no longer shares entries between callers that
present different provider credentials or, with an identity resolver
installed, different principals.

- One developer with one key: same hit rate.
- Deployments where callers send no provider credential and the proxy
supplies the operator's key: same hit rate; they share the `anon`
partition.
- Shared proxy with per-caller keys: entries are no longer shared
between keys. That is the fix.
- Compatible upstream gateways that authenticate by `Cookie` or
`Proxy-Authorization`: entries are no longer shared between sessions.
- Callers whose only credential is a proxy-token `Authorization` header,
or deployments using oauth2, share one partition unless an identity
resolver is installed, because core cannot tell them apart.

## Tests

`tests/test_proxy_response_cache_partition.py`:
- The partition function: differs per credential and is stable for one
credential; every credential and account header shape partitions; other
headers do not fragment the cache; no credential and no principal gives
the shared partition; it is keyed, not a plain hash; a principal
separates callers sharing one credential.
- `SemanticCache` refuses a call without a partition and keeps
partitions apart.
- End to end through the real Anthropic and OpenAI handlers: a second
caller is never served the first caller's cached response, whether the
credential is `x-api-key`/`Authorization`, `Cookie` or
`Proxy-Authorization`. The `Cookie` and `Proxy-Authorization` cases fail
without the latest change. With an identity resolver installed, callers
on one operator key are partitioned. These end-to-end tests fail on
`main`.
- A resolver that raises, or returns an empty principal, yields no
partition, and end to end (both handlers) callers on one operator key
never share a cached response and nothing is stored. These fail without
the fail-closed change. With no resolver installed, the partition is the
credential-only one.
- With the response cache disabled, neither handler calls the identity
resolver.

Existing cache tests now pass a fixed partition. The buffered-stream CCR
test derives the same partition the handler computes.

Manual run: two callers send the identical `/v1/messages` request with
different `x-api-key` values through a proxy in front of a capture
upstream. On `main` the second caller gets the first caller's answer and
only the first key reaches the upstream. On this branch each caller gets
its own answer, and a repeat by the first caller is still a cache hit.

Not tested: multi-worker deployments (the cache and its key are per
process, as before).

## Merge order / conflicts

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863.

Textual conflict: #3860 adds an import next to the one this PR adds in
`handlers/anthropic.py` and `handlers/openai.py`. Whichever merges
second keeps both imports.

Revised 2026-10-04 after review: forwarded `Cookie` and
`Proxy-Authorization` headers are now partition input (via the shared
credential-header rule), so cookie-authenticated callers no longer share
the anonymous partition.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3862"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 15:19:08 -05:00
C0sm0catsandJerrett Davis b84c4c9f71 fix(cache): preserve cache_control schema properties (#3348)
## Description

Preserve legitimate JSON Schema properties named `cache_control` when
building semantic response-cache keys. The normalizer now strips only
Anthropic prompt-cache annotations (`{"type": "ephemeral"}` with
optional `ttl`) instead of every key with that name.

Closes #3345

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Narrow `cache_control` removal to the documented prompt-cache
annotation shape.
- Keep both semantic-cache policy modules synchronized.
- Add regressions proving user-defined schema properties remain in cache
identity.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest -q tests/test_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key_integration.py
42 passed, 1 warning in 1.74s

$ .venv/bin/ruff check <changed files>
All checks passed!

$ .venv/bin/ruff format --check <changed files>
4 files already formatted

$ .venv/bin/mypy headroom/proxy/semantic_cache_key.py headroom/proxy/semantic_cache_key_policy.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Linux, Python 3.14, current `main` plus this patch,
in-memory `SemanticCache`.
- Exact command / steps: instantiated production `SemanticCache`,
computed keys for otherwise-identical tool schemas with and without a
boolean property named `cache_control`, and compared the keys.
- Observed result: the two production cache keys are distinct, so the
schemas no longer collide.

```text
collision=False
keys_distinct=True
```

- Not tested: live Anthropic API traffic; full repository test suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): Semantic response cache key normalization.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Yes, only for tool/schema payloads
containing a non-annotation `cache_control` key.
- Kill switch / disable path: disable semantic caching with
`cache_enabled=False`.
- Unsafe override required: No.
- Qualification impact: cache entries for affected schemas are correctly
partitioned after deployment.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

Documentation is unchanged because this corrects internal cache-key
normalization without changing the public API. The warning above is the
existing Starlette `TestClient` deprecation warning.


## Maintainer refresh (2026-09-16)

Current main removed the unused duplicate `semantic_cache_key.py`; this
refresh preserves that deletion and applies the schema-property fix only
in `semantic_cache_key_policy.py`, which the proxy imports. Both
policy-level and SemanticCache integration regressions remain. All 35
focused cache-key tests pass, and focused Ruff checks pass. Reported
governance checks pass; the local test results are recorded above.

---------

Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
2026-09-24 23:51:28 -05:00
trs-80andClaude Fable 5.1 e9c6276e24 refactor(proxy): delete duplicate semantic_cache_key module (#3472)
## Description

`headroom/proxy/semantic_cache_key.py` (added in #1953) and
`headroom/proxy/semantic_cache_key_policy.py` (added in #1964) were the
same 42-line module: identical ASTs, differing only in two docstrings.
Production `SemanticCache` already imported the `_policy` copy; the
other copy's only consumer was its own test file.

This deletes the copy nothing runs and points its test at the copy that
remains. The retained module is byte-for-byte the one the proxy was
already executing, so no cache key changes.

The module path was never exported from `headroom.proxy` and has no
reference anywhere outside that one test (checked across `headroom/`,
`tests/`, `plugins/`, `sdk/`, `scripts/`, `benchmarks/`, docs, wiki and
CI config), so no compatibility re-export is kept.

Closes nothing; this is the consolidation flagged in the repository
efficiency audit.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache_key.py`: deleted (42 lines).
- `tests/test_proxy_semantic_cache_key_policy.py`: import repointed from
the deleted module to `headroom.proxy.semantic_cache_key_policy`; the
tests themselves are untouched.

Nothing else. In particular `headroom/proxy/semantic_cache.py` is not
touched because it already imported the retained module.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

No new tests: the change deletes a duplicate and repoints one import.
The existing suites for both module paths, the `SemanticCache` key
regression tests and the Anthropic/OpenAI cache-key integration tests
all pass unchanged before and after.

### Test Output

```text
$ pytest tests/test_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key_policy.py \
         tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \
         tests/test_proxy_openai_cache_key_integration.py tests/test_openai_response_cache_key.py \
         tests/test_cache/test_semantic.py -q
67 passed, 1 warning in 4.20s        # identical count on the unchanged base: 67 passed

$ pytest --collect-only -q tests
12549 tests collected in 16.14s     # no module fails to import after the deletion

$ pre-commit run --files tests/test_proxy_semantic_cache_key_policy.py headroom/proxy/semantic_cache.py
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 venv, worktree based on `main` @
`e67b3c8a`.
- Exact command / steps: `git diff --no-index` of the two modules on
`main` (only the two docstring lines differ); `grep -rn
"semantic_cache_key" .` before and after the deletion to enumerate every
reference; the pytest runs above; `compute_semantic_cache_key` evaluated
for representative requests (plain messages, `cache_control` on content
blocks, `system`/`tools`/`temperature` kwargs, a tool schema with a
property named `cache_control`) through both the retained module and the
deleted copy loaded from `main`, with identical hashes.
- Observed result: every semantic-cache key is unchanged; the only
import of the deleted path was the one test file, now repointed; full
test collection succeeds.
- Not tested: no live upstream provider call; no production traffic.
Neither exercises anything this change touches, since the executed code
is identical.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. The semantic response cache key
policy is not rollout-gated.
- Minimum rollout channel: N/A. No rollout-managed feature is touched.
- Stable/default behavior changed: No. The proxy executes the same
module it did before; cache keys, `cache_control` stripping and the
`SemanticCache._compute_key` wrapper are unchanged.
- Kill switch / disable path: N/A. No flag. Reverting the commit
restores the duplicate file.
- Unsafe override required: No.
- Qualification impact: None. No compression decision, cache hit/miss
behavior, CCR, TTL or persistence path is affected.
- Rollback path: `git revert` of this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable. No user-visible surface changes.

## Additional Notes

- Maintainer context on #3348 (preserve `cache_control` schema
properties; open, approved by a collaborator, blocked on required
checks): that PR currently patches both copies. A `git merge-tree` of
the two heads shows exactly one modify/delete conflict on the deleted
file and a clean auto-merge of the test file. Whichever lands first, the
other rebases trivially. If this lands first, #3348 drops its hunk for
the deleted file and its `_policy` and test hunks apply as-is. If #3348
lands first, this PR re-deletes the file and nothing else changes. This
PR deliberately does not absorb #3348's fix, to keep the two reviews
independent.
- Two test files still cover the one remaining module
(`tests/test_semantic_cache_key_policy.py` and
`tests/test_proxy_semantic_cache_key_policy.py`, with three
near-duplicate cases between them). They are left as-is here so #3348's
new test applies cleanly; merging them is a candidate follow-up once
#3348 settles.
- No documentation references the deleted module path, so there is
nothing to update.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 20:41:57 -07:00
Abhay Singh 2cae0f8eaf fix(proxy/cache): strip cache_control from messages in the semantic cache key (#3086)
## Description

The proxy semantic response-cache key (`compute_semantic_cache_key`)
strips `cache_control` from the response-shaping fields (`system`,
`tools`, ...) so that a moved prompt-cache breakpoint does not fragment
the key:

```python
{
    "model": model,
    "messages": messages,  # hashed verbatim
    **{k: strip_cache_control(v) for k, v in key_fields.items()},  # stripped
}
```

But `messages` was hashed **verbatim**. Messages are the primary key
component, and on the Anthropic path they are the most common place a
client (e.g. Claude Code) places and *moves* a `cache_control`
breakpoint between turns (on the last user turn / a `tool_result`
block). So two otherwise-identical requests that differed only in a
message-level breakpoint produced different keys and missed the semantic
cache — the exact fragmentation the `strip_cache_control` helper exists
to prevent, applied to everything except the field that matters most.

The existing tests pin the strip for `system`
(`test_cache_control_breakpoint_move_same_key`) and `tools`
(`test_tools_cache_control_ignored`), but never covered a message-level
breakpoint, so the gap went unnoticed.

## Fix

Apply `strip_cache_control` to `messages` as well. `cache_control` is a
prompt-caching directive for the upstream provider that never changes
the generated completion, so removing the annotation before hashing is
sound: message *content* still differentiates the key, and two requests
that differ only in a `cache_control` breakpoint now share the cache
entry (whose stored response body is identical either way).

The proxy carries two in-sync copies of this pure policy
(`semantic_cache_key_policy.py`, imported by the runtime
`SemanticCache`, and `semantic_cache_key.py`, imported by the policy
test); both are updated identically so they do not diverge.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache_key_policy.py` and
`headroom/proxy/semantic_cache_key.py`: hash
`strip_cache_control(messages)` instead of `messages`, with a docstring
explaining why message-level breakpoints must not fragment the key.
- `tests/test_proxy_semantic_cache_key.py`: added
`test_message_cache_control_breakpoint_move_same_key` (behavioral,
through `SemanticCache._compute_key`) and
`test_message_content_change_still_distinct_key` (guards that stripping
does not collapse genuinely different messages).
- `tests/test_proxy_semantic_cache_key_policy.py`: added
`test_semantic_cache_key_ignores_moved_message_cache_control` at the
pure-policy level.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added

### Test Output

```text
tests/test_proxy_semantic_cache_key.py + tests/test_proxy_semantic_cache_key_policy.py  33 passed
# uvx ruff@0.15.22 check  -> All checks passed!
# uvx mypy@1.20.2 (both policy modules) -> Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted the two policy modules and ran the new
tests to capture the bug (`python -m pytest
tests/test_proxy_semantic_cache_key.py::test_message_cache_control_breakpoint_move_same_key
tests/test_proxy_semantic_cache_key_policy.py::test_semantic_cache_key_ignores_moved_message_cache_control`
-> both failed with two distinct SHA-256 keys for messages that differ
only in a `cache_control` breakpoint); restored the fix; re-ran both key
suites (`python -m pytest tests/test_proxy_semantic_cache_key.py
tests/test_proxy_semantic_cache_key_policy.py` -> 33 passed); ran the
wider `tests/test_cache/` suite and confirmed the only failures
(`test_client_integration.py`) reproduce identically on clean `main` and
are unrelated to this change; then `uvx ruff@0.15.22 format`, `uvx
ruff@0.15.22 check`, and `uvx mypy@1.20.2` on both modules.
- Observed result: before the fix, a request whose last message carries
`cache_control: {type: ephemeral}` hashes to a different key than the
same request without it; after the fix they hash identically (a cache
hit), while messages with different text still hash differently.
- Not tested: a live multi-turn proxy session measuring the hit-rate
improvement (the key contract is verified directly through
`SemanticCache._compute_key` and the pure policy, which is what the
runtime calls).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is the pure semantic-cache key
policy behind `SemanticCache`, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: yes, as a bug fix. Requests that
differ only in a message-level `cache_control` breakpoint now share a
semantic-cache key (a hit) instead of missing. No request that differs
in message content, model, or any shaping field changes key. Because the
cache key changes shape, any entries stored under the old (un-stripped)
keys are simply not reused and age out under the existing TTL/LRU — a
one-time cold start for the affected entries, never a wrong response.
- Kill switch / disable path: the semantic cache itself is already gated
by the existing cache-enable configuration; disabling it bypasses this
path entirely.
- Unsafe override required: no.
- Qualification impact: higher semantic-cache hit rate on the Anthropic
path where clients move `cache_control` breakpoints between turns; no
change to which distinct requests are considered equal beyond ignoring
the caching directive.
- Rollback path: revert this PR; the key returns to hashing messages
verbatim.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

Same class as the `system`/`tools` breakpoint handling already in place
(issue #327 kept the strip from fragmenting the key on a hit); this
extends it to messages, the primary key component. The two in-sync
policy copies are updated together to avoid divergence; consolidating
them into one module is left out of scope for this bug fix.
2026-08-17 20:21:17 -07:00
JD Davis 740fb9bc16 refactor(cache): isolate semantic key policy (#1953)
## Description

Extracts proxy semantic-cache key normalization and hashing into a pure
policy module while preserving `SemanticCache._compute_key` for existing
callers and tests. This separates deterministic cache-key construction
from the async cache adapter and LRU storage concerns.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Added `headroom.proxy.semantic_cache_key` for pure cache-control
stripping and semantic cache key construction.
- Updated `SemanticCache._compute_key` to delegate to the extracted
policy while preserving the local `_strip_cache_control` compatibility
alias.
- Added direct tests for the extracted semantic-cache key policy.
- Kept the LiteLLM callback compatibility shim required for repo-wide
type checking on fresh branches.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
python -m pytest tests/test_proxy_semantic_cache_key_policy.py tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py tests/test_proxy_openai_cache_key_integration.py tests/test_litellm_callback.py tests/test_compress_api.py::TestLiteLLMCallback -q
46 passed in 11.83s

python -m ruff check .
All checks passed!

python -m ruff format --check .
1095 files already formatted

python -m mypy headroom --ignore-missing-imports
Success: no issues found in 409 source files

gitleaks protect --staged --no-banner --redact
no leaks found
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.13, local worktree based on
`headroomlabs/main`.
- Exact command / steps: ran focused semantic cache key tests, handler
cache-key integration tests, LiteLLM callback tests, Ruff lint/format
checks, mypy over `headroom`, and staged gitleaks scan.
- Observed result: all local checks passed; staged secret scan found no
leaks.
- Not tested: full CI matrix and deployment flows; those are covered by
GitHub Actions.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation and changelog updates are not applicable for this internal
refactor. GitHub reported existing Dependabot alerts on the default
branch during push; this PR does not change dependencies, and the staged
secret scan is clean.
2026-07-10 23:49:27 -05:00