Commit Graph
3 Commits
Author SHA1 Message Date
fba09f674c fix(cache): count a tool_result with no text as replayable in the frozen prefix (#4523)
## Description

In Claude Code sessions that load deferred tools, the Anthropic
token-mode pipeline freezes only the messages before the first
ToolSearch result, however far the provider's cache has moved. The log
shows `Pipeline: freezing first 11/166 messages`, and the count stays
there for the rest of the session. Every turn then re-runs compression
over the whole history after that point, only for `finalize_turn` to
overwrite the result with its replay of the provider-confirmed prefix.
Read maturation also trusts that freeze, so it matures Reads the
provider has already cached and the cache re-writes from the Read onward
(#4522).

Root cause: `CompressionCache.compute_frozen_count`
(`headroom/cache/compression_cache.py:293`) stops at a tool_result whose
extracted text is None (`compression_cache.py:323`, "treat as
unstable"). `_extract_text_from_blocks` returns None for a block list
without text (`compression_cache.py:52`): a ToolSearch `tool_reference`
list, or an image. Nothing ever stores such a result or marks it stable,
because `update_from_result` and `mark_stable_from_messages` skip None
too. So it stays a miss for the whole session, and `prepare_turn`'s
`min(tracker_frozen, cache_count)`
(`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to
it.

With this change, a tool_result message with no text in any of its
tool_results counts as stable: the cache has nothing to replay and the
text pipeline has nothing to compress. A message that mixes a textless
first result with a text one still stops the count. The cache keys a
message by its first tool_result, so that text was never stored and
cannot be byte-replayed; fixing that means keying every tool_result,
which belongs in its own change.

What it changes: in such sessions the pipeline freeze follows the
provider-confirmed prefix again, so the pipeline compresses only the new
turn instead of re-running the whole history each turn. Forwarded bytes
inside the confirmed prefix are unchanged, because `finalize_turn`
already replays them. Compression of new content is unchanged.

Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token
mode):
- 299 of the 484 Anthropic requests in its current log froze more than
12 messages below the request size.
- In a sample of that proxy's feed, every long session stopped at a
ToolSearch result. Those holding only `tool_reference` blocks (11/139,
11/166, 11/173) are fixed here. One whose ToolSearch result shares a
message with a text result (13/566 to 13/573) is the mixed case this PR
leaves alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a
tool_result message with no text as stable, through a new
`_carries_tool_result_text` helper (both formats). It still stops at
text it cannot replay, including text beside a textless first result.
- `tests/test_compression_cache.py`: `tool_reference`, image and empty
results no longer stop the count, and a text result beside a textless
one still does.
- `tests/test_session_engine.py`: `prepare_turn` follows the tracker
past a ToolSearch result. The new tests fail on main.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q
90 passed in 4.28s
# without the change, the 4 new tests fail

$ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main>
16 failed, 6 passed
# the same 16 fail on main (f10725e6) on this machine: tests/test_proxy_uds.py (11),
# tests/test_cli/test_wrap_persistent.py (4), test_image_compression.py::TestOnnxRouter::test_full_classify_with_image
# The full suite was not run locally on this branch; CI runs it.

$ ruff check . && ruff format --check .
All checks passed!
1834 files already formatted

$ mypy headroom --ignore-missing-imports
Found 2 errors in 2 files (checked 578 source files)
# the same 2 as main, in files this PR does not touch:
# integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389
```

## Real Behavior Proof

- Environment: macOS 27 on an M2; this branch and upstream main
(f10725e6), each from its own worktree; `headroom proxy` started with
Extra Headroom's flags and env (token mode, read maturation on via the
beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated
HOME, telemetry off), in front of a local mock Anthropic upstream that
reports cache_read for the longest message prefix shared with an earlier
request and cache_write for the rest.
- Exact command / steps: one scripted Claude Code session per build: a
ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB
file, then 8 Bash turns, each a 300-entry JSON listing that the router
compresses. The pipeline freeze per request comes from the `Pipeline:
freezing first N/M` log lines. Each forwarded request is diffed against
the previous one with `cache_control` stripped.
- Observed result: the freeze follows the confirmed prefix instead of
staying at the ToolSearch result, and on the last turn the router
processes 1 content-block message instead of 17.

| Build | Pipeline freeze per request | Router content-block msgs, last
turn | Cached prefix changed | cache_read/forwarded | Cache writes |
tok_saved/tok_before |
  |---|---|---|---|---|---|---|
| main | 2, 2, 2, 2, 2, 2, 2, 2, 2 | 17 | request 8, message 4 (a Read
matured) | 67.0% | 39,360 | 43.2% |
| this PR | 2, 4, 4, 8, 10, 12, 14, 16, 18 | 1 | never | 79.5% | 25,994
| 42.2% |
- Not tested: Linux and Windows; real Anthropic traffic on this branch;
the OpenAI token path and `/v1/compress`, which use the replayable
policy, where a larger count only freezes more and cannot bust (their
suites pass); a session where nothing compresses, where the freeze moves
on only as far as the first uncompressed text result (2 to 4 in the same
session with short Bash output), which is why #4522 is needed as well.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a, not rollout-managed
- Stable/default behavior changed: yes; in token mode the Anthropic
pipeline now freezes past a tool_result with no text, up to the
provider-confirmed prefix, instead of stopping there for the session.
- Kill switch / disable path: none; revert the commit
- Unsafe override required: no
- Qualification impact: none expected; forwarded bytes inside the
confirmed prefix were already replayed by finalize_turn
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Related: #4522 is the fix for the maturation rewrite and works on its
own; this PR removes one of the two reasons the freeze sticks. The other
is a text tool_result that never went through a pipeline run that
changed something: `update_from_result` only marks unchanged results
stable on such a run (`headroom/proxy/handlers/anthropic.py`, `if
result.messages != working_messages`). That is a broader change, left
for its own PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4523"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-28 23:45:04 -05:00
gglucassandClaude Fable 5 aebe98954e fix(cache): keep cached-prefix replay after background recompression (#3380)
## Description

Since the non-inflation size bound added in #3052 (shipped 0.36.x), the
Anthropic proxy token path repeatedly busts the provider prompt cache:
the bound declines the cached-prefix replay exactly when background
compression lands a SMALLER form of an already-forwarded message, so the
forwarded bytes change mid-history and the provider cache misses from
the first changed byte. Steady state on a busy session is a
160-210k-token cache re-write every 1-2 requests (measured across a
0.35.0 -> 0.37.0 swap on identical workload: dollar cache-hit rate 90%
-> 52%, billable input $/M up 2.2x; full data in #3379).

Reworked per review (thanks @chopratejas - the direction is yours):
rather than disabling the size bound wholesale, `overlay_cached_prefix`
now takes `confirmed_frozen_count` and splits at that floor.

- Inside the provider-confirmed prefix: replay is unconditional. The
replay source is exactly what the provider hashed, so changing those
bytes can only bust the cache; an "inflating" replay there bills at the
~0.1x cache-read rate and is still the cheaper request.
- Beyond the floor: the size bound arbitrates each turn exactly as on
main - a shrinking replay repairs freeze drift (#1850), an inflating one
is declined so a fresh compression improvement reaches the wire.
- When the provider count collapses (cold cache, TTL lapse) the floor
collapses with it and every accumulated improvement lands at once: the
natural re-baselining that bounds long-session growth (the #3026
direction of travel) by the TTL-lapse cadence instead of pinning
forever.

The #3052 alignment/append-only guards stay enforced unconditionally.

The OpenAI chat path now carries the same floor (reviewer question: is
the root cause live there too?). It is: the tracker is provider-generic
and is fed OpenAI's own `prompt_tokens_details.cached_tokens`, the
forwarded-bytes feedback loop is identical, both handlers compress
through the same ContentRouter, and OpenAI's automatic prompt caching
busts on the first changed byte exactly like Anthropic's. Exposure is
narrower - the token path's `FREEZE_POLICY_REPLAYABLE` already stops
most recompression of already-forwarded messages - but not zero: an
entry evicted from the local byte-replay cache drops out of that freeze,
and cache mode freezes from the tracker only. Remaining callers with no
provider-confirmed count in scope (sidecar, `openai.py:9940`) pass
nothing and keep today's fully size-bounded behavior; `bedrock.py` /
`gemini.py` never call `finalize_turn` at all, tracked in #3391.

Closes #3379

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `overlay_cached_prefix` gains keyword-only `confirmed_frozen_count:
int | None = None`: an unconditional-replay floor for the
provider-confirmed prefix; the size bound still governs everything
beyond it, and alignment guards are untouched.
- `finalize_turn` forwards the floor (default `None`, so all existing
callers are unchanged).
- The OpenAI chat call site passes the same floor, snapshotted before
`prepare_turn`'s replayable clamp (that count runs over the LOCAL
byte-replay cache and can exceed what the provider confirmed; a cold
prefix zeroes it, so cold re-baselining matches Anthropic's).
- The Anthropic confirmed-clamp call site stashes the pre-clamp tracker
count (the post-`prepare_turn` min under-floors when the local
byte-replay cache evicted entries) and passes it; the stash defaults to
0 on paths that never compute a tracker count (backpressure).
- Tests: floor replays a recompressed confirmed prefix; improvement
beyond the floor lands while the confirmed region replays;
originals-drift beyond the floor is still repaired when the replay
shrinks; alignment guards unaffected by the floor; block-append within
the floor; `finalize_turn` pass-through; the reviewer-requested
multi-turn harness test - a background improvement to an
already-forwarded message never busts the warm cache across 8 turns and
lands on the first cold turn; the reviewer-requested growth bound -
forwarded size grows by only its own delta each warm turn and the first
cold turn re-baselines to exactly the fresh pipeline's output; and an
end-to-end OpenAI chat test asserting the provider-confirmed prefix is
replayed even when the replay inflates (fails without the new argument).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py -q
============================== 58 passed in 0.44s ==============================
$ uvx ruff check headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py headroom/proxy/handlers/anthropic.py tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py
All checks passed!
$ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.12; regression measured on the
released headroom-ai 0.37.0 under real Claude Code traffic through
`headroom proxy` (token mode); this branch validated with the repo test
suite.
- Exact command / steps: diffed per-request PERF logs across a 0.35.0 ->
0.37.0 swap on the same machine and workload (sequences and aggregates
in #3379); then ran the pytest/ruff/mypy commands above on this branch.
A deployment carrying the unconditional-replay behavior on the confirmed
prefix restored the healthy pattern live: the broken wheel capped cached
reads at 157k across 32 large requests (prefix perpetually re-written)
while the patched deployment accumulated to 305k cached reads within 15
minutes.
- Observed result: 58 passed including the new multi-turn harness test;
on stock 0.37.0 every background-compression completion produced a
cache_read collapse to the ~7.7k tools+system head plus a 160-210k
cache_write.
- Not tested: a live OpenAI `cached_tokens` comparison for the OpenAI
half (the Anthropic measurement is what #3379 carries; the OpenAI change
is argued from shared mechanism plus an end-to-end handler test, not
from its own dollar data). A live provider run of this exact branch (the
wheel comparison plus the live patched-deployment measurement isolate
the bound as the trigger; the unit and multi-turn regressions reproduce
decline and fix deterministically). `mypy headroom` was run only on the
two touched modules, not the whole package.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: On the Anthropic proxy
confirmed-clamp path and the OpenAI chat path, both of which now replay
the provider-confirmed prefix unconditionally (the pre-0.36 posture)
while keeping the size bound beyond the confirmed floor. The overlay's
default and every other caller keep the fully size-bounded behavior.
- Kill switch / disable path: remove the `confirmed_frozen_count=`
argument at the two call sites (anthropic.py, openai.py chat); the
parameter defaults to `None`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert the commits; the parameter is additive and
default-preserving.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Lineage for review: #1850 made freeze forward the cached prefix
byte-identical, #3052 added alignment guards plus the size bound after
#3026's misalignment blowup, #3271 unified both modes on the session
engine. The first cut of this PR disabled the bound on the
confirmed-clamp path wholesale; the reviewed rework scopes unconditional
replay to the provider-confirmed prefix, which additionally gives
cold-lapse re-baselining (improvements land whenever the floor
collapses), addressing the growth analysis raised in review. Economics
discussion in the review thread.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-02 16:50:58 +05:30
Tejas ChopraandClaude Fable 5 d12ea50122 feat(proxy): unify proxy and sidecar compression on one session engine (#3271)
> Replaces #3263 (same changeset, squashed to one conventional commit —
after the base PRs squash-merged, the stacked branch's commit history
could not pass the commitlint gate against main, and force-pushing the
original branch was not permitted). #3261 and #3270 (which replaced
#3262) are merged; this is the last piece of the stack.

## Goal

One brain. The cache-management tier — freeze computation, Zone-1 byte
swap, cached-prefix overlay — previously existed twice: inline in the
proxy request handlers, and (as of #3270) in the `/v1/compress` sidecar
path. This PR extracts it into **`headroom/proxy/session_engine.py`**,
invoked by BOTH. Every future cache-management fix lands in both modes
by construction.

## Design

**`prepare_turn(...)` → `TurnPrep`** — freeze +
`mark_stable_from_messages` + `apply_cached`, with two *deliberately
different, documented* freeze policies:
- `FREEZE_POLICY_CONFIRMED_CLAMP`: `min(tracker_frozen, cache_count)` —
never freeze past provider-confirmed (the #327 posture). The Anthropic
proxy passes its already-composed tracker/strict-override value,
reproducing the previous `min()` byte-for-byte.
- `FREEZE_POLICY_REPLAYABLE`: `max(cache_count, explicit)` — freeze
everything locally replayable, because whatever was previously returned
*is* the provider's cache contract; recompressing it (even "better")
busts.

**`finalize_turn(...)` → `TurnFinal`** — the byte-identical
cached-prefix replay (`overlay_cached_prefix`) + conditional token
recount hook.

Run as a **strictly behavior-preserving extraction**: the bar was every
pre-existing test passing *unmodified*, and it held.

## What migrated

| Path | Status |
|---|---|
| `/v1/compress` sidecar turn | ✅ engine (REPLAYABLE); lock, executor
offload, savings accounting, record_returned unchanged |
| `anthropic.py` token-mode pre-block + overlay | ✅ engine
(CONFIRMED_CLAMP); background compression, cold-start fast pass,
`_cold_recompact_active` skip preserved |
| `openai.py` proxy token-mode pre-block + overlay | ✅ engine
(REPLAYABLE — formula-identical to the old bare `compute_frozen_count`);
the added `mark_stable` call means the freeze now survives entry-level
LRU eviction (test-pinned); the router's `_frozen_verdicts` remains the
boundary-message protection |
| `openai.py` cache-mode branch | ⏸ keeps bare `apply_cached` — cache
mode keeps the latest observation mutable by design |

Also fixed for BOTH handlers: overlay replay now runs under backpressure
(shedding it busted every gated session's prompt cache exactly at peak
load), and the inflation guard exempts replayed prefixes.

## Hardening (max-effort review, all applied)

`/v1/usage` applies on the executor under the per-session turn lock with
a timed acquire (503 `session_busy`); registry eviction skips sessions
mid-turn; `peek()` is expiry-aware; silent fallbacks log warnings;
RequestOutcome recorded on session 503s.

## Testing

- `tests/test_session_engine.py`: 13 direct unit tests — both policies,
explicit-pin precedence, REPLAYABLE-without-pin ≡ bare
`compute_frozen_count`, overlay fires/doesn't, recount only on replay,
freeze-survives-entry-eviction.
- Parity bar: full pre-existing suites pass unmodified — cache-stability
(Anthropic + OpenAI), overlay, backpressure (incl.
replay-under-saturation regression), cold-start fast pass, cache-mode,
session-mode byte-stability, compress-API, org-scale, registry. Full
local suite: 11k+ green.
- ruff check/format clean (CI's ruff 0.16.3).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-26 15:57:31 +05:30