Commit Graph
9 Commits
Author SHA1 Message Date
1cf4d053d2 fix(cache): stop Claude Code forks from cold-starting the conversation's lineage (#4043)
### Maintainer qualification refresh (2026-10-09)

Current head: `a91a1ec24ab20b6dcea0f6e7f8d41feb7d0beea0`. The newly
supplied live Claude Code/Anthropic before-after evidence resolves the
previous runtime-proof hold. Clean synchronization with current main
`01d9194` preserves contributor logic. Independently reran all 88
fork-lineage/prefix tests with the existing native extension;
current-head hosted checks are green. Approved for human merge review.
The live capture below remains correctly attributed to the preceding
implementation head, not represented as a new capture of this
synchronization commit. No merge performed.

## Description

Claude Code sends side requests alongside an agent's turns. The subagent
progress summary and the parent session's side query both re-send the
history plus one prompt, with the breakpoint one message earlier, so
they read the prompt cache without writing it. resolve_tracker matched
such a fork as a continuation and replaced the lineage snapshot with it.
The next real turn extends the history before the fork, not the fork,
so it matched no lineage. It started a cold tracker (frozen=0), and the
token-mode pipeline rewrote history the provider had already cached.

Production, 2026-10-08: a review subagent busted Anthropic's cache on
35 of 81 turns. cache_read_input_tokens fell to the 6,065-token system
prompt while 100-150k tokens were re-written. Its 600k-token parent did
the same after its own side queries. In the proxy log, every bust is
the first main request after a fork and shows no frozen prefix.
Fork-free sessions on the same proxy were 96/96 and 130/130 full hits.

A lineage now keeps its last three request snapshots. A request that
extends an earlier one resumes that turn instead of starting cold. Each
response is also recorded against the turn that sent it, through a
request-scoped ContextVar (the same pattern as the tenant key and CCR
store). A fork's response can no longer replace the main line's frozen
prefix or overlay source, whether it lands before or after the
in-flight main turn's response.

Templated fan-out siblings that extend a shared earlier snapshot now
stay on one tracker and each resumes its own turn's state. Splitting
them would mint a new lineage per fork and exhaust the per-session cap.
The fan-out test asserts the per-line state instead of object identity.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: each lineage keeps a
`deque(maxlen=_LINEAGE_HISTORY)` (3) of request snapshots instead of
one. `resolve_tracker` resumes the turn a request extends, and the turn
it resolved is held in a request-scoped `ContextVar` (`_resolved_turn`)
so the response is recorded against that turn.
- `tests/test_anthropic_fork_lineage_cache.py` (new): handler-level
regression. A main turn after a fork (progress summary or side query)
must replay its cached prefix byte for byte, and a fork must not replace
the lineage it branched from, whichever response lands first.
- `tests/test_cache/test_prefix_tracker.py`: the fan-out test asserts
per-line state instead of tracker object identity.

## Testing

- [x] Unit tests pass (`pytest
tests/test_anthropic_fork_lineage_cache.py tests/test_cache/
tests/test_proxy_anthropic_cache_stability.py`: 272 passed, with
`test_cache/test_dynamic_detector.py` run on its own: 57 passed, 2
skipped)
- [x] Linting passes (`ruff check` on the changed files)
- [x] Type checking passes for the changed module (`mypy
headroom/cache/prefix_tracker.py`: no issues)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# new test file, without the fix (parent commit)
FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[summary]
FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[side-query]
FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[fork-first]
FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[main-first]
  AssertionError: main turn 2 re-sent turn 1's history with different bytes
============================== 4 failed ==============================

# with the fix
============================== 4 passed ==============================
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.16, real Claude Code 2.1.295
sessions against the Anthropic API through `headroom proxy --mode token`
(details below).
- Exact command / steps: the scripted Claude Code session described
below, run once on `17abee27` (before) and once on `9b27664a` (after).
- Observed result: main turns right after a fork arrived with a cold
tracker 5/5 before, 0/5 after; cached tokens lost 3,363 before, 0 after.
- Not tested: the full collapse to the system prompt seen in production
fork-heavy sessions (did not reproduce in this short session); a
deployed proxy running this fix.

Live before/after capture on 2026-10-09. Real Claude Code 2.1.295
sessions against the Anthropic API (subscription login,
`claude-sonnet-5-5`), through `headroom proxy --mode token` on Windows
11 / Python 3.13.16, with the native extension built in each tree.

- **Before:** `17abee27` (merge base with `main`)
- **After:** `9b27664a` (this PR)

Both ran the same scripted session: a stream-json SDK client with
`agentProgressSummaries` and `promptSuggestions` on, one general-purpose
subagent that `cat`s ten ~40 KB JSON files one tool call at a time, then
three parent follow-up turns. Each request's kind comes from Claude
Code's own debug log (`[API REQUEST] source=…`). `sdk` and `agent:*` are
main turns; `agent_summary` and `prompt_suggestion` are forks. Token
counts are the upstream `usage` values from the proxy's PERF lines.

| Main turns right after a fork | Before | After |
|---|---|---|
| Count | 5 | 5 |
| Arrived with a cold tracker (no `Pipeline: freezing` line) | **5/5** |
**0/5** |
| Read the full prior prefix | 4/5 | **5/5** |
| Cached tokens lost | 3,363 | 0 |

- **Before**, subagent turn #15 (after its 23-message `agent_summary`
fork): cold, `cache_read` 22,137 against the 25,500 cached by the
previous turn, `cache_creation` 4,902.
- **After**, the matching subagent turn #17 (after a 21-message
`agent_summary` fork): frozen 20/23, `cache_read` 25,311 = 24,192 +
1,119, exactly the previous turn's read + write.

This short session did not reproduce the full collapse to the system
prompt (~6k read) seen in production fork-heavy sessions. In most cold
turns the pipeline happened to regenerate identical bytes. The capture
does show the mechanism deterministically: before, every fork leaves the
next main turn of that conversation starting cold, and one of those cold
turns rewrote 3.4k cached tokens. After, none start cold and every
post-fork turn reads its whole prior prefix.

<details><summary>Per-request tables (before / after)</summary>

**Before (`17abee27`)**

| # | kind | msgs | frozen | cache_read_input_tokens |
cache_creation_input_tokens |
|---|---|---|---|---|---|
| 1 | main (parent) | 2 | - | 0 | 15,072 |
| 2 | main (subagent) | 2 | - | 5,406 | 7,859 |
| 3 | main (subagent) | 5 | 3 | 13,265 | 660 |
| 4 | main (subagent) | 7 | 3 | 13,925 | 1,176 |
| 5 | main (subagent) | 9 | 3 | 15,101 | 1,319 |
| 6 | main (subagent) | 11 | 3 | 16,420 | 1,138 |
| 7 | fork (agent_summary) | 11 | 3 | 16,476 | 0 |
| 8 | main (subagent) <- after fork | 13 | - | 17,558 | 1,127 |
| 9 | main (subagent) | 15 | 3 | 18,685 | 1,134 |
| 10 | main (subagent) | 17 | 3 | 19,819 | 1,126 |
| 11 | main (subagent) | 19 | 3 | 20,945 | 1,136 |
| 12 | main (subagent) | 21 | 3 | 22,081 | 1,133 |
| 13 | main (subagent) | 23 | 3 | 23,214 | 2,286 |
| 14 | fork (agent_summary) | 23 | 22 | 23,321 | 0 |
| 15 | main (subagent) <- after fork | 25 | - | 22,137 | 4,902 |
| 16 | main (subagent) | 27 | 24 | 27,039 | 1,260 |
| 17 | main (subagent) | 29 | 24 | 28,299 | 7,523 |
| 18 | main (parent) | 4 | 3 | 15,072 | 2,204 |
| 19 | fork (prompt_suggestion) | 6 | 5 | 17,276 | 335 |
| 20 | main (parent) <- after fork | 6 | - | 17,611 | 31 |
| 21 | fork (prompt_suggestion) | 8 | 7 | 17,642 | 362 |
| 22 | main (parent) <- after fork | 8 | - | 18,004 | 29 |
| 23 | main (parent) | 10 | 9 | 18,033 | 242 |
| 24 | fork (prompt_suggestion) | 12 | 9 | 18,275 | 128 |
| 25 | main (parent) <- after fork | 12 | - | 18,403 | 18 |
| 26 | fork (prompt_suggestion) | 14 | 9 | 18,421 | 142 |

**After (`9b27664a`)**

| # | kind | msgs | frozen | cache_read_input_tokens |
cache_creation_input_tokens |
|---|---|---|---|---|---|
| 1 | main (parent) | 2 | - | 5,406 | 9,666 |
| 2 | main (subagent) | 2 | - | 10,523 | 2,742 |
| 3 | main (parent) | 4 | 3 | 15,072 | 692 |
| 4 | main (subagent) | 5 | 3 | 13,265 | 1,681 |
| 5 | fork (prompt_suggestion) | 6 | 3 | 15,764 | 25 |
| 6 | main (subagent) | 7 | 3 | 14,946 | 1,246 |
| 7 | main (subagent) | 9 | 3 | 16,192 | 1,138 |
| 8 | main (subagent) | 11 | 3 | 17,330 | 2,303 |
| 9 | main (subagent) | 13 | 3 | 19,633 | 1,167 |
| 10 | main (subagent) | 15 | 3 | 20,800 | 1,137 |
| 11 | main (subagent) | 17 | 3 | 21,937 | 1,129 |
| 12 | main (subagent) | 19 | 3 | 23,066 | 1,126 |
| 13 | main (subagent) | 21 | 20 | 24,192 | 1,119 |
| 14 | main (parent) <- after fork | 6 | 3 | 15,789 | 31 |
| 15 | fork (prompt_suggestion) | 8 | 3 | 15,820 | 108 |
| 16 | fork (agent_summary) | 21 | 20 | 24,248 | 0 |
| 17 | main (subagent) <- after fork | 23 | 20 | 25,311 | 710 |
| 18 | main (parent) <- after fork | 8 | 3 | 15,928 | 851 |
| 19 | fork (prompt_suggestion) | 10 | 3 | 16,779 | 688 |
| 20 | main (parent) <- after fork | 10 | 3 | 17,467 | 29 |
| 21 | main (parent) | 12 | 3 | 17,496 | 130 |
| 22 | main (parent) | 14 | 3 | 17,626 | 249 |
| 23 | fork (prompt_suggestion) | 16 | 3 | 17,875 | 131 |
| 24 | main (parent) <- after fork | 16 | 3 | 18,006 | 18 |
| 25 | fork (prompt_suggestion) | 18 | 3 | 18,024 | 136 |

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, prefix-tracker lineage matching
for requests that extend an earlier snapshot of the same lineage
- Kill switch / disable path: none
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Not covered here: a few small partial misses in production that forks
don't explain, and a synthetic-only risk that an older tool result gets
recompressed once it falls past a lagging frozen boundary.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4043"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-13 16:21:38 -07:00
Mansurshakh JaparovandJD Davis 9553d4292f fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description

Session cleanup currently drops prefix trackers after the default
600-second session lifetime even when Anthropic's request selects a
one-hour cache TTL. Losing the tracker also loses the frozen forwarded
prefix and causes the cache-preserving overlay to replay different
bytes.

Closes #4017.

## Type of Change

- [x] Bug fix
- [x] Documentation update

## Changes Made

- Resolve the request's existing Anthropic cache TTL before selecting
the prefix lineage and pass that TTL through tracker selection/creation.
- Retain each lineage for the greater of the configured session floor
and its observed/current provider TTL plus the existing 60-second safety
margin.
- Keep the lifetime per lineage rather than changing global
configuration: a five-minute session does not inherit another session's
one-hour lifetime.
- Preserve normal expiry beyond the provider lifetime and margin.
- Honor configured cache TTL before a confirmed cache write; discard
expired state before accepting a later request's longer TTL.
- Re-register the live lineage family after lookup/cleanup so recreated
sessions cannot share a tracker's frozen state across sibling
conversations or tool profiles.
- Update existing fake callbacks to the changed tracker contract,
preserve all consumer byte-stability assertions, and remove one
handler-to-store wiring-only test rather than re-pinning its internal
identity assertions.
- Document local tracker retention separately from provider cache
availability.

## Testing

- [x] Unit tests pass (`pytest`), including frozen-prefix byte stability
and independent TTL expiry.
- [x] Changed-file lint, configured pre-commit mypy and manual
production timestamp/overlay checks pass.

### Test Output

```text
Review correction: 273 passed, 2 dependency warnings in 83.79s
{"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true}
```

The mandatory pre-push `make ci-precheck` passed: workspace Rust
fmt/clippy/tests, native extension build/import verification, the
complete configured Python gate, and Commitlint. The native gate
explicitly used the installed CPU `libonnxruntime.so.1.26.0` via
`ORT_DYLIB_PATH`, without skipping cached-model assertions.

```text
[build_rust_extension] headroom._core build + install + verify: OK
================== 177 passed, 4 skipped, 1 warning in 12.87s ==================
✅ ci-precheck PASSED — safe to push.
```

```sh
python -m pytest -q tests/test_cache/test_prefix_tracker.py
python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py
```

**74 passed** in tracker tests; **54 passed** in request/cache tests
(one existing Starlette TestClient deprecation warning). Two new
regressions fail on baseline because the request TTL is not accepted; a
separate production runtime comparison demonstrates the actual pre-fix
loss, not just the missing parameter.

After rebasing onto current main
`1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together
with `tests/test_prefix_tracker_client_rewrote_reply.py`,
`tests/test_cache_mode_delta_marker.py`, and
`tests/test_content_router_mcp_small_result.py`: **150 passed**. The
actual production timestamp/byte-retention smoke was repeated
successfully on the rebased code, covering compatibility with main's
newly merged client-reply normalization.

Review regression and affected HTTP-consumer suite: **273 passed**,
including tracker expiry, Anthropic/OpenAI cache stability,
byte-faithful forwarding, header isolation, buffered timeouts, CCR
injection, and immutable system prompts. All affected two-argument test
callbacks now accept the TTL contract. The obsolete walker-call-order
test was removed; wire-byte assertions remain. The new
configured-TTL/expiry cases fail against the published baseline; the
post-cleanup regression also verifies that an unrelated conversation
gets a different tracker after recreation.

Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and
Commitlint hooks passed. Full-tree Ruff is locally blocked by executable
filesystem permissions on five unchanged OAuth2 plugin files (tracked
100644). A separate full dev-environment mypy run found an unchanged
optional LangChain `truthy-function` error; the configured isolated
pre-commit hook passed. No unrelated source/permissions changed.

## Real Behavior Proof

- Environment: Linux x86_64, CPython 3.13.14; unchanged
`dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real
`SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token
estimator and `overlay_cached_prefix`, no mocked store or mocked clock.
- Exact command / steps: parse `cache_control: {type: ephemeral, ttl:
1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed
frozen message with different original/forwarded prefix bytes; age
`_last_activity` to `time.time()-660` and make periodic cleanup due;
resume the same lineage and compare the tracker, frozen count, and
forwarded-prefix bytes. Also age beyond 3901 seconds and compare
independent 300/3600-second sessions.
- Before: resolved TTL 3600; after 660 seconds, tracker identity
changes, frozen count drops **1→0**, and forwarded-prefix bytes differ.
- Observed result: after the fix, after 660 seconds the same tracker
remains, frozen count stays **1**, and forwarded-prefix bytes are
identical. The one-hour session survives independently, the five-minute
session expires, and the one-hour lineage still expires beyond TTL plus
margin.
- Review runtime proof against published head `e1a107df`: actual stores
and prefix overlays confirm configured TTL retention before the first
cache write, no resurrection of expired state by a later one-hour
request, eventual expiry, and separate trackers after the maintainer's
3901-second cleanup/recreation sequence. On that exact baseline the
unrelated lookup reuses the wrong tracker, although the overlay guard
preserves the particular unrelated input bytes; after the fix both
identity isolation and byte preservation hold. Identical messages under
different tool profiles also receive a cold, separate tracker after the
expired bare-key transition.
- Not tested: Anthropic cloud cache availability, billed savings, or an
actual one-hour wait. The local production state and byte invariants
were exercised by aging actual timestamps, not by a paid upstream
request. Preserving a tracker cannot guarantee that the provider retains
its cache.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; repair of existing TTL-aware
cache-prefix behavior.
- Minimum rollout channel: stable; no rollout-managed feature is
enabled.
- Stable/default behavior changed: eligible local prefix trackers
survive for their provider cache lifetime. Short-TTL/default sessions
remain bounded by their own session floor.
- Kill switch / disable path: the existing prefix-freeze disable path
remains available; no new flag.
- Unsafe override required: no.
- Qualification impact: local Anthropic prefix lineage
selection/retention, not cache pricing or subscription quota policy.
- Rollback path: revert the commit to restore previous retention. No
persistent schema change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] Existing cache TTL parser and safety margin reused; cache docs
updated.
- [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes,
and existing request byte-stability behavior.
- [x] `CHANGELOG.md` not edited; Conventional Commit title supplied.

Remote CI results are not inferred from local verification.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-07 17:46:21 -05:00
16eaec1e95 fix(cache): keep prefix lineage across a replaced system tail (#3934)
## Description


Claude Code can end a `/v1/messages` request with a temporary
`role:"system"` reminder and replace it on the next turn.
`SessionTrackerStore.resolve_tracker` records the full history, so the
stored chain ends in the old reminder and is never a prefix of the next
request: `_classify_history_canonical` returns `diverged` and the
request gets a fresh, empty tracker. The confirmed frozen prefix and the
previously forwarded messages are gone, the #3380 replay has nothing to
replay, and any recompressed history re-writes the provider prompt
cache. The reminder moves every turn, so this repeats for the whole
session.

This change lets lineage resolution set aside that one replaced trailing
system message, only when nothing matched as recorded. It changes
tracker selection only: forwarded bytes, the replay body, and breakpoint
placement are untouched, and the existing replay already stops at the
first changed message, so the new reminder goes out as the client sent
it.

Closes #3933

## Type of Change


- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: new
`_extends_past_system_tail(current, previous)`. It is true when
`previous` ends in a `role:"system"` message, everything before that
message includes user/assistant history and is unchanged in `current`,
and `current` is longer than `previous`. Because `current` must be
longer, siblings that only swap their final instruction stay separate.
- `SessionTrackerStore.resolve_tracker`: after the existing
exact/append, block-append and block-rewrite passes find nothing (and
there are no rewrite candidates, so an ambiguous rewrite tie is not
overridden), Anthropic requests consider chains with the same cache
affinity that pass that check. Only a unique longest candidate is taken;
two equal-length candidates start a fresh lineage, as before. The full
current snapshot is stored as usual.
- `extract_cache_stable_delta` (cache mode's prefix/delta split): a
`diverged` request that replaced the previous trailing system reminder
is split too. Every recorded message before the reminder, plus the model
reply the handlers record after it, must match the current request
position for position (canonical compare, same rule via
`_extends_past_system_tail`). The previously forwarded bytes are
replayed without the old reminder, and everything after it is the delta
to compress. Before this, cache mode tagged these turns
`cache_mode_prefix_mismatch` and forwarded the new messages raw.
- `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns
off both the lineage fallback and the delta split. It uses the same
parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`.
- `tests/test_cache/test_prefix_tracker.py`:
- Tracker retention: the tracker and its confirmed frozen count are
kept, and the existing overlay restores the unchanged prefix.
- A reminder that moves and changes over several turns: full snapshots
are stored.
- Isolation: different cache affinity, OpenAI provider, changed earlier
history, changed leading system message, changed historical system
message, same-length siblings `[user: task, system: A]` vs `[user: task,
system: B]`, ambiguous equal-length siblings, exact-match precedence,
and the kill switch.
- The delta split: with and without the recorded reply, a system message
followed by a client turn is not treated as a reminder, a mismatched
reply is rejected, and the kill switch disables it.
- `tests/test_cache_mode_cold_start.py`: drives
`handle_anthropic_messages` in `mode="cache"` with the real
`SessionTrackerStore` over three turns. The first turn compresses a tool
result and later turns replace the reminder. The test asserts that the
third request replays the second request's compressed prefix
byte-for-byte (ignoring `cache_control`), compresses the new tool
result, and ends in the current reminder.

## Testing

- [x] Unit tests pass (`pytest`) - targeted files below, not the whole
suite
- [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff
format --check`
- [x] Type checking passes (`mypy headroom`) - on
`headroom/cache/prefix_tracker.py`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests with the source at `main` (d5318ac2):

```text
$ python -m pytest tests/test_cache_mode_cold_start.py -q -p no:cacheprovider
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
========================= 1 failed, 2 passed in 1.70s ==========================
```

With the lineage commit only (2c5af818 plus the length fix, before the
delta split), the cache-mode tests fail on the raw new tool result:

```text
E   AssertionError: assert 'verbose tool...ool output 3 ' == 'compressed tool output'
E   TypeError: cannot unpack non-iterable NoneType object
FAILED tests/test_cache_mode_cold_start.py::test_cache_mode_replaced_system_tail_keeps_compressed_prefix
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_cache_delta_replays_prefix_past_replaced_system_tail
========================= 2 failed, 73 passed in 2.83s =========================
```

The same-length sibling test fails at 2c5af818:

```text
FAILED tests/test_cache/test_prefix_tracker.py::TestConversationLineageResolution::test_swapped_final_instruction_is_not_a_continuation
========================= 1 failed, 70 passed in 0.75s =========================
```

With all three commits:

```text
$ python -m pytest tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py \
    tests/test_cache_mode_delta_marker.py tests/test_observed_wire_shapes.py tests/test_proxy_handler_helpers.py \
    tests/test_issue_2671_block_growth_cache.py tests/test_proxy_anthropic_cache_stability.py \
    tests/test_proxy_openai_cache_stability.py -q -p no:cacheprovider
======================= 225 passed, 1 warning in 46.30s ========================

$ ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
All checks passed!
$ ruff format --check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py tests/test_cache_mode_cold_start.py
3 files already formatted
$ mypy headroom/cache/prefix_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 27.0.1, Python 3.11.13, this branch (5b0c20f2) vs
`main` d5318ac2. The proxy was started from each checkout with `python
-m headroom.proxy.server --port <free> --anthropic-api-url
http://127.0.0.1:<stub> --no-rate-limit`, once with
`HEADROOM_MODE=cache` (the default) and once with `HEADROOM_MODE=token`.
The `headroom proxy` CLI refuses to start in this dev venv because the
optional `mcp`, `magika` and `onnxruntime` extras are missing; the
server module itself runs. The upstream is a local stdlib HTTP stub that
records each forwarded body and answers with the next `tool_use`. It
returns Anthropic-shaped `usage` from a simulated prompt cache: the
whole forwarded prompt is cached; the next request reads the leading run
of messages that are byte-identical to the previous forwarded request
(ignoring `cache_control`) and writes the rest. Tokens are approximated
as serialized chars / 3. No real provider was contacted.
- Exact command / steps: `PROOF_MODE=cache|token proof.py <checkout>
<python> <label>` (script below) sends four `claude-fable-5` turns of a
paged tool loop. Each turn adds an assistant `tool_use` and a ~31 KB
JSON `tool_result`, and each request ends in a `role:"system"` reminder
whose text changes every turn. Controls: `-noreminder` sends the same
turns without reminders, and `-mixed` drops only the first turn's
reminder, which is how Claude Code interleaves them.
- Observed result: in cache mode every reminder turn on the branch
replays the previously forwarded history byte-identical and compresses
only the new tool result (simulated read/write per turn: main
10,598/10,581, 21,162/10,581, 31,726/10,580 with nothing compressed;
branch 10,598/3,106, 13,687/3,106, 16,776/3,105); in the mixed run main
re-writes the cache on the first reminder turn (76 read / 31,667 write)
and the branch reads 6,212 / writes 3,106; in token mode main re-writes
the history every reminder turn (read 76/281/486) and the branch reads
3,123/6,212/9,301, matching the no-reminder control.

Simulated cache read / write per turn, and the proxy's `PERF tok_after`
(tokens forwarded):

  | Mode, run | Turn 1 | Turn 2 | Turn 3 | Turn 4 |
  | --- | --- | --- | --- | --- |
| cache, main | 0 / 10,615 (10,886) | 10,598 / 10,581 (21,747) | 21,162
/ 10,581 (32,608) | 31,726 / 10,580 (43,469) |
| cache, branch before delta split (2c5af818) | 0 / 10,615 (10,886) |
10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726 / 10,580
(43,469) |
| cache, branch | 0 / 10,615 (10,886) | 10,598 / 3,106 (15,122) | 13,687
/ 3,106 (19,358) | 16,776 / 3,105 (23,594) |
| cache, branch, `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` | 0 / 10,615
(10,886) | 10,598 / 10,581 (21,747) | 21,162 / 10,581 (32,608) | 31,726
/ 10,580 (43,469) |
| cache, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |
| cache mixed, main | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) | 76 /
31,667 (32,608) | 31,726 / 10,580 (43,469) |
| cache mixed, branch before delta split (2c5af818) | 0 / 3,123 (4,253)
| 3,123 / 3,106 (8,498) | 6,212 / 10,581 (19,359) | 16,776 / 10,580
(30,220) |
| cache mixed, branch | 0 / 3,123 (4,253) | 3,123 / 3,106 (8,498) |
6,212 / 3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main | 0 / 3,140 (4,262) | 76 / 3,271 (4,429) | 281 / 3,271
(4,595) | 486 / 3,270 (4,762) |
| token, branch | 0 / 3,140 (4,262) | 3,123 / 3,106 (8,498) | 6,212 /
3,106 (12,734) | 9,301 / 3,105 (16,970) |
| token, main, no reminders (control) | 0 / 3,123 (4,253) | 3,123 /
3,089 (8,489) | 6,212 / 3,089 (12,725) | 9,301 / 3,088 (16,961) |

  How to read it:
- **Cache mode, main:** for a session that always ends in a reminder,
the cache is stable but nothing is ever compressed. Each turn gets a
fresh tracker, and the cold-start branch freezes the whole request
because its last message is not a user turn.
- **Cache mode, mixed run on main:** the first reminder turn after a
compressed turn re-sends the compressed history raw and re-writes the
cache.
- **Lineage commit alone (2c5af818):** kept the history stable, because
the cached-prefix replay in `finalize_turn` restores it. New tool
results still went out raw (`cache_mode_prefix_mismatch`).
- **With the delta split:** they are compressed (31,572 -> 9,187 bytes
each), and the mixed run matches the no-reminder control.
- **Every forwarded request on the branch** ends in that turn's own
reminder text, and the old reminder is not replayed.
- **Turn 1 of the always-reminder cache run** is forwarded raw on both
main and the branch. A cold request whose last message is a system
message is frozen whole by `_strict_previous_turn_frozen_count`. This PR
does not change that.
- Downstream production evidence (Headroom Desktop on the 0.37.0 wheel,
`claude-fable-5`, token mode, provider-reported `usage`, no request
content): one long Claude Code session went 0 read / 490,996 write
(cold), then 490,996 read / 12,509 write on an 862-message request
ending in the reminder, then 7,259 read / 491,260 write on the next
864-message request, although its first 861 messages were unchanged. 27
such continuations in that window wrote 15,183,730 cache tokens and read
195,993. After the lineage fix was applied downstream, a live run of 13
requests (9 replaced the reminder) read 664,386 and wrote 49,646 cache
tokens (92.99% reuse).
- Not tested: no real Anthropic request or live provider cache billing
for this branch; details below.
- No real Anthropic request was made from this branch, so live provider
cache billing was not measured for this PR. The cache numbers above come
from the stub's simulation.
- The downstream numbers come from a vendored lineage patch on the
0.37.0 wheel in token mode. That patch has the same selection rules but
not this exact code, and it has no cache-mode delta split, which is new
here and has not run against a live provider.
  - Not exercised end to end:
- streaming responses (they record the reply the same way, but were not
driven here)
    - the Bedrock backend path
    - the `headroom proxy` CLI entry point with all extras installed
- models that relocate the reminder into the top-level `system` field.
There the reminder is part of the system segment and this change has no
effect.
- OpenAI paths are unchanged by design and are covered only by their
existing tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is a fix in lineage selection
and in cache mode's prefix/delta split, not a rollout feature in
`headroom/rollout.py`.
- Minimum rollout channel: n/a (applies on every channel).
- Stable/default behavior changed: yes, for Anthropic requests whose
previous request ended in a `role:"system"` message that the client
replaced:
- **Token mode:** they keep their existing prefix tracker, so the
confirmed prefix is replayed byte-stable instead of being recompressed.
That history is therefore no longer recompressed on those turns.
- **Cache mode (default):** the same request is split into the
previously forwarded prefix plus a compressed delta instead of being
forwarded with its new messages raw. Compression on those turns goes up,
cache reads stay at the replayed prefix, and a session that alternates
reminder and non-reminder turns no longer re-writes its cache on the
first reminder turn.
- **Unchanged:** OpenAI, conversations without a trailing system
message, siblings that only swap their final instruction, and every case
where an existing relation already matches.
- Kill switch / disable path: `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (or
`false`/`no`/`off`), read per request. It disables both the lineage
fallback and the cache-mode split. In the proof, the cache-mode run with
it set matches `main` exactly.
- Unsafe override required: no.
- Qualification impact: expect higher provider cache reads and lower
cache writes on Claude Code sessions that send trailing system reminders
in token mode, and more compression of new messages on those turns in
cache mode. This was measured here only against a stub; live provider
billing for this branch was not measured.
- Rollback path: set `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` and restart
the proxy, or revert the three commits. No persisted state or schema
changes.

<details>
<summary>proof.py</summary>

```python
"""Boot the proxy (`python -m headroom.proxy.server`) from a given checkout
against a local stub Anthropic upstream and replay a Claude Code shaped
conversation (paged tool calls, ~31 KB JSON tool results) whose trailing
role:"system" reminder is replaced every turn.

Usage: PROOF_MODE=cache|token proof.py <checkout-dir> <python> <label>
Label suffix "-noreminder" drops the reminders (control); "-mixed" drops only
the first one. Reports, per turn, how many previously forwarded history
messages were forwarded again byte-identical (cache_control ignored), the
newest tool result's size before/after the proxy, and the stub's simulated
prompt-cache read/write. No real provider is contacted: the upstream is
127.0.0.1.
"""

import json
import os
import socket
import subprocess
import sys
import threading
import time
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

checkout, python, label = sys.argv[1], sys.argv[2], sys.argv[3]
received: list[dict] = []
usage: list[tuple[int, int]] = []


def strip(x):
    """Drop cache_control markers: the proxy re-places them every turn and the
    provider does not hash them into the cached prefix."""
    if isinstance(x, dict):
        return {k: strip(v) for k, v in x.items() if k != "cache_control"}
    if isinstance(x, list):
        return [strip(v) for v in x]
    return x


class Stub(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["content-length"])))
        received.append(body)
        # Minimal provider prompt-cache simulation: the whole forwarded prompt
        # is cached (client-style breakpoint on the newest message); this turn
        # reads the leading run of messages byte-identical to the previous
        # forwarded request, and writes the rest. Tokens ~= serialized chars / 3.
        def toks(x):
            return len(json.dumps(strip(x), sort_keys=True)) // 3

        msgs = body["messages"]
        total = toks(body.get("system")) + sum(toks(m) for m in msgs)
        read = 0
        if len(received) > 1 and strip(received[-2].get("system")) == strip(body.get("system")):
            prev = received[-2]["messages"]
            k = 0
            while k < min(len(prev), len(msgs)) and strip(prev[k]) == strip(msgs[k]):
                k += 1
            if k:
                read = toks(body.get("system")) + sum(toks(m) for m in msgs[:k])
        usage.append((read, total - read))
        payload = json.dumps(
            {
                "id": f"msg_{len(received)}",
                "type": "message",
                "role": "assistant",
                "model": body["model"],
                "content": assistant(len(received) + 1)["content"],
                "stop_reason": "tool_use",
                "stop_sequence": None,
                "usage": {
                    "input_tokens": 10,
                    "output_tokens": 2,
                    "cache_creation_input_tokens": total - read,
                    "cache_read_input_tokens": read,
                },
            }
        ).encode()
        self.send_response(200)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args):
        pass


def assistant(k: int) -> dict:
    return {"role": "assistant", "content": [
        {"type": "tool_use", "id": f"toolu_{k}", "name": "list_services", "input": {"page": k}}]}


def tool_result(k: int) -> dict:
    rows = [{"id": i, "name": f"service-{k}-{i}", "status": "ok" if (i + k) % 7 else "degraded",
             "latency_ms": 20 + (i * 37 + k) % 400, "region": ["eu", "us", "ap"][i % 3]}
            for i in range(300)]
    return {"role": "user", "content": [
        {"type": "tool_result", "tool_use_id": f"toolu_{k}", "content": json.dumps(rows)}]}


def free_port() -> int:
    with socket.socket() as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]


stub = ThreadingHTTPServer(("127.0.0.1", free_port()), Stub)
threading.Thread(target=stub.serve_forever, daemon=True).start()
port = free_port()
env = {
    **os.environ,
    "PYTHONPATH": checkout,
    "HEADROOM_BEACON": "off",
    "HEADROOM_MODE": os.environ.get("PROOF_MODE", "token"),
    "LITELLM_LOCAL_MODEL_COST_MAP": "True",
    "HEADROOM_DATA_DIR": os.path.join(os.path.dirname(__file__), f"data-{label}"),
}
env.pop("GROK_CLI_CHAT_PROXY_BASE_URL", None)
log = open(os.path.join(os.path.dirname(__file__), f"proxy-{label}.log"), "w")
proxy = subprocess.Popen(
    [
        python, "-m", "headroom.proxy.server", "--port", str(port),
        "--anthropic-api-url", f"http://127.0.0.1:{stub.server_address[1]}",
        "--no-rate-limit",
    ],
    env=env, stdout=log, stderr=subprocess.STDOUT,
)
try:
    for _ in range(240):
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=1)
            break
        except Exception:
            time.sleep(0.5)
    else:
        raise SystemExit("proxy did not come up")

    turns = []
    convo = [{"role": "user", "content": "Page through the services and find the degraded ones."}]
    for n in range(1, 5):
        convo = convo + [assistant(n), tool_result(n)]
        turns.append(convo + [{"role": "system", "content": f"Reminder variant {n}."}])

    if label.endswith("noreminder"):
        # Control: the same conversation without the trailing system reminder.
        turns = [t[:-1] for t in turns]
    elif label.endswith("mixed"):
        # Claude Code sends reminders on some turns only: the first request
        # has none (so it is compressed and that form is cached), later ones do.
        turns[0] = turns[0][:-1]
    previous = None
    for n, messages in enumerate(turns, 1):
        req = urllib.request.Request(
            f"http://127.0.0.1:{port}/v1/messages",
            data=json.dumps({"model": "claude-fable-5", "max_tokens": 64,
                             "system": "You are a coding agent.",
                             "messages": messages}).encode(),
            headers={"content-type": "application/json", "x-api-key": "stub-key",
                     "anthropic-version": "2023-06-01"},
        )
        urllib.request.urlopen(req, timeout=120).read()
        forwarded = received[-1]["messages"]
        newest = max(i for i, m in enumerate(messages) if m["role"] == "user")
        fresh = (f"newest tool_result {len(json.dumps(messages[newest]))}->"
                 f"{len(json.dumps(forwarded[newest]))} bytes")
        cache = f"simulated cache read={usage[-1][0]} write={usage[-1][1]}"
        if previous is None:
            print(f"[{label}] turn {n}: cold; {fresh}; {cache}")
        else:
            # The previous turn's trailing system reminder is replaced this turn.
            limit = len(previous) - (previous[-1]["role"] == "system")
            same = 0
            while same < limit and strip(forwarded[same]) == strip(previous[same]):
                same += 1
            print(f"[{label}] turn {n}: {same}/{limit} previously forwarded history messages "
                  f"byte-identical; {fresh}; {cache}")
        previous = forwarded
    with open(os.path.join(os.path.dirname(__file__), f"forwarded-{label}.json"), "w") as f:
        json.dump(received, f, indent=1)
finally:
    proxy.terminate()
    proxy.wait(timeout=30)
    stub.shutdown()
```

</details>

## Review Readiness


- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist


- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)


N/A

## Additional Notes


- Headroom Desktop has shipped this as a downstream patch since
2026-09-07 (a wrapper around `resolve_tracker` with the same guards).
This PR expresses it inside `resolve_tracker` instead, without mutating
the lineage family.
- Why Anthropic only: the shape was observed from Claude Code on
`/v1/messages`; the OpenAI path keeps its current behaviour until there
is evidence of the same pattern there.
- Why the current history must extend past the old reminder (not just
equal the history before it): a request that drops the reminder and adds
nothing is not a continuation we have seen, and keeping the condition
strict keeps sibling retries apart.
- Documentation: the new env var is described in the `resolve_tracker`
docstring; I did not find a docs page listing the prefix-tracker env
vars (`HEADROOM_STABLE_BOUNDARY_BREAKPOINT` is not listed either). Happy
to add one if there is a preferred place.
- `prefix_tracker.py` changed in #3895 after the version the downstream
patch targeted; the bug reproduces unchanged on `main` d5318ac2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3934"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-04 13:58:14 -07:00
Andrei BoldyrevandJerrettDavis 7bfb1d7f38 fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)
## Description

Running headroom as the proxy for Claude Code destroys Anthropic
prompt-cache
reuse (#2085: ~4.4x cache-creation inflation, 2.5–3x net cost). Tracing
live
Claude Code traffic through the proxy shows **two independent
session-identity
defects**, both of which orphan or thrash the frozen-prefix state; this
PR
fixes both.

### Defect 1: `<system-reminder>` turns rotate the fallback session id
mid-conversation

Claude Code interleaves reminder turns into the history as actual
`role:"system"` messages (hook output, skills lists, file-truncation
notices).
`compute_session_id` hashed **every** system message, so the id rotated
each
time a reminder landed. Live trace (subagent reading two 80KB files; sid
changes exactly when the truncation reminder appears, and the tracker
restarts
at turn 0):

```
REQ#2 sid=68d4ee666990 nmsg=3   [0]SYSTEM<<top-level system>> [1]user [2]SYSTEM<<skills reminder>>
REQ#3 sid=6944948c9fb2 nmsg=6   ... [5]SYSTEM<<Truncated: PARTIAL view ...>>   <- id rotated
```

Everything keyed on the session id is orphaned at that moment: the
prefix
tracker (freeze never survives past a reminder-bearing turn),
beta-header
stickiness, the CCR and memory-tool registries, and the compression
cache.

**Fix:** hash only the **leading run** of system messages (everything
before
the first non-system turn) — the top-level system prompt on the
Anthropic path
(folded in as the synthetic first message), the conventional leading
system
message(s) on the OpenAI path. Stable for the life of a conversation;
mid-history system turns are content, not identity.

### Defect 2: conversations sharing a (now stable) id thrash one tracker

With ids stable, the fallback tuple `model + system prompt` is identical
across every same-type parallel subagent (and any sessions reusing one
system
prompt) — all of them collapse onto one `PrefixCacheTracker`, and their
interleaved histories cross-contaminate the freeze state: the forwarded
prefix
is byte-unstable on nearly every turn and the provider cache is
re-written
instead of read. Reproduced against the real code paths (script below):

```
1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True
2) single conversation, legacy       : stable prefix on 4/4 later turns, trackers=1
2) interleaved (subagents), legacy   : stable prefix on 0/9 later turns, trackers=1
2) interleaved, lineage resolution   : stable prefix on 8/8 later turns, trackers=2
```

**Fix:** `SessionTrackerStore.resolve_tracker` — within a session id,
reuse
the tracker whose previous request messages are a prefix of the incoming
history (client histories are append-only, so a conversation's next
request
always extends its previous one); a diverging or rewritten history
(client-side compaction) starts a fresh lineage. Matching uses the
repo's
existing canonical cross-turn equivalence
(`_canonicalize_for_prefix_compare`,
the same one the cache-stable delta path uses) on the **original client
bytes**, so moved cache breakpoints, string<->block sugar, transport
annotations, or a tail-mutating `pre_compress` hook never read as a
rewrite.
Byte-identical histories (templated fan-outs before they diverge)
intentionally share a tracker — their provider cache line is identical
too.

### Both fixes together, on live Claude Code traffic (sonnet, 2 parallel
Explore agents)

```
main conversation: sid=5b7e245a...  one tracker, turns 0->4, id stable across reminders
agents (collide):  sid=2bdffc9e...  -> lineage bare  (alpha) turns 0->1->2
                                    -> lineage "~1"  (beta)  turns 0->1->2
```

Before: the agents' ids rotated per reminder (every tracker stuck at
turn 0),
and whenever they did share an id they thrashed one tracker (`0/9`
stable
prefixes in the repro).

### Why not key the session id on conversation content?

Draft #1912 folds the first user turn into the fallback id; this change
composes with it, but identity-level keying alone can't close #2085:
identical
first turns (templated fan-outs) still collide, and everything keyed on
the
session id rotates with it when the client rewrites history. The
"session"
(client/workspace grouping) and the "conversation" (positional cache
lineage)
are different identities; only the tracker holds positional per-turn
state
that thrashes under collision — beta stickiness is a monotone union and
the
compression cache is content-addressed — so lineage resolution lives one
level below the session id and leaves the id semantics (and every other
consumer) untouched.

## Changes Made

- `headroom/cache/prefix_tracker.py`:
- `compute_session_id`: harvest only the leading system run (defect 1).
- `SessionTrackerStore.resolve_tracker`: conversation-lineage resolution
    (defect 2). First lineage lives under the bare session id —
single-conversation sessions behave byte-identically to before; degrades
to `get_or_create` when messages are absent or prefix freeze is
disabled.
- Lineages are capped per session id
(`PrefixFreezeConfig.max_lineages_per_session`, default 32). **Over-cap
  conversations share one overflow tracker instead of evicting an
established lineage** — any eviction policy degrades every conversation
  once the working set exceeds the cap (under round-robin the victim is
always the conversation about to arrive), while overflow sharing
degrades
only the over-cap tail, to exactly the pre-lineage shared behavior; `0`
  disables lineage splitting. Chains are stored as structural snapshots
  that normalize `NaN` (`json.loads` accepts bare NaN, and `NaN != NaN`
would read a byte-identical resend as a rewrite). Synthetic lineage keys
use a `\x00` separator, which cannot appear in an HTTP header value, so
  they can never collide with a client-supplied `x-headroom-session-id`.
- `headroom/proxy/handlers/anthropic.py`, `openai.py`: the session id
and
  the lineage both derive from the **same original client bytes** (a
turn-dependent hook rewrite can no longer rotate one without the other);
anthropic folds in its synthetic system message so explicit-header
clients
with different system prompts stay separate. Plus a docstring correction
in `streaming.py` that falsely claimed its coarse mid-turn key "mirrors"
  `compute_session_id`.
- `tests/test_cache/test_prefix_tracker.py`: 24 new test cases —
  reminder-rotation regression; interleaved isolation + per-conversation
turn state; identical-first-turn share-then-split; cache_control
movement
(3 cases); representation churn (string<->block sugar / streaming
`index`
  / Bedrock cachePoint); rewritten history → fresh lineage (compacted /
  middle-edited / truncated); legacy no-messages / freeze-disabled /
  empty-canonical fallbacks; NaN-in-tool-payload stability; overflow
  sharing, established-lineages-survive-cap, and a cap+1 round-robin
no-cliff guard; TTL cleanup; session-id-not-rotated-by-lineage guard.
One
  existing test renamed (`uses_all_system_messages` →
  `distinguishes_leading_system_run`) to match the new contract.
- Three SimpleNamespace stub stores in existing tests gained a
  `resolve_tracker` field (handlers call it unconditionally — a silent
`hasattr` fallback would degrade to the pre-fix behavior with no
signal).
One of them is the cold-start fast-pass suite (#2073), which landed
while
  this branch was in review.
- `CHANGELOG.md` entry.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Testing

- [x] Unit tests pass (`pytest`) — 11 failed, 8652 passed, 528 skipped
in 4:37 (the 11 are pre-existing on unmodified `main` — verified by
rerunning the same node ids on a clean checkout:
gh-CLI/onnx/PID-reuse/deadline flakes and order-dependent cases, none
touching session/cache/proxy paths)
- [x] Linting passes (`ruff check .`) — All checks passed (ruff 0.15.17,
CI-pinned; `ruff format --check .` clean)
- [x] Type checking passes (`mypy headroom`) — Success: no issues found
in 471 source files
- [x] New tests added for new functionality — 24 test cases; the
rotation/isolation/no-cliff ones fail on `main`
- [x] Manual testing performed — live Claude Code end-to-end, below

### Test Output

```text
$ python -m pytest tests/ -q
11 failed, 8652 passed, 528 skipped, 5857 warnings in 276.68s (0:04:36)
# same 11 fail on unmodified main (env/order-dependent: test_wrap_claude_base_url pid-reuse,
# copilot_auth gh-cli fallback, image_compression onnx, content_router deadline, rtk/output-shaper/dedup order flakes)

$ python -m pytest tests/test_cache/test_prefix_tracker.py -q
63 passed

$ uvx ruff@0.15.17 check . && uvx ruff@0.15.17 format --check .
All checks passed! / 1208 files already formatted

$ mypy headroom
Success: no issues found in 471 source files

$ python repro_2085.py
1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True
2) single conversation, legacy       : stable prefix on 4/4 later turns, trackers=1
2) interleaved (subagents), legacy   : stable prefix on 0/9 later turns, trackers=1
2) interleaved, lineage resolution   : stable prefix on 8/8 later turns, trackers=2
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.13, `uv sync --extra dev --extra
proxy`;
  real Claude Code CLI pointed at the proxy via
  `ANTHROPIC_BASE_URL=http://127.0.0.1:8790`, real Anthropic backend.
- Exact command / steps: ran Claude Code sessions that launch 2–3
parallel Explore subagents
(each reading multi-KB JSON files, several tool-loop turns each), with
an
observability wrapper printing each request's resolved session id,
tracker
  identity, and turn counter inside the proxy.
- Observed result: on `main`, subagent session ids rotate on
reminder-bearing turns
(trackers permanently stuck at turn 0); when conversations do share an
id
  they share one tracker whose turn counter interleaves all of them.
  On this branch: ids stable for the life of each conversation;
colliding subagents resolve to separate lineages (`bare`, `~1`) with
clean
per-conversation turn progressions (trace above). Unit-level repro shows
  forwarded-prefix stability going 0/9 → 8/8 for the interleaved shape.
- Not tested: reporter-scale cache-economics (his 4.4x needs his
long-session
workload against a paid backend); happy to coordinate with
@RomanAlexanderW
on a before/after — the number to watch is the cache-read ratio in
Claude
  Code transcripts recovering toward ~96%.

<details>
<summary>repro_2085.py</summary>

```python
"""Repro for #2085: concurrent conversations sharing a fallback session id
(same model + system prompt — e.g. a Claude Code session and its parallel
subagents) collapse onto one PrefixCacheTracker and thrash its frozen-prefix
state -> byte-unstable forwarded prefixes -> the provider prompt cache is
re-written on nearly every call. Uses headroom's real code paths.

Run from the repo root: python ../repro_2085.py
"""

from headroom.cache.prefix_tracker import PrefixFreezeConfig, SessionTrackerStore

MODEL = "claude-sonnet-5"
# Claude Code system prompt: long, static, identical across the main session
# and every parallel subagent of the same type.
SYSTEM = ("You are Claude Code, Anthropic's official CLI for Claude. " * 40)[:2000]


def convo(name: str, turns: int) -> list[dict]:
    msgs = [{"role": "system", "content": SYSTEM}]
    for t in range(turns):
        msgs.append({"role": "user", "content": f"[{name}] user turn {t}: " + ("x" * 800)})
        msgs.append(
            {"role": "assistant", "content": f"[{name}] tool_result {t}: " + ('{"data": 1}' * 200)}
        )
    return msgs


class _Req:  # request stub: no x-headroom-session-id header
    headers: dict = {}


# --- Part 1: identity collision (real derivation) ----------------------------
store = SessionTrackerStore(PrefixFreezeConfig())
id_a = store.compute_session_id(_Req(), MODEL, convo("A", 3))
id_b = store.compute_session_id(_Req(), MODEL, convo("B", 5))
print(f"1) fallback session ids: A={id_a} B={id_b} -> COLLIDE={id_a == id_b}")

# --- Part 2: interleaved conversations thrash the freeze state ---------------


def run(interleave: bool, lineage_resolution: bool) -> tuple[int, int, int]:
    store = SessionTrackerStore(PrefixFreezeConfig())
    stable_turns = 0
    later_turns = 0
    seq = []
    for t in range(1, 6):
        seq.append(("A", convo("A", t)))
        if interleave:
            seq.append(("B", convo("B", t)))
    for _name, msgs in seq:
        sid = store.compute_session_id(_Req(), MODEL, msgs)
        if lineage_resolution:
            tracker = store.resolve_tracker(sid, "anthropic", messages=msgs)
        else:
            tracker = store.get_or_create(sid, "anthropic")
        if tracker._turn_number > 0:
            later_turns += 1
            if tracker._forwarded_prefix_stable(msgs):
                stable_turns += 1
        tracker.update_from_response(
            cache_read_tokens=5000 * len(msgs),
            cache_write_tokens=2000,
            messages=msgs,
        )
    return stable_turns, later_turns, store.active_sessions


for label, interleave, fixed in (
    ("single conversation, legacy       ", False, False),
    ("interleaved (subagents), legacy   ", True, False),
    ("interleaved, lineage resolution   ", True, True),
):
    stable, later, sessions = run(interleave, fixed)
    print(f"2) {label}: stable prefix on {stable}/{later} later turns, trackers={sessions}")
```
</details>

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (CHANGELOG
only — no docs describe the tracker store)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Addresses the session-identity mechanisms of #2085; intentionally does
not
`Closes` it — the reporter should confirm the cache-read ratio recovers
on
  live traffic first.
- Composes with draft #1912 (first-user-turn fallback id).
- Known bounded tradeoffs (all strictly milder than the per-turn thrash
this
fixes): a fork-style branch that resends a parent's full history adopts
the
parent's lineage, costing the parent one cold restart at its next turn;
a
request that aborts before the response and is retried with different
bytes
starts a fresh lineage; history truncation/tail-edit starts a fresh
lineage
  even though the shorter provider prefix may still be warm.
- Hot-path cost, measured on a 199-message/2.1MB agentic history:
canonical
projection 0.21ms + structural snapshot 0.92ms + match loop 0.06ms with
  32 candidate lineages (2.27ms absolute worst case) ≈ **1.3ms per
  request** — same order as the handler's existing request deepcopy
(0.80ms) and below one `json.dumps` of the body (2.9ms). Chain memory is
structure-only (~180-330KB per lineage; message strings are shared with
  state the tracker already retains).
- Known semantic shift to flag: hashing only the leading system run
means
  conversations distinguished ONLY by mid-list system messages (e.g.
clients injecting a per-conversation system context late in the list)
now
share a fallback id. The tracker is protected by lineage resolution; the
  residual sharing concentrates in the CCR sticky-tool registry and the
monotone beta union — the same pre-existing class as same-system-prompt
  conversations today. Happy to file the CCR-stickiness scoping as a
  follow-up.
- Out of scope, observed while tracing: `SessionCcrTracker.has_done_ccr`
  mildly cross-contaminates conversations sharing an id (monotone, no
  thrash) — can file separately if useful.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:42:20 +00:00
Abhay Singh ec6e60ea3e fix(proxy/anthropic): scope session id by top-level system prompt (#2070)
## Description

`SessionTrackerStore.compute_session_id`
(`headroom/cache/prefix_tracker.py`) computes a fallback
session id (when no `x-headroom-session-id` header is present) from
`model` + system-prompt text.
But it harvests system text **only** from `messages` entries with `role
== "system"`:

```python
for msg in messages:
    if msg.get("role") == "system":
        ...  # collect system text
system_content = json.dumps(system_parts, ...)
key = f"{model}:{system_content}"
```

Anthropic's `/v1/messages` carries the system prompt as a **top-level**
`body["system"]` field —
it never sends `role:"system"` entries inside `messages`. And
`x-headroom-session-id` is a
Headroom-internal header no client sends. So for every genuine Anthropic
request `system_parts`
is empty and the id collapses to `md5(f"{model}:[]")` — **every
conversation on the same model
shares one session id**, and therefore one `PrefixCacheTracker` and all
session-sticky state.

The colliding state cross-contaminates across conversations
(`anthropic.py:1052`):
- sticky `headroom_retrieve` / memory tools keyed purely on `session_id`
(no content guard) get
injected into another conversation's tool list — busting its tools cache
and adding tools its
  client never requested;
- sticky `anthropic-beta` header tokens leak across conversations;
- `frozen_message_count` and the per-session compression cache
cross-contaminate.

(The sibling `StreamingMixin._get_session_key` already reads
`body.get("system")` and its docstring
claims to mirror `compute_session_id` — which it did not.)

Closes: no issue filed — found while auditing the session/prefix
tracker.

## Fix

Add an optional `system` parameter to `compute_session_id` and fold its
text (a plain string or a
list of `{"type":"text"}` blocks) into the hash. The Anthropic handler
passes `body.get("system")`.
OpenAI callers don't pass it (defaults to `None`), so their behavior is
unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cache/prefix_tracker.py`: `compute_session_id` accepts an
optional `system` and folds it into the id.
- `headroom/proxy/handlers/anthropic.py`: pass
`system=body.get("system")` when computing the session id.
- `tests/test_cache/test_prefix_tracker.py`: add
`test_compute_session_id_distinguishes_top_level_system` (distinct
systems → distinct ids; list-form == string-form; `system=None`
unchanged).

## Testing

- [x] New regression test added
(`tests/test_cache/test_prefix_tracker.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).

```text
$ uvx ruff@0.15.17 check headroom/cache/prefix_tracker.py headroom/proxy/handlers/anthropic.py tests/test_cache/test_prefix_tracker.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the hash logic with
a dependency-free script and left the full pytest to CI.
- Exact command / steps: computed ids for two conversations with the
same model and messages but different top-level `system` prompts,
through the old (never-folds-system) and new logic.
- Observed result: the old logic collapses both to one id (the leak);
the new logic separates them, folds list-form system the same as
string-form, and leaves the `system=None` (OpenAI) path unchanged:

```text
OLD: A=97d8857ba27010bb  B=97d8857ba27010bb  same=True
NEW: A=1e838c0f6e3980a6  B=18ec49bfa8240852  same=False
SESSION-ID SYSTEM FIX VERIFIED (old collapses Anthropic convos; new separates them)
```

- Not tested: a full two-conversation proxy run asserting no sticky-tool
leakage (needs the heavy stack). The fix is confined to
`compute_session_id` + the one handler call site, and the new test
drives the method directly. Full local `pytest` deferred to CI (OOM, per
above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Backward-compatible: the new `system` parameter defaults to `None`, so
the OpenAI call sites (`openai.py`) need no change and their session ids
are identical.
- @JerrettDavis tagging you — this one lets one Anthropic conversation's
sticky tools/headers leak into another on the same model, so it seemed
worth surfacing. Thanks!
2026-07-12 19:40:58 -04:00
Rod Boev 0f606b6281 fix(cache): avoid fallback session collisions (#1827)
## Description

Cache-mode session tracking currently collapses unrelated conversations
when they share a large static first system prompt. The fallback
session-id hash ignores later system messages entirely, so dynamic
per-conversation context can get cut out of the key and two different
sessions reuse the same `PrefixCacheTracker`. This hashes the full
ordered system-text payload instead, while leaving explicit
`x-headroom-session-id` overrides untouched. Refs #1808.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Collected all system-text content when building the fallback cache
session id.
- Stopped truncating fallback session-id input to the first 500
characters of the first system message.
- Added a regression that proves two conversations with different later
system context no longer collide.
- Added a preservation test that appending only non-system turns keeps
the same fallback session id.
- Applied the pinned Ruff formatter to three pre-existing files on the
current base so the repo-wide lint job passes unchanged semantics.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_cache/test_prefix_tracker.py -q`)
- [x] Linting passes (`uv run ruff check
headroom/cache/prefix_tracker.py
tests/test_cache/test_prefix_tracker.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_cache/test_prefix_tracker.py -q
40 passed, 1 warning in 0.15s

uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py
All checks passed!

uv run ruff check .
All checks passed!

uv run ruff format --check .
1046 files already formatted
```

## Real Behavior Proof

- Environment: Windows, project `uv` environment, focused cache-tracker
regression.
- Exact command / steps: run `tests/test_cache/test_prefix_tracker.py`
on `origin/main` with the new collision regression present, then rerun
the same file on this branch.
- Observed result: base returns the same session id for two
conversations that differ only in a later system message and fails
`assert id_a != id_b`; head passes the focused file and keeps the
fallback session id stable when only non-system turns are appended.
- Not tested: live proxy traffic through a real agentic client.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

This is only the session-collision half of #1808. The
duplicate-response-header fix stays separate so this PR can reference
the issue without claiming the whole bug report is resolved. The extra
formatting-only diff comes from the current base failing the pinned
full-repo Ruff format check.
2026-07-07 23:26:36 -05:00
Lakshya Sharma 4658721ea0 feat(cache): attribute prompt-cache misses to TTL lapse vs prefix change (#1313) (#1343)
## Description

A low prompt-cache hit rate is hard to act on without knowing *why*
turns miss. Two very different causes need very different responses:

- **TTL lapse** — the session went idle longer than the provider's cache
lifetime, so the entry expired. The fix is a longer TTL (e.g.
Anthropic's 1h breakpoint instead of the 5m default).
- **Prefix change** — the cacheable message prefix shifted, so the new
request couldn't match the cached key. A longer TTL won't help here at
all.

Right now those look identical from the dashboard (just "cache_read was
0"). This adds the attribution so a user can actually decide 5m vs 1h.

Closes #1313

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

`PrefixCacheTracker` already kept the previous turn's forwarded messages
and a per-turn activity timestamp, so the signal was already there — it
just wasn't being read.

- **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn
expected a cached prefix (non-zero cached tokens last turn) but read 0
this turn, returns `ttl_expiry` if the idle gap exceeded the provider
cache TTL, else `prefix_change` if the forwarded prefix differs from
last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed,
a coincident content change is moot, and the 5m-vs-1h decision is
exactly what the TTL signal answers. A 1h-breakpoint session can widen
the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and
hits return `is_miss=False`.
- **Anthropic handlers (streaming + non-streaming)** — classify BEFORE
`update_from_response` overwrites the last-turn state the classifier
reads, then record the reason.
- **`prometheus_metrics.py`** — a per-provider/per-reason counter,
`record_cache_miss_attribution()`, reset handling, and a
`headroom_cache_miss_attribution_total{provider,reason}` export series.
- **`cost.py`** — `build_prefix_cache_stats()` aggregates a
`miss_attribution` block (per-provider + totals, with the ttl/prefix
split as a % of *attributed* misses, so `unknown` doesn't dilute the
headline).
- **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix
change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix
change" headline.

Scoped to Anthropic for this first cut (where the tracker is fully
wired); OpenAI/Gemini can follow once the shape is proven.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_cache/test_prefix_tracker.py -q
38 passed
# 29 existing + 9 new classifier tests (TestClassifyCacheMiss).

$ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q
5 passed, 8 deselected
# new: counter bucketing, stats aggregation, empty case, /metrics export, reset.
```

The full `test_proxy_cache_ttl_metrics.py` /
`test_proxy_dashboard_stats_cache.py` files have some failures in this
sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) —
those spin up the proxy server / Rust `_core` extension, which isn't
built here. I confirmed via `git stash` that they fail identically on
`main` without my changes, so they're pre-existing and unrelated. My
additions to the stats dict are purely additive and don't break any
passing assertion.

## Real Behavior Proof

- Environment: Windows 11, Python 3.10. The Rust `_core` extension and a
live proxy aren't available in this checkout.
- Exact command / steps: drove `classify_cache_miss()` through every
branch with a faithful warm-then-miss sequence; drove
`record_cache_miss_attribution()` → `build_prefix_cache_stats()` →
`export()` end to end.
- Observed result: classifier returns
`cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL
wins the tie when both signals fire, a growing (append-only) prefix is
treated as stable, and the 1h override widens the window. The stats
builder produces `miss_attribution.totals`
(`ttl_expiry`/`prefix_change`/`unknown`/`total` +
`ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and
`by_provider`; `/metrics` emits
`headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`.
- Not tested: a live Anthropic session through the running proxy with a
real idle-then-resume to confirm the handler wiring fires end-to-end. I
verified the handler integration by reading scope/order (classify before
`update_from_response`, `provider_name`/`self.metrics` in scope) and
unit-tested every layer it calls, but didn't exercise the actual server
loop.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- The classifier is intentionally pure (takes the cache-read result +
current forwarded messages + an optional idle override) so it's
order-independent and unit-testable without a live tracker clock.
- No README/docs change yet — this surfaces in the dashboard and
`/metrics`, which are self-describing; happy to add a docs page if you'd
like one.
- CHANGELOG.md isn't touched — release-please generates it from the
`feat(cache):` commit subject.
- Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a
per-provider breakdown row in the dashboard panel (the stats already
carry `by_provider`).
2026-06-24 09:50:34 -05:00
Tejas Chopra 22dad133e2 fix: eliminate prefix cache busts from frozen count underestimation
Root cause: CompressionCache.compute_frozen_count() stopped at the first
tool_result not in its cache, capping frozen_message_count at 2. Tool
results excluded by content_router (Read/Glob) or skipped (ratio too
high) never entered the cache, so every subsequent message was eligible
for recompression — causing 192 cache busts per session.

Four fixes:
1. Add _stable_hashes set to CompressionCache so excluded/skipped
   tool_results don't block the frozen count walk
2. Fix _estimate_message_tokens to count tool_result content and
   tool_use input fields (were counted as 0 tokens in Anthropic format)
3. Fix streaming handler to include assistant response and
   original_messages in prefix tracker updates (parity with non-streaming)
4. TTL-aware batch recompression: defer first-time compressions within
   the 5-min cache TTL window, batching them at the boundary to trade
   many small busts for one
2026-04-07 17:10:20 -07:00
chopratejas 8f438b0674 Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude
- It will also install rtk-ai locally
- rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code
2026-03-11 23:01:42 -07:00