mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
feat/https-compose-caddy
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1cf4d053d2 |
fix(cache): stop Claude Code forks from cold-starting the conversation's lineage (#4043)
### Maintainer qualification refresh (2026-10-09) Current head: `a91a1ec24ab20b6dcea0f6e7f8d41feb7d0beea0`. The newly supplied live Claude Code/Anthropic before-after evidence resolves the previous runtime-proof hold. Clean synchronization with current main `01d9194` preserves contributor logic. Independently reran all 88 fork-lineage/prefix tests with the existing native extension; current-head hosted checks are green. Approved for human merge review. The live capture below remains correctly attributed to the preceding implementation head, not represented as a new capture of this synchronization commit. No merge performed. ## Description Claude Code sends side requests alongside an agent's turns. The subagent progress summary and the parent session's side query both re-send the history plus one prompt, with the breakpoint one message earlier, so they read the prompt cache without writing it. resolve_tracker matched such a fork as a continuation and replaced the lineage snapshot with it. The next real turn extends the history before the fork, not the fork, so it matched no lineage. It started a cold tracker (frozen=0), and the token-mode pipeline rewrote history the provider had already cached. Production, 2026-10-08: a review subagent busted Anthropic's cache on 35 of 81 turns. cache_read_input_tokens fell to the 6,065-token system prompt while 100-150k tokens were re-written. Its 600k-token parent did the same after its own side queries. In the proxy log, every bust is the first main request after a fork and shows no frozen prefix. Fork-free sessions on the same proxy were 96/96 and 130/130 full hits. A lineage now keeps its last three request snapshots. A request that extends an earlier one resumes that turn instead of starting cold. Each response is also recorded against the turn that sent it, through a request-scoped ContextVar (the same pattern as the tenant key and CCR store). A fork's response can no longer replace the main line's frozen prefix or overlay source, whether it lands before or after the in-flight main turn's response. Templated fan-out siblings that extend a shared earlier snapshot now stay on one tracker and each resumes its own turn's state. Splitting them would mint a new lineage per fork and exhaust the per-session cap. The fan-out test asserts the per-line state instead of object identity. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: each lineage keeps a `deque(maxlen=_LINEAGE_HISTORY)` (3) of request snapshots instead of one. `resolve_tracker` resumes the turn a request extends, and the turn it resolved is held in a request-scoped `ContextVar` (`_resolved_turn`) so the response is recorded against that turn. - `tests/test_anthropic_fork_lineage_cache.py` (new): handler-level regression. A main turn after a fork (progress summary or side query) must replay its cached prefix byte for byte, and a fork must not replace the lineage it branched from, whichever response lands first. - `tests/test_cache/test_prefix_tracker.py`: the fan-out test asserts per-line state instead of tracker object identity. ## Testing - [x] Unit tests pass (`pytest tests/test_anthropic_fork_lineage_cache.py tests/test_cache/ tests/test_proxy_anthropic_cache_stability.py`: 272 passed, with `test_cache/test_dynamic_detector.py` run on its own: 57 passed, 2 skipped) - [x] Linting passes (`ruff check` on the changed files) - [x] Type checking passes for the changed module (`mypy headroom/cache/prefix_tracker.py`: no issues) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # new test file, without the fix (parent commit) FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[summary] FAILED tests/test_anthropic_fork_lineage_cache.py::test_main_turn_after_a_fork_replays_its_cached_prefix[side-query] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[fork-first] FAILED tests/test_anthropic_fork_lineage_cache.py::test_fork_does_not_replace_the_lineage_it_branched_from[main-first] AssertionError: main turn 2 re-sent turn 1's history with different bytes ============================== 4 failed ============================== # with the fix ============================== 4 passed ============================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.16, real Claude Code 2.1.295 sessions against the Anthropic API through `headroom proxy --mode token` (details below). - Exact command / steps: the scripted Claude Code session described below, run once on `17abee27` (before) and once on `9b27664a` (after). - Observed result: main turns right after a fork arrived with a cold tracker 5/5 before, 0/5 after; cached tokens lost 3,363 before, 0 after. - Not tested: the full collapse to the system prompt seen in production fork-heavy sessions (did not reproduce in this short session); a deployed proxy running this fix. Live before/after capture on 2026-10-09. Real Claude Code 2.1.295 sessions against the Anthropic API (subscription login, `claude-sonnet-5-5`), through `headroom proxy --mode token` on Windows 11 / Python 3.13.16, with the native extension built in each tree. - **Before:** `17abee27` (merge base with `main`) - **After:** `9b27664a` (this PR) Both ran the same scripted session: a stream-json SDK client with `agentProgressSummaries` and `promptSuggestions` on, one general-purpose subagent that `cat`s ten ~40 KB JSON files one tool call at a time, then three parent follow-up turns. Each request's kind comes from Claude Code's own debug log (`[API REQUEST] source=…`). `sdk` and `agent:*` are main turns; `agent_summary` and `prompt_suggestion` are forks. Token counts are the upstream `usage` values from the proxy's PERF lines. | Main turns right after a fork | Before | After | |---|---|---| | Count | 5 | 5 | | Arrived with a cold tracker (no `Pipeline: freezing` line) | **5/5** | **0/5** | | Read the full prior prefix | 4/5 | **5/5** | | Cached tokens lost | 3,363 | 0 | - **Before**, subagent turn #15 (after its 23-message `agent_summary` fork): cold, `cache_read` 22,137 against the 25,500 cached by the previous turn, `cache_creation` 4,902. - **After**, the matching subagent turn #17 (after a 21-message `agent_summary` fork): frozen 20/23, `cache_read` 25,311 = 24,192 + 1,119, exactly the previous turn's read + write. This short session did not reproduce the full collapse to the system prompt (~6k read) seen in production fork-heavy sessions. In most cold turns the pipeline happened to regenerate identical bytes. The capture does show the mechanism deterministically: before, every fork leaves the next main turn of that conversation starting cold, and one of those cold turns rewrote 3.4k cached tokens. After, none start cold and every post-fork turn reads its whole prior prefix. <details><summary>Per-request tables (before / after)</summary> **Before (`17abee27`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 0 | 15,072 | | 2 | main (subagent) | 2 | - | 5,406 | 7,859 | | 3 | main (subagent) | 5 | 3 | 13,265 | 660 | | 4 | main (subagent) | 7 | 3 | 13,925 | 1,176 | | 5 | main (subagent) | 9 | 3 | 15,101 | 1,319 | | 6 | main (subagent) | 11 | 3 | 16,420 | 1,138 | | 7 | fork (agent_summary) | 11 | 3 | 16,476 | 0 | | 8 | main (subagent) <- after fork | 13 | - | 17,558 | 1,127 | | 9 | main (subagent) | 15 | 3 | 18,685 | 1,134 | | 10 | main (subagent) | 17 | 3 | 19,819 | 1,126 | | 11 | main (subagent) | 19 | 3 | 20,945 | 1,136 | | 12 | main (subagent) | 21 | 3 | 22,081 | 1,133 | | 13 | main (subagent) | 23 | 3 | 23,214 | 2,286 | | 14 | fork (agent_summary) | 23 | 22 | 23,321 | 0 | | 15 | main (subagent) <- after fork | 25 | - | 22,137 | 4,902 | | 16 | main (subagent) | 27 | 24 | 27,039 | 1,260 | | 17 | main (subagent) | 29 | 24 | 28,299 | 7,523 | | 18 | main (parent) | 4 | 3 | 15,072 | 2,204 | | 19 | fork (prompt_suggestion) | 6 | 5 | 17,276 | 335 | | 20 | main (parent) <- after fork | 6 | - | 17,611 | 31 | | 21 | fork (prompt_suggestion) | 8 | 7 | 17,642 | 362 | | 22 | main (parent) <- after fork | 8 | - | 18,004 | 29 | | 23 | main (parent) | 10 | 9 | 18,033 | 242 | | 24 | fork (prompt_suggestion) | 12 | 9 | 18,275 | 128 | | 25 | main (parent) <- after fork | 12 | - | 18,403 | 18 | | 26 | fork (prompt_suggestion) | 14 | 9 | 18,421 | 142 | **After (`9b27664a`)** | # | kind | msgs | frozen | cache_read_input_tokens | cache_creation_input_tokens | |---|---|---|---|---|---| | 1 | main (parent) | 2 | - | 5,406 | 9,666 | | 2 | main (subagent) | 2 | - | 10,523 | 2,742 | | 3 | main (parent) | 4 | 3 | 15,072 | 692 | | 4 | main (subagent) | 5 | 3 | 13,265 | 1,681 | | 5 | fork (prompt_suggestion) | 6 | 3 | 15,764 | 25 | | 6 | main (subagent) | 7 | 3 | 14,946 | 1,246 | | 7 | main (subagent) | 9 | 3 | 16,192 | 1,138 | | 8 | main (subagent) | 11 | 3 | 17,330 | 2,303 | | 9 | main (subagent) | 13 | 3 | 19,633 | 1,167 | | 10 | main (subagent) | 15 | 3 | 20,800 | 1,137 | | 11 | main (subagent) | 17 | 3 | 21,937 | 1,129 | | 12 | main (subagent) | 19 | 3 | 23,066 | 1,126 | | 13 | main (subagent) | 21 | 20 | 24,192 | 1,119 | | 14 | main (parent) <- after fork | 6 | 3 | 15,789 | 31 | | 15 | fork (prompt_suggestion) | 8 | 3 | 15,820 | 108 | | 16 | fork (agent_summary) | 21 | 20 | 24,248 | 0 | | 17 | main (subagent) <- after fork | 23 | 20 | 25,311 | 710 | | 18 | main (parent) <- after fork | 8 | 3 | 15,928 | 851 | | 19 | fork (prompt_suggestion) | 10 | 3 | 16,779 | 688 | | 20 | main (parent) <- after fork | 10 | 3 | 17,467 | 29 | | 21 | main (parent) | 12 | 3 | 17,496 | 130 | | 22 | main (parent) | 14 | 3 | 17,626 | 249 | | 23 | fork (prompt_suggestion) | 16 | 3 | 17,875 | 131 | | 24 | main (parent) <- after fork | 16 | 3 | 18,006 | 18 | | 25 | fork (prompt_suggestion) | 18 | 3 | 18,024 | 136 | </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, prefix-tracker lineage matching for requests that extend an earlier snapshot of the same lineage - Kill switch / disable path: none - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Not covered here: a few small partial misses in production that forks don't explain, and a synthetic-only risk that an older tool result gets recompressed once it falls past a lagging frozen boundary. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4043" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
9553d4292f |
fix(cache): retain prefix trackers for the provider cache lifetime (#4023)
## Description Session cleanup currently drops prefix trackers after the default 600-second session lifetime even when Anthropic's request selects a one-hour cache TTL. Losing the tracker also loses the frozen forwarded prefix and causes the cache-preserving overlay to replay different bytes. Closes #4017. ## Type of Change - [x] Bug fix - [x] Documentation update ## Changes Made - Resolve the request's existing Anthropic cache TTL before selecting the prefix lineage and pass that TTL through tracker selection/creation. - Retain each lineage for the greater of the configured session floor and its observed/current provider TTL plus the existing 60-second safety margin. - Keep the lifetime per lineage rather than changing global configuration: a five-minute session does not inherit another session's one-hour lifetime. - Preserve normal expiry beyond the provider lifetime and margin. - Honor configured cache TTL before a confirmed cache write; discard expired state before accepting a later request's longer TTL. - Re-register the live lineage family after lookup/cleanup so recreated sessions cannot share a tracker's frozen state across sibling conversations or tool profiles. - Update existing fake callbacks to the changed tracker contract, preserve all consumer byte-stability assertions, and remove one handler-to-store wiring-only test rather than re-pinning its internal identity assertions. - Document local tracker retention separately from provider cache availability. ## Testing - [x] Unit tests pass (`pytest`), including frozen-prefix byte stability and independent TTL expiry. - [x] Changed-file lint, configured pre-commit mypy and manual production timestamp/overlay checks pass. ### Test Output ```text Review correction: 273 passed, 2 dependency warnings in 83.79s {"expires_beyond_provider_ttl_and_margin": true, "five_minute_session_evicted": true, "forwarded_prefix_byte_exact": true, "initial_frozen": 1, "one_hour_session_retained": true, "resolved_ttl_seconds": 3600, "resumed_frozen": 1, "same_tracker_after_660s": true} ``` The mandatory pre-push `make ci-precheck` passed: workspace Rust fmt/clippy/tests, native extension build/import verification, the complete configured Python gate, and Commitlint. The native gate explicitly used the installed CPU `libonnxruntime.so.1.26.0` via `ORT_DYLIB_PATH`, without skipping cached-model assertions. ```text [build_rust_extension] headroom._core build + install + verify: OK ================== 177 passed, 4 skipped, 1 warning in 12.87s ================== ✅ ci-precheck PASSED — safe to push. ``` ```sh python -m pytest -q tests/test_cache/test_prefix_tracker.py python -m pytest -q tests/test_proxy_anthropic_cache_stability.py tests/test_cache_prefix_overlay.py tests/test_cache_ttl_preserved.py tests/test_cold_prefix.py ``` **74 passed** in tracker tests; **54 passed** in request/cache tests (one existing Starlette TestClient deprecation warning). Two new regressions fail on baseline because the request TTL is not accepted; a separate production runtime comparison demonstrates the actual pre-fix loss, not just the missing parameter. After rebasing onto current main `1ad92f0cbc3dea27676e253588100f64a2af6994`, ran those suites together with `tests/test_prefix_tracker_client_rewrote_reply.py`, `tests/test_cache_mode_delta_marker.py`, and `tests/test_content_router_mcp_small_result.py`: **150 passed**. The actual production timestamp/byte-retention smoke was repeated successfully on the rebased code, covering compatibility with main's newly merged client-reply normalization. Review regression and affected HTTP-consumer suite: **273 passed**, including tracker expiry, Anthropic/OpenAI cache stability, byte-faithful forwarding, header isolation, buffered timeouts, CCR injection, and immutable system prompts. All affected two-argument test callbacks now accept the TTL contract. The obsolete walker-call-order test was removed; wire-byte assertions remain. The new configured-TTL/expiry cases fail against the published baseline; the post-cleanup regression also verifies that an unrelated conversation gets a different tracker after recreation. Changed-file Ruff, formatting, installed pre-commit `mypy headroom`, and Commitlint hooks passed. Full-tree Ruff is locally blocked by executable filesystem permissions on five unchanged OAuth2 plugin files (tracked 100644). A separate full dev-environment mypy run found an unchanged optional LangChain `truthy-function` error; the configured isolated pre-commit hook passed. No unrelated source/permissions changed. ## Real Behavior Proof - Environment: Linux x86_64, CPython 3.13.14; unchanged `dea9092e19385f286b7f67a64f4d85de2708d756` and this branch; real `SessionTrackerStore`, `PrefixCacheTracker`, request TTL parser, token estimator and `overlay_cached_prefix`, no mocked store or mocked clock. - Exact command / steps: parse `cache_control: {type: ephemeral, ttl: 1h}` for `claude-sonnet-4-5`; establish one real cache-write-confirmed frozen message with different original/forwarded prefix bytes; age `_last_activity` to `time.time()-660` and make periodic cleanup due; resume the same lineage and compare the tracker, frozen count, and forwarded-prefix bytes. Also age beyond 3901 seconds and compare independent 300/3600-second sessions. - Before: resolved TTL 3600; after 660 seconds, tracker identity changes, frozen count drops **1→0**, and forwarded-prefix bytes differ. - Observed result: after the fix, after 660 seconds the same tracker remains, frozen count stays **1**, and forwarded-prefix bytes are identical. The one-hour session survives independently, the five-minute session expires, and the one-hour lineage still expires beyond TTL plus margin. - Review runtime proof against published head `e1a107df`: actual stores and prefix overlays confirm configured TTL retention before the first cache write, no resurrection of expired state by a later one-hour request, eventual expiry, and separate trackers after the maintainer's 3901-second cleanup/recreation sequence. On that exact baseline the unrelated lookup reuses the wrong tracker, although the overlay guard preserves the particular unrelated input bytes; after the fix both identity isolation and byte preservation hold. Identical messages under different tool profiles also receive a cold, separate tracker after the expired bare-key transition. - Not tested: Anthropic cloud cache availability, billed savings, or an actual one-hour wait. The local production state and byte invariants were exercised by aging actual timestamps, not by a paid upstream request. Preserving a tracker cannot guarantee that the provider retains its cache. ## Runtime Rollout Safety - Rollout-managed feature(s): none; repair of existing TTL-aware cache-prefix behavior. - Minimum rollout channel: stable; no rollout-managed feature is enabled. - Stable/default behavior changed: eligible local prefix trackers survive for their provider cache lifetime. Short-TTL/default sessions remain bounded by their own session floor. - Kill switch / disable path: the existing prefix-freeze disable path remains available; no new flag. - Unsafe override required: no. - Qualification impact: local Anthropic prefix lineage selection/retention, not cache pricing or subscription quota policy. - Rollback path: revert the commit to restore previous retention. No persistent schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Existing cache TTL parser and safety margin reused; cache docs updated. - [x] Regressions cover frozen-byte retention, expiry, mixed lifetimes, and existing request byte-stability behavior. - [x] `CHANGELOG.md` not edited; Conventional Commit title supplied. Remote CI results are not inferred from local verification. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
16eaec1e95 |
fix(cache): keep prefix lineage across a replaced system tail (#3934)
## Description Claude Code can end a `/v1/messages` request with a temporary `role:"system"` reminder and replace it on the next turn. `SessionTrackerStore.resolve_tracker` records the full history, so the stored chain ends in the old reminder and is never a prefix of the next request: `_classify_history_canonical` returns `diverged` and the request gets a fresh, empty tracker. The confirmed frozen prefix and the previously forwarded messages are gone, the #3380 replay has nothing to replay, and any recompressed history re-writes the provider prompt cache. The reminder moves every turn, so this repeats for the whole session. This change lets lineage resolution set aside that one replaced trailing system message, only when nothing matched as recorded. It changes tracker selection only: forwarded bytes, the replay body, and breakpoint placement are untouched, and the existing replay already stops at the first changed message, so the new reminder goes out as the client sent it. Closes #3933 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: new `_extends_past_system_tail(current, previous)`. It is true when `previous` ends in a `role:"system"` message, everything before that message includes user/assistant history and is unchanged in `current`, and `current` is longer than `previous`. Because `current` must be longer, siblings that only swap their final instruction stay separate. - `SessionTrackerStore.resolve_tracker`: after the existing exact/append, block-append and block-rewrite passes find nothing (and there are no rewrite candidates, so an ambiguous rewrite tie is not overridden), Anthropic requests consider chains with the same cache affinity that pass that check. Only a unique longest candidate is taken; two equal-length candidates start a fresh lineage, as before. The full current snapshot is stored as usual. - `extract_cache_stable_delta` (cache mode's prefix/delta split): a `diverged` request that replaced the previous trailing system reminder is split too. Every recorded message before the reminder, plus the model reply the handlers record after it, must match the current request position for position (canonical compare, same rule via `_extends_past_system_tail`). The previously forwarded bytes are replayed without the old reminder, and everything after it is the delta to compress. Before this, cache mode tagged these turns `cache_mode_prefix_mismatch` and forwarded the new messages raw. - `HEADROOM_TRANSIENT_SYSTEM_LINEAGE=0` (also `false`/`no`/`off`) turns off both the lineage fallback and the delta split. It uses the same parsing as `HEADROOM_STABLE_BOUNDARY_BREAKPOINT`. - `tests/test_cache/test_prefix_tracker.py`: - Tracker retention: the tracker and its confirmed frozen count are kept, and the existing overlay restores the unchanged prefix. - A reminder that moves and changes over several turns: full snapshots are stored. - Isolation: different cache affinity, OpenAI provider, changed earlier history, changed leading system message, changed historical system message, same-length siblings `[user: task, system: A]` vs `[user: task, system: B]`, ambiguous equal-length siblings, exact-match precedence, and the kill switch. - The delta split: with and without the recorded reply, a system message followed by a client turn is not treated as a reminder, a mismatched reply is rejected, and the kill switch disables it. - `tests/test_cache_mode_cold_start.py`: drives `handle_anthropic_messages` in `mode="cache"` with the real `SessionTrackerStore` over three turns. The first turn compresses a tool result and later turns replace the reminder. The test asserts that the third request replays the second request's compressed prefix byte-for-byte (ignoring `cache_control`), compresses the new tool result, and ends in the current reminder. ## Testing - [x] Unit tests pass (`pytest`) - targeted files below, not the whole suite - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - on `headroom/cache/prefix_tracker.py` - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New tests with the source at `main` ( |
||
|
|
7bfb1d7f38 |
fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)
## Description Running headroom as the proxy for Claude Code destroys Anthropic prompt-cache reuse (#2085: ~4.4x cache-creation inflation, 2.5–3x net cost). Tracing live Claude Code traffic through the proxy shows **two independent session-identity defects**, both of which orphan or thrash the frozen-prefix state; this PR fixes both. ### Defect 1: `<system-reminder>` turns rotate the fallback session id mid-conversation Claude Code interleaves reminder turns into the history as actual `role:"system"` messages (hook output, skills lists, file-truncation notices). `compute_session_id` hashed **every** system message, so the id rotated each time a reminder landed. Live trace (subagent reading two 80KB files; sid changes exactly when the truncation reminder appears, and the tracker restarts at turn 0): ``` REQ#2 sid=68d4ee666990 nmsg=3 [0]SYSTEM<<top-level system>> [1]user [2]SYSTEM<<skills reminder>> REQ#3 sid=6944948c9fb2 nmsg=6 ... [5]SYSTEM<<Truncated: PARTIAL view ...>> <- id rotated ``` Everything keyed on the session id is orphaned at that moment: the prefix tracker (freeze never survives past a reminder-bearing turn), beta-header stickiness, the CCR and memory-tool registries, and the compression cache. **Fix:** hash only the **leading run** of system messages (everything before the first non-system turn) — the top-level system prompt on the Anthropic path (folded in as the synthetic first message), the conventional leading system message(s) on the OpenAI path. Stable for the life of a conversation; mid-history system turns are content, not identity. ### Defect 2: conversations sharing a (now stable) id thrash one tracker With ids stable, the fallback tuple `model + system prompt` is identical across every same-type parallel subagent (and any sessions reusing one system prompt) — all of them collapse onto one `PrefixCacheTracker`, and their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider cache is re-written instead of read. Reproduced against the real code paths (script below): ``` 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` **Fix:** `SessionTrackerStore.resolve_tracker` — within a session id, reuse the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one); a diverging or rewritten history (client-side compaction) starts a fresh lineage. Matching uses the repo's existing canonical cross-turn equivalence (`_canonicalize_for_prefix_compare`, the same one the cache-stable delta path uses) on the **original client bytes**, so moved cache breakpoints, string<->block sugar, transport annotations, or a tail-mutating `pre_compress` hook never read as a rewrite. Byte-identical histories (templated fan-outs before they diverge) intentionally share a tracker — their provider cache line is identical too. ### Both fixes together, on live Claude Code traffic (sonnet, 2 parallel Explore agents) ``` main conversation: sid=5b7e245a... one tracker, turns 0->4, id stable across reminders agents (collide): sid=2bdffc9e... -> lineage bare (alpha) turns 0->1->2 -> lineage "~1" (beta) turns 0->1->2 ``` Before: the agents' ids rotated per reminder (every tracker stuck at turn 0), and whenever they did share an id they thrashed one tracker (`0/9` stable prefixes in the repro). ### Why not key the session id on conversation content? Draft #1912 folds the first user turn into the fallback id; this change composes with it, but identity-level keying alone can't close #2085: identical first turns (templated fan-outs) still collide, and everything keyed on the session id rotates with it when the client rewrites history. The "session" (client/workspace grouping) and the "conversation" (positional cache lineage) are different identities; only the tracker holds positional per-turn state that thrashes under collision — beta stickiness is a monotone union and the compression cache is content-addressed — so lineage resolution lives one level below the session id and leaves the id semantics (and every other consumer) untouched. ## Changes Made - `headroom/cache/prefix_tracker.py`: - `compute_session_id`: harvest only the leading system run (defect 1). - `SessionTrackerStore.resolve_tracker`: conversation-lineage resolution (defect 2). First lineage lives under the bare session id — single-conversation sessions behave byte-identically to before; degrades to `get_or_create` when messages are absent or prefix freeze is disabled. - Lineages are capped per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32). **Over-cap conversations share one overflow tracker instead of evicting an established lineage** — any eviction policy degrades every conversation once the working set exceeds the cap (under round-robin the victim is always the conversation about to arrive), while overflow sharing degrades only the over-cap tail, to exactly the pre-lineage shared behavior; `0` disables lineage splitting. Chains are stored as structural snapshots that normalize `NaN` (`json.loads` accepts bare NaN, and `NaN != NaN` would read a byte-identical resend as a rewrite). Synthetic lineage keys use a `\x00` separator, which cannot appear in an HTTP header value, so they can never collide with a client-supplied `x-headroom-session-id`. - `headroom/proxy/handlers/anthropic.py`, `openai.py`: the session id and the lineage both derive from the **same original client bytes** (a turn-dependent hook rewrite can no longer rotate one without the other); anthropic folds in its synthetic system message so explicit-header clients with different system prompts stay separate. Plus a docstring correction in `streaming.py` that falsely claimed its coarse mid-turn key "mirrors" `compute_session_id`. - `tests/test_cache/test_prefix_tracker.py`: 24 new test cases — reminder-rotation regression; interleaved isolation + per-conversation turn state; identical-first-turn share-then-split; cache_control movement (3 cases); representation churn (string<->block sugar / streaming `index` / Bedrock cachePoint); rewritten history → fresh lineage (compacted / middle-edited / truncated); legacy no-messages / freeze-disabled / empty-canonical fallbacks; NaN-in-tool-payload stability; overflow sharing, established-lineages-survive-cap, and a cap+1 round-robin no-cliff guard; TTL cleanup; session-id-not-rotated-by-lineage guard. One existing test renamed (`uses_all_system_messages` → `distinguishes_leading_system_run`) to match the new contract. - Three SimpleNamespace stub stores in existing tests gained a `resolve_tracker` field (handlers call it unconditionally — a silent `hasattr` fallback would degrade to the pre-fix behavior with no signal). One of them is the cold-start fast-pass suite (#2073), which landed while this branch was in review. - `CHANGELOG.md` entry. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Testing - [x] Unit tests pass (`pytest`) — 11 failed, 8652 passed, 528 skipped in 4:37 (the 11 are pre-existing on unmodified `main` — verified by rerunning the same node ids on a clean checkout: gh-CLI/onnx/PID-reuse/deadline flakes and order-dependent cases, none touching session/cache/proxy paths) - [x] Linting passes (`ruff check .`) — All checks passed (ruff 0.15.17, CI-pinned; `ruff format --check .` clean) - [x] Type checking passes (`mypy headroom`) — Success: no issues found in 471 source files - [x] New tests added for new functionality — 24 test cases; the rotation/isolation/no-cliff ones fail on `main` - [x] Manual testing performed — live Claude Code end-to-end, below ### Test Output ```text $ python -m pytest tests/ -q 11 failed, 8652 passed, 528 skipped, 5857 warnings in 276.68s (0:04:36) # same 11 fail on unmodified main (env/order-dependent: test_wrap_claude_base_url pid-reuse, # copilot_auth gh-cli fallback, image_compression onnx, content_router deadline, rtk/output-shaper/dedup order flakes) $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 63 passed $ uvx ruff@0.15.17 check . && uvx ruff@0.15.17 format --check . All checks passed! / 1208 files already formatted $ mypy headroom Success: no issues found in 471 source files $ python repro_2085.py 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, `uv sync --extra dev --extra proxy`; real Claude Code CLI pointed at the proxy via `ANTHROPIC_BASE_URL=http://127.0.0.1:8790`, real Anthropic backend. - Exact command / steps: ran Claude Code sessions that launch 2–3 parallel Explore subagents (each reading multi-KB JSON files, several tool-loop turns each), with an observability wrapper printing each request's resolved session id, tracker identity, and turn counter inside the proxy. - Observed result: on `main`, subagent session ids rotate on reminder-bearing turns (trackers permanently stuck at turn 0); when conversations do share an id they share one tracker whose turn counter interleaves all of them. On this branch: ids stable for the life of each conversation; colliding subagents resolve to separate lineages (`bare`, `~1`) with clean per-conversation turn progressions (trace above). Unit-level repro shows forwarded-prefix stability going 0/9 → 8/8 for the interleaved shape. - Not tested: reporter-scale cache-economics (his 4.4x needs his long-session workload against a paid backend); happy to coordinate with @RomanAlexanderW on a before/after — the number to watch is the cache-read ratio in Claude Code transcripts recovering toward ~96%. <details> <summary>repro_2085.py</summary> ```python """Repro for #2085: concurrent conversations sharing a fallback session id (same model + system prompt — e.g. a Claude Code session and its parallel subagents) collapse onto one PrefixCacheTracker and thrash its frozen-prefix state -> byte-unstable forwarded prefixes -> the provider prompt cache is re-written on nearly every call. Uses headroom's real code paths. Run from the repo root: python ../repro_2085.py """ from headroom.cache.prefix_tracker import PrefixFreezeConfig, SessionTrackerStore MODEL = "claude-sonnet-5" # Claude Code system prompt: long, static, identical across the main session # and every parallel subagent of the same type. SYSTEM = ("You are Claude Code, Anthropic's official CLI for Claude. " * 40)[:2000] def convo(name: str, turns: int) -> list[dict]: msgs = [{"role": "system", "content": SYSTEM}] for t in range(turns): msgs.append({"role": "user", "content": f"[{name}] user turn {t}: " + ("x" * 800)}) msgs.append( {"role": "assistant", "content": f"[{name}] tool_result {t}: " + ('{"data": 1}' * 200)} ) return msgs class _Req: # request stub: no x-headroom-session-id header headers: dict = {} # --- Part 1: identity collision (real derivation) ---------------------------- store = SessionTrackerStore(PrefixFreezeConfig()) id_a = store.compute_session_id(_Req(), MODEL, convo("A", 3)) id_b = store.compute_session_id(_Req(), MODEL, convo("B", 5)) print(f"1) fallback session ids: A={id_a} B={id_b} -> COLLIDE={id_a == id_b}") # --- Part 2: interleaved conversations thrash the freeze state --------------- def run(interleave: bool, lineage_resolution: bool) -> tuple[int, int, int]: store = SessionTrackerStore(PrefixFreezeConfig()) stable_turns = 0 later_turns = 0 seq = [] for t in range(1, 6): seq.append(("A", convo("A", t))) if interleave: seq.append(("B", convo("B", t))) for _name, msgs in seq: sid = store.compute_session_id(_Req(), MODEL, msgs) if lineage_resolution: tracker = store.resolve_tracker(sid, "anthropic", messages=msgs) else: tracker = store.get_or_create(sid, "anthropic") if tracker._turn_number > 0: later_turns += 1 if tracker._forwarded_prefix_stable(msgs): stable_turns += 1 tracker.update_from_response( cache_read_tokens=5000 * len(msgs), cache_write_tokens=2000, messages=msgs, ) return stable_turns, later_turns, store.active_sessions for label, interleave, fixed in ( ("single conversation, legacy ", False, False), ("interleaved (subagents), legacy ", True, False), ("interleaved, lineage resolution ", True, True), ): stable, later, sessions = run(interleave, fixed) print(f"2) {label}: stable prefix on {stable}/{later} later turns, trackers={sessions}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (CHANGELOG only — no docs describe the tracker store) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Addresses the session-identity mechanisms of #2085; intentionally does not `Closes` it — the reporter should confirm the cache-read ratio recovers on live traffic first. - Composes with draft #1912 (first-user-turn fallback id). - Known bounded tradeoffs (all strictly milder than the per-turn thrash this fixes): a fork-style branch that resends a parent's full history adopts the parent's lineage, costing the parent one cold restart at its next turn; a request that aborts before the response and is retried with different bytes starts a fresh lineage; history truncation/tail-edit starts a fresh lineage even though the shorter provider prefix may still be warm. - Hot-path cost, measured on a 199-message/2.1MB agentic history: canonical projection 0.21ms + structural snapshot 0.92ms + match loop 0.06ms with 32 candidate lineages (2.27ms absolute worst case) ≈ **1.3ms per request** — same order as the handler's existing request deepcopy (0.80ms) and below one `json.dumps` of the body (2.9ms). Chain memory is structure-only (~180-330KB per lineage; message strings are shared with state the tracker already retains). - Known semantic shift to flag: hashing only the leading system run means conversations distinguished ONLY by mid-list system messages (e.g. clients injecting a per-conversation system context late in the list) now share a fallback id. The tracker is protected by lineage resolution; the residual sharing concentrates in the CCR sticky-tool registry and the monotone beta union — the same pre-existing class as same-system-prompt conversations today. Happy to file the CCR-stickiness scoping as a follow-up. - Out of scope, observed while tracing: `SessionCcrTracker.has_done_ccr` mildly cross-contaminates conversations sharing an id (monotone, no thrash) — can file separately if useful. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
ec6e60ea3e |
fix(proxy/anthropic): scope session id by top-level system prompt (#2070)
## Description
`SessionTrackerStore.compute_session_id`
(`headroom/cache/prefix_tracker.py`) computes a fallback
session id (when no `x-headroom-session-id` header is present) from
`model` + system-prompt text.
But it harvests system text **only** from `messages` entries with `role
== "system"`:
```python
for msg in messages:
if msg.get("role") == "system":
... # collect system text
system_content = json.dumps(system_parts, ...)
key = f"{model}:{system_content}"
```
Anthropic's `/v1/messages` carries the system prompt as a **top-level**
`body["system"]` field —
it never sends `role:"system"` entries inside `messages`. And
`x-headroom-session-id` is a
Headroom-internal header no client sends. So for every genuine Anthropic
request `system_parts`
is empty and the id collapses to `md5(f"{model}:[]")` — **every
conversation on the same model
shares one session id**, and therefore one `PrefixCacheTracker` and all
session-sticky state.
The colliding state cross-contaminates across conversations
(`anthropic.py:1052`):
- sticky `headroom_retrieve` / memory tools keyed purely on `session_id`
(no content guard) get
injected into another conversation's tool list — busting its tools cache
and adding tools its
client never requested;
- sticky `anthropic-beta` header tokens leak across conversations;
- `frozen_message_count` and the per-session compression cache
cross-contaminate.
(The sibling `StreamingMixin._get_session_key` already reads
`body.get("system")` and its docstring
claims to mirror `compute_session_id` — which it did not.)
Closes: no issue filed — found while auditing the session/prefix
tracker.
## Fix
Add an optional `system` parameter to `compute_session_id` and fold its
text (a plain string or a
list of `{"type":"text"}` blocks) into the hash. The Anthropic handler
passes `body.get("system")`.
OpenAI callers don't pass it (defaults to `None`), so their behavior is
unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/prefix_tracker.py`: `compute_session_id` accepts an
optional `system` and folds it into the id.
- `headroom/proxy/handlers/anthropic.py`: pass
`system=body.get("system")` when computing the session id.
- `tests/test_cache/test_prefix_tracker.py`: add
`test_compute_session_id_distinguishes_top_level_system` (distinct
systems → distinct ids; list-form == string-form; `system=None`
unchanged).
## Testing
- [x] New regression test added
(`tests/test_cache/test_prefix_tracker.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/prefix_tracker.py headroom/proxy/handlers/anthropic.py tests/test_cache/test_prefix_tracker.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the hash logic with
a dependency-free script and left the full pytest to CI.
- Exact command / steps: computed ids for two conversations with the
same model and messages but different top-level `system` prompts,
through the old (never-folds-system) and new logic.
- Observed result: the old logic collapses both to one id (the leak);
the new logic separates them, folds list-form system the same as
string-form, and leaves the `system=None` (OpenAI) path unchanged:
```text
OLD: A=97d8857ba27010bb B=97d8857ba27010bb same=True
NEW: A=1e838c0f6e3980a6 B=18ec49bfa8240852 same=False
SESSION-ID SYSTEM FIX VERIFIED (old collapses Anthropic convos; new separates them)
```
- Not tested: a full two-conversation proxy run asserting no sticky-tool
leakage (needs the heavy stack). The fix is confined to
`compute_session_id` + the one handler call site, and the new test
drives the method directly. Full local `pytest` deferred to CI (OOM, per
above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Backward-compatible: the new `system` parameter defaults to `None`, so
the OpenAI call sites (`openai.py`) need no change and their session ids
are identical.
- @JerrettDavis tagging you — this one lets one Anthropic conversation's
sticky tools/headers leak into another on the same model, so it seemed
worth surfacing. Thanks!
|
||
|
|
0f606b6281 |
fix(cache): avoid fallback session collisions (#1827)
## Description Cache-mode session tracking currently collapses unrelated conversations when they share a large static first system prompt. The fallback session-id hash ignores later system messages entirely, so dynamic per-conversation context can get cut out of the key and two different sessions reuse the same `PrefixCacheTracker`. This hashes the full ordered system-text payload instead, while leaving explicit `x-headroom-session-id` overrides untouched. Refs #1808. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Collected all system-text content when building the fallback cache session id. - Stopped truncating fallback session-id input to the first 500 characters of the first system message. - Added a regression that proves two conversations with different later system context no longer collide. - Added a preservation test that appending only non-system turns keeps the same fallback session id. - Applied the pinned Ruff formatter to three pre-existing files on the current base so the repo-wide lint job passes unchanged semantics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cache/test_prefix_tracker.py -q`) - [x] Linting passes (`uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_cache/test_prefix_tracker.py -q 40 passed, 1 warning in 0.15s uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py All checks passed! uv run ruff check . All checks passed! uv run ruff format --check . 1046 files already formatted ``` ## Real Behavior Proof - Environment: Windows, project `uv` environment, focused cache-tracker regression. - Exact command / steps: run `tests/test_cache/test_prefix_tracker.py` on `origin/main` with the new collision regression present, then rerun the same file on this branch. - Observed result: base returns the same session id for two conversations that differ only in a later system message and fails `assert id_a != id_b`; head passes the focused file and keeps the fallback session id stable when only non-system turns are appended. - Not tested: live proxy traffic through a real agentic client. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This is only the session-collision half of #1808. The duplicate-response-header fix stays separate so this PR can reference the issue without claiming the whole bug report is resolved. The extra formatting-only diff comes from the current base failing the pinned full-repo Ruff format check. |
||
|
|
4658721ea0 |
feat(cache): attribute prompt-cache misses to TTL lapse vs prefix change (#1313) (#1343)
## Description A low prompt-cache hit rate is hard to act on without knowing *why* turns miss. Two very different causes need very different responses: - **TTL lapse** — the session went idle longer than the provider's cache lifetime, so the entry expired. The fix is a longer TTL (e.g. Anthropic's 1h breakpoint instead of the 5m default). - **Prefix change** — the cacheable message prefix shifted, so the new request couldn't match the cached key. A longer TTL won't help here at all. Right now those look identical from the dashboard (just "cache_read was 0"). This adds the attribution so a user can actually decide 5m vs 1h. Closes #1313 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `PrefixCacheTracker` already kept the previous turn's forwarded messages and a per-turn activity timestamp, so the signal was already there — it just wasn't being read. - **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn expected a cached prefix (non-zero cached tokens last turn) but read 0 this turn, returns `ttl_expiry` if the idle gap exceeded the provider cache TTL, else `prefix_change` if the forwarded prefix differs from last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed, a coincident content change is moot, and the 5m-vs-1h decision is exactly what the TTL signal answers. A 1h-breakpoint session can widen the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and hits return `is_miss=False`. - **Anthropic handlers (streaming + non-streaming)** — classify BEFORE `update_from_response` overwrites the last-turn state the classifier reads, then record the reason. - **`prometheus_metrics.py`** — a per-provider/per-reason counter, `record_cache_miss_attribution()`, reset handling, and a `headroom_cache_miss_attribution_total{provider,reason}` export series. - **`cost.py`** — `build_prefix_cache_stats()` aggregates a `miss_attribution` block (per-provider + totals, with the ttl/prefix split as a % of *attributed* misses, so `unknown` doesn't dilute the headline). - **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix change" headline. Scoped to Anthropic for this first cut (where the tracker is fully wired); OpenAI/Gemini can follow once the shape is proven. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 38 passed # 29 existing + 9 new classifier tests (TestClassifyCacheMiss). $ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q 5 passed, 8 deselected # new: counter bucketing, stats aggregation, empty case, /metrics export, reset. ``` The full `test_proxy_cache_ttl_metrics.py` / `test_proxy_dashboard_stats_cache.py` files have some failures in this sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) — those spin up the proxy server / Rust `_core` extension, which isn't built here. I confirmed via `git stash` that they fail identically on `main` without my changes, so they're pre-existing and unrelated. My additions to the stats dict are purely additive and don't break any passing assertion. ## Real Behavior Proof - Environment: Windows 11, Python 3.10. The Rust `_core` extension and a live proxy aren't available in this checkout. - Exact command / steps: drove `classify_cache_miss()` through every branch with a faithful warm-then-miss sequence; drove `record_cache_miss_attribution()` → `build_prefix_cache_stats()` → `export()` end to end. - Observed result: classifier returns `cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL wins the tie when both signals fire, a growing (append-only) prefix is treated as stable, and the 1h override widens the window. The stats builder produces `miss_attribution.totals` (`ttl_expiry`/`prefix_change`/`unknown`/`total` + `ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and `by_provider`; `/metrics` emits `headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`. - Not tested: a live Anthropic session through the running proxy with a real idle-then-resume to confirm the handler wiring fires end-to-end. I verified the handler integration by reading scope/order (classify before `update_from_response`, `provider_name`/`self.metrics` in scope) and unit-tested every layer it calls, but didn't exercise the actual server loop. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The classifier is intentionally pure (takes the cache-read result + current forwarded messages + an optional idle override) so it's order-independent and unit-testable without a live tracker clock. - No README/docs change yet — this surfaces in the dashboard and `/metrics`, which are self-describing; happy to add a docs page if you'd like one. - CHANGELOG.md isn't touched — release-please generates it from the `feat(cache):` commit subject. - Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a per-provider breakdown row in the dashboard panel (the stats already carry `by_provider`). |
||
|
|
22dad133e2 |
fix: eliminate prefix cache busts from frozen count underestimation
Root cause: CompressionCache.compute_frozen_count() stopped at the first tool_result not in its cache, capping frozen_message_count at 2. Tool results excluded by content_router (Read/Glob) or skipped (ratio too high) never entered the cache, so every subsequent message was eligible for recompression — causing 192 cache busts per session. Four fixes: 1. Add _stable_hashes set to CompressionCache so excluded/skipped tool_results don't block the frozen count walk 2. Fix _estimate_message_tokens to count tool_result content and tool_use input fields (were counted as 0 tokens in Anthropic format) 3. Fix streaming handler to include assistant response and original_messages in prefix tracker updates (parity with non-streaming) 4. TTL-aware batch recompression: defer first-time compressions within the 5-min cache TTL window, batching them at the boundary to trade many small busts for one |
||
|
|
8f438b0674 |
Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude - It will also install rtk-ai locally - rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code |