mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
fix/plugin-launcher-isolated-python
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
fba09f674c |
fix(cache): count a tool_result with no text as replayable in the frozen prefix (#4523)
## Description In Claude Code sessions that load deferred tools, the Anthropic token-mode pipeline freezes only the messages before the first ToolSearch result, however far the provider's cache has moved. The log shows `Pipeline: freezing first 11/166 messages`, and the count stays there for the rest of the session. Every turn then re-runs compression over the whole history after that point, only for `finalize_turn` to overwrite the result with its replay of the provider-confirmed prefix. Read maturation also trusts that freeze, so it matures Reads the provider has already cached and the cache re-writes from the Read onward (#4522). Root cause: `CompressionCache.compute_frozen_count` (`headroom/cache/compression_cache.py:293`) stops at a tool_result whose extracted text is None (`compression_cache.py:323`, "treat as unstable"). `_extract_text_from_blocks` returns None for a block list without text (`compression_cache.py:52`): a ToolSearch `tool_reference` list, or an image. Nothing ever stores such a result or marks it stable, because `update_from_result` and `mark_stable_from_messages` skip None too. So it stays a miss for the whole session, and `prepare_turn`'s `min(tracker_frozen, cache_count)` (`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to it. With this change, a tool_result message with no text in any of its tool_results counts as stable: the cache has nothing to replay and the text pipeline has nothing to compress. A message that mixes a textless first result with a text one still stops the count. The cache keys a message by its first tool_result, so that text was never stored and cannot be byte-replayed; fixing that means keying every tool_result, which belongs in its own change. What it changes: in such sessions the pipeline freeze follows the provider-confirmed prefix again, so the pipeline compresses only the new turn instead of re-running the whole history each turn. Forwarded bytes inside the confirmed prefix are unchanged, because `finalize_turn` already replays them. Compression of new content is unchanged. Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token mode): - 299 of the 484 Anthropic requests in its current log froze more than 12 messages below the request size. - In a sample of that proxy's feed, every long session stopped at a ToolSearch result. Those holding only `tool_reference` blocks (11/139, 11/166, 11/173) are fixed here. One whose ToolSearch result shares a message with a text result (13/566 to 13/573) is the mixed case this PR leaves alone. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a tool_result message with no text as stable, through a new `_carries_tool_result_text` helper (both formats). It still stops at text it cannot replay, including text beside a textless first result. - `tests/test_compression_cache.py`: `tool_reference`, image and empty results no longer stop the count, and a text result beside a textless one still does. - `tests/test_session_engine.py`: `prepare_turn` follows the tracker past a ToolSearch result. The new tests fail on main. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q 90 passed in 4.28s # without the change, the 4 new tests fail $ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main> 16 failed, 6 passed # the same 16 fail on main ( |
||
|
|
aebe98954e |
fix(cache): keep cached-prefix replay after background recompression (#3380)
## Description Since the non-inflation size bound added in #3052 (shipped 0.36.x), the Anthropic proxy token path repeatedly busts the provider prompt cache: the bound declines the cached-prefix replay exactly when background compression lands a SMALLER form of an already-forwarded message, so the forwarded bytes change mid-history and the provider cache misses from the first changed byte. Steady state on a busy session is a 160-210k-token cache re-write every 1-2 requests (measured across a 0.35.0 -> 0.37.0 swap on identical workload: dollar cache-hit rate 90% -> 52%, billable input $/M up 2.2x; full data in #3379). Reworked per review (thanks @chopratejas - the direction is yours): rather than disabling the size bound wholesale, `overlay_cached_prefix` now takes `confirmed_frozen_count` and splits at that floor. - Inside the provider-confirmed prefix: replay is unconditional. The replay source is exactly what the provider hashed, so changing those bytes can only bust the cache; an "inflating" replay there bills at the ~0.1x cache-read rate and is still the cheaper request. - Beyond the floor: the size bound arbitrates each turn exactly as on main - a shrinking replay repairs freeze drift (#1850), an inflating one is declined so a fresh compression improvement reaches the wire. - When the provider count collapses (cold cache, TTL lapse) the floor collapses with it and every accumulated improvement lands at once: the natural re-baselining that bounds long-session growth (the #3026 direction of travel) by the TTL-lapse cadence instead of pinning forever. The #3052 alignment/append-only guards stay enforced unconditionally. The OpenAI chat path now carries the same floor (reviewer question: is the root cause live there too?). It is: the tracker is provider-generic and is fed OpenAI's own `prompt_tokens_details.cached_tokens`, the forwarded-bytes feedback loop is identical, both handlers compress through the same ContentRouter, and OpenAI's automatic prompt caching busts on the first changed byte exactly like Anthropic's. Exposure is narrower - the token path's `FREEZE_POLICY_REPLAYABLE` already stops most recompression of already-forwarded messages - but not zero: an entry evicted from the local byte-replay cache drops out of that freeze, and cache mode freezes from the tracker only. Remaining callers with no provider-confirmed count in scope (sidecar, `openai.py:9940`) pass nothing and keep today's fully size-bounded behavior; `bedrock.py` / `gemini.py` never call `finalize_turn` at all, tracked in #3391. Closes #3379 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `overlay_cached_prefix` gains keyword-only `confirmed_frozen_count: int | None = None`: an unconditional-replay floor for the provider-confirmed prefix; the size bound still governs everything beyond it, and alignment guards are untouched. - `finalize_turn` forwards the floor (default `None`, so all existing callers are unchanged). - The OpenAI chat call site passes the same floor, snapshotted before `prepare_turn`'s replayable clamp (that count runs over the LOCAL byte-replay cache and can exceed what the provider confirmed; a cold prefix zeroes it, so cold re-baselining matches Anthropic's). - The Anthropic confirmed-clamp call site stashes the pre-clamp tracker count (the post-`prepare_turn` min under-floors when the local byte-replay cache evicted entries) and passes it; the stash defaults to 0 on paths that never compute a tracker count (backpressure). - Tests: floor replays a recompressed confirmed prefix; improvement beyond the floor lands while the confirmed region replays; originals-drift beyond the floor is still repaired when the replay shrinks; alignment guards unaffected by the floor; block-append within the floor; `finalize_turn` pass-through; the reviewer-requested multi-turn harness test - a background improvement to an already-forwarded message never busts the warm cache across 8 turns and lands on the first cold turn; the reviewer-requested growth bound - forwarded size grows by only its own delta each warm turn and the first cold turn re-baselines to exactly the fresh pipeline's output; and an end-to-end OpenAI chat test asserting the provider-confirmed prefix is replayed even when the replay inflates (fails without the new argument). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py -q ============================== 58 passed in 0.44s ============================== $ uvx ruff check headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py headroom/proxy/handlers/anthropic.py tests/test_cache_prefix_overlay.py tests/test_session_engine.py tests/test_cross_turn_cache_safety.py All checks passed! $ uv run --frozen --extra dev mypy headroom/cache/prefix_tracker.py headroom/proxy/session_engine.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.12; regression measured on the released headroom-ai 0.37.0 under real Claude Code traffic through `headroom proxy` (token mode); this branch validated with the repo test suite. - Exact command / steps: diffed per-request PERF logs across a 0.35.0 -> 0.37.0 swap on the same machine and workload (sequences and aggregates in #3379); then ran the pytest/ruff/mypy commands above on this branch. A deployment carrying the unconditional-replay behavior on the confirmed prefix restored the healthy pattern live: the broken wheel capped cached reads at 157k across 32 large requests (prefix perpetually re-written) while the patched deployment accumulated to 305k cached reads within 15 minutes. - Observed result: 58 passed including the new multi-turn harness test; on stock 0.37.0 every background-compression completion produced a cache_read collapse to the ~7.7k tools+system head plus a 160-210k cache_write. - Not tested: a live OpenAI `cached_tokens` comparison for the OpenAI half (the Anthropic measurement is what #3379 carries; the OpenAI change is argued from shared mechanism plus an end-to-end handler test, not from its own dollar data). A live provider run of this exact branch (the wheel comparison plus the live patched-deployment measurement isolate the bound as the trigger; the unit and multi-turn regressions reproduce decline and fix deterministically). `mypy headroom` was run only on the two touched modules, not the whole package. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: On the Anthropic proxy confirmed-clamp path and the OpenAI chat path, both of which now replay the provider-confirmed prefix unconditionally (the pre-0.36 posture) while keeping the size bound beyond the confirmed floor. The overlay's default and every other caller keep the fully size-bounded behavior. - Kill switch / disable path: remove the `confirmed_frozen_count=` argument at the two call sites (anthropic.py, openai.py chat); the parameter defaults to `None`. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert the commits; the parameter is additive and default-preserving. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes Lineage for review: #1850 made freeze forward the cached prefix byte-identical, #3052 added alignment guards plus the size bound after #3026's misalignment blowup, #3271 unified both modes on the session engine. The first cut of this PR disabled the bound on the confirmed-clamp path wholesale; the reviewed rework scopes unconditional replay to the provider-confirmed prefix, which additionally gives cold-lapse re-baselining (improvements land whenever the floor collapses), addressing the growth analysis raised in review. Economics discussion in the review thread. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
d12ea50122 |
feat(proxy): unify proxy and sidecar compression on one session engine (#3271)
> Replaces #3263 (same changeset, squashed to one conventional commit — after the base PRs squash-merged, the stacked branch's commit history could not pass the commitlint gate against main, and force-pushing the original branch was not permitted). #3261 and #3270 (which replaced #3262) are merged; this is the last piece of the stack. ## Goal One brain. The cache-management tier — freeze computation, Zone-1 byte swap, cached-prefix overlay — previously existed twice: inline in the proxy request handlers, and (as of #3270) in the `/v1/compress` sidecar path. This PR extracts it into **`headroom/proxy/session_engine.py`**, invoked by BOTH. Every future cache-management fix lands in both modes by construction. ## Design **`prepare_turn(...)` → `TurnPrep`** — freeze + `mark_stable_from_messages` + `apply_cached`, with two *deliberately different, documented* freeze policies: - `FREEZE_POLICY_CONFIRMED_CLAMP`: `min(tracker_frozen, cache_count)` — never freeze past provider-confirmed (the #327 posture). The Anthropic proxy passes its already-composed tracker/strict-override value, reproducing the previous `min()` byte-for-byte. - `FREEZE_POLICY_REPLAYABLE`: `max(cache_count, explicit)` — freeze everything locally replayable, because whatever was previously returned *is* the provider's cache contract; recompressing it (even "better") busts. **`finalize_turn(...)` → `TurnFinal`** — the byte-identical cached-prefix replay (`overlay_cached_prefix`) + conditional token recount hook. Run as a **strictly behavior-preserving extraction**: the bar was every pre-existing test passing *unmodified*, and it held. ## What migrated | Path | Status | |---|---| | `/v1/compress` sidecar turn | ✅ engine (REPLAYABLE); lock, executor offload, savings accounting, record_returned unchanged | | `anthropic.py` token-mode pre-block + overlay | ✅ engine (CONFIRMED_CLAMP); background compression, cold-start fast pass, `_cold_recompact_active` skip preserved | | `openai.py` proxy token-mode pre-block + overlay | ✅ engine (REPLAYABLE — formula-identical to the old bare `compute_frozen_count`); the added `mark_stable` call means the freeze now survives entry-level LRU eviction (test-pinned); the router's `_frozen_verdicts` remains the boundary-message protection | | `openai.py` cache-mode branch | ⏸ keeps bare `apply_cached` — cache mode keeps the latest observation mutable by design | Also fixed for BOTH handlers: overlay replay now runs under backpressure (shedding it busted every gated session's prompt cache exactly at peak load), and the inflation guard exempts replayed prefixes. ## Hardening (max-effort review, all applied) `/v1/usage` applies on the executor under the per-session turn lock with a timed acquire (503 `session_busy`); registry eviction skips sessions mid-turn; `peek()` is expiry-aware; silent fallbacks log warnings; RequestOutcome recorded on session 503s. ## Testing - `tests/test_session_engine.py`: 13 direct unit tests — both policies, explicit-pin precedence, REPLAYABLE-without-pin ≡ bare `compute_frozen_count`, overlay fires/doesn't, recount only on replay, freeze-survives-entry-eviction. - Parity bar: full pre-existing suites pass unmodified — cache-stability (Anthropic + OpenAI), overlay, backpressure (incl. replay-under-saturation regression), cold-start fast pass, cache-mode, session-mode byte-stability, compress-API, org-scale, registry. Full local suite: 11k+ green. - ruff check/format clean (CI's ruff 0.16.3). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |