mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:07:29 +08:00
## Description In Claude Code sessions that load deferred tools, the Anthropic token-mode pipeline freezes only the messages before the first ToolSearch result, however far the provider's cache has moved. The log shows `Pipeline: freezing first 11/166 messages`, and the count stays there for the rest of the session. Every turn then re-runs compression over the whole history after that point, only for `finalize_turn` to overwrite the result with its replay of the provider-confirmed prefix. Read maturation also trusts that freeze, so it matures Reads the provider has already cached and the cache re-writes from the Read onward (#4522). Root cause: `CompressionCache.compute_frozen_count` (`headroom/cache/compression_cache.py:293`) stops at a tool_result whose extracted text is None (`compression_cache.py:323`, "treat as unstable"). `_extract_text_from_blocks` returns None for a block list without text (`compression_cache.py:52`): a ToolSearch `tool_reference` list, or an image. Nothing ever stores such a result or marks it stable, because `update_from_result` and `mark_stable_from_messages` skip None too. So it stays a miss for the whole session, and `prepare_turn`'s `min(tracker_frozen, cache_count)` (`headroom/proxy/session_engine.py:134`) clamps every turn's freeze to it. With this change, a tool_result message with no text in any of its tool_results counts as stable: the cache has nothing to replay and the text pipeline has nothing to compress. A message that mixes a textless first result with a text one still stops the count. The cache keys a message by its first tool_result, so that text was never stored and cannot be byte-replayed; fixing that means keying every tool_result, which belongs in its own change. What it changes: in such sessions the pipeline freeze follows the provider-confirmed prefix again, so the pipeline compresses only the new turn instead of re-running the whole history each turn. Forwarded bytes inside the confirmed prefix are unchanged, because `finalize_turn` already replays them. Compression of new content is unchanged. Evidence from one Extra Headroom user's macOS proxy (wheel 0.41.0, token mode): - 299 of the 484 Anthropic requests in its current log froze more than 12 messages below the request size. - In a sample of that proxy's feed, every long session stopped at a ToolSearch result. Those holding only `tool_reference` blocks (11/139, 11/166, 11/173) are fixed here. One whose ToolSearch result shares a message with a text result (13/566 to 13/573) is the mixed case this PR leaves alone. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/compression_cache.py`: `compute_frozen_count` treats a tool_result message with no text as stable, through a new `_carries_tool_result_text` helper (both formats). It still stops at text it cannot replay, including text beside a textless first result. - `tests/test_compression_cache.py`: `tool_reference`, image and empty results no longer stop the count, and a text result beside a textless one still does. - `tests/test_session_engine.py`: `prepare_turn` follows the tracker past a ToolSearch result. The new tests fail on main. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev --extra proxy pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_token_headroom_mode.py -q 90 passed in 4.28s # without the change, the 4 new tests fail $ uv run --frozen --extra dev --extra proxy pytest <the 22 tests that fail in a full local run on main> 16 failed, 6 passed # the same 16 fail on main (f10725e6) on this machine: tests/test_proxy_uds.py (11), # tests/test_cli/test_wrap_persistent.py (4), test_image_compression.py::TestOnnxRouter::test_full_classify_with_image # The full suite was not run locally on this branch; CI runs it. $ ruff check . && ruff format --check . All checks passed! 1834 files already formatted $ mypy headroom --ignore-missing-imports Found 2 errors in 2 files (checked 578 source files) # the same 2 as main, in files this PR does not touch: # integrations/langchain/chat_model.py:380, memory/adapters/embedders.py:389 ``` ## Real Behavior Proof - Environment: macOS 27 on an M2; this branch and upstream main (f10725e6), each from its own worktree; `headroom proxy` started with Extra Headroom's flags and env (token mode, read maturation on via the beta channel, `--no-http2 --no-rate-limit --log-messages`, isolated HOME, telemetry off), in front of a local mock Anthropic upstream that reports cache_read for the longest message prefix shared with an earlier request and cache_write for the rest. - Exact command / steps: one scripted Claude Code session per build: a ToolSearch answered with a `tool_reference` block, a Read of a 7.7 KB file, then 8 Bash turns, each a 300-entry JSON listing that the router compresses. The pipeline freeze per request comes from the `Pipeline: freezing first N/M` log lines. Each forwarded request is diffed against the previous one with `cache_control` stripped. - Observed result: the freeze follows the confirmed prefix instead of staying at the ToolSearch result, and on the last turn the router processes 1 content-block message instead of 17. | Build | Pipeline freeze per request | Router content-block msgs, last turn | Cached prefix changed | cache_read/forwarded | Cache writes | tok_saved/tok_before | |---|---|---|---|---|---|---| | main | 2, 2, 2, 2, 2, 2, 2, 2, 2 | 17 | request 8, message 4 (a Read matured) | 67.0% | 39,360 | 43.2% | | this PR | 2, 4, 4, 8, 10, 12, 14, 16, 18 | 1 | never | 79.5% | 25,994 | 42.2% | - Not tested: Linux and Windows; real Anthropic traffic on this branch; the OpenAI token path and `/v1/compress`, which use the replayable policy, where a larger count only freezes more and cannot bust (their suites pass); a session where nothing compresses, where the freeze moves on only as far as the first uncompressed text result (2 to 4 in the same session with short Bash output), which is why #4522 is needed as well. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, not rollout-managed - Stable/default behavior changed: yes; in token mode the Anthropic pipeline now freezes past a tool_result with no text, up to the provider-confirmed prefix, instead of stopping there for the session. - Kill switch / disable path: none; revert the commit - Unsafe override required: no - Qualification impact: none expected; forwarded bytes inside the confirmed prefix were already replayed by finalize_turn - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Related: #4522 is the fix for the maturation rewrite and works on its own; this PR removes one of the two reasons the freeze sticks. The other is a text tool_result that never went through a pipeline run that changed something: `update_from_result` only marks unchanged results stable on such a run (`headroom/proxy/handlers/anthropic.py`, `if result.messages != working_messages`). That is a broader change, left for its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4523" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
279 lines
11 KiB
Python
279 lines
11 KiB
Python
"""Unit tests for the shared session-turn engine (headroom/proxy/session_engine).
|
|
|
|
The engine is the single cache-management brain for the proxy request paths
|
|
and the sidecar /v1/compress path; these tests pin its two freeze policies
|
|
and the overlay finalization directly, without an HTTP harness.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.compression_cache import CompressionCache
|
|
from headroom.proxy.session_engine import (
|
|
FREEZE_POLICY_CONFIRMED_CLAMP,
|
|
FREEZE_POLICY_REPLAYABLE,
|
|
finalize_turn,
|
|
prepare_turn,
|
|
)
|
|
|
|
|
|
def _tool_msg(content: str, call_id: str = "c1") -> dict:
|
|
return {"role": "tool", "tool_call_id": call_id, "content": content}
|
|
|
|
|
|
def _history_with_cached_tool(
|
|
cache: CompressionCache, original: str, compressed: str
|
|
) -> list[dict]:
|
|
"""A 3-message history whose tool result has a cached compressed form."""
|
|
cache.store_compressed(cache.content_hash(original), compressed, tokens_saved=10)
|
|
return [
|
|
{"role": "user", "content": "get items"},
|
|
{"role": "assistant", "content": "calling"},
|
|
_tool_msg(original),
|
|
]
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# prepare_turn: freeze policies #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def test_sidecar_policy_freezes_full_replayable_prefix() -> None:
|
|
cache = CompressionCache()
|
|
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
|
|
messages.append({"role": "user", "content": "next"})
|
|
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
|
|
# user, assistant, cached tool are all stable; the trailing message is
|
|
# always excluded by compute_frozen_count.
|
|
assert prep.frozen_message_count == 3
|
|
# The swap replaced the tool result with its cached compressed form.
|
|
assert prep.pipeline_input[2]["content"] == "[compressed]"
|
|
# The caller's list is never mutated.
|
|
assert messages[2]["content"].startswith("ORIGINAL")
|
|
|
|
|
|
def test_sidecar_policy_explicit_pin_wins_when_larger() -> None:
|
|
cache = CompressionCache()
|
|
messages = [
|
|
{"role": "user", "content": "a"},
|
|
_tool_msg("never seen before " * 50), # not in cache -> derived stops here
|
|
{"role": "user", "content": "next"},
|
|
]
|
|
derived = cache.compute_frozen_count(messages)
|
|
assert derived == 1 # only the leading plain message
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE, explicit_frozen=2)
|
|
assert prep.frozen_message_count == 2
|
|
|
|
|
|
def test_sidecar_policy_derived_wins_when_explicit_smaller() -> None:
|
|
cache = CompressionCache()
|
|
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
|
|
messages.append({"role": "user", "content": "next"})
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE, explicit_frozen=1)
|
|
assert prep.frozen_message_count == 3
|
|
|
|
|
|
def test_proxy_policy_clamps_by_cache_count() -> None:
|
|
"""Provider says 5 messages are cached, but local state can only replay 3:
|
|
freezing past the replayable bound would forward raw bytes."""
|
|
cache = CompressionCache()
|
|
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
|
|
messages.append(_tool_msg("uncached " * 50, "c2"))
|
|
messages.append({"role": "user", "content": "next"})
|
|
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=5)
|
|
assert prep.frozen_message_count == 3
|
|
|
|
|
|
def test_proxy_policy_follows_the_tracker_past_a_tool_search_result() -> None:
|
|
"""A ToolSearch result holds only tool_reference blocks. It used to stop
|
|
the cache count, so the freeze sat there for the whole session and the
|
|
pipeline re-ran everything after it on every turn."""
|
|
cache = CompressionCache()
|
|
search = {
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "ts",
|
|
"content": [{"type": "tool_reference", "tool_name": "mcp__notes__search"}],
|
|
}
|
|
],
|
|
}
|
|
messages = [{"role": "user", "content": "find the notes tool"}, search]
|
|
messages += [{"role": "assistant", "content": f"step {i}"} for i in range(6)]
|
|
messages.append({"role": "user", "content": "next"})
|
|
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=8)
|
|
assert prep.frozen_message_count == 8
|
|
|
|
|
|
def test_proxy_policy_clamps_by_tracker() -> None:
|
|
"""Local state could replay 3, but the provider only confirmed 1: content
|
|
past the confirmed prefix stays compressible (the #327 posture)."""
|
|
cache = CompressionCache()
|
|
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
|
|
messages.append({"role": "user", "content": "next"})
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=1)
|
|
assert prep.frozen_message_count == 1
|
|
|
|
|
|
def test_proxy_policy_none_tracker_freezes_nothing() -> None:
|
|
cache = CompressionCache()
|
|
messages = _history_with_cached_tool(cache, "ORIGINAL " * 100, "[compressed]")
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_CONFIRMED_CLAMP, tracker_frozen=None)
|
|
assert prep.frozen_message_count == 0
|
|
|
|
|
|
def test_unknown_policy_rejected() -> None:
|
|
cache = CompressionCache()
|
|
with pytest.raises(ValueError):
|
|
prepare_turn(cache, [], policy="wat")
|
|
|
|
|
|
def test_prepare_marks_frozen_tool_results_stable() -> None:
|
|
cache = CompressionCache()
|
|
original = "ORIGINAL " * 100
|
|
messages = _history_with_cached_tool(cache, original, "[compressed]")
|
|
messages.append({"role": "user", "content": "next"})
|
|
prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
|
|
assert cache.content_hash(original) in cache._stable_hashes
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# finalize_turn: overlay + recount hook #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def _prev_pair() -> tuple[list[dict], list[dict]]:
|
|
prev_original = [
|
|
{"role": "user", "content": "ORIGINAL " * 100},
|
|
{"role": "assistant", "content": "ok"},
|
|
]
|
|
prev_returned = [
|
|
{"role": "user", "content": "[returned-form]"},
|
|
{"role": "assistant", "content": "ok"},
|
|
]
|
|
return prev_original, prev_returned
|
|
|
|
|
|
def test_finalize_replays_previous_returned_prefix() -> None:
|
|
prev_original, prev_returned = _prev_pair()
|
|
current = prev_original + [{"role": "user", "content": "next"}]
|
|
# The pipeline "drifted": it emitted the raw original for message 0.
|
|
drifted = [dict(m) for m in current]
|
|
|
|
counted: list[int] = []
|
|
|
|
def _count(msgs: list[dict]) -> int:
|
|
counted.append(len(json.dumps(msgs)))
|
|
return 42
|
|
|
|
turn = finalize_turn(drifted, current, prev_original, prev_returned, count_tokens=_count)
|
|
assert turn.replayed
|
|
assert turn.messages[0]["content"] == "[returned-form]"
|
|
assert turn.messages[-1]["content"] == "next"
|
|
assert turn.tokens == 42
|
|
assert len(counted) == 1
|
|
|
|
|
|
def test_finalize_noop_without_prev_snapshots() -> None:
|
|
current = [{"role": "user", "content": "hi"}]
|
|
calls: list[int] = []
|
|
turn = finalize_turn(current, current, [], [], count_tokens=lambda m: calls.append(1) or 1)
|
|
assert not turn.replayed
|
|
assert turn.messages == current
|
|
assert turn.tokens is None
|
|
assert not calls # count_tokens only runs when the overlay fired
|
|
|
|
|
|
def test_finalize_count_hook_failure_falls_back() -> None:
|
|
prev_original, prev_returned = _prev_pair()
|
|
current = prev_original + [{"role": "user", "content": "next"}]
|
|
|
|
def _boom(_msgs: list[dict]) -> int:
|
|
raise RuntimeError("tokenizer down")
|
|
|
|
turn = finalize_turn(
|
|
[dict(m) for m in current], current, prev_original, prev_returned, count_tokens=_boom
|
|
)
|
|
assert turn.replayed
|
|
assert turn.tokens is None
|
|
|
|
|
|
def test_finalize_declines_inflating_replay_by_default() -> None:
|
|
prev_original, prev_returned = _prev_pair()
|
|
current = prev_original + [{"role": "user", "content": "next"}]
|
|
# The pipeline recompressed message 0 SMALLER than the returned form.
|
|
recompressed = [
|
|
{"role": "user", "content": "[t]"},
|
|
{"role": "assistant", "content": "ok"},
|
|
{"role": "user", "content": "next"},
|
|
]
|
|
turn = finalize_turn(recompressed, current, prev_original, prev_returned)
|
|
assert not turn.replayed
|
|
assert turn.messages == recompressed
|
|
|
|
|
|
def test_finalize_confirmed_floor_replays_confirmed_bytes() -> None:
|
|
prev_original, prev_returned = _prev_pair()
|
|
current = prev_original + [{"role": "user", "content": "next"}]
|
|
recompressed = [
|
|
{"role": "user", "content": "[t]"},
|
|
{"role": "assistant", "content": "ok"},
|
|
{"role": "user", "content": "next"},
|
|
]
|
|
turn = finalize_turn(
|
|
recompressed, current, prev_original, prev_returned, confirmed_frozen_count=2
|
|
)
|
|
assert turn.replayed
|
|
assert turn.messages[0]["content"] == "[returned-form]"
|
|
assert turn.messages[-1]["content"] == "next"
|
|
|
|
|
|
# --------------------------------------------------------------------------- #
|
|
# OpenAI proxy token-path migration: formula identity + marking benefit. #
|
|
# --------------------------------------------------------------------------- #
|
|
|
|
|
|
def test_replayable_without_pin_equals_bare_cache_count() -> None:
|
|
"""The OpenAI proxy token path historically froze on compute_frozen_count
|
|
alone; REPLAYABLE with no explicit pin must be formula-identical, so its
|
|
migration onto the engine is a pure extraction."""
|
|
cache = CompressionCache()
|
|
messages = _history_with_cached_tool(cache, "AAAA " * 50, "[c1]")
|
|
messages.append(_tool_msg("uncached content", call_id="c2"))
|
|
messages.append({"role": "user", "content": "next"})
|
|
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
|
|
assert prep.frozen_message_count == cache.compute_frozen_count(messages)
|
|
# And that count stops at the uncached tool_result (index 3).
|
|
assert prep.frozen_message_count == 3
|
|
|
|
|
|
def test_marking_preserves_freeze_across_entry_eviction() -> None:
|
|
"""The one real benefit mark_stable_from_messages adds on the migrated
|
|
path: an in-prefix tool_result stays stable via `_stable_hashes` even
|
|
after its compressed ENTRY is evicted by the per-cache LRU, so the frozen
|
|
count does not collapse at that position on the next turn."""
|
|
cache = CompressionCache(max_entries=100)
|
|
original = "BBBB " * 50
|
|
messages = _history_with_cached_tool(cache, original, "[c1]")
|
|
messages.append({"role": "user", "content": "next"})
|
|
|
|
prep = prepare_turn(cache, messages, policy=FREEZE_POLICY_REPLAYABLE)
|
|
assert prep.frozen_message_count == 3 # tool in prefix, marked stable
|
|
|
|
# Simulate entry LRU turnover: the compressed entry disappears.
|
|
h = cache.content_hash(original)
|
|
with cache._lock:
|
|
cache._cache.pop(h, None)
|
|
|
|
# Without marking, the frozen count would collapse to 2 here; the
|
|
# stable-hash record keeps the position frozen.
|
|
assert cache.compute_frozen_count(messages) == 3
|