mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(router): compress tool_result blocks in mutable window with trailing messages (#4717)
## Description
In Anthropic conversation structures with client tools (such as Claude
Code or custom tool harnesses), user tool_result content blocks are
frequently accompanied by trailing follow-up prompt text or subsequent
turn messages within the current mutable turn. When
prefix_replay_guaranteed=True was active,
ContentRouter._process_content_blocks() evaluated fresh turn status
strictly via messages_from_end == 1. Because the tool execution message
was not the final element in the array (messages_from_end > 1), the
router treated it as frozen prompt cache history, leaving large
tool_result blocks uncompressed.
This change determines freshness based on membership in the mutable turn
suffix (i >= mutable_turn_start, where mutable_turn_start =
max(frozen_message_count, last_assistant_index + 1)). Tool results in
the current un-forwarded turn are now compressed into dense
representations while preserving prompt cache integrity for all frozen
prefix messages (i < frozen_message_count) and leaving preceding
completed turns untouched.
Closes #4708
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Calculated last_assistant_index and mutable_turn_start =
max(frozen_message_count, last_assistant_index + 1) in
ContentRouter.apply().
- Passed in_mutable_window=(i >= mutable_turn_start) to
ContentRouter._process_content_blocks().
- Updated fresh_turn calculation to evaluate in_mutable_window when
present, releasing tool_result blocks in the mutable turn suffix even
when trailing prompt messages exist.
- Added regression tests in tests/test_content_router.py covering tool
result compression with trailing messages and verifying frozen prefix
immutability under frozen_message_count.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_content_router.py tests/test_transforms_content_router.py
======================== 107 passed, 1 skipped in 111.42s ======================
$ uv run ruff check headroom/transforms/content_router.py tests/test_content_router.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.9, native PyO3 Rust extension
built via maturin, model claude-3-7-sonnet-20250219.
- Exact command / steps: Ran ContentRouter with 500 log lines followed
by trailing message under prefix_replay_guaranteed.
- Observed result: Tokens before: 3070 -> Tokens after: 127 (saved 2943
tokens, 95.8% reduction on repetitive logs). FATAL disk full preserved.
Frozen prefix unchanged under frozen_message_count=3.
- Not tested: Live proxy network transmission to production Anthropic
API endpoints.
## Runtime Rollout Safety
- Rollout-managed feature(s): Router transform logic for Anthropic
content blocks.
- Minimum rollout channel: N/A
- Stable/default behavior changed: No (affects only Anthropic tool
blocks in the mutable turn suffix under prefix_replay_guaranteed=True;
preserves existing cache contract).
- Kill switch / disable path: Omitting prefix_replay_guaranteed or
specifying frozen_message_count.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Clean git revert of commit fc595554.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit CHANGELOG.md — it is generated by release-please
from my Conventional Commit PR title (a CI guard enforces this)
## Additional Notes
Co-authored-by: MartinMayday <MartinMayday@users.noreply.github.com>
Co-authored-by: MartinMayday <MartinMayday@users.noreply.github.com>
This commit is contained in:
co-authored by
MartinMayday
parent
3ba422361e
commit
0511bcd331
@@ -6559,6 +6559,14 @@ class ContentRouter(Transform):
|
||||
),
|
||||
num_messages,
|
||||
)
|
||||
last_assistant_index = max(
|
||||
(
|
||||
idx
|
||||
for idx, msg in enumerate(messages)
|
||||
if isinstance(msg, dict) and msg.get("role") == "assistant"
|
||||
),
|
||||
default=-1,
|
||||
)
|
||||
model_limit = kwargs.get("model_limit", 0)
|
||||
|
||||
# Adaptive Read protection: protect a fraction of recent messages
|
||||
@@ -6668,6 +6676,7 @@ class ContentRouter(Transform):
|
||||
analysis_intent = self._detect_analysis_intent(messages)
|
||||
|
||||
frozen_message_count = kwargs.get("frozen_message_count", 0)
|
||||
mutable_turn_start = max(frozen_message_count, last_assistant_index + 1)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Two-pass parallel compression.
|
||||
@@ -6823,6 +6832,7 @@ class ContentRouter(Transform):
|
||||
compress_assistant_text_blocks=compress_assistant_text_blocks,
|
||||
prefix_replay_guaranteed=prefix_replay_guaranteed,
|
||||
protect_prompt_text=prompt_turn,
|
||||
in_mutable_window=(i >= mutable_turn_start),
|
||||
)
|
||||
result_slots[i] = transformed_message
|
||||
route_counts["content_blocks"] += 1
|
||||
@@ -7885,6 +7895,7 @@ class ContentRouter(Transform):
|
||||
compress_assistant_text_blocks: bool = False,
|
||||
prefix_replay_guaranteed: bool = False,
|
||||
protect_prompt_text: bool = False,
|
||||
in_mutable_window: bool | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Process content blocks (Anthropic format) for compression.
|
||||
|
||||
@@ -7987,14 +7998,13 @@ class ContentRouter(Transform):
|
||||
else:
|
||||
protect_text_blocks = True
|
||||
|
||||
# The final user/tool message is this turn's fresh content: not yet
|
||||
# forwarded, so its cache_control marker names a key the provider has
|
||||
# not written. Everything earlier may already be cached under its
|
||||
# marker and stays byte-exact (contract 1). Only tool_result blocks
|
||||
# are released; a marked text block is the user's prompt.
|
||||
fresh_turn = (
|
||||
prefix_replay_guaranteed and messages_from_end == 1 and role in ("user", "tool")
|
||||
)
|
||||
# User/tool messages in the mutable turn suffix are this turn's
|
||||
# fresh content: not yet forwarded, so a cache_control marker names a key
|
||||
# the provider has not written. Everything earlier may already be cached
|
||||
# under its marker and stays byte-exact (contract 1). Only tool_result
|
||||
# blocks are released; a marked text block is the user's prompt.
|
||||
is_fresh = in_mutable_window if in_mutable_window is not None else (messages_from_end == 1)
|
||||
fresh_turn = prefix_replay_guaranteed and is_fresh and role in ("user", "tool")
|
||||
|
||||
for block in content_blocks:
|
||||
if not isinstance(block, dict):
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.transforms.content_router import (
|
||||
CompressionStrategy,
|
||||
ContentRouter,
|
||||
ContentRouterConfig,
|
||||
)
|
||||
|
||||
|
||||
class _TestTokenizer:
|
||||
def count_tokens(self, text: str) -> int:
|
||||
return max(1, len(text) // 4)
|
||||
|
||||
def count_text(self, text: str) -> int:
|
||||
return max(1, len(text) // 4)
|
||||
|
||||
def count_message_tokens(self, msg: dict) -> int:
|
||||
return 10
|
||||
|
||||
|
||||
def test_tool_result_compressed_with_trailing_messages_in_mutable_window(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
|
||||
def fake_compress(content, context="", bias=1.0, precomputed_detection=None):
|
||||
return SimpleNamespace(
|
||||
compressed=content[: len(content) // 2] + "[compressed]",
|
||||
compression_ratio=0.5,
|
||||
strategy_used=CompressionStrategy.LOG,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(router, "compress", fake_compress)
|
||||
tool_output = "tool output line " * 200
|
||||
|
||||
# Issue #4708: Anthropic tool_result block followed by trailing message in the mutable turn.
|
||||
messages = [
|
||||
{"role": "user", "content": "read the module"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": "t1", "name": "bash", "input": {}}],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "t1",
|
||||
"content": tool_output,
|
||||
"cache_control": {"type": "ephemeral"},
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "what failed in the test above?"},
|
||||
]
|
||||
|
||||
result = router.apply(
|
||||
messages,
|
||||
_TestTokenizer(),
|
||||
model_limit=100_000,
|
||||
prefix_replay_guaranteed=True,
|
||||
)
|
||||
compressed_block = result.messages[2]["content"][0]
|
||||
assert compressed_block["content"].endswith("[compressed]"), (
|
||||
"tool_result followed by trailing message must be compressed in mutable turn"
|
||||
)
|
||||
assert compressed_block["cache_control"] == {"type": "ephemeral"}
|
||||
|
||||
|
||||
def test_tool_result_in_frozen_prefix_not_mutated(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
|
||||
def fake_compress(content, context="", bias=1.0, precomputed_detection=None):
|
||||
return SimpleNamespace(
|
||||
compressed=content[: len(content) // 2] + "[compressed]",
|
||||
compression_ratio=0.5,
|
||||
strategy_used=CompressionStrategy.LOG,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(router, "compress", fake_compress)
|
||||
tool_output = "tool output line " * 200
|
||||
|
||||
# Turn 1 tool_result is part of frozen prefix (frozen_message_count=3).
|
||||
messages = [
|
||||
{"role": "user", "content": "read the module"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": "t1", "name": "bash", "input": {}}],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "t1",
|
||||
"content": tool_output,
|
||||
"cache_control": {"type": "ephemeral"},
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "assistant", "content": "tests failed with error"},
|
||||
{"role": "user", "content": "now fix it"},
|
||||
]
|
||||
|
||||
result = router.apply(
|
||||
messages,
|
||||
_TestTokenizer(),
|
||||
model_limit=100_000,
|
||||
frozen_message_count=3,
|
||||
prefix_replay_guaranteed=True,
|
||||
)
|
||||
# Message 2 is frozen (index 2 < frozen_message_count=3), must not be mutated
|
||||
assert result.messages[2]["content"][0]["content"] == tool_output
|
||||
Reference in New Issue
Block a user