fix(router): compress tool_result blocks in mutable window with trailing messages (#4717)

## Description

In Anthropic conversation structures with client tools (such as Claude
Code or custom tool harnesses), user tool_result content blocks are
frequently accompanied by trailing follow-up prompt text or subsequent
turn messages within the current mutable turn. When
prefix_replay_guaranteed=True was active,
ContentRouter._process_content_blocks() evaluated fresh turn status
strictly via messages_from_end == 1. Because the tool execution message
was not the final element in the array (messages_from_end > 1), the
router treated it as frozen prompt cache history, leaving large
tool_result blocks uncompressed.

This change determines freshness based on membership in the mutable turn
suffix (i >= mutable_turn_start, where mutable_turn_start =
max(frozen_message_count, last_assistant_index + 1)). Tool results in
the current un-forwarded turn are now compressed into dense
representations while preserving prompt cache integrity for all frozen
prefix messages (i < frozen_message_count) and leaving preceding
completed turns untouched.

Closes #4708

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Calculated last_assistant_index and mutable_turn_start =
max(frozen_message_count, last_assistant_index + 1) in
ContentRouter.apply().
- Passed in_mutable_window=(i >= mutable_turn_start) to
ContentRouter._process_content_blocks().
- Updated fresh_turn calculation to evaluate in_mutable_window when
present, releasing tool_result blocks in the mutable turn suffix even
when trailing prompt messages exist.
- Added regression tests in tests/test_content_router.py covering tool
result compression with trailing messages and verifying frozen prefix
immutability under frozen_message_count.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_content_router.py tests/test_transforms_content_router.py
======================== 107 passed, 1 skipped in 111.42s ======================

$ uv run ruff check headroom/transforms/content_router.py tests/test_content_router.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.9, native PyO3 Rust extension
built via maturin, model claude-3-7-sonnet-20250219.
- Exact command / steps: Ran ContentRouter with 500 log lines followed
by trailing message under prefix_replay_guaranteed.
- Observed result: Tokens before: 3070 -> Tokens after: 127 (saved 2943
tokens, 95.8% reduction on repetitive logs). FATAL disk full preserved.
Frozen prefix unchanged under frozen_message_count=3.
- Not tested: Live proxy network transmission to production Anthropic
API endpoints.

## Runtime Rollout Safety

- Rollout-managed feature(s): Router transform logic for Anthropic
content blocks.
- Minimum rollout channel: N/A
- Stable/default behavior changed: No (affects only Anthropic tool
blocks in the mutable turn suffix under prefix_replay_guaranteed=True;
preserves existing cache contract).
- Kill switch / disable path: Omitting prefix_replay_guaranteed or
specifying frozen_message_count.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Clean git revert of commit fc595554.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit CHANGELOG.md — it is generated by release-please
from my Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

Co-authored-by: MartinMayday <MartinMayday@users.noreply.github.com>

Co-authored-by: MartinMayday <MartinMayday@users.noreply.github.com>
This commit is contained in:
Mohit Sharma
2026-11-09 14:52:37 -06:00
committed by GitHub
co-authored by MartinMayday
parent 3ba422361e
commit 0511bcd331
2 changed files with 137 additions and 8 deletions
+18 -8
View File
@@ -6559,6 +6559,14 @@ class ContentRouter(Transform):
),
num_messages,
)
last_assistant_index = max(
(
idx
for idx, msg in enumerate(messages)
if isinstance(msg, dict) and msg.get("role") == "assistant"
),
default=-1,
)
model_limit = kwargs.get("model_limit", 0)
# Adaptive Read protection: protect a fraction of recent messages
@@ -6668,6 +6676,7 @@ class ContentRouter(Transform):
analysis_intent = self._detect_analysis_intent(messages)
frozen_message_count = kwargs.get("frozen_message_count", 0)
mutable_turn_start = max(frozen_message_count, last_assistant_index + 1)
# ------------------------------------------------------------------
# Two-pass parallel compression.
@@ -6823,6 +6832,7 @@ class ContentRouter(Transform):
compress_assistant_text_blocks=compress_assistant_text_blocks,
prefix_replay_guaranteed=prefix_replay_guaranteed,
protect_prompt_text=prompt_turn,
in_mutable_window=(i >= mutable_turn_start),
)
result_slots[i] = transformed_message
route_counts["content_blocks"] += 1
@@ -7885,6 +7895,7 @@ class ContentRouter(Transform):
compress_assistant_text_blocks: bool = False,
prefix_replay_guaranteed: bool = False,
protect_prompt_text: bool = False,
in_mutable_window: bool | None = None,
) -> dict[str, Any]:
"""Process content blocks (Anthropic format) for compression.
@@ -7987,14 +7998,13 @@ class ContentRouter(Transform):
else:
protect_text_blocks = True
# The final user/tool message is this turn's fresh content: not yet
# forwarded, so its cache_control marker names a key the provider has
# not written. Everything earlier may already be cached under its
# marker and stays byte-exact (contract 1). Only tool_result blocks
# are released; a marked text block is the user's prompt.
fresh_turn = (
prefix_replay_guaranteed and messages_from_end == 1 and role in ("user", "tool")
)
# User/tool messages in the mutable turn suffix are this turn's
# fresh content: not yet forwarded, so a cache_control marker names a key
# the provider has not written. Everything earlier may already be cached
# under its marker and stays byte-exact (contract 1). Only tool_result
# blocks are released; a marked text block is the user's prompt.
is_fresh = in_mutable_window if in_mutable_window is not None else (messages_from_end == 1)
fresh_turn = prefix_replay_guaranteed and is_fresh and role in ("user", "tool")
for block in content_blocks:
if not isinstance(block, dict):
+119
View File
@@ -0,0 +1,119 @@
from __future__ import annotations
from types import SimpleNamespace
import pytest
from headroom.transforms.content_router import (
CompressionStrategy,
ContentRouter,
ContentRouterConfig,
)
class _TestTokenizer:
def count_tokens(self, text: str) -> int:
return max(1, len(text) // 4)
def count_text(self, text: str) -> int:
return max(1, len(text) // 4)
def count_message_tokens(self, msg: dict) -> int:
return 10
def test_tool_result_compressed_with_trailing_messages_in_mutable_window(
monkeypatch: pytest.MonkeyPatch,
) -> None:
router = ContentRouter(ContentRouterConfig())
def fake_compress(content, context="", bias=1.0, precomputed_detection=None):
return SimpleNamespace(
compressed=content[: len(content) // 2] + "[compressed]",
compression_ratio=0.5,
strategy_used=CompressionStrategy.LOG,
)
monkeypatch.setattr(router, "compress", fake_compress)
tool_output = "tool output line " * 200
# Issue #4708: Anthropic tool_result block followed by trailing message in the mutable turn.
messages = [
{"role": "user", "content": "read the module"},
{
"role": "assistant",
"content": [{"type": "tool_use", "id": "t1", "name": "bash", "input": {}}],
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "t1",
"content": tool_output,
"cache_control": {"type": "ephemeral"},
}
],
},
{"role": "user", "content": "what failed in the test above?"},
]
result = router.apply(
messages,
_TestTokenizer(),
model_limit=100_000,
prefix_replay_guaranteed=True,
)
compressed_block = result.messages[2]["content"][0]
assert compressed_block["content"].endswith("[compressed]"), (
"tool_result followed by trailing message must be compressed in mutable turn"
)
assert compressed_block["cache_control"] == {"type": "ephemeral"}
def test_tool_result_in_frozen_prefix_not_mutated(
monkeypatch: pytest.MonkeyPatch,
) -> None:
router = ContentRouter(ContentRouterConfig())
def fake_compress(content, context="", bias=1.0, precomputed_detection=None):
return SimpleNamespace(
compressed=content[: len(content) // 2] + "[compressed]",
compression_ratio=0.5,
strategy_used=CompressionStrategy.LOG,
)
monkeypatch.setattr(router, "compress", fake_compress)
tool_output = "tool output line " * 200
# Turn 1 tool_result is part of frozen prefix (frozen_message_count=3).
messages = [
{"role": "user", "content": "read the module"},
{
"role": "assistant",
"content": [{"type": "tool_use", "id": "t1", "name": "bash", "input": {}}],
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "t1",
"content": tool_output,
"cache_control": {"type": "ephemeral"},
}
],
},
{"role": "assistant", "content": "tests failed with error"},
{"role": "user", "content": "now fix it"},
]
result = router.apply(
messages,
_TestTokenizer(),
model_limit=100_000,
frozen_message_count=3,
prefix_replay_guaranteed=True,
)
# Message 2 is frozen (index 2 < frozen_message_count=3), must not be mutated
assert result.messages[2]["content"][0]["content"] == tool_output