mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description When the model calls `headroom_retrieve` in parallel with client tools, `CCRResponseHandler.handle_response` hands the whole turn back for the client to resolve (#839). The client has no `headroom_retrieve`, so it can never resolve that call: Claude Code answers `Error: No such tool available: headroom_retrieve`, and the model never gets the content it asked for. Parallel calls are the normal shape for agentic clients. A Claude Code subagent that greps and reads several files in one turn hits this regularly, and users report the error as "happens sometimes, usually with subagents". Measured on one machine's Claude Code transcripts (`~/.claude/projects`): 501 leaked `headroom_retrieve` calls since 2026-09-21. 484 of them were mixed turns (retrieve plus at least one client tool in the same assistant message), and 435 of those were in subagents. The other 17 were lone retrieves on the #3071 streaming path, which this PR does not touch. The client tools in a mixed turn have not run yet. This PR drops them from the turn, serves the retrieval in the continuation exactly as for a lone call, and lets the model decide again there, with the retrieved content in hand. The continuation is what the client receives, whatever it does: re-issue the client calls, change them, or withdraw them because the retrieved content says not to. The client never sees `headroom_retrieve` and never runs a call the model withdrew. Only when the continuation request fails does the client get the model's own mixed turn back (the #839 behaviour), since the model made no newer decision; a continuation that succeeded is kept even if a later retrieval round fails. A `headroom_retrieve` without a valid hash is never treated as a client call: such a turn is handed back unchanged. This applies to Anthropic and OpenAI chat only. In the Responses API a `function_call` can be the item a `reasoning` item requires next, and a Gemini `functionCall` part can carry the turn's thought signature, so dropping a sibling there could make the continuation invalid. Those providers keep the #839 pass-through unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `ccr/tool_calls.py`: new `drop_tool_calls(response, provider, calls)`. It returns a shallow copy of the response without the given tool-call objects, matched by identity. It handles Anthropic and OpenAI chat and returns the response unchanged for any other provider. - `ccr/response_handler.py`: on a mixed turn, `handle_response` trims the client calls and continues the normal retrieval loop. A successful continuation is returned as is. If the continuation request for a mixed round fails, it returns the original mixed turn instead of the trimmed one. When the provider is not trimmable, or one of the extra calls is a `headroom_retrieve` without a valid hash, it keeps the old skip-and-return. The model's own response object is not mutated. - Tests: the #839 handler test now asserts the new behaviour for Anthropic and OpenAI chat: one continuation carrying only the retrieve pair, and the re-issuing continuation returned. New handler tests return the continuation when it withdraws the client call in text, changes its arguments, or re-issues it unchanged; return the original turn when the continuation request fails; and keep a successful continuation when a later retrieval round fails. A malformed `headroom_retrieve` keeps the turn unchanged with no continuation. The Responses pass-through is pinned as unchanged. On the streaming SSE path, a re-issued client tool streams without `headroom_retrieve`, a continuation that withdraws the client tool streams the continuation, and a failed continuation streams both original blocks as a 200. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/ -q -k "ccr or CCR or retrieve or mixed" 1168 passed, 24 skipped, 15064 deselected, 3 warnings in 178.17s $ uv run --frozen --extra dev pytest <every test file naming headroom_retrieve, CCR_TOOL_NAME or ccr_response_handler> # at03e8d8761101 passed, 63 skipped, 3 warnings in 86.75s # against main (tests kept): the serve tests fail FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_serves_retrieval FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_serves_retrieval_openai FAILED tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py::test_mixed_ccr_and_client_tool_serves_retrieval_then_streams_reissued_tool # against95dc4975(handed the original turn back unless every client call was re-issued unchanged): the continuation tests fail FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_returns_the_continuation[withdrawn] FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_returns_the_continuation[changed] FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_later_failure_keeps_the_continuation FAILED tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py::test_mixed_turn_streams_the_continuation_or_the_turn_on_failure[withdrawn] # against the first commit of this PR (trimmed turn returned on a failed continuation, malformed retrieve trimmed) FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_mixed_tools_failed_continuation_returns_turn FAILED tests/test_ccr_response_handler.py::TestCCRResponseHandling::test_handle_response_malformed_ccr_call_keeps_turn $ uv run --frozen --extra dev ruff check <changed files> && ruff format --check <changed files> All checks passed! $ uv run --frozen --extra dev mypy headroom/ccr/tool_calls.py headroom/ccr/response_handler.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS, the released `headroom-ai==0.39.0` wheel (byte-identical `response_handler.py` and `tool_calls.py` to main3c418f6f) with this logic applied to the installed `handle_response`, the way Headroom Desktop ships it. - Exact command / steps: ran `CCRResponseHandler().handle_response` on a stored hash with an Anthropic turn `[tool_use headroom_retrieve, tool_use Read]` and a fake continuation that re-issues the Read, then one that answers in text, then the same turn shaped as Responses `function_call` items. - Observed result: re-issued: one continuation carrying only the retrieve `tool_use` and a `tool_result` with the original content, and the client gets the re-issued `[Read]`. Answered in text: the client gets that text, no calls. The reviewer's loopback-HTTP reproducer (#issuecomment-6062651317) on03e8d876, with its fallback assertion inverted: a cancelling text continuation returns no call, a changed `write_file(new.txt)` returns that call, an unchanged re-issue returns it, and the original `write_file(old.txt)` is never handed back. Without the change, no continuation and the client gets `[headroom_retrieve, Read]`. The Responses turn comes back unchanged in every case. - Not tested: a live Anthropic call where the trimmed assistant turn also carries a signed `thinking` block. Only `tool_use` blocks are removed and thinking blocks are kept unmodified, which is the shape the lone-retrieve continuation already sends. ## Runtime Rollout Safety - Rollout-managed feature(s): none; CCR response handling (`ccr_handle_responses`) is existing behaviour. - Minimum rollout channel: n/a - Stable/default behavior changed: yes; a mixed retrieve plus client-tool turn on Anthropic or OpenAI chat is now served by a continuation instead of being returned with an unresolvable `headroom_retrieve`. - Kill switch / disable path: `--no-ccr` (`ccr_handle_responses=False`) disables CCR response handling entirely, as before. - Unsafe override required: no - Qualification impact: none on compression; one extra upstream continuation per mixed turn, the same cost a lone retrieve already pays. If the continuation request fails, the turn costs what it does today plus that failed request. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - The model does not see the client calls it dropped: the continuation carries the trimmed assistant turn (any text and thinking, plus the retrieve) and its tool_result. If its text announced a client call, the model re-issues it in the continuation, or decides not to. - If the model calls a mixed turn again in the last allowed retrieval round, the handler returns that turn as before (`RESIDUAL_CCR_SKIPPED_MIXED`). - Responses and Gemini still leak a mixed `headroom_retrieve` to the client. Fixing them needs the reasoning item or thought-signature pairing handled, which is out of scope here. - Headroom Desktop vendors this logic against 0.39.0 until a release carries it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
1044 lines
34 KiB
Python
1044 lines
34 KiB
Python
"""Tests for CCR response handler.
|
|
|
|
These tests verify that:
|
|
1. CCR tool calls are correctly detected in responses
|
|
2. Retrieval execution works for both full and search modes
|
|
3. Continuation flow handles multiple rounds
|
|
4. Provider-specific formats are handled correctly
|
|
5. Streaming buffer detection works
|
|
"""
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from headroom.cache.compression_store import (
|
|
get_compression_store,
|
|
reset_compression_store,
|
|
)
|
|
from headroom.ccr.response_handler import (
|
|
CCRResponseHandler,
|
|
CCRToolCall,
|
|
CCRToolResult,
|
|
ResponseHandlerConfig,
|
|
StreamingCCRBuffer,
|
|
)
|
|
from headroom.ccr.tool_injection import CCR_TOOL_NAME
|
|
|
|
|
|
class TestCCRToolCallDetection:
|
|
"""Test detection of CCR tool calls in responses."""
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_store(self):
|
|
"""Reset global store before each test."""
|
|
reset_compression_store()
|
|
yield
|
|
reset_compression_store()
|
|
|
|
def test_detect_anthropic_ccr_tool_call(self):
|
|
"""Detect CCR tool call in Anthropic format."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"content": [
|
|
{"type": "text", "text": "Let me retrieve that data."},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": "abc123"},
|
|
},
|
|
]
|
|
}
|
|
|
|
assert handler.has_ccr_tool_calls(response, "anthropic")
|
|
|
|
def test_detect_openai_ccr_tool_call(self):
|
|
"""Detect CCR tool call in OpenAI format."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"choices": [
|
|
{
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": "Let me retrieve that data.",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_123",
|
|
"type": "function",
|
|
"function": {
|
|
"name": CCR_TOOL_NAME,
|
|
"arguments": '{"hash": "abc123"}',
|
|
},
|
|
}
|
|
],
|
|
}
|
|
}
|
|
]
|
|
}
|
|
|
|
assert handler.has_ccr_tool_calls(response, "openai")
|
|
|
|
def test_no_ccr_tool_call_anthropic(self):
|
|
"""No false positive when no CCR tool call present."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"content": [
|
|
{"type": "text", "text": "Here is the data."},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": "some_other_tool",
|
|
"input": {"param": "value"},
|
|
},
|
|
]
|
|
}
|
|
|
|
assert not handler.has_ccr_tool_calls(response, "anthropic")
|
|
|
|
def test_no_ccr_tool_call_openai(self):
|
|
"""No false positive when no CCR tool call present in OpenAI format."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"choices": [
|
|
{
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": "Here is the data.",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_123",
|
|
"type": "function",
|
|
"function": {
|
|
"name": "other_tool",
|
|
"arguments": '{"param": "value"}',
|
|
},
|
|
}
|
|
],
|
|
}
|
|
}
|
|
]
|
|
}
|
|
|
|
assert not handler.has_ccr_tool_calls(response, "openai")
|
|
|
|
def test_text_only_response(self):
|
|
"""No false positive for text-only responses."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {"content": [{"type": "text", "text": "Just plain text."}]}
|
|
|
|
assert not handler.has_ccr_tool_calls(response, "anthropic")
|
|
|
|
def test_empty_response(self):
|
|
"""Handle empty response gracefully."""
|
|
handler = CCRResponseHandler()
|
|
|
|
assert not handler.has_ccr_tool_calls({}, "anthropic")
|
|
assert not handler.has_ccr_tool_calls({"content": []}, "anthropic")
|
|
|
|
|
|
class TestCCRToolCallParsing:
|
|
"""Test parsing of CCR tool calls."""
|
|
|
|
def test_parse_anthropic_full_retrieval(self):
|
|
"""Parse full retrieval call from Anthropic format."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": "abc123def456abc123def456"},
|
|
}
|
|
]
|
|
}
|
|
|
|
ccr_calls, other_calls = handler._parse_ccr_tool_calls(response, "anthropic")
|
|
|
|
assert len(ccr_calls) == 1
|
|
assert ccr_calls[0].tool_call_id == "tool_123"
|
|
assert ccr_calls[0].hash_key == "abc123def456abc123def456"
|
|
assert not hasattr(ccr_calls[0], "query")
|
|
assert len(other_calls) == 0
|
|
|
|
def test_parse_anthropic_retrieval_ignores_query(self):
|
|
"""Retrieval parses the hash; any legacy ``query`` input is ignored."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_456",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": "def456abc123def456abc123", "query": "authentication error"},
|
|
}
|
|
]
|
|
}
|
|
|
|
ccr_calls, other_calls = handler._parse_ccr_tool_calls(response, "anthropic")
|
|
|
|
assert len(ccr_calls) == 1
|
|
assert ccr_calls[0].hash_key == "def456abc123def456abc123"
|
|
assert not hasattr(ccr_calls[0], "query")
|
|
|
|
def test_parse_mixed_tool_calls(self):
|
|
"""Parse response with both CCR and other tool calls."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_1",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": "abc123def456abc123def456"},
|
|
},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_2",
|
|
"name": "read_file",
|
|
"input": {"path": "/etc/config"},
|
|
},
|
|
]
|
|
}
|
|
|
|
ccr_calls, other_calls = handler._parse_ccr_tool_calls(response, "anthropic")
|
|
|
|
assert len(ccr_calls) == 1
|
|
assert len(other_calls) == 1
|
|
assert other_calls[0]["name"] == "read_file"
|
|
|
|
|
|
class TestCCRRetrievalExecution:
|
|
"""Test CCR retrieval execution."""
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_store(self):
|
|
"""Reset global store before each test."""
|
|
reset_compression_store()
|
|
yield
|
|
reset_compression_store()
|
|
|
|
def test_full_retrieval_success(self):
|
|
"""Successfully retrieve full content."""
|
|
store = get_compression_store()
|
|
original = json.dumps([{"id": i} for i in range(100)])
|
|
compressed = json.dumps([{"id": i} for i in range(10)])
|
|
|
|
hash_key = store.store(
|
|
original=original,
|
|
compressed=compressed,
|
|
original_item_count=100,
|
|
compressed_item_count=10,
|
|
)
|
|
|
|
handler = CCRResponseHandler()
|
|
call = CCRToolCall(tool_call_id="test_id", hash_key=hash_key)
|
|
|
|
result = handler._execute_retrieval(call)
|
|
|
|
assert result.success
|
|
assert result.items_retrieved == 100
|
|
|
|
# Check content structure
|
|
content = json.loads(result.content)
|
|
assert content["hash"] == hash_key
|
|
assert "original_content" in content
|
|
|
|
def test_retrieval_returns_full_content_for_cached_hash(self):
|
|
"""Retrieval always returns the full original content (never empty)."""
|
|
store = get_compression_store()
|
|
items = [
|
|
{"id": 1, "text": "Python programming language tutorial"},
|
|
{"id": 2, "text": "JavaScript web development framework"},
|
|
{"id": 3, "text": "Python data science machine learning"},
|
|
{"id": 4, "text": "Ruby programming language basics"},
|
|
{"id": 5, "text": "Python web framework django flask"},
|
|
]
|
|
original = json.dumps(items)
|
|
compressed = json.dumps(items[:1])
|
|
|
|
hash_key = store.store(
|
|
original=original,
|
|
compressed=compressed,
|
|
original_item_count=5,
|
|
compressed_item_count=1,
|
|
)
|
|
|
|
handler = CCRResponseHandler()
|
|
call = CCRToolCall(tool_call_id="test_id", hash_key=hash_key)
|
|
|
|
result = handler._execute_retrieval(call)
|
|
|
|
assert result.success
|
|
assert result.items_retrieved == 5
|
|
|
|
content = json.loads(result.content)
|
|
assert content["hash"] == hash_key
|
|
# Full content is always returned — the complete original round-trips.
|
|
assert json.loads(content["original_content"]) == items
|
|
|
|
def test_retrieval_nonexistent_hash(self):
|
|
"""Handle retrieval of nonexistent hash."""
|
|
handler = CCRResponseHandler()
|
|
call = CCRToolCall(tool_call_id="test_id", hash_key="nonexistent123")
|
|
|
|
result = handler._execute_retrieval(call)
|
|
|
|
assert not result.success
|
|
assert result.items_retrieved == 0
|
|
|
|
content = json.loads(result.content)
|
|
assert "error" in content
|
|
|
|
|
|
class TestCCRToolResultMessage:
|
|
"""Test tool result message creation."""
|
|
|
|
def test_anthropic_tool_result_format(self):
|
|
"""Create tool result message in Anthropic format."""
|
|
handler = CCRResponseHandler()
|
|
results = [
|
|
CCRToolResult(
|
|
tool_call_id="tool_123",
|
|
content='{"data": "retrieved"}',
|
|
success=True,
|
|
items_retrieved=10,
|
|
)
|
|
]
|
|
|
|
message = handler._create_tool_result_message(results, "anthropic")
|
|
|
|
assert message["role"] == "user"
|
|
assert len(message["content"]) == 1
|
|
assert message["content"][0]["type"] == "tool_result"
|
|
assert message["content"][0]["tool_use_id"] == "tool_123"
|
|
|
|
def test_openai_tool_result_format(self):
|
|
"""Create tool result messages in OpenAI format."""
|
|
handler = CCRResponseHandler()
|
|
results = [
|
|
CCRToolResult(
|
|
tool_call_id="call_123",
|
|
content='{"data": "retrieved"}',
|
|
success=True,
|
|
),
|
|
CCRToolResult(
|
|
tool_call_id="call_456",
|
|
content='{"data": "more data"}',
|
|
success=True,
|
|
),
|
|
]
|
|
|
|
message = handler._create_tool_result_message(results, "openai")
|
|
|
|
assert "_openai_tool_results" in message
|
|
assert len(message["_openai_tool_results"]) == 2
|
|
assert message["_openai_tool_results"][0]["role"] == "tool"
|
|
|
|
|
|
class TestCCRResponseHandling:
|
|
"""Test the full response handling flow."""
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_store(self):
|
|
"""Reset global store before each test."""
|
|
reset_compression_store()
|
|
yield
|
|
reset_compression_store()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_no_ccr(self):
|
|
"""Handle response with no CCR calls (pass-through)."""
|
|
handler = CCRResponseHandler()
|
|
response = {"content": [{"type": "text", "text": "Just text."}]}
|
|
|
|
async def mock_api_call(messages, tools):
|
|
return {"content": [{"type": "text", "text": "Response"}]}
|
|
|
|
result = await handler.handle_response(response, [], None, mock_api_call, "anthropic")
|
|
|
|
# Should return original response unchanged
|
|
assert result == response
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_with_ccr(self):
|
|
"""Handle response containing CCR tool call."""
|
|
store = get_compression_store()
|
|
original = json.dumps([{"id": i} for i in range(50)])
|
|
hash_key = store.store(
|
|
original=original,
|
|
compressed="[]",
|
|
original_item_count=50,
|
|
)
|
|
|
|
handler = CCRResponseHandler()
|
|
|
|
# Initial response with CCR tool call
|
|
initial_response = {
|
|
"content": [
|
|
{"type": "text", "text": "Let me get that data."},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
},
|
|
]
|
|
}
|
|
|
|
# Final response after tool result
|
|
final_response = {"content": [{"type": "text", "text": "Here is all 50 items of data."}]}
|
|
|
|
call_count = 0
|
|
|
|
async def mock_api_call(messages, tools):
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return final_response
|
|
|
|
result = await handler.handle_response(
|
|
initial_response,
|
|
[{"role": "user", "content": "Get me the data"}],
|
|
None,
|
|
mock_api_call,
|
|
"anthropic",
|
|
)
|
|
|
|
# Should have made continuation call
|
|
assert call_count == 1
|
|
# Should return final response
|
|
assert result == final_response
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_continuation_failure_logs_cause_for_empty_str_exception(self, caplog):
|
|
"""A continuation exception whose str() is empty (e.g.
|
|
httpx.TimeoutException(''), a bare Exception()) must still log its
|
|
cause. Interpolating str(e) alone produced a log line with nothing
|
|
after the colon, losing the type entirely (#3129)."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(
|
|
original=json.dumps([{"id": i} for i in range(50)]),
|
|
compressed="[]",
|
|
original_item_count=50,
|
|
)
|
|
|
|
handler = CCRResponseHandler()
|
|
initial_response = {
|
|
"content": [
|
|
{"type": "text", "text": "Let me get that data."},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
},
|
|
]
|
|
}
|
|
|
|
async def failing_api_call(messages, tools):
|
|
raise Exception("") # empty str(e), non-empty repr()
|
|
|
|
with caplog.at_level("ERROR", logger="headroom.ccr.response_handler"):
|
|
result = await handler.handle_response(
|
|
initial_response,
|
|
[{"role": "user", "content": "Get me the data"}],
|
|
None,
|
|
failing_api_call,
|
|
"anthropic",
|
|
)
|
|
|
|
# Degrades to the current response rather than raising.
|
|
assert result == initial_response
|
|
# The cause survives: the type name appears even though str(e) is "".
|
|
failure_logs = [r.getMessage() for r in caplog.records if "Continuation" in r.getMessage()]
|
|
assert failure_logs
|
|
assert "Exception" in failure_logs[0]
|
|
# Regression guard: never a bare "...failed: " with nothing after it.
|
|
assert not failure_logs[0].rstrip().endswith("failed:")
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_max_rounds(self):
|
|
"""Respects max retrieval rounds limit."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
|
|
config = ResponseHandlerConfig(max_retrieval_rounds=2)
|
|
handler = CCRResponseHandler(config)
|
|
|
|
# Response that always has CCR tool call (simulating infinite loop)
|
|
ccr_response = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
}
|
|
]
|
|
}
|
|
|
|
call_count = 0
|
|
|
|
async def mock_api_call(messages, tools):
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return ccr_response
|
|
|
|
await handler.handle_response(ccr_response, [], None, mock_api_call, "anthropic")
|
|
|
|
# Should stop after max rounds
|
|
assert call_count == 2
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_disabled(self):
|
|
"""Disabled handler returns response unchanged."""
|
|
config = ResponseHandlerConfig(enabled=False)
|
|
handler = CCRResponseHandler(config)
|
|
|
|
response = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": "abc123"},
|
|
}
|
|
]
|
|
}
|
|
|
|
async def mock_api_call(messages, tools):
|
|
raise AssertionError("Should not be called")
|
|
|
|
result = await handler.handle_response(response, [], None, mock_api_call, "anthropic")
|
|
|
|
assert result == response
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_mixed_tools_serves_retrieval(self):
|
|
"""CCR called alongside a client tool: serve the retrieval now.
|
|
|
|
The client has no headroom_retrieve, so handing the turn back made
|
|
Claude Code answer "No such tool available: headroom_retrieve" and the
|
|
model never got the content. The client tool has not run yet, so it is
|
|
dropped from the continuation and the model re-issues it there.
|
|
"""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
|
|
handler = CCRResponseHandler()
|
|
|
|
mixed_response = {
|
|
"content": [
|
|
{"type": "text", "text": "Reading both."},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "ccr_call",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "user_call",
|
|
"name": "read_file",
|
|
"input": {"path": "/etc/config"},
|
|
},
|
|
],
|
|
"stop_reason": "tool_use",
|
|
}
|
|
reissued = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "user_call_2",
|
|
"name": "read_file",
|
|
"input": {"path": "/etc/config"},
|
|
}
|
|
],
|
|
"stop_reason": "tool_use",
|
|
}
|
|
sent = []
|
|
|
|
async def mock_api_call(messages, tools):
|
|
sent.append(messages)
|
|
return reissued
|
|
|
|
result = await handler.handle_response(mixed_response, [], None, mock_api_call, "anthropic")
|
|
|
|
assert len(sent) == 1
|
|
assistant, tool_result = sent[0]
|
|
assert [b.get("id") for b in assistant["content"]] == [None, "ccr_call"]
|
|
assert tool_result["content"][0]["tool_use_id"] == "ccr_call"
|
|
assert "[1,2,3]" in tool_result["content"][0]["content"]
|
|
# The client gets the re-issued client tool, never headroom_retrieve.
|
|
assert result is reissued
|
|
assert handler.residual_ccr_status(result, "anthropic") == "resolved"
|
|
# The model's own response is not mutated.
|
|
assert len(mixed_response["content"]) == 3
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_mixed_tools_serves_retrieval_openai(self):
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
|
|
handler = CCRResponseHandler()
|
|
|
|
def call(call_id, name, arguments):
|
|
return {
|
|
"id": call_id,
|
|
"type": "function",
|
|
"function": {"name": name, "arguments": arguments},
|
|
}
|
|
|
|
mixed_response = {
|
|
"choices": [
|
|
{
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
call("call_user", "read_file", '{"path": "/etc/config"}'),
|
|
call("call_ccr", CCR_TOOL_NAME, json.dumps({"hash": hash_key})),
|
|
],
|
|
},
|
|
"finish_reason": "tool_calls",
|
|
}
|
|
]
|
|
}
|
|
final = {
|
|
"choices": [
|
|
{
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [call("call_user_2", "read_file", '{"path":"/etc/config"}')],
|
|
},
|
|
"finish_reason": "tool_calls",
|
|
}
|
|
]
|
|
}
|
|
sent = []
|
|
|
|
async def mock_api_call(messages, tools):
|
|
sent.append(messages)
|
|
return final
|
|
|
|
result = await handler.handle_response(mixed_response, [], None, mock_api_call, "openai")
|
|
|
|
assert result is final
|
|
assistant, tool_result = sent[0]
|
|
assert [c["id"] for c in assistant["tool_calls"]] == ["call_ccr"]
|
|
assert tool_result["tool_call_id"] == "call_ccr"
|
|
|
|
@staticmethod
|
|
def _mixed_turn(hash_key, client_input):
|
|
return {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "ccr_call",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
},
|
|
{
|
|
"type": "tool_use",
|
|
"id": "user_call",
|
|
"name": "write_file",
|
|
"input": client_input,
|
|
},
|
|
],
|
|
"stop_reason": "tool_use",
|
|
}
|
|
|
|
@pytest.mark.asyncio
|
|
@pytest.mark.parametrize(
|
|
"continuation",
|
|
[
|
|
# Withdraws the client call once it has the content.
|
|
{
|
|
"content": [{"type": "text", "text": "Do not overwrite old.txt."}],
|
|
"stop_reason": "end_turn",
|
|
},
|
|
# Changes it.
|
|
{
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "user_call_2",
|
|
"name": "write_file",
|
|
"input": {"path": "new.txt", "content": "safe output"},
|
|
}
|
|
],
|
|
"stop_reason": "tool_use",
|
|
},
|
|
# Re-issues it unchanged.
|
|
{
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "user_call_2",
|
|
"name": "write_file",
|
|
"input": {"path": "old.txt", "content": "overwrite"},
|
|
}
|
|
],
|
|
"stop_reason": "tool_use",
|
|
},
|
|
],
|
|
ids=["withdrawn", "changed", "reissued"],
|
|
)
|
|
async def test_handle_response_mixed_tools_returns_the_continuation(self, continuation):
|
|
"""The continuation is the model's newer decision, made with the
|
|
retrieved content: the client gets it, never the dropped call."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
mixed_response = self._mixed_turn(hash_key, {"path": "old.txt", "content": "overwrite"})
|
|
|
|
async def mock_api_call(messages, tools):
|
|
return continuation
|
|
|
|
result = await CCRResponseHandler().handle_response(
|
|
mixed_response, [], None, mock_api_call, "anthropic"
|
|
)
|
|
|
|
assert result is continuation
|
|
assert "user_call" not in [b.get("id") for b in result["content"]]
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_mixed_tools_failed_continuation_returns_turn(self):
|
|
"""No newer decision: the client gets the model's own turn, client
|
|
call included, as before (#839)."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
mixed_response = self._mixed_turn(hash_key, {"path": "/tmp/out"})
|
|
|
|
async def mock_api_call(messages, tools):
|
|
raise RuntimeError("upstream down")
|
|
|
|
result = await CCRResponseHandler().handle_response(
|
|
mixed_response, [], None, mock_api_call, "anthropic"
|
|
)
|
|
|
|
assert result is mixed_response
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_mixed_tools_later_failure_keeps_the_continuation(self):
|
|
"""A continuation that succeeded supersedes the mixed turn even when a
|
|
later retrieval round fails."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
mixed_response = self._mixed_turn(hash_key, {"path": "/tmp/out"})
|
|
retrieve_again = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "ccr_call_2",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
}
|
|
],
|
|
"stop_reason": "tool_use",
|
|
}
|
|
calls = 0
|
|
|
|
async def mock_api_call(messages, tools):
|
|
nonlocal calls
|
|
calls += 1
|
|
if calls > 1:
|
|
raise RuntimeError("upstream down")
|
|
return retrieve_again
|
|
|
|
result = await CCRResponseHandler().handle_response(
|
|
mixed_response, [], None, mock_api_call, "anthropic"
|
|
)
|
|
|
|
assert calls == 2
|
|
assert result is retrieve_again
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_malformed_ccr_call_keeps_turn(self):
|
|
"""A headroom_retrieve without a valid hash is not a client call to
|
|
drop: the turn is handed back unchanged."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
mixed_response = self._mixed_turn(hash_key, {})
|
|
mixed_response["content"][1]["name"] = CCR_TOOL_NAME
|
|
|
|
async def mock_api_call(messages, tools):
|
|
raise AssertionError("no continuation when a CCR call is malformed")
|
|
|
|
result = await CCRResponseHandler().handle_response(
|
|
mixed_response, [], None, mock_api_call, "anthropic"
|
|
)
|
|
|
|
assert result is mixed_response
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handle_response_mixed_tools_responses_still_skips(self):
|
|
"""Responses API: a function_call can be the item a reasoning item
|
|
requires next, so a sibling is not dropped there; the turn is handed
|
|
back unchanged as before (#839)."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
|
|
handler = CCRResponseHandler()
|
|
mixed_response = {
|
|
"output": [
|
|
{"type": "reasoning", "id": "rs_1", "summary": []},
|
|
{
|
|
"type": "function_call",
|
|
"call_id": "call_user",
|
|
"name": "read_file",
|
|
"arguments": "{}",
|
|
},
|
|
{
|
|
"type": "function_call",
|
|
"call_id": "call_ccr",
|
|
"name": CCR_TOOL_NAME,
|
|
"arguments": json.dumps({"hash": hash_key}),
|
|
},
|
|
]
|
|
}
|
|
|
|
async def mock_api_call(messages, tools):
|
|
raise AssertionError("no continuation for a Responses mixed turn")
|
|
|
|
result = await handler.handle_response(
|
|
mixed_response, [], None, mock_api_call, "openai_responses"
|
|
)
|
|
|
|
assert result is mixed_response
|
|
|
|
|
|
class TestCCRResponseHandlerStats:
|
|
"""Test handler statistics."""
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_store(self):
|
|
"""Reset global store before each test."""
|
|
reset_compression_store()
|
|
yield
|
|
reset_compression_store()
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_retrieval_count_tracking(self):
|
|
"""Track total retrieval count."""
|
|
store = get_compression_store()
|
|
hash_key = store.store(original="[1,2,3]", compressed="[]")
|
|
|
|
handler = CCRResponseHandler()
|
|
|
|
initial_response = {
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "tool_123",
|
|
"name": CCR_TOOL_NAME,
|
|
"input": {"hash": hash_key},
|
|
}
|
|
]
|
|
}
|
|
|
|
final_response = {"content": [{"type": "text", "text": "Done"}]}
|
|
|
|
async def mock_api_call(messages, tools):
|
|
return final_response
|
|
|
|
await handler.handle_response(initial_response, [], None, mock_api_call, "anthropic")
|
|
|
|
stats = handler.get_stats()
|
|
assert stats["total_retrievals"] == 1
|
|
|
|
|
|
class TestStreamingCCRBuffer:
|
|
"""Test streaming buffer for CCR detection."""
|
|
|
|
def test_buffer_accumulation(self):
|
|
"""Buffer accumulates chunks."""
|
|
buffer = StreamingCCRBuffer()
|
|
|
|
buffer.add_chunk(b"part1")
|
|
buffer.add_chunk(b"part2")
|
|
buffer.add_chunk(b"part3")
|
|
|
|
assert buffer.get_accumulated() == b"part1part2part3"
|
|
|
|
def test_detect_ccr_tool_in_stream(self):
|
|
"""Detect CCR tool call in streaming chunks."""
|
|
buffer = StreamingCCRBuffer()
|
|
|
|
# Simulate streaming response with tool_use
|
|
chunk1 = b'{"type":"content_block_start","content_block":{"type":"tool_use"'
|
|
chunk2 = f',"name":"{CCR_TOOL_NAME}"'.encode()
|
|
|
|
detected = buffer.add_chunk(chunk1)
|
|
assert not detected # Not complete yet
|
|
|
|
detected = buffer.add_chunk(chunk2)
|
|
assert detected # Now detected
|
|
|
|
assert buffer.detected_ccr
|
|
|
|
def test_no_false_positive_detection(self):
|
|
"""No false positive for non-CCR tool calls."""
|
|
buffer = StreamingCCRBuffer()
|
|
|
|
chunk = b'{"type":"content_block_start","content_block":{"type":"tool_use","name":"other_tool"}}'
|
|
|
|
detected = buffer.add_chunk(chunk)
|
|
assert not detected
|
|
assert not buffer.detected_ccr
|
|
|
|
def test_buffer_clear(self):
|
|
"""Buffer clears state correctly."""
|
|
buffer = StreamingCCRBuffer()
|
|
buffer.add_chunk(b"data")
|
|
buffer.detected_ccr = True
|
|
|
|
buffer.clear()
|
|
|
|
assert buffer.get_accumulated() == b""
|
|
assert not buffer.detected_ccr
|
|
|
|
|
|
class TestResponseHandlerConfig:
|
|
"""Test response handler configuration."""
|
|
|
|
def test_default_config(self):
|
|
"""Default config values."""
|
|
config = ResponseHandlerConfig()
|
|
|
|
assert config.enabled is True
|
|
assert config.max_retrieval_rounds == 3
|
|
assert config.strip_ccr_from_response is True
|
|
assert config.continuation_timeout_ms == 120000
|
|
|
|
def test_custom_config(self):
|
|
"""Custom config values."""
|
|
config = ResponseHandlerConfig(
|
|
enabled=False,
|
|
max_retrieval_rounds=5,
|
|
)
|
|
|
|
assert config.enabled is False
|
|
assert config.max_retrieval_rounds == 5
|
|
|
|
|
|
class TestCCRToolCallDataClass:
|
|
"""Test CCRToolCall dataclass."""
|
|
|
|
def test_full_retrieval_call(self):
|
|
"""Create full retrieval call."""
|
|
call = CCRToolCall(
|
|
tool_call_id="test_123",
|
|
hash_key="abc123",
|
|
)
|
|
|
|
assert call.tool_call_id == "test_123"
|
|
assert call.hash_key == "abc123"
|
|
assert not hasattr(call, "query")
|
|
|
|
|
|
class TestCCRToolResultDataClass:
|
|
"""Test CCRToolResult dataclass."""
|
|
|
|
def test_successful_result(self):
|
|
"""Create successful result."""
|
|
result = CCRToolResult(
|
|
tool_call_id="test_123",
|
|
content='{"data": "content"}',
|
|
success=True,
|
|
items_retrieved=50,
|
|
)
|
|
|
|
assert result.success
|
|
assert result.items_retrieved == 50
|
|
assert not hasattr(result, "was_search")
|
|
|
|
def test_failed_result(self):
|
|
"""Create failed result."""
|
|
result = CCRToolResult(
|
|
tool_call_id="test_789",
|
|
content='{"error": "not found"}',
|
|
success=False,
|
|
)
|
|
|
|
assert not result.success
|
|
assert result.items_retrieved == 0
|
|
|
|
|
|
class TestExtractAssistantMessage:
|
|
"""Test extraction of assistant messages from responses."""
|
|
|
|
def test_extract_anthropic_message(self):
|
|
"""Extract assistant message from Anthropic response."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"content": [
|
|
{"type": "text", "text": "Hello"},
|
|
{"type": "tool_use", "id": "123", "name": "test", "input": {}},
|
|
]
|
|
}
|
|
|
|
message = handler._extract_assistant_message(response, "anthropic")
|
|
|
|
assert message["role"] == "assistant"
|
|
assert message["content"] == response["content"]
|
|
|
|
def test_extract_openai_message(self):
|
|
"""Extract assistant message from OpenAI response."""
|
|
handler = CCRResponseHandler()
|
|
|
|
response = {
|
|
"choices": [
|
|
{
|
|
"message": {
|
|
"role": "assistant",
|
|
"content": "Hello",
|
|
"tool_calls": [{"id": "123"}],
|
|
}
|
|
}
|
|
]
|
|
}
|
|
|
|
message = handler._extract_assistant_message(response, "openai")
|
|
|
|
assert message["role"] == "assistant"
|
|
assert message["content"] == "Hello"
|
|
assert message["tool_calls"] == [{"id": "123"}]
|
|
|
|
|
|
class TestExtractAssistantMessageEdgeCases:
|
|
"""Regression: `_extract_assistant_message` must not crash on an empty or
|
|
malformed OpenAI `choices` array (OpenAI-compatible gateways can send
|
|
`choices: []` or `[null]` on content-filtered / usage-only responses)."""
|
|
|
|
def test_openai_empty_choices_does_not_crash(self):
|
|
handler = CCRResponseHandler()
|
|
msg = handler._extract_assistant_message({"choices": []}, "openai")
|
|
assert msg == {"role": "assistant", "content": None, "tool_calls": None}
|
|
|
|
def test_openai_null_first_choice_does_not_crash(self):
|
|
handler = CCRResponseHandler()
|
|
msg = handler._extract_assistant_message({"choices": [None]}, "openai")
|
|
assert msg == {"role": "assistant", "content": None, "tool_calls": None}
|
|
|
|
def test_openai_absent_choices_does_not_crash(self):
|
|
handler = CCRResponseHandler()
|
|
msg = handler._extract_assistant_message({}, "openai")
|
|
assert msg == {"role": "assistant", "content": None, "tool_calls": None}
|
|
|
|
def test_openai_normal_choice_still_extracts(self):
|
|
handler = CCRResponseHandler()
|
|
resp = {"choices": [{"message": {"content": "hi", "tool_calls": [{"id": "1"}]}}]}
|
|
msg = handler._extract_assistant_message(resp, "openai")
|
|
assert msg == {"role": "assistant", "content": "hi", "tool_calls": [{"id": "1"}]}
|