mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description When the model calls one of Headroom's injected memory tools on a buffered Anthropic turn (a `stream:false` request, or a streaming request that CCR switched to buffered), Headroom runs the tool itself and then asks the model to continue. That follow-up request repeats the model's whole turn, but it only includes answers for the memory calls. So if the model called a memory tool **and** one of the client's own tools in the same turn (parallel tool use makes this common), the client tool was left unanswered. Anthropic rejects that with a 400 (`tool_use ids were found without tool_result blocks`), the 400 replaced the perfectly good first response, and the client's turn failed. The tool id in the error was never sent to the client, so it had no way to repair its history. The log still said `continuation complete`, which hid the failure. The streaming path already handles this correctly since #3947: it only continues when the turn called nothing but memory tools, and otherwise gives the turn back to the client. This PR applies the same rule to the buffered path. Closes #4009 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/anthropic.py`, buffered memory block: - Continue only when every `tool_use` in the model's turn has a `tool_result`. - Otherwise, return the turn to the client with the proxy's own memory calls removed (the client never declared those tools and they already ran). The client's own tool calls stay, so it can run them as usual. A memory tool the client declared itself is left alone, same as on the streaming path. - Build the continuation from `body["messages"]` (what was actually sent upstream) instead of `optimized_messages`. They normally match, but later steps such as the system-role relocation (#765) only update `body["messages"]`. - Log a warning with the upstream status when the continuation fails, instead of `continuation complete`. - `tests/test_anthropic_buffered_memory_continuation.py` (new): drives the real app with a faked upstream. - A turn with a memory call plus a `Bash` call: no continuation is sent, the memory tool still runs, and the client receives only the `Bash` call. Fails on `main` (a continuation is sent). - A memory-only turn: continues as before, with a result for every call, on top of the messages that were sent. - A continuation that returns 400 is logged as a failure. Fails on `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output Before the fix (new tests on unmodified `main`): ```text FAILED tests/test_anthropic_buffered_memory_continuation.py::test_mixed_turn_returns_client_tool_without_continuation assert len(sent) == 1 E AssertionError: assert 2 == 1 FAILED tests/test_anthropic_buffered_memory_continuation.py::test_failed_continuation_is_not_logged_as_complete ======================== 2 failed, 1 passed in 15.54s ========================= ``` After the fix: ```text $ pytest tests/test_anthropic_buffered_memory_continuation.py tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_memory_tool_stream.py -q ============================= 70 passed in 18.42s ============================= $ ruff check headroom/proxy/handlers/anthropic.py tests/test_anthropic_buffered_memory_continuation.py All checks passed! $ ruff format --check headroom/proxy/handlers/anthropic.py tests/test_anthropic_buffered_memory_continuation.py 2 files already formatted $ mypy headroom --ignore-missing-imports headroom\fileperms.py:92: error: Module has no attribute "fchmod"; maybe "chmod"? [attr-defined] Found 1 error in 1 file (checked 562 source files) ``` The one mypy error is Windows-only (`os.fchmod` does not exist there) and is unrelated to this change; it does not appear on Linux CI. I also ran every `tests/*memory*`, `*anthropic*`, `*buffered*`, `*ccr*` file plus `tests/test_proxy/` (1705 passed). The failures in that run (`test_wrap_code_memory.py`, `test_orphan_watchdog.py`, an import error in `test_anthropic_buffered_ccr_usage.py`) fail the same way on unmodified `main` on my Windows machine. `test_cc_switch_reconciler.py` failed once under xdist and passed on three serial reruns; it is timing-sensitive and does not touch this code. ## Real Behavior Proof - Environment: Windows 11, Python 3.13, headroom at this branch, full FastAPI app via `TestClient` with the upstream call faked (memory handler faked to behave like the real one: results only for memory tools). - Exact command / steps: `pytest tests/test_anthropic_buffered_memory_continuation.py -q`, which sends a `stream:false` `/v1/messages` request with memory tools injected, where the upstream answers with `[memory_search tool_use, Bash tool_use]`. - Observed result: on `main` Headroom sends a second upstream request whose user turn answers only `toolu_mem`, leaving `toolu_bash` unanswered (the exact shape Anthropic rejects in #4009). With this change no second request is sent, `memory_search` still runs, and the client gets back a response containing only the `Bash` tool call. - Not tested: against a live Anthropic or Bedrock-backed gateway, and the reporter's exact 214-message session. The issue's error points at `messages.210` while the request had 214 messages; Headroom appends the continuation after the full list, so I could not explain that index from Headroom's code (possibly the gateway re-indexing). ## Runtime Rollout Safety - Rollout-managed feature(s): None. This changes the existing memory-tool handling on buffered Anthropic turns. - Minimum rollout channel: stable - Stable/default behavior changed: Yes, only for turns that previously failed: when the model calls an injected memory tool together with a client tool, the turn now goes back to the client instead of producing an upstream 400. Memory-only turns behave as before. - Kill switch / disable path: Disable memory tool injection (`--no-memory-tools` / memory `inject_tools` off), which skips this block entirely. - Unsafe override required: No - Qualification impact: None - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` (it is generated by release-please from my Conventional Commit PR title) ## Additional Notes - No docs change: this is internal proxy behavior with no user-facing setting. - Follow-up, not in this PR: the OpenAI Chat Completions memory path (`openai.py`, around the `Memory: Handled ... tool call(s) with continuation` log) builds its continuation the same way. It does not break the client, because it checks the continuation's status and falls back to the original response, but the continuation itself is still malformed on a mixed turn and wastes a request. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4013" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
278bbc8771
commit
87e05b68b8
@@ -4731,18 +4731,62 @@ class AnthropicHandlerMixin:
|
||||
request_context=memory_request_ctx,
|
||||
)
|
||||
|
||||
if tool_results:
|
||||
turn_content = resp_json.get("content") or []
|
||||
answered = {result.get("tool_use_id") for result in tool_results}
|
||||
unanswered = [
|
||||
block
|
||||
for block in turn_content
|
||||
if isinstance(block, dict)
|
||||
and block.get("type") == "tool_use"
|
||||
and block.get("id") not in answered
|
||||
]
|
||||
if tool_results and unanswered:
|
||||
# The continuation replays this whole turn, and
|
||||
# Anthropic rejects any tool_use without a result
|
||||
# (#4009). Only the client can answer its own
|
||||
# tools, so the turn goes back to it, minus the
|
||||
# memory calls it never declared and that already
|
||||
# ran here (same rule as the streaming path, #3947).
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Turn also called a client "
|
||||
"tool; returning the turn to the client"
|
||||
)
|
||||
resp_json = {
|
||||
**resp_json,
|
||||
"content": [
|
||||
block
|
||||
for block in turn_content
|
||||
if not (
|
||||
isinstance(block, dict)
|
||||
and block.get("type") == "tool_use"
|
||||
and block.get("name") in server_memory_tool_names
|
||||
)
|
||||
],
|
||||
}
|
||||
response = httpx.Response(
|
||||
status_code=200,
|
||||
content=json.dumps(resp_json).encode(),
|
||||
headers={
|
||||
key: value
|
||||
for key, value in response.headers.items()
|
||||
if key.lower()
|
||||
not in ("content-encoding", "content-length")
|
||||
},
|
||||
)
|
||||
elif tool_results:
|
||||
# Create continuation messages
|
||||
assistant_msg = {
|
||||
"role": "assistant",
|
||||
"content": resp_json.get("content", []),
|
||||
"content": turn_content,
|
||||
}
|
||||
user_msg = {
|
||||
"role": "user",
|
||||
"content": tool_results,
|
||||
}
|
||||
|
||||
continuation_messages = optimized_messages + [
|
||||
# body["messages"], not optimized_messages: it is
|
||||
# what this turn actually sent upstream.
|
||||
continuation_messages = body["messages"] + [
|
||||
assistant_msg,
|
||||
user_msg,
|
||||
]
|
||||
@@ -4763,9 +4807,16 @@ class AnthropicHandlerMixin:
|
||||
# Update response with continuation
|
||||
resp_json = cont_response.json()
|
||||
response = cont_response
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Tool calls handled, continuation complete"
|
||||
)
|
||||
if cont_response.status_code >= 400:
|
||||
logger.warning(
|
||||
f"[{request_id}] Memory: Continuation failed with "
|
||||
f"upstream status {cont_response.status_code}"
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"[{request_id}] Memory: Tool calls handled, "
|
||||
"continuation complete"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Buffered memory continuations answer every tool_use they replay (GH #4009).
|
||||
|
||||
On a buffered Anthropic turn the proxy runs its injected memory tools and sends
|
||||
the model a continuation. That continuation replays the whole assistant turn,
|
||||
so a client tool called alongside a memory tool used to go upstream with no
|
||||
``tool_result`` and the provider rejected it with a 400.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
MEMORY_TOOLS = ("memory_save", "memory_search")
|
||||
|
||||
|
||||
class _MemoryHandler:
|
||||
def __init__(self) -> None:
|
||||
self.config = type(
|
||||
"MemoryConfig",
|
||||
(),
|
||||
{"inject_context": False, "inject_tools": True, "project_root_override": ""},
|
||||
)()
|
||||
self.initialized = False
|
||||
self.backend = None
|
||||
self.executed: list[str] = []
|
||||
|
||||
def compute_memory_tool_definitions(self, provider: str) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{"name": name, "description": name, "input_schema": {"type": "object"}}
|
||||
for name in MEMORY_TOOLS
|
||||
]
|
||||
|
||||
def get_beta_headers(self) -> dict[str, str]:
|
||||
return {}
|
||||
|
||||
def has_memory_tool_calls(self, response: dict[str, Any], provider: str) -> bool:
|
||||
return bool(self._memory_calls(response))
|
||||
|
||||
async def handle_memory_tool_calls(
|
||||
self, response: dict[str, Any], user_id: str, provider: str, **kwargs: Any
|
||||
) -> list[dict[str, Any]]:
|
||||
# Like the real handler: results only for memory tools.
|
||||
results = []
|
||||
for block in self._memory_calls(response):
|
||||
self.executed.append(block["name"])
|
||||
results.append({"type": "tool_result", "tool_use_id": block["id"], "content": "ok"})
|
||||
return results
|
||||
|
||||
@staticmethod
|
||||
def _memory_calls(response: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
return [
|
||||
block
|
||||
for block in response.get("content") or []
|
||||
if block.get("type") == "tool_use" and block.get("name") in MEMORY_TOOLS
|
||||
]
|
||||
|
||||
|
||||
def _message(content: list[dict[str, Any]], stop_reason: str) -> dict[str, Any]:
|
||||
return {
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"content": content,
|
||||
"stop_reason": stop_reason,
|
||||
"usage": {"input_tokens": 10, "output_tokens": 5},
|
||||
}
|
||||
|
||||
|
||||
MEMORY_CALL = {
|
||||
"type": "tool_use",
|
||||
"id": "toolu_mem",
|
||||
"name": "memory_search",
|
||||
"input": {"query": "deploy region"},
|
||||
}
|
||||
CLIENT_CALL = {"type": "tool_use", "id": "toolu_bash", "name": "Bash", "input": {"command": "ls"}}
|
||||
|
||||
|
||||
def _run(upstream: list[httpx.Response]) -> tuple[httpx.Response, list[dict], _MemoryHandler]:
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
image_optimize=False,
|
||||
)
|
||||
sent: list[dict] = []
|
||||
handler = _MemoryHandler()
|
||||
with TestClient(create_app(config)) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy.memory_handler = handler
|
||||
|
||||
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
||||
sent.append(json.loads(json.dumps(body)))
|
||||
return upstream[len(sent) - 1]
|
||||
|
||||
proxy._retry_request = _fake_retry # type: ignore[assignment]
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
headers={
|
||||
"x-api-key": "test-key",
|
||||
"anthropic-version": "2023-06-01",
|
||||
"x-headroom-user-id": "u1",
|
||||
},
|
||||
json={
|
||||
"model": "claude-sonnet-4-6",
|
||||
"max_tokens": 64,
|
||||
"tools": [{"name": "Bash", "input_schema": {"type": "object"}}],
|
||||
"messages": [{"role": "user", "content": "check the deploy region"}],
|
||||
},
|
||||
)
|
||||
return response, sent, handler
|
||||
|
||||
|
||||
def test_mixed_turn_returns_client_tool_without_continuation() -> None:
|
||||
first = _message([MEMORY_CALL, CLIENT_CALL], "tool_use")
|
||||
response, sent, handler = _run([httpx.Response(200, json=first)])
|
||||
|
||||
assert response.status_code == 200, response.text
|
||||
# No continuation: it could not answer the Bash call.
|
||||
assert len(sent) == 1
|
||||
assert handler.executed == ["memory_search"]
|
||||
# The client gets its own tool call back, not the proxy's.
|
||||
assert [block["name"] for block in response.json()["content"]] == ["Bash"]
|
||||
|
||||
|
||||
def test_memory_only_turn_continues_with_a_result_for_every_call() -> None:
|
||||
first = _message([MEMORY_CALL], "tool_use")
|
||||
final = _message([{"type": "text", "text": "us-east-1"}], "end_turn")
|
||||
response, sent, _ = _run([httpx.Response(200, json=first), httpx.Response(200, json=final)])
|
||||
|
||||
assert len(sent) == 2
|
||||
assistant, user = sent[1]["messages"][-2:]
|
||||
assert assistant == {"role": "assistant", "content": [MEMORY_CALL]}
|
||||
assert [r["tool_use_id"] for r in user["content"]] == ["toolu_mem"]
|
||||
assert sent[1]["messages"][:-2] == sent[0]["messages"]
|
||||
assert response.json()["content"] == [{"type": "text", "text": "us-east-1"}]
|
||||
|
||||
|
||||
def test_failed_continuation_is_not_logged_as_complete(caplog: pytest.LogCaptureFixture) -> None:
|
||||
first = _message([MEMORY_CALL], "tool_use")
|
||||
rejected = httpx.Response(
|
||||
400, json={"type": "error", "error": {"type": "invalid_request_error", "message": "no"}}
|
||||
)
|
||||
with caplog.at_level(logging.INFO, logger="headroom.proxy"):
|
||||
_run([httpx.Response(200, json=first), rejected])
|
||||
|
||||
assert "Memory: Continuation failed with upstream status 400" in caplog.text
|
||||
assert "continuation complete" not in caplog.text
|
||||
Reference in New Issue
Block a user