mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy/anthropic): keep tool_result blocks first when neutralizing headroom_retrieve history (#3874)
## Description `strip_unsupported_ccr_retrieve_blocks` (#2876) neutralizes a `headroom_retrieve` `tool_use` / `tool_result` pair into text blocks in place whenever this turn's `tools` array does not declare `headroom_retrieve`. When the retrieve call was one of several **parallel** tool calls and its `tool_result` was not the last block of the user turn, the repair leaves a text block in front of a surviving sibling `tool_result`, and Anthropic rejects the whole request: ``` 400 messages.N: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_... Each `tool_use` block must have a corresponding `tool_result` block in the next message. ``` Claude Code surfaces this as `API Error: 400 due to tool use concurrency issues`. Because a resume replays the same transcript, the session stays broken until the tool is declared again. Minimal shape: ``` before assistant: [tool_use headroom_retrieve (A), tool_use Bash (B)] user: [tool_result A ("<tool_use_error>Error: No such tool available: headroom_retrieve</tool_use_error>"), tool_result B] after user: [text "<tool_use_error>...", tool_result B] <- invalid: B's result no longer leads the turn ``` Fix: after neutralizing, stable-partition each repaired message so the surviving `tool_result` blocks come first and the neutralized text follows. Order within each group is kept. Only messages the repair actually touched are rebuilt; untouched messages (and the original `messages` object when nothing changed) are returned as before. Assistant turns carry no `tool_result` blocks, so the partition is a no-op there and signed thinking blocks keep their coordinates (#3457). When it fires on current `main`: the repair runs whenever `headroom_retrieve` is absent from the outbound tools, i.e. `HEADROOM_CCR_TOOL_INJECTION=lazy`, bypass / optimization-off requests (`allow_eager=False`), and tool-less side-requests. On wheels before #3810 (eager injection) it also ran on the first request of every long session after a proxy restart, because the session branch of `apply_session_sticky_ccr_tool` gates on the per-process `SessionCcrTracker` and ignores `history_has_ccr_reference` while the tracker is cold. That is how it was found: a 399-message Claude Code transcript from a 0.39.0 user. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: `strip_unsupported_ccr_retrieve_blocks` stable-partitions a repaired message's content so `tool_result` blocks lead it. - `tests/test_ccr_retrieve_history_repair.py`: `test_parallel_retrieve_keeps_sibling_tool_result_first` covers `headroom_retrieve` issued in parallel with `Bash`, its result listed first, plus a trailing user text block. It fails on `main` (text before the Bash `tool_result`) and passes here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_ccr_retrieve_history_repair.py tests/test_proxy tests/test_ccr_tool_injection_cache_stability.py -q \ --deselect tests/test_proxy/test_bedrock_passthrough.py::test_oversized_body_returns_413_and_never_forwards 353 passed, 1 deselected, 2 warnings in 43.63s $ uvx ruff check headroom/proxy/helpers.py tests/test_ccr_retrieve_history_repair.py All checks passed! $ uvx ruff format --check headroom/proxy/helpers.py tests/test_ccr_retrieve_history_repair.py 2 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/helpers.py Success: no issues found in 1 source file ``` The deselected bedrock test (`'invalid_request_error' == 'request_too_large'`) fails identically on `main` at8ff46dc9(CI run 36634641584, shard 3) and is unrelated to this change. ## Real Behavior Proof - Environment: macOS 15, Python 3.11 worktree venv at upstream main8ff46dc9plus this commit; the real 399-message Claude Code request body from the incident (not synthetic). - Exact command / steps: ran `apply_session_sticky_ccr_tool`, `strip_unsupported_tool_search_blocks` and `strip_unsupported_ccr_retrieve_blocks` over the captured messages with a cold tracker, in three modes (eager + allow_eager, lazy, eager + allow_eager=False), then checked every assistant `tool_use` turn for a following user turn whose leading `tool_result` blocks cover all its ids (the rule Anthropic enforces). - Observed result: on main, lazy and allow_eager=False both neutralize 2 blocks and leave `messages[256] = [text, tool_result]`, a violation for `toolu_012Zqt7AE3Xy4ctFGWSjAPw7` (eager + allow_eager injects the tool, so no repair and no violation). With this change all three modes report no violations. Replaying the same request through the 0.39.0 handler stage with a cold tracker (`tool_injection_decision=skip`, 2 blocks neutralized) produces the identical violation, matching the 400 the user hit in production. - Not tested: a live resend of the repaired body to api.anthropic.com; validity was checked with the structural rule quoted in the 400, not by the API. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: Stable. - Stable/default behavior changed: Yes, only for requests the CCR history repair already rewrites, and only when a neutralized `tool_result` preceded another `tool_result`; those requests used to 400 and now validate. All other requests are byte-identical. - Kill switch / disable path: Not applicable; the repair only runs when `headroom_retrieve` is absent from the outbound tools. - Unsafe override required: No. - Qualification impact: None. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Documentation: N/A, internal repair helper. - Deliberately left out: making the session branch of `apply_session_sticky_ccr_tool` honour `history_has_ccr_reference` while the `SessionCcrTracker` is cold (the sessionless branch already does, #2440). It would avoid the repair on the first request after a restart under `lazy` mode, but it changes the tools array on that request, which is the head of Anthropic's cache key, so it deserves its own review of the cache trade. This PR only makes the repair's output valid, which is needed regardless of how often it runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
7d856b81f5
commit
ccd9fffce9
@@ -4286,7 +4286,19 @@ def strip_unsupported_ccr_retrieve_blocks(messages: Any, tools: Any) -> tuple[An
|
||||
if touched:
|
||||
changed = True
|
||||
repaired = dict(message)
|
||||
repaired["content"] = new_content
|
||||
# Anthropic requires a user turn's tool_result blocks to lead its
|
||||
# content. When headroom_retrieve ran in parallel with another tool,
|
||||
# neutralizing its result in place leaves [text, tool_result(sibling)]
|
||||
# and the request 400s ("tool_use ids were found without tool_result
|
||||
# blocks immediately after"). Stable-partition so surviving
|
||||
# tool_results stay first; a no-op for assistant turns, which carry none.
|
||||
repaired["content"] = [
|
||||
b for b in new_content if isinstance(b, dict) and b.get("type") == "tool_result"
|
||||
] + [
|
||||
b
|
||||
for b in new_content
|
||||
if not (isinstance(b, dict) and b.get("type") == "tool_result")
|
||||
]
|
||||
out.append(repaired)
|
||||
else:
|
||||
out.append(message)
|
||||
|
||||
@@ -117,3 +117,36 @@ def test_result_falls_back_to_placeholder_without_text() -> None:
|
||||
out, n = strip_unsupported_ccr_retrieve_blocks(messages, tools=[])
|
||||
assert n == 2
|
||||
assert out[1]["content"][0] == {"type": "text", "text": "[headroom_retrieve result omitted]"}
|
||||
|
||||
|
||||
def test_parallel_retrieve_keeps_sibling_tool_result_first() -> None:
|
||||
# headroom_retrieve issued in parallel with Bash, its result listed first.
|
||||
# Neutralizing in place produced user [text, tool_result(Bash)], which
|
||||
# Anthropic rejects: "`tool_use` ids were found without `tool_result`
|
||||
# blocks immediately after".
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "tool_use", "id": "toolu_ccr", "name": CCR_TOOL_NAME, "input": {}},
|
||||
{"type": "tool_use", "id": "toolu_bash", "name": "Bash", "input": {}},
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "tool_result", "tool_use_id": "toolu_ccr", "content": "expanded"},
|
||||
{"type": "tool_result", "tool_use_id": "toolu_bash", "content": "ok"},
|
||||
{"type": "text", "text": "keep going"},
|
||||
],
|
||||
},
|
||||
]
|
||||
out, n = strip_unsupported_ccr_retrieve_blocks(messages, tools=[{"name": "Bash"}])
|
||||
|
||||
assert n == 2
|
||||
assert [b["type"] for b in out[0]["content"]] == ["text", "tool_use"]
|
||||
assert out[1]["content"] == [
|
||||
{"type": "tool_result", "tool_use_id": "toolu_bash", "content": "ok"},
|
||||
{"type": "text", "text": "expanded"},
|
||||
{"type": "text", "text": "keep going"},
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user