fix(proxy/anthropic): keep tool_result blocks first when neutralizing headroom_retrieve history (#3874)

## Description

`strip_unsupported_ccr_retrieve_blocks` (#2876) neutralizes a
`headroom_retrieve` `tool_use` / `tool_result` pair into text blocks in
place whenever this turn's `tools` array does not declare
`headroom_retrieve`. When the retrieve call was one of several
**parallel** tool calls and its `tool_result` was not the last block of
the user turn, the repair leaves a text block in front of a surviving
sibling `tool_result`, and Anthropic rejects the whole request:

```
400 messages.N: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_... Each `tool_use` block must have a corresponding `tool_result` block in the next message.
```

Claude Code surfaces this as `API Error: 400 due to tool use concurrency
issues`. Because a resume replays the same transcript, the session stays
broken until the tool is declared again.

Minimal shape:

```
before  assistant: [tool_use headroom_retrieve (A), tool_use Bash (B)]
        user:      [tool_result A ("<tool_use_error>Error: No such tool available: headroom_retrieve</tool_use_error>"), tool_result B]

after   user:      [text "<tool_use_error>...", tool_result B]      <- invalid: B's result no longer leads the turn
```

Fix: after neutralizing, stable-partition each repaired message so the
surviving `tool_result` blocks come first and the neutralized text
follows. Order within each group is kept. Only messages the repair
actually touched are rebuilt; untouched messages (and the original
`messages` object when nothing changed) are returned as before.
Assistant turns carry no `tool_result` blocks, so the partition is a
no-op there and signed thinking blocks keep their coordinates (#3457).

When it fires on current `main`: the repair runs whenever
`headroom_retrieve` is absent from the outbound tools, i.e.
`HEADROOM_CCR_TOOL_INJECTION=lazy`, bypass / optimization-off requests
(`allow_eager=False`), and tool-less side-requests. On wheels before
#3810 (eager injection) it also ran on the first request of every long
session after a proxy restart, because the session branch of
`apply_session_sticky_ccr_tool` gates on the per-process
`SessionCcrTracker` and ignores `history_has_ccr_reference` while the
tracker is cold. That is how it was found: a 399-message Claude Code
transcript from a 0.39.0 user.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: `strip_unsupported_ccr_retrieve_blocks`
stable-partitions a repaired message's content so `tool_result` blocks
lead it.
- `tests/test_ccr_retrieve_history_repair.py`:
`test_parallel_retrieve_keeps_sibling_tool_result_first` covers
`headroom_retrieve` issued in parallel with `Bash`, its result listed
first, plus a trailing user text block. It fails on `main` (text before
the Bash `tool_result`) and passes here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_ccr_retrieve_history_repair.py tests/test_proxy tests/test_ccr_tool_injection_cache_stability.py -q \
    --deselect tests/test_proxy/test_bedrock_passthrough.py::test_oversized_body_returns_413_and_never_forwards
353 passed, 1 deselected, 2 warnings in 43.63s

$ uvx ruff check headroom/proxy/helpers.py tests/test_ccr_retrieve_history_repair.py
All checks passed!
$ uvx ruff format --check headroom/proxy/helpers.py tests/test_ccr_retrieve_history_repair.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/helpers.py
Success: no issues found in 1 source file
```

The deselected bedrock test (`'invalid_request_error' ==
'request_too_large'`) fails identically on `main` at 8ff46dc9 (CI run
36634641584, shard 3) and is unrelated to this change.

## Real Behavior Proof

- Environment: macOS 15, Python 3.11 worktree venv at upstream main
8ff46dc9 plus this commit; the real 399-message Claude Code request body
from the incident (not synthetic).
- Exact command / steps: ran `apply_session_sticky_ccr_tool`,
`strip_unsupported_tool_search_blocks` and
`strip_unsupported_ccr_retrieve_blocks` over the captured messages with
a cold tracker, in three modes (eager + allow_eager, lazy, eager +
allow_eager=False), then checked every assistant `tool_use` turn for a
following user turn whose leading `tool_result` blocks cover all its ids
(the rule Anthropic enforces).
- Observed result: on main, lazy and allow_eager=False both neutralize 2
blocks and leave `messages[256] = [text, tool_result]`, a violation for
`toolu_012Zqt7AE3Xy4ctFGWSjAPw7` (eager + allow_eager injects the tool,
so no repair and no violation). With this change all three modes report
no violations. Replaying the same request through the 0.39.0 handler
stage with a cold tracker (`tool_injection_decision=skip`, 2 blocks
neutralized) produces the identical violation, matching the 400 the user
hit in production.
- Not tested: a live resend of the repaired body to api.anthropic.com;
validity was checked with the structural rule quoted in the 400, not by
the API.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, only for requests the CCR
history repair already rewrites, and only when a neutralized
`tool_result` preceded another `tool_result`; those requests used to 400
and now validate. All other requests are byte-identical.
- Kill switch / disable path: Not applicable; the repair only runs when
`headroom_retrieve` is absent from the outbound tools.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation: N/A, internal repair helper.
- Deliberately left out: making the session branch of
`apply_session_sticky_ccr_tool` honour `history_has_ccr_reference` while
the `SessionCcrTracker` is cold (the sessionless branch already does,
#2440). It would avoid the repair on the first request after a restart
under `lazy` mode, but it changes the tools array on that request, which
is the head of Anthropic's cache key, so it deserves its own review of
the cache trade. This PR only makes the repair's output valid, which is
needed regardless of how often it runs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-09-30 11:05:43 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent 7d856b81f5
commit ccd9fffce9
2 changed files with 46 additions and 1 deletions
+13 -1
View File
@@ -4286,7 +4286,19 @@ def strip_unsupported_ccr_retrieve_blocks(messages: Any, tools: Any) -> tuple[An
if touched:
changed = True
repaired = dict(message)
repaired["content"] = new_content
# Anthropic requires a user turn's tool_result blocks to lead its
# content. When headroom_retrieve ran in parallel with another tool,
# neutralizing its result in place leaves [text, tool_result(sibling)]
# and the request 400s ("tool_use ids were found without tool_result
# blocks immediately after"). Stable-partition so surviving
# tool_results stay first; a no-op for assistant turns, which carry none.
repaired["content"] = [
b for b in new_content if isinstance(b, dict) and b.get("type") == "tool_result"
] + [
b
for b in new_content
if not (isinstance(b, dict) and b.get("type") == "tool_result")
]
out.append(repaired)
else:
out.append(message)
+33
View File
@@ -117,3 +117,36 @@ def test_result_falls_back_to_placeholder_without_text() -> None:
out, n = strip_unsupported_ccr_retrieve_blocks(messages, tools=[])
assert n == 2
assert out[1]["content"][0] == {"type": "text", "text": "[headroom_retrieve result omitted]"}
def test_parallel_retrieve_keeps_sibling_tool_result_first() -> None:
# headroom_retrieve issued in parallel with Bash, its result listed first.
# Neutralizing in place produced user [text, tool_result(Bash)], which
# Anthropic rejects: "`tool_use` ids were found without `tool_result`
# blocks immediately after".
messages = [
{
"role": "assistant",
"content": [
{"type": "tool_use", "id": "toolu_ccr", "name": CCR_TOOL_NAME, "input": {}},
{"type": "tool_use", "id": "toolu_bash", "name": "Bash", "input": {}},
],
},
{
"role": "user",
"content": [
{"type": "tool_result", "tool_use_id": "toolu_ccr", "content": "expanded"},
{"type": "tool_result", "tool_use_id": "toolu_bash", "content": "ok"},
{"type": "text", "text": "keep going"},
],
},
]
out, n = strip_unsupported_ccr_retrieve_blocks(messages, tools=[{"name": "Bash"}])
assert n == 2
assert [b["type"] for b in out[0]["content"]] == ["text", "tool_use"]
assert out[1]["content"] == [
{"type": "tool_result", "tool_use_id": "toolu_bash", "content": "ok"},
{"type": "text", "text": "expanded"},
{"type": "text", "text": "keep going"},
]