fix(backends/litellm): stop dropping image blocks from /v1/messages requests (#3753)

## Description

Thanks for the LiteLLM backend and the careful converter work in #3586.
`_convert_messages_for_litellm` had no `image` branch: text+image turns
lost the image and image-only turns were dropped. User-turn images now
map to `image_url` parts, as `AnyLLMBackend._convert_content_blocks`
already does.

Fixes #3752

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- `headroom/backends/litellm.py`: `_anthropic_image_to_openai()`; user
turns with an image send ordered text + image parts.
- `tests/test_backend_bugs.py`: `TestConvertMessagesImageBlocks` (3
tests).

| Case | Before | After |
|---|---|---|
| User text + image | ❌ image dropped | ✅ `[image_url, text]` → Converse
`[image, text]` |
| Image-only user turn | ❌ `""`, turn removed | ✅ `[image_url]` |
| Text-only turns, tool_use / tool_result, thinking | ✅ | ✅ unchanged |
| Assistant-turn image | dropped, 200 | ✅ unchanged (mapping it makes
litellm's Bedrock transform raise, so it stays dropped) |
| `/v1/chat/completions`, Rust Bedrock proxy | ✅ | ✅ not touched |

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_backend_bugs.py -k Image        # source reverted to main
E   AssertionError: assert 'What color?' == [{'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0KGgo='}}, {'type': 'text', 'text': 'What color?'}]
E   AssertionError: assert '' == [{'type': 'image_url', 'image_url': {'url': 'https://x/a.png'}}]
2 failed, 1 passed
$ pytest tests/test_backend_bugs.py -k Image        # this branch
3 passed
$ pytest <17 test files that import the LiteLLM backend>
main: 231 passed, 3 skipped   branch: 234 passed, 3 skipped
$ ruff check . && ruff format --check .
All checks passed!  1631 files already formatted
$ mypy headroom --ignore-missing-imports             # mypy 1.20.2, as in CI
Success: no issues found in 538 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, litellm 1.102.1, `headroom proxy
--backend bedrock --region us-east-1`, Bedrock API key.
- Exact command / steps: the issue's request and the variants below, on
main then this branch.
- Observed result: every model below answers red 3/3 here and 0/3 on
main; Converse now gets the `image` block.
- Not tested: `tool_result` images (follow-up), Rust Bedrock proxy,
non-Bedrock providers.

```text
# POST /v1/messages, 64x64 red PNG + "What single color is this image? One word.", 3 tries per row (one main answer shown).
# text+img = one user turn; img-only = image as the last user turn. Converse body captured at httpx.
                                    main                                               this branch
us.openai.gpt-6-sol       text+img  0/3 red ('Unknown')                   user:[text]  3/3 red      user:[image, text]
us.openai.gpt-6-sol       img-only  0/3 red ('Please upload the image.')  turn gone    3/3 red      user:[image]
us.openai.gpt-6-luna      text+img  0/3 red ('White')                                  3/3 red
us.openai.gpt-6-luna      img-only  0/3 red ('Unclear.')                               3/3 red
us.openai.gpt-6-astra     text+img  0/3 red ('Unknown')                                3/3 red
us.openai.gpt-6-astra     img-only  0/3 red ('Missing.')                               3/3 red
global.openai.gpt-6-astra text+img  0/3 red ('Image?')                                 3/3 red
global.openai.gpt-6-astra img-only  0/3 red ('Please upload the image.')               3/3 red
us.openai.gpt-5.6-sol     text+img  0/3 red ('Unknown')                                3/3 red
us.openai.gpt-5.6-sol     img-only  0/3 red ('Upload')                                 3/3 red
claude-haiku-4-5          text+img  0/3 red ('Blue.')                                  3/3 red
claude-haiku-4-5          img-only  0/3 red ('')                                       3/3 red
stream=true, GPT-6 Sol/Luna/Astra   'Unknown' / 'White' / 'Unknown'                    'Red' / 'Red' / 'Red'
assistant-turn image, GPT-6 x3      200 'OK'                                           200 'OK'
url-source image, GPT-6 Sol/Astra   'No image attached.' / 'No image is attached.'     'Community Savings' / 'Yes, COMMUNITY SAVINGS.'
# url-source row: source.type=url pointing at this repo's headroom-savings.png, asked for the word in its title.
```

## Runtime Rollout Safety

<details><summary>No rollout gate</summary>

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: user-turn images on the LiteLLM
`/v1/messages` path are now forwarded.
- Kill switch / disable path: none.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

</details>

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

<details><summary>7 of 8 checked (docs left open: no docs
change)</summary>

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

</details>

## Screenshots (if applicable)

N/A

## Additional Notes

No docs change.

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Lucas Kim
2026-09-24 17:14:04 -05:00
committed by GitHub
co-authored by Claude JD Davis
parent 55c78deaa8
commit 577336d2b6
2 changed files with 95 additions and 0 deletions
+30
View File
@@ -588,6 +588,23 @@ def _parse_tool_arguments(arguments: Any) -> Any:
return arguments
def _anthropic_image_to_openai(block: dict[str, Any]) -> dict[str, Any] | None:
"""Convert an Anthropic ``image`` block to an OpenAI ``image_url`` part.
Same mapping as ``AnyLLMBackend._convert_content_blocks``. Returns None for a
source type it does not know, so that block is skipped as before.
"""
source = block.get("source") or {}
if source.get("type") == "base64":
media_type = source.get("media_type", "image/png")
url = f"data:{media_type};base64,{source.get('data', '')}"
elif source.get("type") == "url":
url = source.get("url", "")
else:
return None
return {"type": "image_url", "image_url": {"url": url}}
def _is_anthropic_family_model(litellm_model: str) -> bool:
"""True when the resolved litellm target speaks the Anthropic Messages
dialect: the Anthropic API, Bedrock-Claude, Vertex-Claude, Azure-Claude.
@@ -884,6 +901,9 @@ class LiteLLMBackend(Backend):
tool_use_blocks = []
tool_result_blocks = []
thinking_blocks: list[dict[str, Any]] = []
# Ordered text + image parts; only used when the turn has an image.
parts: list[dict[str, Any]] = []
has_image = False
for block in content:
if not isinstance(block, dict):
@@ -891,6 +911,12 @@ class LiteLLMBackend(Backend):
block_type = block.get("type", "")
if block_type == "text":
text_parts.append(block.get("text", ""))
parts.append({"type": "text", "text": block.get("text", "")})
elif block_type == "image":
image_part = _anthropic_image_to_openai(block)
if image_part:
parts.append(image_part)
has_image = True
elif block_type == "tool_use":
tool_use_blocks.append(block)
elif block_type == "tool_result":
@@ -966,6 +992,10 @@ class LiteLLMBackend(Backend):
"role": role,
"content": "\n".join(text_parts) if text_parts else "",
}
# User turns only: litellm's Bedrock transform raises on an
# assistant-turn image, which this code has always dropped.
if has_image and role == "user":
simple_msg["content"] = parts
if preserve_thinking and thinking_blocks and role == "assistant":
simple_msg["thinking_blocks"] = thinking_blocks
converted.append(simple_msg)
+65
View File
@@ -399,6 +399,71 @@ class TestConvertMessagesToolBlocks:
assert converted[0]["content"] == "42"
class TestConvertMessagesImageBlocks:
"""Anthropic image blocks must survive _convert_messages_for_litellm."""
PNG_B64 = "iVBORw0KGgo="
def _make_backend(self):
with patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}):
return LiteLLMBackend(provider="bedrock")
def test_text_and_image_turn_keeps_image(self):
"""The image reaches the Bedrock Converse body instead of being dropped."""
from litellm.litellm_core_utils.prompt_templates.factory import (
_bedrock_converse_messages_pt,
)
backend = self._make_backend()
image = {
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": self.PNG_B64},
}
messages = [
{"role": "user", "content": [image, {"type": "text", "text": "What color?"}]},
]
converted = backend._convert_messages_for_litellm(messages)
assert converted[0]["content"] == [
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{self.PNG_B64}"}},
{"type": "text", "text": "What color?"},
]
converse = _bedrock_converse_messages_pt(
messages=converted, model="us.openai.gpt-6-sol", llm_provider="bedrock_converse"
)
assert [list(block) for block in converse[0]["content"]] == [["image"], ["text"]]
def test_image_only_turn_is_not_emptied(self):
"""An image-only turn must not become "" (litellm then drops the turn)."""
backend = self._make_backend()
messages = [
{"role": "user", "content": [{"type": "text", "text": "Describe the next image."}]},
{"role": "assistant", "content": "Send it."},
{
"role": "user",
"content": [{"type": "image", "source": {"type": "url", "url": "https://x/a.png"}}],
},
]
converted = backend._convert_messages_for_litellm(messages)
# Text-only block lists still flatten to a string, as before.
assert converted[0]["content"] == "Describe the next image."
assert converted[2]["content"] == [
{"type": "image_url", "image_url": {"url": "https://x/a.png"}},
]
def test_assistant_turn_image_is_still_dropped(self):
"""Assistant-turn images keep the old text-only content (litellm's Bedrock
transform raises on them)."""
backend = self._make_backend()
image = {"type": "image", "source": {"type": "url", "url": "https://x/a.png"}}
messages = [
{"role": "assistant", "content": [image, {"type": "text", "text": "Here is one."}]},
]
converted = backend._convert_messages_for_litellm(messages)
assert converted[0]["content"] == "Here is one."
# =============================================================================
# Streaming tool_calls (GitHub Issue — Bug 1)
# =============================================================================