mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(backends/litellm): stop dropping image blocks from /v1/messages requests (#3753)
## Description Thanks for the LiteLLM backend and the careful converter work in #3586. `_convert_messages_for_litellm` had no `image` branch: text+image turns lost the image and image-only turns were dropped. User-turn images now map to `image_url` parts, as `AnyLLMBackend._convert_content_blocks` already does. Fixes #3752 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - `headroom/backends/litellm.py`: `_anthropic_image_to_openai()`; user turns with an image send ordered text + image parts. - `tests/test_backend_bugs.py`: `TestConvertMessagesImageBlocks` (3 tests). | Case | Before | After | |---|---|---| | User text + image | ❌ image dropped | ✅ `[image_url, text]` → Converse `[image, text]` | | Image-only user turn | ❌ `""`, turn removed | ✅ `[image_url]` | | Text-only turns, tool_use / tool_result, thinking | ✅ | ✅ unchanged | | Assistant-turn image | dropped, 200 | ✅ unchanged (mapping it makes litellm's Bedrock transform raise, so it stays dropped) | | `/v1/chat/completions`, Rust Bedrock proxy | ✅ | ✅ not touched | ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_backend_bugs.py -k Image # source reverted to main E AssertionError: assert 'What color?' == [{'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0KGgo='}}, {'type': 'text', 'text': 'What color?'}] E AssertionError: assert '' == [{'type': 'image_url', 'image_url': {'url': 'https://x/a.png'}}] 2 failed, 1 passed $ pytest tests/test_backend_bugs.py -k Image # this branch 3 passed $ pytest <17 test files that import the LiteLLM backend> main: 231 passed, 3 skipped branch: 234 passed, 3 skipped $ ruff check . && ruff format --check . All checks passed! 1631 files already formatted $ mypy headroom --ignore-missing-imports # mypy 1.20.2, as in CI Success: no issues found in 538 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, litellm 1.102.1, `headroom proxy --backend bedrock --region us-east-1`, Bedrock API key. - Exact command / steps: the issue's request and the variants below, on main then this branch. - Observed result: every model below answers red 3/3 here and 0/3 on main; Converse now gets the `image` block. - Not tested: `tool_result` images (follow-up), Rust Bedrock proxy, non-Bedrock providers. ```text # POST /v1/messages, 64x64 red PNG + "What single color is this image? One word.", 3 tries per row (one main answer shown). # text+img = one user turn; img-only = image as the last user turn. Converse body captured at httpx. main this branch us.openai.gpt-6-sol text+img 0/3 red ('Unknown') user:[text] 3/3 red user:[image, text] us.openai.gpt-6-sol img-only 0/3 red ('Please upload the image.') turn gone 3/3 red user:[image] us.openai.gpt-6-luna text+img 0/3 red ('White') 3/3 red us.openai.gpt-6-luna img-only 0/3 red ('Unclear.') 3/3 red us.openai.gpt-6-astra text+img 0/3 red ('Unknown') 3/3 red us.openai.gpt-6-astra img-only 0/3 red ('Missing.') 3/3 red global.openai.gpt-6-astra text+img 0/3 red ('Image?') 3/3 red global.openai.gpt-6-astra img-only 0/3 red ('Please upload the image.') 3/3 red us.openai.gpt-5.6-sol text+img 0/3 red ('Unknown') 3/3 red us.openai.gpt-5.6-sol img-only 0/3 red ('Upload') 3/3 red claude-haiku-4-5 text+img 0/3 red ('Blue.') 3/3 red claude-haiku-4-5 img-only 0/3 red ('') 3/3 red stream=true, GPT-6 Sol/Luna/Astra 'Unknown' / 'White' / 'Unknown' 'Red' / 'Red' / 'Red' assistant-turn image, GPT-6 x3 200 'OK' 200 'OK' url-source image, GPT-6 Sol/Astra 'No image attached.' / 'No image is attached.' 'Community Savings' / 'Yes, COMMUNITY SAVINGS.' # url-source row: source.type=url pointing at this repo's headroom-savings.png, asked for the word in its title. ``` ## Runtime Rollout Safety <details><summary>No rollout gate</summary> - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: user-turn images on the LiteLLM `/v1/messages` path are now forwarded. - Kill switch / disable path: none. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist <details><summary>7 of 8 checked (docs left open: no docs change)</summary> - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` </details> ## Screenshots (if applicable) N/A ## Additional Notes No docs change. Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude
JD Davis
parent
55c78deaa8
commit
577336d2b6
@@ -588,6 +588,23 @@ def _parse_tool_arguments(arguments: Any) -> Any:
|
||||
return arguments
|
||||
|
||||
|
||||
def _anthropic_image_to_openai(block: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Convert an Anthropic ``image`` block to an OpenAI ``image_url`` part.
|
||||
|
||||
Same mapping as ``AnyLLMBackend._convert_content_blocks``. Returns None for a
|
||||
source type it does not know, so that block is skipped as before.
|
||||
"""
|
||||
source = block.get("source") or {}
|
||||
if source.get("type") == "base64":
|
||||
media_type = source.get("media_type", "image/png")
|
||||
url = f"data:{media_type};base64,{source.get('data', '')}"
|
||||
elif source.get("type") == "url":
|
||||
url = source.get("url", "")
|
||||
else:
|
||||
return None
|
||||
return {"type": "image_url", "image_url": {"url": url}}
|
||||
|
||||
|
||||
def _is_anthropic_family_model(litellm_model: str) -> bool:
|
||||
"""True when the resolved litellm target speaks the Anthropic Messages
|
||||
dialect: the Anthropic API, Bedrock-Claude, Vertex-Claude, Azure-Claude.
|
||||
@@ -884,6 +901,9 @@ class LiteLLMBackend(Backend):
|
||||
tool_use_blocks = []
|
||||
tool_result_blocks = []
|
||||
thinking_blocks: list[dict[str, Any]] = []
|
||||
# Ordered text + image parts; only used when the turn has an image.
|
||||
parts: list[dict[str, Any]] = []
|
||||
has_image = False
|
||||
|
||||
for block in content:
|
||||
if not isinstance(block, dict):
|
||||
@@ -891,6 +911,12 @@ class LiteLLMBackend(Backend):
|
||||
block_type = block.get("type", "")
|
||||
if block_type == "text":
|
||||
text_parts.append(block.get("text", ""))
|
||||
parts.append({"type": "text", "text": block.get("text", "")})
|
||||
elif block_type == "image":
|
||||
image_part = _anthropic_image_to_openai(block)
|
||||
if image_part:
|
||||
parts.append(image_part)
|
||||
has_image = True
|
||||
elif block_type == "tool_use":
|
||||
tool_use_blocks.append(block)
|
||||
elif block_type == "tool_result":
|
||||
@@ -966,6 +992,10 @@ class LiteLLMBackend(Backend):
|
||||
"role": role,
|
||||
"content": "\n".join(text_parts) if text_parts else "",
|
||||
}
|
||||
# User turns only: litellm's Bedrock transform raises on an
|
||||
# assistant-turn image, which this code has always dropped.
|
||||
if has_image and role == "user":
|
||||
simple_msg["content"] = parts
|
||||
if preserve_thinking and thinking_blocks and role == "assistant":
|
||||
simple_msg["thinking_blocks"] = thinking_blocks
|
||||
converted.append(simple_msg)
|
||||
|
||||
@@ -399,6 +399,71 @@ class TestConvertMessagesToolBlocks:
|
||||
assert converted[0]["content"] == "42"
|
||||
|
||||
|
||||
class TestConvertMessagesImageBlocks:
|
||||
"""Anthropic image blocks must survive _convert_messages_for_litellm."""
|
||||
|
||||
PNG_B64 = "iVBORw0KGgo="
|
||||
|
||||
def _make_backend(self):
|
||||
with patch("headroom.backends.litellm._fetch_bedrock_inference_profiles", return_value={}):
|
||||
return LiteLLMBackend(provider="bedrock")
|
||||
|
||||
def test_text_and_image_turn_keeps_image(self):
|
||||
"""The image reaches the Bedrock Converse body instead of being dropped."""
|
||||
from litellm.litellm_core_utils.prompt_templates.factory import (
|
||||
_bedrock_converse_messages_pt,
|
||||
)
|
||||
|
||||
backend = self._make_backend()
|
||||
image = {
|
||||
"type": "image",
|
||||
"source": {"type": "base64", "media_type": "image/png", "data": self.PNG_B64},
|
||||
}
|
||||
messages = [
|
||||
{"role": "user", "content": [image, {"type": "text", "text": "What color?"}]},
|
||||
]
|
||||
converted = backend._convert_messages_for_litellm(messages)
|
||||
|
||||
assert converted[0]["content"] == [
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{self.PNG_B64}"}},
|
||||
{"type": "text", "text": "What color?"},
|
||||
]
|
||||
converse = _bedrock_converse_messages_pt(
|
||||
messages=converted, model="us.openai.gpt-6-sol", llm_provider="bedrock_converse"
|
||||
)
|
||||
assert [list(block) for block in converse[0]["content"]] == [["image"], ["text"]]
|
||||
|
||||
def test_image_only_turn_is_not_emptied(self):
|
||||
"""An image-only turn must not become "" (litellm then drops the turn)."""
|
||||
backend = self._make_backend()
|
||||
messages = [
|
||||
{"role": "user", "content": [{"type": "text", "text": "Describe the next image."}]},
|
||||
{"role": "assistant", "content": "Send it."},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "image", "source": {"type": "url", "url": "https://x/a.png"}}],
|
||||
},
|
||||
]
|
||||
converted = backend._convert_messages_for_litellm(messages)
|
||||
|
||||
# Text-only block lists still flatten to a string, as before.
|
||||
assert converted[0]["content"] == "Describe the next image."
|
||||
assert converted[2]["content"] == [
|
||||
{"type": "image_url", "image_url": {"url": "https://x/a.png"}},
|
||||
]
|
||||
|
||||
def test_assistant_turn_image_is_still_dropped(self):
|
||||
"""Assistant-turn images keep the old text-only content (litellm's Bedrock
|
||||
transform raises on them)."""
|
||||
backend = self._make_backend()
|
||||
image = {"type": "image", "source": {"type": "url", "url": "https://x/a.png"}}
|
||||
messages = [
|
||||
{"role": "assistant", "content": [image, {"type": "text", "text": "Here is one."}]},
|
||||
]
|
||||
converted = backend._convert_messages_for_litellm(messages)
|
||||
assert converted[0]["content"] == "Here is one."
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Streaming tool_calls (GitHub Issue — Bug 1)
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user