mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(backends/litellm): report requested model in OpenAI streaming chunks (#2690)
## Description `stream_openai_message` in the LiteLLM backend forwards each LiteLLM chunk to the client with `chunk.model_dump()` untouched, so the `model` field carries the LiteLLM-mapped provider slug the backend was called with (for example `openrouter/qwen3`, or `bedrock/us.anthropic.claude-...`) rather than the model the client requested. The non-streaming sibling `send_openai_message` already rewrites the response `model` to `original_model` (the requested name). So the two paths disagree on the same request: an OpenAI-format client that keys cost or telemetry on the `model` field sees the requested name for a buffered reply, but an unrecognized provider slug for every streamed chunk. A client that prices or attributes by model name then reads that slug as an unknown model on streamed turns, while real OpenAI streaming reports the requested model in every chunk. The fix rewrites the chunk `model` to `original_model` before serializing, matching the non-streaming path and real OpenAI streaming behavior. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/backends/litellm.py` (`stream_openai_message`): set `chunk_dict["model"] = original_model` (when the chunk carries a model field) before yielding, mirroring `send_openai_message`. - `tests/test_litellm_openai_passthrough.py`: added `test_streaming_chunks_report_requested_model_not_mapped_slug`, which streams a chunk tagged `openrouter/qwen3` and asserts the client sees `qwen3`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Fail-before (source fix stashed, new test kept): tests/test_litellm_openai_passthrough.py::test_streaming_chunks_report_requested_model_not_mapped_slug FAILED assert payload["model"] == "qwen3" AssertionError: assert 'openrouter/qwen3' == 'qwen3' # Pass-after (fix applied): tests/test_litellm_openai_passthrough.py ..... 5 passed in 4.94s # Related streaming suites: tests/test_backend_anyllm.py tests/test_backend_streaming_cache_metrics.py 20 passed # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/backends/litellm.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, litellm installed in the project venv, pytest 9.1.1 with pytest-asyncio 1.4.0 (asyncio_mode=auto per pyproject). - Exact command / steps: confirmed the non-streaming path rewrites the model by reading `send_openai_message` (`"model": original_model` in the response dict), then fail-before with `git stash push headroom/backends/litellm.py` and `python -m pytest tests/test_litellm_openai_passthrough.py -k requested_model` (the chunk reaches the client as `openrouter/qwen3`), and pass-after with `git stash pop` and rerunning the file (5 passed). - Observed result: streamed OpenAI chunks now report the requested model (`qwen3`) rather than the LiteLLM-mapped slug (`openrouter/qwen3`), so the streaming and non-streaming responses agree on the model field. - Not tested: a live streamed call through a real provider (no provider credentials in this environment). The mapped-slug behavior is reproduced with a mock chunk that mirrors what LiteLLM tags chunks with, and the non-streaming rewrite it matches is existing, tested code. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Docs and manual testing are N/A: this aligns the streaming model field with the already-correct non-streaming path, no API surface change. The rewrite is guarded on the chunk actually carrying a `model` field, so chunks that omit it (if any) are left untouched. Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -1979,6 +1979,15 @@ class LiteLLMBackend(Backend):
|
||||
|
||||
async for chunk in response:
|
||||
chunk_dict = chunk.model_dump(exclude_none=True, exclude_unset=True)
|
||||
# Report the model the client requested, not the LiteLLM-mapped
|
||||
# provider slug (e.g. "openrouter/qwen3",
|
||||
# "bedrock/us.anthropic.claude-..."). send_openai_message already
|
||||
# rewrites the model to original_model on the non-streaming path;
|
||||
# without this the streaming and non-streaming responses disagree
|
||||
# and OpenAI clients that key cost/telemetry on the model field
|
||||
# see an unrecognized name for every streamed request.
|
||||
if "model" in chunk_dict:
|
||||
chunk_dict["model"] = original_model
|
||||
yield f"data: {json.dumps(chunk_dict)}\n\n"
|
||||
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
@@ -105,6 +105,40 @@ async def test_chat_template_kwargs_forwarded_streaming() -> None:
|
||||
assert chunks[-1] == "data: [DONE]\n\n"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_streaming_chunks_report_requested_model_not_mapped_slug() -> None:
|
||||
"""Streamed chunks must carry the client's requested model, not the
|
||||
LiteLLM-mapped provider slug (matches the non-streaming path)."""
|
||||
import json
|
||||
|
||||
backend = make_backend() # provider="openrouter"
|
||||
|
||||
# LiteLLM tags each chunk with the mapped model it was called with
|
||||
# ("openrouter/qwen3"); the client asked for "qwen3".
|
||||
stream = FakeAsyncStream(
|
||||
[
|
||||
SimpleNamespace(
|
||||
model_dump=lambda **kwargs: {
|
||||
"id": "chunk1",
|
||||
"model": "openrouter/qwen3",
|
||||
"choices": [{"index": 0, "delta": {"content": "hi"}}],
|
||||
}
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
with patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp:
|
||||
mock_acomp.return_value = stream
|
||||
|
||||
chunks = [
|
||||
chunk async for chunk in backend.stream_openai_message(request_body(model="qwen3"), {})
|
||||
]
|
||||
|
||||
data_chunks = [c for c in chunks if c.startswith("data: ") and "[DONE]" not in c]
|
||||
payload = json.loads(data_chunks[0][len("data: ") :])
|
||||
assert payload["model"] == "qwen3"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_standard_only_body_has_no_extra_body() -> None:
|
||||
backend = make_backend()
|
||||
|
||||
Reference in New Issue
Block a user