fix(backends/litellm): report requested model in OpenAI streaming chunks (#2690)

## Description

`stream_openai_message` in the LiteLLM backend forwards each LiteLLM
chunk to the client with `chunk.model_dump()` untouched, so the `model`
field carries the LiteLLM-mapped provider slug the backend was called
with (for example `openrouter/qwen3`, or
`bedrock/us.anthropic.claude-...`) rather than the model the client
requested.

The non-streaming sibling `send_openai_message` already rewrites the
response `model` to `original_model` (the requested name). So the two
paths disagree on the same request: an OpenAI-format client that keys
cost or telemetry on the `model` field sees the requested name for a
buffered reply, but an unrecognized provider slug for every streamed
chunk. A client that prices or attributes by model name then reads that
slug as an unknown model on streamed turns, while real OpenAI streaming
reports the requested model in every chunk.

The fix rewrites the chunk `model` to `original_model` before
serializing, matching the non-streaming path and real OpenAI streaming
behavior.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/backends/litellm.py` (`stream_openai_message`): set
`chunk_dict["model"] = original_model` (when the chunk carries a model
field) before yielding, mirroring `send_openai_message`.
- `tests/test_litellm_openai_passthrough.py`: added
`test_streaming_chunks_report_requested_model_not_mapped_slug`, which
streams a chunk tagged `openrouter/qwen3` and asserts the client sees
`qwen3`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Fail-before (source fix stashed, new test kept):
tests/test_litellm_openai_passthrough.py::test_streaming_chunks_report_requested_model_not_mapped_slug FAILED
  assert payload["model"] == "qwen3"
  AssertionError: assert 'openrouter/qwen3' == 'qwen3'

# Pass-after (fix applied):
tests/test_litellm_openai_passthrough.py .....
5 passed in 4.94s

# Related streaming suites:
tests/test_backend_anyllm.py tests/test_backend_streaming_cache_metrics.py
20 passed

# uvx ruff@0.15.17 check  -> All checks passed!
# uvx mypy@1.20.2 headroom/backends/litellm.py -> Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, litellm installed in the
project venv, pytest 9.1.1 with pytest-asyncio 1.4.0 (asyncio_mode=auto
per pyproject).
- Exact command / steps: confirmed the non-streaming path rewrites the
model by reading `send_openai_message` (`"model": original_model` in the
response dict), then fail-before with `git stash push
headroom/backends/litellm.py` and `python -m pytest
tests/test_litellm_openai_passthrough.py -k requested_model` (the chunk
reaches the client as `openrouter/qwen3`), and pass-after with `git
stash pop` and rerunning the file (5 passed).
- Observed result: streamed OpenAI chunks now report the requested model
(`qwen3`) rather than the LiteLLM-mapped slug (`openrouter/qwen3`), so
the streaming and non-streaming responses agree on the model field.
- Not tested: a live streamed call through a real provider (no provider
credentials in this environment). The mapped-slug behavior is reproduced
with a mock chunk that mirrors what LiteLLM tags chunks with, and the
non-streaming rewrite it matches is existing, tested code.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Docs and manual testing are N/A: this aligns the streaming model field
with the already-correct non-streaming path, no API surface change. The
rewrite is guarded on the chunk actually carrying a `model` field, so
chunks that omit it (if any) are left untouched.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Abhay Singh
2026-10-01 16:29:59 -05:00
committed by GitHub
co-authored by JD Davis
parent ffc359976b
commit 8538a83153
2 changed files with 43 additions and 0 deletions
+9
View File
@@ -1979,6 +1979,15 @@ class LiteLLMBackend(Backend):
async for chunk in response:
chunk_dict = chunk.model_dump(exclude_none=True, exclude_unset=True)
# Report the model the client requested, not the LiteLLM-mapped
# provider slug (e.g. "openrouter/qwen3",
# "bedrock/us.anthropic.claude-..."). send_openai_message already
# rewrites the model to original_model on the non-streaming path;
# without this the streaming and non-streaming responses disagree
# and OpenAI clients that key cost/telemetry on the model field
# see an unrecognized name for every streamed request.
if "model" in chunk_dict:
chunk_dict["model"] = original_model
yield f"data: {json.dumps(chunk_dict)}\n\n"
yield "data: [DONE]\n\n"
+34
View File
@@ -105,6 +105,40 @@ async def test_chat_template_kwargs_forwarded_streaming() -> None:
assert chunks[-1] == "data: [DONE]\n\n"
@pytest.mark.asyncio
async def test_streaming_chunks_report_requested_model_not_mapped_slug() -> None:
"""Streamed chunks must carry the client's requested model, not the
LiteLLM-mapped provider slug (matches the non-streaming path)."""
import json
backend = make_backend() # provider="openrouter"
# LiteLLM tags each chunk with the mapped model it was called with
# ("openrouter/qwen3"); the client asked for "qwen3".
stream = FakeAsyncStream(
[
SimpleNamespace(
model_dump=lambda **kwargs: {
"id": "chunk1",
"model": "openrouter/qwen3",
"choices": [{"index": 0, "delta": {"content": "hi"}}],
}
),
]
)
with patch("headroom.backends.litellm.acompletion", new_callable=AsyncMock) as mock_acomp:
mock_acomp.return_value = stream
chunks = [
chunk async for chunk in backend.stream_openai_message(request_body(model="qwen3"), {})
]
data_chunks = [c for c in chunks if c.startswith("data: ") and "[DONE]" not in c]
payload = json.loads(data_chunks[0][len("data: ") :])
assert payload["model"] == "qwen3"
@pytest.mark.asyncio
async def test_standard_only_body_has_no_extra_body() -> None:
backend = make_backend()