fix(backends/litellm): keep cache_control on text blocks for Bedrock prompt caching (#4565)

## Description

Thanks for #2144 and #2225, which keep `cache_control` on `tool_result`
blocks and the `system` field. Text blocks are the one marker class
still lost on the LiteLLM path: `_convert_messages_for_litellm` joins a
text-only turn into a single string, so a breakpoint on a user prompt
never becomes a Bedrock Converse `cachePoint`. This keeps it there, only
on turns that carry a marker.

The [comment at
L1087-L1092](https://github.com/headroomlabs-ai/headroom/blob/a2d4fe72/headroom/backends/litellm.py#L1087-L1092)
still credits #1390 with this case, but #1390 was closed as superseded
by those two (eilhamv flagged the gap on #2151).

Related to #1345

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/backends/litellm.py` (+9/−1): text parts keep
`cache_control`; a marked turn is sent as parts.
- `tests/test_bedrock_tool_result_cache_and_streaming_stats.py` (+34): 2
tests beside the `tool_result` ones.

| Case (litellm 1.101.3 Converse transform, sync and async) | Before |
After |
|---|---|---|
| User turn, marked text block | ❌ joined string, 0 `cachePoint` | ✅
`[text, cachePoint, text]` |
| User text + image, text marked | ❌ 0 `cachePoint` | ✅ `[image, text,
cachePoint]` |
| Assistant turn, signed thinking + marked text | ❌ 0 `cachePoint` | ✅
`[reasoningContent, text, cachePoint]` (an assistant image is still
dropped) |
| Unmarked text-only turns | joined string | ✅ unchanged |
| `tool_result`, `tool_use`, `system` paths | ✅ | ✅ unchanged |
| Non-Bedrock targets (litellm OpenAI transform) | string | ✅ text list,
litellm strips `cache_control` |

## Testing

<details><summary>5 of 5 checked</summary>

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

</details>

### Test Output

```text
$ pytest tests/test_bedrock_tool_result_cache_and_streaming_stats.py -k TextBlockCacheControl   # source reverted to main
E   AssertionError: assert [{'text': 'St...\nQuestion?'}] == [{'text': 'St... 'Question?'}]
E   AssertionError: assert 'Here is one.' == [{'type': 'text', 'text': 'Here is one.', 'cache_control': {'type': 'ephemeral'}}]
2 failed, 16 deselected
$ pytest tests/test_bedrock_tool_result_cache_and_streaming_stats.py   # this branch
18 passed
$ pytest <the 21 test files that import the LiteLLM backend>
source reverted: 2 failed, 541 passed, 12 skipped    this branch: 543 passed, 12 skipped
$ ruff check <changed files> && ruff format --check <changed files>   # ruff 0.17.0
All checks passed!  2 files already formatted
$ mypy headroom --ignore-missing-imports   # mypy 1.20.2
Success: no issues found in 579 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, litellm 1.101.3 (uv.lock), main at
`a2d4fe72`, Bedrock API key,
`us.anthropic.claude-haiku-4-5-20251001-v1:0`.
- Exact command / steps: `headroom proxy --backend bedrock --region
us-east-1` (default `--mode cache`), then this request twice per mode
with a fresh prefix, on main and on this branch:

```json
{"model": "claude-haiku-4-5-20251001", "max_tokens": 8, "stream": false,
 "messages": [{"role": "user", "content": [
   {"type": "text", "text": "<~10.8k-token prefix>", "cache_control": {"type": "ephemeral"}},
   {"type": "text", "text": "Reply with the word ok."}]}]}
```

- Observed result: main never touches the cache; this branch writes the
prefix on call 1 and reads it on call 2 (8/8 HTTP 200).
- Not tested: live Claude Code; other providers live; tool definitions
(`_convert_anthropic_tool` still drops their `cache_control`).

```text
                       call 1                                   call 2
main    non-stream     input=10824 cache_write=0     cache_read=0   input=10824 cache_write=0 cache_read=0
main    stream=true    input=10822 cache_write=0     cache_read=0   input=10822 cache_write=0 cache_read=0
branch  non-stream     input=9     cache_write=10811 cache_read=0   input=9     cache_write=0 cache_read=10811
branch  stream=true    input=9     cache_write=10812 cache_read=0   input=9     cache_write=0 cache_read=10812
```

## Runtime Rollout Safety

<details><summary>No rollout gate</summary>

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, marked text-only turns become
text parts.
- Kill switch / disable path: none.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

</details>

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4565"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Main CI unblock (maintainer requested)

Also repairs main test shard 3 ([failing
job](https://github.com/headroomlabs-ai/headroom/actions/runs/41756408770/job/126336450772)).
LiteLLM 1.101.3 no longer prices the retired `groq/llama-guard-3-8b`
fixture. Replace that catalog-dependent example with a controlled bare
catalog entry while exercising the real gateway-prefixed provider
lookup. The assertion still rejects the unknown-model fallback; existing
live-catalog provider cases remain. Production pricing is unchanged.

Validation: reproduced the exact original failure with locked LiteLLM
1.101.3; all 10 pricing tests and 54 combined
pricing/backend/cost-card/provider-matrix tests pass after the fix.
Disabling gateway unwrapping makes the new regression fail, confirming
coverage is retained. Ruff lint/format and whitespace checks pass.
Hosted qualification is pending on the added commit.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Lucas Kim
2026-11-03 20:54:33 -06:00
committed by GitHub
co-authored by Claude JD Davis
parent 4d7765c84b
commit 23a5bb1c81
3 changed files with 62 additions and 7 deletions
+9 -1
View File
@@ -1037,7 +1037,10 @@ class LiteLLMBackend(Backend):
block_type = block.get("type", "")
if block_type == "text":
text_parts.append(block.get("text", ""))
parts.append({"type": "text", "text": block.get("text", "")})
text_part: dict[str, Any] = {"type": "text", "text": block.get("text", "")}
if "cache_control" in block:
text_part["cache_control"] = block["cache_control"]
parts.append(text_part)
elif block_type == "image":
image_part = _anthropic_image_to_openai(block)
if image_part:
@@ -1131,6 +1134,11 @@ class LiteLLMBackend(Backend):
# assistant-turn image, which this code has always dropped.
if has_image and role == "user":
simple_msg["content"] = parts
elif any("cache_control" in p for p in parts):
# A joined string can't carry the breakpoint: litellm's Bedrock
# Converse transform only emits a cachePoint for a marked content
# part, as with the tool_result and system paths.
simple_msg["content"] = [p for p in parts if p["type"] == "text"]
if preserve_thinking and thinking_blocks and role == "assistant":
simple_msg["thinking_blocks"] = thinking_blocks
converted.append(simple_msg)
@@ -114,6 +114,40 @@ class TestToolResultCacheControlPreserved:
assert converted[0]["cache_control"] == {"type": "ephemeral"}
class TestTextBlockCacheControlPreserved:
"""Plain text blocks must keep cache_control too: a new user prompt puts the
breakpoint on a text block, and #1390 (which covered this branch) was closed."""
def test_marked_text_block_becomes_a_converse_cache_point(self):
from litellm.litellm_core_utils.prompt_templates.factory import (
_bedrock_converse_messages_pt,
)
backend = _backend()
marked = {"type": "text", "text": "Stable prefix.", "cache_control": {"type": "ephemeral"}}
messages = [{"role": "user", "content": [marked, {"type": "text", "text": "Question?"}]}]
converted = backend._convert_messages_for_litellm(messages)
converse = _bedrock_converse_messages_pt(
messages=converted,
model="global.anthropic.claude-sonnet-5",
llm_provider="bedrock_converse",
)
assert converse[0]["content"] == [
{"text": "Stable prefix."},
{"cachePoint": {"type": "default"}},
{"text": "Question?"},
]
def test_marked_assistant_turn_still_drops_its_image(self):
backend = _backend()
image = {"type": "image", "source": {"type": "url", "url": "https://x/a.png"}}
marked = {"type": "text", "text": "Here is one.", "cache_control": {"type": "ephemeral"}}
messages = [{"role": "assistant", "content": [image, marked]}]
converted = backend._convert_messages_for_litellm(messages)
assert converted[0]["content"] == [marked]
class TestStreamingCacheStatsCompletion:
"""stream_message must request usage and re-surface real cache stats."""
+19 -6
View File
@@ -41,12 +41,6 @@ def test_unwrapped_model_forms_drops_leading_segments() -> None:
("bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0", 3.00, 15.00),
("bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", 3.00, 15.00),
("vertex_ai/claude-sonnet-4-5", 3.00, 15.00),
# Any gateway-prefixed name here must be one litellm still prices:
# when it prunes a model the unwrap finds nothing and _get_pricing
# silently returns the $2.50/$10.00 GPT-4o default, which is what
# this test exists to catch. litellm dropped
# groq/llama-3.3-70b-versatile on 2026-09-23; see issue #3732.
("groq/llama-guard-3-8b", 0.20, 0.20),
# Non-OpenAI models reachable through the OpenAI-compatible passthrough.
("gemini-2.5-flash", 0.30, 2.50),
("deepseek-chat", 0.28, 0.42),
@@ -60,6 +54,25 @@ def test_provider_prices_models_its_table_never_covered(
assert (round(got_in, 2), round(got_out, 2)) == (want_in, want_out)
def test_gateway_pricing_uses_unwrapped_catalog_entry(monkeypatch: pytest.MonkeyPatch) -> None:
"""Retiring a vendor model must not break the gateway-resolution regression."""
bare_model = "headroom-pricing-fixture"
monkeypatch.setattr(
litellm,
"model_cost",
{
bare_model: {
"input_cost_per_token": 0.20 / 1_000_000,
"output_cost_per_token": 0.20 / 1_000_000,
}
},
)
# Only the unwrapped key exists. Returning the unknown-model default instead
# of looking through the gateway prefix must still fail this assertion.
assert OpenAIProvider()._get_pricing(f"groq/{bare_model}") == pytest.approx((0.20, 0.20))
def test_explicit_config_outranks_litellm() -> None:
"""A configured price is a decision, not a guess."""
provider = OpenAIProvider()