mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description Thanks for #2144 and #2225, which keep `cache_control` on `tool_result` blocks and the `system` field. Text blocks are the one marker class still lost on the LiteLLM path: `_convert_messages_for_litellm` joins a text-only turn into a single string, so a breakpoint on a user prompt never becomes a Bedrock Converse `cachePoint`. This keeps it there, only on turns that carry a marker. The [comment at L1087-L1092](https://github.com/headroomlabs-ai/headroom/blob/a2d4fe72/headroom/backends/litellm.py#L1087-L1092) still credits #1390 with this case, but #1390 was closed as superseded by those two (eilhamv flagged the gap on #2151). Related to #1345 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/backends/litellm.py` (+9/−1): text parts keep `cache_control`; a marked turn is sent as parts. - `tests/test_bedrock_tool_result_cache_and_streaming_stats.py` (+34): 2 tests beside the `tool_result` ones. | Case (litellm 1.101.3 Converse transform, sync and async) | Before | After | |---|---|---| | User turn, marked text block | ❌ joined string, 0 `cachePoint` | ✅ `[text, cachePoint, text]` | | User text + image, text marked | ❌ 0 `cachePoint` | ✅ `[image, text, cachePoint]` | | Assistant turn, signed thinking + marked text | ❌ 0 `cachePoint` | ✅ `[reasoningContent, text, cachePoint]` (an assistant image is still dropped) | | Unmarked text-only turns | joined string | ✅ unchanged | | `tool_result`, `tool_use`, `system` paths | ✅ | ✅ unchanged | | Non-Bedrock targets (litellm OpenAI transform) | string | ✅ text list, litellm strips `cache_control` | ## Testing <details><summary>5 of 5 checked</summary> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed </details> ### Test Output ```text $ pytest tests/test_bedrock_tool_result_cache_and_streaming_stats.py -k TextBlockCacheControl # source reverted to main E AssertionError: assert [{'text': 'St...\nQuestion?'}] == [{'text': 'St... 'Question?'}] E AssertionError: assert 'Here is one.' == [{'type': 'text', 'text': 'Here is one.', 'cache_control': {'type': 'ephemeral'}}] 2 failed, 16 deselected $ pytest tests/test_bedrock_tool_result_cache_and_streaming_stats.py # this branch 18 passed $ pytest <the 21 test files that import the LiteLLM backend> source reverted: 2 failed, 541 passed, 12 skipped this branch: 543 passed, 12 skipped $ ruff check <changed files> && ruff format --check <changed files> # ruff 0.17.0 All checks passed! 2 files already formatted $ mypy headroom --ignore-missing-imports # mypy 1.20.2 Success: no issues found in 579 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, litellm 1.101.3 (uv.lock), main at `a2d4fe72`, Bedrock API key, `us.anthropic.claude-haiku-4-5-20251001-v1:0`. - Exact command / steps: `headroom proxy --backend bedrock --region us-east-1` (default `--mode cache`), then this request twice per mode with a fresh prefix, on main and on this branch: ```json {"model": "claude-haiku-4-5-20251001", "max_tokens": 8, "stream": false, "messages": [{"role": "user", "content": [ {"type": "text", "text": "<~10.8k-token prefix>", "cache_control": {"type": "ephemeral"}}, {"type": "text", "text": "Reply with the word ok."}]}]} ``` - Observed result: main never touches the cache; this branch writes the prefix on call 1 and reads it on call 2 (8/8 HTTP 200). - Not tested: live Claude Code; other providers live; tool definitions (`_convert_anthropic_tool` still drops their `cache_control`). ```text call 1 call 2 main non-stream input=10824 cache_write=0 cache_read=0 input=10824 cache_write=0 cache_read=0 main stream=true input=10822 cache_write=0 cache_read=0 input=10822 cache_write=0 cache_read=0 branch non-stream input=9 cache_write=10811 cache_read=0 input=9 cache_write=0 cache_read=10811 branch stream=true input=9 cache_write=10812 cache_read=0 input=9 cache_write=0 cache_read=10812 ``` ## Runtime Rollout Safety <details><summary>No rollout gate</summary> - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, marked text-only turns become text parts. - Kill switch / disable path: none. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4565" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Main CI unblock (maintainer requested) Also repairs main test shard 3 ([failing job](https://github.com/headroomlabs-ai/headroom/actions/runs/41756408770/job/126336450772)). LiteLLM 1.101.3 no longer prices the retired `groq/llama-guard-3-8b` fixture. Replace that catalog-dependent example with a controlled bare catalog entry while exercising the real gateway-prefixed provider lookup. The assertion still rejects the unknown-model fallback; existing live-catalog provider cases remain. Production pricing is unchanged. Validation: reproduced the exact original failure with locked LiteLLM 1.101.3; all 10 pricing tests and 54 combined pricing/backend/cost-card/provider-matrix tests pass after the fix. Disabling gateway unwrapping makes the new regression fail, confirming coverage is retained. Ruff lint/format and whitespace checks pass. Hosted qualification is pending on the added commit. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
106 lines
3.9 KiB
Python
106 lines
3.9 KiB
Python
"""Pricing comes from LiteLLM; the built-in table is only a fallback.
|
|
|
|
`_PRICING` used to be authoritative, with no LiteLLM lookup in front of it. That
|
|
is how it went ~18 months stale and priced `gpt-4.1-nano` 300x over. The table is
|
|
still needed -- the `litellm` dependency is gated `python_version < '3.14'`, and
|
|
LiteLLM does not know every model -- but it must not outrank the live source.
|
|
|
|
Resolution order (mirroring `get_context_limit`, so limits and prices agree):
|
|
|
|
1. explicit user config (`HEADROOM_MODEL_LIMITS` / `models.json`)
|
|
2. LiteLLM
|
|
3. built-in table -> family pattern -> unknown default
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.pricing.litellm_model_resolution import unwrapped_model_forms
|
|
from headroom.providers.openai import OpenAIProvider
|
|
|
|
litellm = pytest.importorskip("litellm")
|
|
|
|
|
|
def test_unwrapped_model_forms_drops_leading_segments() -> None:
|
|
"""Pure function: no gateway-prefix list to maintain."""
|
|
assert unwrapped_model_forms("bedrock/anthropic.claude-x") == ("anthropic.claude-x",)
|
|
assert unwrapped_model_forms("accounts/fireworks/models/kimi-k2") == (
|
|
"fireworks/models/kimi-k2",
|
|
"models/kimi-k2",
|
|
"kimi-k2",
|
|
)
|
|
assert unwrapped_model_forms("gpt-4o") == ()
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("model", "want_in", "want_out"),
|
|
[
|
|
# Gateway-routed names. litellm.model_cost keys the UNWRAPPED form, so
|
|
# these all returned None (-> $2.50/$10.00 unknown default) before.
|
|
("bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0", 3.00, 15.00),
|
|
("bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", 3.00, 15.00),
|
|
("vertex_ai/claude-sonnet-4-5", 3.00, 15.00),
|
|
# Non-OpenAI models reachable through the OpenAI-compatible passthrough.
|
|
("gemini-2.5-flash", 0.30, 2.50),
|
|
("deepseek-chat", 0.28, 0.42),
|
|
],
|
|
)
|
|
def test_provider_prices_models_its_table_never_covered(
|
|
model: str, want_in: float, want_out: float
|
|
) -> None:
|
|
got_in, got_out = OpenAIProvider()._get_pricing(model)
|
|
|
|
assert (round(got_in, 2), round(got_out, 2)) == (want_in, want_out)
|
|
|
|
|
|
def test_gateway_pricing_uses_unwrapped_catalog_entry(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Retiring a vendor model must not break the gateway-resolution regression."""
|
|
bare_model = "headroom-pricing-fixture"
|
|
monkeypatch.setattr(
|
|
litellm,
|
|
"model_cost",
|
|
{
|
|
bare_model: {
|
|
"input_cost_per_token": 0.20 / 1_000_000,
|
|
"output_cost_per_token": 0.20 / 1_000_000,
|
|
}
|
|
},
|
|
)
|
|
|
|
# Only the unwrapped key exists. Returning the unknown-model default instead
|
|
# of looking through the gateway prefix must still fail this assertion.
|
|
assert OpenAIProvider()._get_pricing(f"groq/{bare_model}") == pytest.approx((0.20, 0.20))
|
|
|
|
|
|
def test_explicit_config_outranks_litellm() -> None:
|
|
"""A configured price is a decision, not a guess."""
|
|
provider = OpenAIProvider()
|
|
provider._pricing_overrides["gpt-4o"] = (99.0, 111.0)
|
|
|
|
assert provider._get_pricing("gpt-4o") == (99.0, 111.0)
|
|
|
|
|
|
def test_falls_back_to_the_builtin_table_without_litellm(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Offline / Python >= 3.14 installs must still get sane numbers.
|
|
|
|
Pinned because the fallback is exactly where the table's correctness still
|
|
matters -- it is the only thing those installs see.
|
|
"""
|
|
import headroom.pricing.litellm_pricing as lp
|
|
|
|
monkeypatch.setattr(lp, "LITELLM_AVAILABLE", False)
|
|
|
|
provider = OpenAIProvider()
|
|
assert provider._get_pricing("gpt-4.1-nano") == (0.10, 0.40)
|
|
assert provider._get_pricing("gpt-4") == (30.00, 60.00)
|
|
assert provider._get_pricing("o3") == (2.00, 8.00)
|
|
|
|
|
|
def test_unknown_model_still_returns_a_usable_default() -> None:
|
|
"""Never raise, never return None, for a model nobody knows."""
|
|
got = OpenAIProvider()._get_pricing("totally-made-up-model-xyz")
|
|
|
|
assert got is not None
|
|
assert got[0] > 0
|