mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): read stream cache usage from message_delta; keep Fireworks sessions on one replica (#4174)
## Description
Two fixes for Claude Code (or any Anthropic-format client) running
through Headroom to Anthropic-compatible hosts such as Fireworks.
1. **Stream cache usage was read from the wrong event.** Fireworks'
`/v1/messages` stream sends zeros in `message_start` and the real
`input_tokens` / `cache_read_input_tokens` only in `message_delta`. The
SSE parser only read them from `message_start`, so every streamed turn
was logged as `cache_read=0`. The prefix tracker uses that number to
decide how much history to freeze, so it froze nothing and could rewrite
history Fireworks had already cached. `message_delta` usage is
cumulative, so the parser now takes its counts when they are larger.
2. **Session affinity for Fireworks.** Fireworks documents
`x-session-affinity` as the routing hint for keeping prompt-cache hits
on one replica. Requests to `api.fireworks.ai` now carry Headroom's
session id in that header. No other host gets it, and a value the client
set wins.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- `headroom/proxy/handlers/streaming.py`: also read `input_tokens`,
`cache_read_input_tokens`, `cache_creation_input_tokens` from
`message_delta` usage, keeping the larger value.
- `headroom/proxy/session_affinity.py` (new): host → affinity-header
map, currently only `api.fireworks.ai` → `x-session-affinity`.
- `headroom/proxy/handlers/anthropic.py`: add the affinity header on the
direct path, right after the upstream URL is built. The Bedrock, Vertex
and translated backends are not affected.
- Tests:
`test_streaming_parser_reads_cache_usage_sent_only_in_message_delta`
fails without the fix and passes with it.
`tests/test_session_affinity.py` covers four things through the real
proxy: Fireworks gets one stable affinity value per conversation,
Anthropic and other upstreams get no header, and a client-set header is
kept.
## Testing
- [x] Unit tests pass (`pytest`): the tests above plus the neighbouring
proxy, streaming, cache and backend suites
- [x] Linting passes (`ruff check .`, `ruff format --check .`, ruff
0.16.8)
- [ ] Type checking passes (`mypy headroom`): one error, in a file this
PR does not touch (see notes)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check . && ruff format --check .
All checks passed!
1779 files already formatted
$ pytest tests/test_session_affinity.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_anthropic_*.py \
tests/*stream* tests/*prefix_tracker* tests/*cache_stab* tests/*copilot* tests/*bedrock* tests/*vertex* ...
1326 passed, 3 skipped
$ mypy headroom --ignore-missing-imports
headroom/integrations/langchain/chat_model.py:380: error: Function "RunnableBinding" could always be true in boolean context [truthy-function]
```
## Real Behavior Proof
- Environment: macOS, real Claude Code (`claude -p`) → Headroom proxy →
`https://api.fireworks.ai/inference`
(`accounts/fireworks/models/kimi-k3`). A local relay in front of
Headroom swaps Claude Code's model name for the Fireworks one and
records the provider usage.
- Exact steps: run a small coding task on `pallets/itsdangerous@2.2.0`
(add a method, write tests, run pytest) with
`ANTHROPIC_TARGET_API_URL=https://api.fireworks.ai/inference`.
- Observed result:
- **Before:** on the same requests where Fireworks reported
`cache_read_input_tokens` of 25461 and 29213, Headroom's `PERF` lines
logged `cache_read=0` every time.
- **After:** Headroom logs the provider's real counts, e.g.
`cache_read=8192`. The task completed with all tests passing. With
compression on, every turn that hit the cache reused the full previous
prompt minus one token, so Headroom caused no prefix breaks.
- **Affinity:** the header is harmless to Anthropic: `api.anthropic.com`
returned 200 with and without it, and Headroom never sends it there
anyway. On Fireworks' `/v1/messages` with Kimi K3, 8-turn probes with
and without the header gave the same cache-hit pattern, and a live
Claude Code run measured 77.7% with it vs 78–79% without. **No hit-rate
improvement was measured on this endpoint and model.** It follows
Fireworks' documented guidance and may matter more on dedicated
multi-replica deployments.
- Not tested: dedicated Fireworks deployments; Fireworks'
OpenAI-compatible endpoint (the header is only added on the Anthropic
path).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed:
- Streamed Anthropic-format responses whose real counts arrive in
`message_delta` now report them. Anthropic's own stream puts them in
`message_start`, so it is unaffected; the larger-value rule never lowers
a count.
- Requests to `api.fireworks.ai` gain one header.
- Kill switch / disable path: a client-set `x-session-affinity`
overrides Headroom's value. There is no env toggle.
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- The mypy error in `headroom/integrations/langchain/chat_model.py:380`
exists on `main` without this change; this PR does not touch that file.
- Two related concerns are in one PR by request. They touch separate
code and can be reverted independently.
- Related: vLLM's Anthropic endpoint only reports cached tokens when
started with `--enable-prompt-tokens-details`. Worth a docs note for
self-hosted users in a follow-up.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4174"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
976aa714ef
commit
30b22df7fd
@@ -19,6 +19,7 @@ from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from headroom.proxy.session_affinity import session_affinity_headers
|
||||
from headroom.proxy.stage_timer import StageTimer, emit_stage_timings_log
|
||||
from headroom.proxy.tool_schema_savings_policy import without_deferral_flags
|
||||
|
||||
@@ -3927,6 +3928,9 @@ class AnthropicHandlerMixin:
|
||||
request.url.path,
|
||||
request.url.query,
|
||||
)
|
||||
# Keep a conversation on one replica of providers that route on a
|
||||
# header (Fireworks), so each turn finds the previous turn's cache.
|
||||
headers.update(session_affinity_headers(url, session_id, headers))
|
||||
|
||||
try:
|
||||
ccr_handler_config = getattr(self.ccr_response_handler, "config", None)
|
||||
|
||||
@@ -372,6 +372,27 @@ class StreamingMixin:
|
||||
delta_usage = _sse_dict(data.get("usage"))
|
||||
if delta_usage:
|
||||
usage_found["output_tokens"] = delta_usage.get("output_tokens", 0)
|
||||
# Usage here is cumulative. Some Anthropic-compatible
|
||||
# upstreams (Fireworks) send zeros in message_start and
|
||||
# the real input and cache counts only here; without
|
||||
# them the prefix tracker sees no cache and freezes
|
||||
# nothing.
|
||||
for key in (
|
||||
"input_tokens",
|
||||
"cache_read_input_tokens",
|
||||
"cache_creation_input_tokens",
|
||||
):
|
||||
value = usage_int(delta_usage.get(key))
|
||||
if value > usage_int(usage_found.get(key)):
|
||||
usage_found[key] = value
|
||||
# Their TTL split too, or one-hour writes price as five-minute.
|
||||
delta_5m, delta_1h = self._extract_anthropic_cache_ttl_metrics(delta_usage)
|
||||
for key, value in (
|
||||
("cache_creation_ephemeral_5m_input_tokens", delta_5m),
|
||||
("cache_creation_ephemeral_1h_input_tokens", delta_1h),
|
||||
):
|
||||
if value > usage_int(usage_found.get(key)):
|
||||
usage_found[key] = value
|
||||
|
||||
elif provider == "openai":
|
||||
chunk_usage = data.get("usage")
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Session-affinity headers for hosted providers that route on them.
|
||||
|
||||
Some providers run many replicas, each with its own prompt cache, and pick one
|
||||
per request. Without a hint, consecutive turns of one conversation can land on
|
||||
different replicas and miss the cache the previous turn built. Providers that
|
||||
route on a header get Headroom's session id in it, so a conversation stays on
|
||||
one replica.
|
||||
|
||||
Only hosts listed here receive the header; every other upstream (Anthropic,
|
||||
OpenAI, Bedrock, Vertex, ...) is left untouched. A header the client already
|
||||
set wins.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
# Host -> header it routes on. Fireworks: docs.fireworks.ai prompt caching.
|
||||
AFFINITY_HEADERS: dict[str, str] = {"api.fireworks.ai": "x-session-affinity"}
|
||||
|
||||
|
||||
def session_affinity_headers(
|
||||
url: str, session_id: str | None, headers: Mapping[str, str]
|
||||
) -> dict[str, str]:
|
||||
"""Headers to add for ``url``: the provider's affinity header, or nothing."""
|
||||
name = AFFINITY_HEADERS.get(urlsplit(url).hostname or "")
|
||||
if name is None or not session_id or any(k.lower() == name for k in headers):
|
||||
return {}
|
||||
return {name: session_id}
|
||||
@@ -518,3 +518,66 @@ def test_prefix_cache_stats_prices_xai_cache_reads_with_grok_models(
|
||||
|
||||
# No catalog cache rates: xAI's own fallback ratio, not Anthropic's 0.1.
|
||||
assert stats["by_provider"]["xai"]["savings_usd"] == round(1_000_000 * 3e-6 * (1 - 0.16), 4)
|
||||
|
||||
|
||||
def test_streaming_parser_reads_cache_usage_sent_only_in_message_delta() -> None:
|
||||
# Fireworks' Anthropic endpoint sends zeros in message_start and the real
|
||||
# input and cache counts in message_delta. The prefix tracker freezes history
|
||||
# by cache_read, so dropping them makes it treat a warm cache as cold.
|
||||
from headroom.proxy.server import HeadroomProxy, ProxyConfig
|
||||
|
||||
proxy = HeadroomProxy(
|
||||
ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
)
|
||||
)
|
||||
chunk = (
|
||||
b'data: {"type":"message_start","message":{"usage":{"input_tokens":0,'
|
||||
b'"output_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}\n\n'
|
||||
b'data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":'
|
||||
b'{"input_tokens":1,"output_tokens":5,"cache_read_input_tokens":2802,'
|
||||
b'"cache_creation_input_tokens":0}}\n\n'
|
||||
)
|
||||
usage = proxy._parse_sse_usage_from_buffer({"sse_buffer": bytearray(chunk)}, "anthropic")
|
||||
|
||||
assert usage["cache_read_input_tokens"] == 2802
|
||||
assert usage["input_tokens"] == 1
|
||||
assert usage["output_tokens"] == 5
|
||||
|
||||
|
||||
def test_streaming_parser_reads_ttl_buckets_sent_only_in_message_delta() -> None:
|
||||
# One-hour cache writes reported only in message_delta must stay one-hour,
|
||||
# or the cost tracker prices them as five-minute writes.
|
||||
from headroom.proxy.server import HeadroomProxy, ProxyConfig
|
||||
|
||||
proxy = HeadroomProxy(
|
||||
ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
)
|
||||
)
|
||||
chunk = (
|
||||
b'data: {"type":"message_start","message":{"usage":{"input_tokens":0,'
|
||||
b'"cache_creation_input_tokens":0}}}\n\n'
|
||||
b'data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":'
|
||||
b'{"input_tokens":5,"output_tokens":2,"cache_creation_input_tokens":1000,'
|
||||
b'"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":1000}}}\n\n'
|
||||
)
|
||||
usage = proxy._parse_sse_usage_from_buffer({"sse_buffer": bytearray(chunk)}, "anthropic")
|
||||
|
||||
assert usage["cache_creation_input_tokens"] == 1000
|
||||
assert usage["cache_creation_ephemeral_1h_input_tokens"] == 1000
|
||||
assert usage["cache_creation_ephemeral_5m_input_tokens"] == 0
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Session-affinity header: sent to providers that route on it, and nowhere else."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
REPLY = {
|
||||
"id": "m1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "ok"}],
|
||||
"stop_reason": "end_turn",
|
||||
"usage": {"input_tokens": 3, "output_tokens": 1},
|
||||
}
|
||||
|
||||
|
||||
def forwarded_headers(upstream: str, extra: dict[str, str] | None = None) -> list[dict]:
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
anthropic_api_url=upstream,
|
||||
)
|
||||
sent: list[dict] = []
|
||||
|
||||
async def fake_upstream(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001, ANN003
|
||||
sent.append({k.lower(): v for k, v in headers.items()})
|
||||
return httpx.Response(200, json=REPLY)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
client.app.state.proxy._retry_request = fake_upstream
|
||||
for _ in range(2):
|
||||
response = client.post(
|
||||
"/v1/messages",
|
||||
headers={"x-api-key": "k", "anthropic-version": "2023-06-01", **(extra or {})},
|
||||
json={
|
||||
"model": "kimi",
|
||||
"max_tokens": 5,
|
||||
"system": "You are a coding agent.",
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
return sent
|
||||
|
||||
|
||||
def test_fireworks_gets_one_stable_affinity_per_conversation() -> None:
|
||||
first, second = forwarded_headers("https://api.fireworks.ai/inference")
|
||||
assert first["x-session-affinity"]
|
||||
assert first["x-session-affinity"] == second["x-session-affinity"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("upstream", ["https://api.anthropic.com", "http://vllm.internal:8000"])
|
||||
def test_other_upstreams_get_no_affinity_header(upstream: str) -> None:
|
||||
assert all("x-session-affinity" not in h for h in forwarded_headers(upstream))
|
||||
|
||||
|
||||
def test_client_affinity_header_is_kept() -> None:
|
||||
sent = forwarded_headers("https://api.fireworks.ai/inference", {"X-Session-Affinity": "mine"})
|
||||
assert all(h["x-session-affinity"] == "mine" for h in sent)
|
||||
Reference in New Issue
Block a user