mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
Two fixes for Claude Code (or any Anthropic-format client) running
through Headroom to Anthropic-compatible hosts such as Fireworks.
1. **Stream cache usage was read from the wrong event.** Fireworks'
`/v1/messages` stream sends zeros in `message_start` and the real
`input_tokens` / `cache_read_input_tokens` only in `message_delta`. The
SSE parser only read them from `message_start`, so every streamed turn
was logged as `cache_read=0`. The prefix tracker uses that number to
decide how much history to freeze, so it froze nothing and could rewrite
history Fireworks had already cached. `message_delta` usage is
cumulative, so the parser now takes its counts when they are larger.
2. **Session affinity for Fireworks.** Fireworks documents
`x-session-affinity` as the routing hint for keeping prompt-cache hits
on one replica. Requests to `api.fireworks.ai` now carry Headroom's
session id in that header. No other host gets it, and a value the client
set wins.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- `headroom/proxy/handlers/streaming.py`: also read `input_tokens`,
`cache_read_input_tokens`, `cache_creation_input_tokens` from
`message_delta` usage, keeping the larger value.
- `headroom/proxy/session_affinity.py` (new): host → affinity-header
map, currently only `api.fireworks.ai` → `x-session-affinity`.
- `headroom/proxy/handlers/anthropic.py`: add the affinity header on the
direct path, right after the upstream URL is built. The Bedrock, Vertex
and translated backends are not affected.
- Tests:
`test_streaming_parser_reads_cache_usage_sent_only_in_message_delta`
fails without the fix and passes with it.
`tests/test_session_affinity.py` covers four things through the real
proxy: Fireworks gets one stable affinity value per conversation,
Anthropic and other upstreams get no header, and a client-set header is
kept.
## Testing
- [x] Unit tests pass (`pytest`): the tests above plus the neighbouring
proxy, streaming, cache and backend suites
- [x] Linting passes (`ruff check .`, `ruff format --check .`, ruff
0.16.8)
- [ ] Type checking passes (`mypy headroom`): one error, in a file this
PR does not touch (see notes)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check . && ruff format --check .
All checks passed!
1779 files already formatted
$ pytest tests/test_session_affinity.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_anthropic_*.py \
tests/*stream* tests/*prefix_tracker* tests/*cache_stab* tests/*copilot* tests/*bedrock* tests/*vertex* ...
1326 passed, 3 skipped
$ mypy headroom --ignore-missing-imports
headroom/integrations/langchain/chat_model.py:380: error: Function "RunnableBinding" could always be true in boolean context [truthy-function]
```
## Real Behavior Proof
- Environment: macOS, real Claude Code (`claude -p`) → Headroom proxy →
`https://api.fireworks.ai/inference`
(`accounts/fireworks/models/kimi-k3`). A local relay in front of
Headroom swaps Claude Code's model name for the Fireworks one and
records the provider usage.
- Exact steps: run a small coding task on `pallets/itsdangerous@2.2.0`
(add a method, write tests, run pytest) with
`ANTHROPIC_TARGET_API_URL=https://api.fireworks.ai/inference`.
- Observed result:
- **Before:** on the same requests where Fireworks reported
`cache_read_input_tokens` of 25461 and 29213, Headroom's `PERF` lines
logged `cache_read=0` every time.
- **After:** Headroom logs the provider's real counts, e.g.
`cache_read=8192`. The task completed with all tests passing. With
compression on, every turn that hit the cache reused the full previous
prompt minus one token, so Headroom caused no prefix breaks.
- **Affinity:** the header is harmless to Anthropic: `api.anthropic.com`
returned 200 with and without it, and Headroom never sends it there
anyway. On Fireworks' `/v1/messages` with Kimi K3, 8-turn probes with
and without the header gave the same cache-hit pattern, and a live
Claude Code run measured 77.7% with it vs 78–79% without. **No hit-rate
improvement was measured on this endpoint and model.** It follows
Fireworks' documented guidance and may matter more on dedicated
multi-replica deployments.
- Not tested: dedicated Fireworks deployments; Fireworks'
OpenAI-compatible endpoint (the header is only added on the Anthropic
path).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed:
- Streamed Anthropic-format responses whose real counts arrive in
`message_delta` now report them. Anthropic's own stream puts them in
`message_start`, so it is unaffected; the larger-value rule never lowers
a count.
- Requests to `api.fireworks.ai` gain one header.
- Kill switch / disable path: a client-set `x-session-affinity`
overrides Headroom's value. There is no env toggle.
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- The mypy error in `headroom/integrations/langchain/chat_model.py:380`
exists on `main` without this change; this PR does not touch that file.
- Two related concerns are in one PR by request. They touch separate
code and can be reverted independently.
- Related: vLLM's Anthropic endpoint only reports cached tokens when
started with `--enable-prompt-tokens-details`. Worth a docs note for
self-hosted users in a follow-up.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4174"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
584 lines
22 KiB
Python
584 lines
22 KiB
Python
"""Tests for observed Anthropic cache TTL bucket metrics."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
import pytest
|
|
|
|
from headroom.observability import reset_headroom_tracing, reset_otel_metrics
|
|
from headroom.proxy.cost import CostTracker, build_prefix_cache_stats
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
from headroom.proxy.savings_tracker import SavingsTracker
|
|
|
|
|
|
def test_prometheus_metrics_tracks_observed_ttl_buckets() -> None:
|
|
metrics = PrometheusMetrics()
|
|
|
|
asyncio.run(
|
|
metrics.record_request(
|
|
provider="anthropic",
|
|
model="claude-opus-4-6",
|
|
input_tokens=100,
|
|
output_tokens=20,
|
|
tokens_saved=5,
|
|
latency_ms=10.0,
|
|
cache_read_tokens=40,
|
|
cache_write_tokens=60,
|
|
cache_write_5m_tokens=10,
|
|
cache_write_1h_tokens=50,
|
|
)
|
|
)
|
|
|
|
stats = metrics.cache_by_provider["anthropic"]
|
|
assert stats["cache_write_5m_tokens"] == 10
|
|
assert stats["cache_write_1h_tokens"] == 50
|
|
assert stats["cache_write_5m_requests"] == 1
|
|
assert stats["cache_write_1h_requests"] == 1
|
|
|
|
|
|
def test_cost_tracker_exposes_observed_ttl_buckets_per_model() -> None:
|
|
tracker = CostTracker()
|
|
tracker.record_tokens(
|
|
"claude-opus-4-6",
|
|
tokens_saved=10,
|
|
tokens_sent=90,
|
|
cache_read_tokens=40,
|
|
cache_write_tokens=60,
|
|
cache_write_5m_tokens=10,
|
|
cache_write_1h_tokens=50,
|
|
uncached_tokens=20,
|
|
)
|
|
|
|
stats = tracker.stats()
|
|
assert stats["cache_write_5m_tokens"] == 10
|
|
assert stats["cache_write_1h_tokens"] == 50
|
|
assert stats["per_model"]["claude-opus-4-6"]["cache_write_5m_tokens"] == 10
|
|
assert stats["per_model"]["claude-opus-4-6"]["cache_write_1h_tokens"] == 50
|
|
|
|
|
|
def test_prefix_cache_stats_include_observed_ttl_mix() -> None:
|
|
metrics = PrometheusMetrics()
|
|
provider_stats = metrics.cache_by_provider["anthropic"]
|
|
provider_stats["requests"] = 2
|
|
provider_stats["hit_requests"] = 1
|
|
provider_stats["cache_read_tokens"] = 40
|
|
provider_stats["cache_write_tokens"] = 60
|
|
provider_stats["cache_write_5m_tokens"] = 15
|
|
provider_stats["cache_write_1h_tokens"] = 45
|
|
provider_stats["cache_write_5m_requests"] = 1
|
|
provider_stats["cache_write_1h_requests"] = 1
|
|
|
|
stats = build_prefix_cache_stats(metrics, None)
|
|
anthropic = stats["by_provider"]["anthropic"]
|
|
|
|
assert anthropic["observed_ttl_buckets"]["5m"]["tokens"] == 15
|
|
assert anthropic["observed_ttl_buckets"]["1h"]["tokens"] == 45
|
|
assert anthropic["observed_ttl_mix"]["5m_pct"] == 25.0
|
|
assert anthropic["observed_ttl_mix"]["1h_pct"] == 75.0
|
|
assert stats["totals"]["observed_ttl_buckets"]["5m"]["tokens"] == 15
|
|
assert stats["totals"]["observed_ttl_buckets"]["1h"]["tokens"] == 45
|
|
|
|
|
|
def test_prefix_cache_stats_subtracts_write_premium_from_provider_net_savings(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
metrics = PrometheusMetrics()
|
|
metrics.cache_by_provider["anthropic"].update(
|
|
{
|
|
"requests": 2,
|
|
"hit_requests": 1,
|
|
"cache_read_tokens": 40,
|
|
"cache_write_tokens": 60,
|
|
"cache_write_5m_tokens": 60,
|
|
"cache_write_1h_tokens": 0,
|
|
"cache_write_5m_requests": 1,
|
|
"cache_write_1h_requests": 0,
|
|
}
|
|
)
|
|
|
|
tracker = CostTracker()
|
|
tracker._tokens_sent_by_model.update({"claude-opus-4-6": 1})
|
|
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 100.0)
|
|
|
|
stats = build_prefix_cache_stats(metrics, tracker)
|
|
|
|
anthropic = stats["by_provider"]["anthropic"]
|
|
|
|
assert anthropic["savings_usd"] == 0.0036
|
|
assert anthropic["write_premium_usd"] == 0.0015
|
|
assert anthropic["net_savings_usd"] == 0.0021
|
|
|
|
|
|
def test_prefix_cache_stats_prices_by_most_used_model(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""Cache-read savings must be valued with the highest-volume model's price,
|
|
not whichever model was recorded first. A Claude Code session sends Haiku
|
|
(titles) and Sonnet (main loop); pricing the savings at Haiku's rate because
|
|
it was inserted first understates the dashboard figure ~3.75x."""
|
|
prices = {"claude-haiku-4-5": 0.80, "claude-sonnet-4-5": 3.00}
|
|
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, m: prices.get(m))
|
|
|
|
def _savings(tokens_by_model: dict[str, int]) -> float:
|
|
metrics = PrometheusMetrics()
|
|
# Use a large read count so the reported savings_usd (rounded to 4 dp)
|
|
# stays exact and the price ratio is not lost to rounding.
|
|
metrics.cache_by_provider["anthropic"].update(
|
|
{
|
|
"requests": 1,
|
|
"hit_requests": 1,
|
|
"cache_read_tokens": 1_000_000,
|
|
"cache_write_tokens": 0,
|
|
"cache_write_5m_tokens": 0,
|
|
"cache_write_1h_tokens": 0,
|
|
"cache_write_5m_requests": 0,
|
|
"cache_write_1h_requests": 0,
|
|
}
|
|
)
|
|
tracker = CostTracker()
|
|
tracker._tokens_sent_by_model.update(tokens_by_model)
|
|
stats = build_prefix_cache_stats(metrics, tracker)
|
|
return stats["by_provider"]["anthropic"]["savings_usd"]
|
|
|
|
# Haiku recorded first, but Sonnet carries the higher token volume.
|
|
haiku_first = _savings({"claude-haiku-4-5": 500, "claude-sonnet-4-5": 50_000})
|
|
sonnet_only = _savings({"claude-sonnet-4-5": 50_000})
|
|
haiku_only = _savings({"claude-haiku-4-5": 500})
|
|
|
|
# Priced by Sonnet regardless of insertion order, not by first-seen Haiku.
|
|
assert haiku_first == sonnet_only
|
|
assert haiku_first > haiku_only
|
|
assert haiku_only == pytest.approx(sonnet_only * 0.80 / 3.00)
|
|
|
|
|
|
def test_prefix_cache_stats_subtracts_write_premium_from_total_net_savings(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
metrics = PrometheusMetrics()
|
|
metrics.cache_by_provider["anthropic"].update(
|
|
{
|
|
"requests": 2,
|
|
"hit_requests": 1,
|
|
"cache_read_tokens": 40,
|
|
"cache_write_tokens": 60,
|
|
"cache_write_5m_tokens": 60,
|
|
"cache_write_1h_tokens": 0,
|
|
"cache_write_5m_requests": 1,
|
|
"cache_write_1h_requests": 0,
|
|
}
|
|
)
|
|
metrics.cache_by_provider["openai"].update(
|
|
{
|
|
"requests": 1,
|
|
"hit_requests": 1,
|
|
"cache_read_tokens": 20,
|
|
"cache_write_tokens": 10,
|
|
"cache_write_5m_tokens": 0,
|
|
"cache_write_1h_tokens": 10,
|
|
"cache_write_5m_requests": 0,
|
|
"cache_write_1h_requests": 1,
|
|
}
|
|
)
|
|
|
|
tracker = CostTracker()
|
|
tracker._tokens_sent_by_model.update({"claude-opus-4-6": 1, "gpt-4o": 1})
|
|
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 100.0)
|
|
|
|
stats = build_prefix_cache_stats(metrics, tracker)
|
|
|
|
openai = stats["by_provider"]["openai"]
|
|
|
|
assert openai["write_premium_usd"] == 0.0
|
|
assert openai["net_savings_usd"] == openai["savings_usd"]
|
|
assert stats["totals"]["savings_usd"] == 0.0046
|
|
assert stats["totals"]["write_premium_usd"] == 0.0015
|
|
assert stats["totals"]["net_savings_usd"] == 0.0031
|
|
|
|
|
|
def test_prefix_cache_stats_keeps_net_equal_without_write_premium(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
metrics = PrometheusMetrics()
|
|
metrics.cache_by_provider["openai"].update(
|
|
{
|
|
"requests": 1,
|
|
"hit_requests": 1,
|
|
"cache_read_tokens": 20,
|
|
"cache_write_tokens": 0,
|
|
"cache_write_5m_tokens": 0,
|
|
"cache_write_1h_tokens": 0,
|
|
"cache_write_5m_requests": 0,
|
|
"cache_write_1h_requests": 0,
|
|
}
|
|
)
|
|
|
|
tracker = CostTracker()
|
|
tracker._tokens_sent_by_model.update({"gpt-4o": 1})
|
|
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 100.0)
|
|
|
|
stats = build_prefix_cache_stats(metrics, tracker)
|
|
|
|
openai = stats["by_provider"]["openai"]
|
|
|
|
assert openai["savings_usd"] == 0.001
|
|
assert openai["write_premium_usd"] == 0.0
|
|
assert openai["net_savings_usd"] == openai["savings_usd"]
|
|
|
|
|
|
def test_prometheus_metrics_export_includes_extended_fields(tmp_path) -> None:
|
|
metrics = PrometheusMetrics(
|
|
savings_tracker=SavingsTracker(path=str(tmp_path / "proxy_savings.json"))
|
|
)
|
|
|
|
asyncio.run(
|
|
metrics.record_request(
|
|
provider="anthropic",
|
|
model="claude-opus-4-6",
|
|
input_tokens=100,
|
|
output_tokens=20,
|
|
tokens_saved=5,
|
|
latency_ms=12.5,
|
|
overhead_ms=3.0,
|
|
ttfb_ms=9.0,
|
|
pipeline_timing={"router": 4.5},
|
|
waste_signals={"json_bloat": 7},
|
|
cache_read_tokens=40,
|
|
cache_write_tokens=60,
|
|
cache_write_5m_tokens=10,
|
|
cache_write_1h_tokens=50,
|
|
uncached_input_tokens=20,
|
|
)
|
|
)
|
|
asyncio.run(metrics.record_cache_bust(11))
|
|
|
|
exported = asyncio.run(metrics.export())
|
|
|
|
assert "headroom_requests_total 1" in exported
|
|
assert "headroom_tokens_saved_total 5" in exported
|
|
assert "headroom_persistent_savings_requests_total 1" in exported
|
|
assert "headroom_persistent_savings_tokens_saved_total 5" in exported
|
|
assert "headroom_persistent_savings_input_tokens_total 100" in exported
|
|
assert "headroom_latency_ms_count 1" in exported
|
|
assert 'headroom_transform_timing_ms_sum{transform="router"} 4.5' in exported
|
|
assert 'headroom_waste_signal_tokens_total{signal="json_bloat"} 7' in exported
|
|
assert 'headroom_cache_write_ttl_tokens_total{provider="anthropic",ttl="5m"} 10' in exported
|
|
assert 'headroom_provider_cache_hit_requests_total{provider="anthropic"} 1' in exported
|
|
assert "headroom_cache_bust_tokens_lost_total 11" in exported
|
|
|
|
|
|
def test_prometheus_export_includes_persistent_savings_after_restart(tmp_path) -> None:
|
|
savings_path = tmp_path / "proxy_savings.json"
|
|
metrics = PrometheusMetrics(savings_tracker=SavingsTracker(path=str(savings_path)))
|
|
|
|
asyncio.run(
|
|
metrics.record_request(
|
|
provider="openai",
|
|
model="gpt-4o",
|
|
input_tokens=120,
|
|
output_tokens=20,
|
|
tokens_saved=40,
|
|
latency_ms=12.5,
|
|
)
|
|
)
|
|
|
|
reloaded = PrometheusMetrics(savings_tracker=SavingsTracker(path=str(savings_path)))
|
|
exported = asyncio.run(reloaded.export())
|
|
|
|
assert "headroom_tokens_saved_total 0" in exported
|
|
assert "headroom_requests_total 0" in exported
|
|
assert "headroom_persistent_savings_requests_total 1" in exported
|
|
assert "headroom_persistent_savings_tokens_saved_total 40" in exported
|
|
assert "headroom_persistent_savings_input_tokens_total 120" in exported
|
|
|
|
|
|
def test_streaming_parser_extracts_anthropic_ttl_bucket_usage() -> None:
|
|
from headroom.proxy.server import HeadroomProxy, ProxyConfig
|
|
|
|
proxy = HeadroomProxy(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
|
|
chunk = (
|
|
b'data: {"type":"message_start","message":{"usage":{"input_tokens":12,'
|
|
b'"cache_read_input_tokens":3,"cache_creation_input_tokens":9,'
|
|
b'"cache_creation":{"ephemeral_5m_input_tokens":4,"ephemeral_1h_input_tokens":5}}}}\n\n'
|
|
)
|
|
usage = proxy._parse_sse_usage_from_buffer({"sse_buffer": bytearray(chunk)}, "anthropic")
|
|
|
|
assert usage is not None
|
|
assert usage["cache_creation_ephemeral_5m_input_tokens"] == 4
|
|
assert usage["cache_creation_ephemeral_1h_input_tokens"] == 5
|
|
|
|
|
|
def test_stats_endpoint_reports_observed_ttl_buckets() -> None:
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
|
|
proxy = app.state.proxy
|
|
provider_stats = proxy.metrics.cache_by_provider["anthropic"]
|
|
provider_stats["requests"] = 1
|
|
provider_stats["hit_requests"] = 1
|
|
provider_stats["cache_read_tokens"] = 30
|
|
provider_stats["cache_write_tokens"] = 70
|
|
provider_stats["cache_write_5m_tokens"] = 20
|
|
provider_stats["cache_write_1h_tokens"] = 50
|
|
provider_stats["cache_write_5m_requests"] = 1
|
|
provider_stats["cache_write_1h_requests"] = 1
|
|
|
|
with TestClient(app) as client:
|
|
response = client.get("/stats")
|
|
|
|
assert response.status_code == 200
|
|
prefix_cache = response.json()["prefix_cache"]
|
|
anthropic = prefix_cache["by_provider"]["anthropic"]
|
|
assert anthropic["observed_ttl_buckets"]["5m"]["tokens"] == 20
|
|
assert anthropic["observed_ttl_buckets"]["1h"]["tokens"] == 50
|
|
assert prefix_cache["totals"]["observed_ttl_mix"]["active_buckets"] == ["5m", "1h"]
|
|
|
|
|
|
def test_stats_endpoint_reports_otel_configuration(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
reset_otel_metrics()
|
|
monkeypatch.setenv("HEADROOM_OTEL_METRICS_ENABLED", "1")
|
|
monkeypatch.setenv("HEADROOM_OTEL_METRICS_EXPORTER", "console")
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
|
|
with TestClient(app) as client:
|
|
response = client.get("/stats")
|
|
|
|
assert response.status_code == 200
|
|
otel = response.json()["otel"]
|
|
assert otel["configured"] is True
|
|
assert otel["enabled"] is True
|
|
assert otel["service_name"] == "headroom-proxy"
|
|
assert otel["exporter"] == "console"
|
|
|
|
|
|
def test_stats_endpoint_reports_langfuse_configuration(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
reset_headroom_tracing()
|
|
monkeypatch.setenv("HEADROOM_LANGFUSE_ENABLED", "1")
|
|
monkeypatch.setenv("LANGFUSE_PUBLIC_KEY", "pk-lf-test")
|
|
monkeypatch.setenv("LANGFUSE_SECRET_KEY", "sk-lf-test")
|
|
monkeypatch.setenv("LANGFUSE_BASE_URL", "https://cloud.langfuse.com")
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
|
|
with TestClient(app) as client:
|
|
response = client.get("/stats")
|
|
|
|
assert response.status_code == 200
|
|
langfuse = response.json()["langfuse"]
|
|
assert langfuse["configured"] is True
|
|
assert langfuse["enabled"] is True
|
|
assert langfuse["service_name"] == "headroom-proxy"
|
|
assert langfuse["endpoint"] == "https://cloud.langfuse.com/api/public/otel/v1/traces"
|
|
|
|
|
|
# --- Cache-miss attribution (#1313) ---
|
|
|
|
|
|
def test_record_cache_miss_attribution_buckets_by_provider_and_reason() -> None:
|
|
metrics = PrometheusMetrics()
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "ttl_expiry"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "ttl_expiry"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "prefix_change"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "unknown"))
|
|
|
|
buckets = metrics.cache_miss_attribution_by_provider["anthropic"]
|
|
assert buckets["ttl_expiry"] == 2
|
|
assert buckets["prefix_change"] == 1
|
|
assert buckets["unknown"] == 1
|
|
|
|
|
|
def test_prefix_cache_stats_include_miss_attribution() -> None:
|
|
metrics = PrometheusMetrics()
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "ttl_expiry"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "ttl_expiry"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "prefix_change"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "unknown"))
|
|
|
|
stats = build_prefix_cache_stats(metrics, None)
|
|
ma = stats["miss_attribution"]
|
|
|
|
assert ma["totals"]["ttl_expiry"] == 2
|
|
assert ma["totals"]["prefix_change"] == 1
|
|
assert ma["totals"]["unknown"] == 1
|
|
assert ma["totals"]["total"] == 4
|
|
# Percentages are over attributed (non-unknown) misses: 2 / 3, 1 / 3.
|
|
assert ma["totals"]["ttl_expiry_pct"] == 66.7
|
|
assert ma["totals"]["prefix_change_pct"] == 33.3
|
|
assert ma["by_provider"]["anthropic"]["total"] == 4
|
|
|
|
|
|
def test_prefix_cache_stats_miss_attribution_empty_when_no_misses() -> None:
|
|
metrics = PrometheusMetrics()
|
|
stats = build_prefix_cache_stats(metrics, None)
|
|
ma = stats["miss_attribution"]
|
|
assert ma["totals"]["total"] == 0
|
|
assert ma["totals"]["ttl_expiry_pct"] == 0.0
|
|
assert ma["by_provider"] == {}
|
|
|
|
|
|
def test_prometheus_export_includes_miss_attribution() -> None:
|
|
metrics = PrometheusMetrics()
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "ttl_expiry"))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "prefix_change"))
|
|
|
|
exported = asyncio.run(metrics.export())
|
|
|
|
assert (
|
|
'headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"} 1'
|
|
in exported
|
|
)
|
|
assert (
|
|
'headroom_cache_miss_attribution_total{provider="anthropic",reason="prefix_change"} 1'
|
|
in exported
|
|
)
|
|
|
|
|
|
def test_reset_runtime_clears_miss_attribution() -> None:
|
|
metrics = PrometheusMetrics()
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "ttl_expiry"))
|
|
asyncio.run(metrics.reset_runtime())
|
|
assert dict(metrics.cache_miss_attribution_by_provider) == {}
|
|
|
|
|
|
def test_prefix_cache_stats_prices_xai_cache_reads_with_grok_models(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
) -> None:
|
|
"""Grok traffic books under ``xai``; its cache reads must be priced with a
|
|
Grok model, not dropped to $0 because no model matched the provider."""
|
|
metrics = PrometheusMetrics()
|
|
metrics.cache_by_provider["xai"].update(
|
|
{"requests": 1, "hit_requests": 1, "cache_read_tokens": 1_000_000}
|
|
)
|
|
tracker = CostTracker()
|
|
tracker._tokens_sent_by_model.update({"grok-4": 1})
|
|
monkeypatch.setattr(CostTracker, "_get_list_price", lambda _self, _model: 3.0)
|
|
monkeypatch.setattr(CostTracker, "_get_cache_prices", lambda _self, _model: None)
|
|
|
|
stats = build_prefix_cache_stats(metrics, tracker)
|
|
|
|
# No catalog cache rates: xAI's own fallback ratio, not Anthropic's 0.1.
|
|
assert stats["by_provider"]["xai"]["savings_usd"] == round(1_000_000 * 3e-6 * (1 - 0.16), 4)
|
|
|
|
|
|
def test_streaming_parser_reads_cache_usage_sent_only_in_message_delta() -> None:
|
|
# Fireworks' Anthropic endpoint sends zeros in message_start and the real
|
|
# input and cache counts in message_delta. The prefix tracker freezes history
|
|
# by cache_read, so dropping them makes it treat a warm cache as cold.
|
|
from headroom.proxy.server import HeadroomProxy, ProxyConfig
|
|
|
|
proxy = HeadroomProxy(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
chunk = (
|
|
b'data: {"type":"message_start","message":{"usage":{"input_tokens":0,'
|
|
b'"output_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}\n\n'
|
|
b'data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":'
|
|
b'{"input_tokens":1,"output_tokens":5,"cache_read_input_tokens":2802,'
|
|
b'"cache_creation_input_tokens":0}}\n\n'
|
|
)
|
|
usage = proxy._parse_sse_usage_from_buffer({"sse_buffer": bytearray(chunk)}, "anthropic")
|
|
|
|
assert usage["cache_read_input_tokens"] == 2802
|
|
assert usage["input_tokens"] == 1
|
|
assert usage["output_tokens"] == 5
|
|
|
|
|
|
def test_streaming_parser_reads_ttl_buckets_sent_only_in_message_delta() -> None:
|
|
# One-hour cache writes reported only in message_delta must stay one-hour,
|
|
# or the cost tracker prices them as five-minute writes.
|
|
from headroom.proxy.server import HeadroomProxy, ProxyConfig
|
|
|
|
proxy = HeadroomProxy(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
chunk = (
|
|
b'data: {"type":"message_start","message":{"usage":{"input_tokens":0,'
|
|
b'"cache_creation_input_tokens":0}}}\n\n'
|
|
b'data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":'
|
|
b'{"input_tokens":5,"output_tokens":2,"cache_creation_input_tokens":1000,'
|
|
b'"cache_creation":{"ephemeral_5m_input_tokens":0,"ephemeral_1h_input_tokens":1000}}}\n\n'
|
|
)
|
|
usage = proxy._parse_sse_usage_from_buffer({"sse_buffer": bytearray(chunk)}, "anthropic")
|
|
|
|
assert usage["cache_creation_input_tokens"] == 1000
|
|
assert usage["cache_creation_ephemeral_1h_input_tokens"] == 1000
|
|
assert usage["cache_creation_ephemeral_5m_input_tokens"] == 0
|