mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
`/stats` only exposes aggregate cost, so a tool cannot attribute spend
to one Claude Code session. This adds `cost_usd` and `turn_id` to each
`recent_requests` entry so a statusline HUD can sum cost per turn or
time window without shelling out to `ccusage`.
`cost_usd` prices the input count the cost tracker bills (the provider
reported prompt count, else the local count) plus output, at LiteLLM
list price. It does not apply cache read or write discounts, and `null`
means unknown, never `$0`.
Closes #1079
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- `request_cost_usd` in `headroom/proxy/cost.py` prices
`provider_input_tokens` when the provider reported it, else
`input_tokens_optimized`, and returns `null` when output usage is
unknown.
- `RequestLog` keeps `provider_input_tokens`, set in
`emit_request_outcome`.
- `/stats` `recent_requests` entries carry `cost_usd` and `turn_id`.
- Regression tests for a provider and local count mismatch, unknown
output usage, and grouping by `turn_id` past the 10 row `request_logs`
tail.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests scripts/tests --splits 4 --group N # Docker python:3.12-slim, maturin --profile ci wheel
shard 1: 4195 passed, 157 skipped
shard 2: 4134 passed, 217 skipped, 1 failed # test_install_supervisor_*: getpass needs $USER in the container, passes with USER set
shard 3: 4257 passed, 89 skipped, 6 failed # test_plugin_hook_launcher (4), test_cc_switch_reconciler (2): identical failures on unmodified main in the same container
shard 4: 4183 passed, 167 skipped
$ pytest tests/test_request_cost_usd.py -q
13 passed # the 4 new regression tests fail on 3b4176df
$ ruff check . && ruff format --check . # ruff 0.17.0
All checks passed! / 1803 files already formatted
$ mypy headroom --ignore-missing-imports # mypy 1.20.2
Success: no issues found in 572 source files
```
## Real Behavior Proof
- Environment: Docker `python:3.12-slim`, wheel built from this branch
with `maturin build --profile ci` and installed in a clean venv, real
`headroom proxy` against a local mock Anthropic and OpenAI upstream that
reports 800 input and 50 output tokens per call.
- Exact command / steps: Four agent loop turns of three calls each
(Anthropic streamed, Anthropic non streamed, OpenAI non streamed, OpenAI
streamed), then `GET /stats` from loopback.
- Observed result: All 12 rows carry `turn_id`, including the 2 beyond
the 10 row `request_logs` tail, and price the provider's 800 input
tokens where the local `input_tokens_optimized` was 17 to 97.
`sum(cost_usd)` is `$0.0198`, equal to the cost tracker's
`total_cost_usd`. The same run on `3b4176df` priced the local count
(`$0.0058`) with no `turn_id`.
- Not tested: Live provider traffic. Cache discounts, which `cost_usd`
does not apply by design. A `null` output count, which live outcomes do
not produce today and the unit tests cover.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No. Two additive keys on the loopback
only `recent_requests` entries, and one additive `provider_input_tokens`
field on `RequestLog` and the JSONL request log.
- Kill switch / disable path: Not applicable. With cost tracking
disabled (`cost_tracking_enabled=False`), `cost_usd` is `null`.
- Unsafe override required: No.
- Qualification impact: None. No new endpoint, header, flag, or
dependency.
- Rollback path: Revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
No docs page lists the `recent_requests` fields, so there is no docs
change.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/1565"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
### Maintainer current-main requalification
Exact head `dc8a867422890ad3f8af681764ad4b386daed62b`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.
Clean current-main integration,six intended Python files; contributor
request_cost_usd helper AST unchanged.
Fresh125tests:123pass,1Windows/POSIXdirectory-fsyncskip,1failure,1warning.
Failure unpinned claude-sonnet-5 offlinepositive-cache-price assertion
ASTidenticalmain; reused recorded exact0ddmain failure control
frompr2245 (not rerun), not PR blocker. Actual unmocked outcome
emitter/CostTracker/RequestLogger/localASGI /stats: provider800vs
local600 input kept end-to-end; modelgpt-4o-mini list-price
estimate0.00015 perrow,20pricedrows across5turn totals0.0006each despite
rawlogtail10;
unknownoutput=>null,reportedzero=>input-onlypositive0.00012. No pricing
or handler mocks in probe. Initial stateless/non-loopbackHost harness
requests correctly lacked private recent rows; configured
logging-enabled ownedtempworkspace and loopbackpeer+Host, no
privacy/productlogic changed. This is synthetic usage/listprice, not
provider traffic, invoice/cache-discount or realtokenizer proof.
Existing author/4Devinthreads adjudicated:
provider/local,unknownoutput,turngrouping are addressed in later source;
no duplicate feedback or feature-permission request. All6Python
Rufflint/format,offline lock/whitespace pass. Full
nativeinputset/hashes/artifactchecksum verified before3982DLL reuse,no
freshbuild. Exact published-head hosted qualification pending before
formal approval; maintainers handle CI. No PR merge.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
---------
Signed-off-by: Krishnachaitanyakc <krishnabkc15@gmail.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
JD Davis
Claude Opus 5.5
parent
3912c04053
commit
84cedbf2b4
+38
-1
@@ -12,6 +12,7 @@ import importlib.util
|
||||
import logging
|
||||
import math
|
||||
from collections import deque
|
||||
from collections.abc import Mapping
|
||||
from datetime import datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any, NamedTuple
|
||||
|
||||
@@ -544,6 +545,42 @@ def build_prefix_cache_stats(
|
||||
}
|
||||
|
||||
|
||||
def request_cost_usd(cost_tracker: CostTracker | None, log: Mapping[str, Any]) -> float | None:
|
||||
"""Best-effort actual-cost estimate (USD) for one logged request (issue #1079).
|
||||
|
||||
Prices the same input count the cost tracker bills — the provider-reported
|
||||
``provider_input_tokens`` when the provider reported one, else the local
|
||||
post-compression ``input_tokens_optimized`` — plus ``output_tokens``, via the
|
||||
existing ``CostTracker.estimate_cost`` (LiteLLM list price). Best-effort: it
|
||||
does NOT apply cache-read/cache-write discounts. The log keeps cache read and
|
||||
write counts, but not the inferred-write flag or the 5m/1h write split that
|
||||
cache-aware pricing needs (pricing an inferred OpenAI write as a write would
|
||||
overcharge), so treat the value as a list-price estimate, not an exact bill.
|
||||
|
||||
Returns ``None`` — never a fabricated number — when cost tracking is
|
||||
disabled, the entry lacks a model, input tokens or output tokens, or pricing
|
||||
is unavailable, so downstream tooling can distinguish "unknown" from "$0". A
|
||||
missing output count is unknown, not zero: pricing it as zero would publish
|
||||
an input-only figure that looks complete. Exception-isolated so a single bad
|
||||
entry never breaks the ``/stats`` payload.
|
||||
"""
|
||||
if cost_tracker is None:
|
||||
return None
|
||||
model = log.get("model")
|
||||
input_tokens = log.get("provider_input_tokens") or log.get("input_tokens_optimized")
|
||||
output_tokens = log.get("output_tokens")
|
||||
if not model or input_tokens is None or output_tokens is None:
|
||||
return None
|
||||
try:
|
||||
return cost_tracker.estimate_cost(
|
||||
model=model,
|
||||
input_tokens=int(input_tokens),
|
||||
output_tokens=int(output_tokens),
|
||||
)
|
||||
except Exception: # noqa: BLE001 — never break /stats on a pricing error
|
||||
return None
|
||||
|
||||
|
||||
def merge_cost_stats(
|
||||
cost_stats: dict | None,
|
||||
cache_stats: dict,
|
||||
@@ -649,7 +686,7 @@ def build_session_summary(
|
||||
def _entry_has_number(entry: Any, attr: str) -> bool:
|
||||
value = getattr(entry, attr, None)
|
||||
return (
|
||||
isinstance(value, (int, float))
|
||||
isinstance(value, int | float)
|
||||
and not isinstance(value, bool)
|
||||
and math.isfinite(float(value))
|
||||
)
|
||||
|
||||
@@ -97,6 +97,12 @@ class RequestLog:
|
||||
cache_write_tokens: int = 0
|
||||
uncached_input_tokens: int = 0
|
||||
|
||||
# The provider's own prompt-token count, when it reported one (0 otherwise).
|
||||
# ``input_tokens_optimized`` is the local tokenizer's count, which can differ;
|
||||
# this is the figure the cost tracker bills, so per-request cost prices it
|
||||
# in preference to the local count (#1079).
|
||||
provider_input_tokens: int = 0
|
||||
|
||||
# Waste signals detected in original messages
|
||||
waste_signals: dict[str, int] | None = None
|
||||
|
||||
|
||||
@@ -791,6 +791,7 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
||||
cache_read_tokens=outcome.cache_read_tokens,
|
||||
cache_write_tokens=outcome.cache_write_tokens,
|
||||
uncached_input_tokens=outcome.uncached_input_tokens,
|
||||
provider_input_tokens=outcome.provider_input_tokens,
|
||||
transforms_applied=list(outcome.transforms_applied),
|
||||
savings_breakdown=savings_breakdown,
|
||||
waste_signals=outcome.waste_signals,
|
||||
|
||||
@@ -133,6 +133,7 @@ from headroom.proxy.cost import (
|
||||
build_prefix_cache_stats,
|
||||
build_session_summary,
|
||||
merge_cost_stats,
|
||||
request_cost_usd,
|
||||
)
|
||||
from headroom.proxy.helpers import (
|
||||
COMPRESSION_CACHE_MAX_ENTRIES,
|
||||
@@ -5051,6 +5052,11 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
"total_latency_ms": _recent_request_optional_number(log, "total_latency_ms"),
|
||||
"has_exact_tokens": token_accounting_status == "complete",
|
||||
"token_accounting_status": token_accounting_status,
|
||||
# Best-effort list-price estimate; null means unavailable,
|
||||
# never fabricated zero. ``turn_id`` lets a consumer group
|
||||
# every priced row by turn, not only the raw log tail.
|
||||
"cost_usd": request_cost_usd(proxy.cost_tracker, log),
|
||||
"turn_id": log.get("turn_id"),
|
||||
"transforms_applied": log.get("transforms_applied", []),
|
||||
"savings_breakdown": log.get("savings_breakdown", []),
|
||||
"waste_signals": log.get("waste_signals"),
|
||||
|
||||
@@ -351,6 +351,9 @@ def test_savings_tracker_save_fsyncs_parent_directory(tmp_path, monkeypatch):
|
||||
# The file fsync persists contents, but the rename isn't durable until the
|
||||
# parent directory is fsynced too — without it a crash can drop the last
|
||||
# save. Assert a directory fd is fsynced on save. (FP4b)
|
||||
if os.name == "nt":
|
||||
pytest.skip("directory fsync is POSIX-specific")
|
||||
|
||||
path = tmp_path / "proxy_savings.json"
|
||||
tracker = SavingsTracker(path=str(path))
|
||||
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
"""Tests for per-request ``cost_usd`` in ``/stats`` recent_requests (issue #1079).
|
||||
|
||||
The proxy exposes a per-request billed-cost estimate so tooling (e.g. claude-hud)
|
||||
can attribute cost to a Claude Code session by summing over turn_id / time window,
|
||||
without an external ``ccusage`` subprocess.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from types import SimpleNamespace
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.cost import CostTracker, request_cost_usd
|
||||
from headroom.proxy.models import RequestLog
|
||||
from headroom.proxy.outcome import RequestOutcome, emit_request_outcome
|
||||
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
||||
from headroom.proxy.request_logger import RequestLogger
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
|
||||
def _tracker(estimate):
|
||||
"""A stand-in CostTracker exposing only ``estimate_cost``."""
|
||||
return SimpleNamespace(estimate_cost=estimate)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unit: request_cost_usd helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_none_when_cost_tracker_missing() -> None:
|
||||
log = {
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"input_tokens_optimized": 600,
|
||||
"output_tokens": 50,
|
||||
}
|
||||
assert request_cost_usd(None, log) is None
|
||||
|
||||
|
||||
def test_none_when_model_missing() -> None:
|
||||
tracker = _tracker(lambda **_: 0.01)
|
||||
assert request_cost_usd(tracker, {"input_tokens_optimized": 600, "output_tokens": 50}) is None
|
||||
|
||||
|
||||
def test_none_when_input_tokens_missing() -> None:
|
||||
tracker = _tracker(lambda **_: 0.01)
|
||||
assert request_cost_usd(tracker, {"model": "claude-3-5-sonnet-20241022"}) is None
|
||||
|
||||
|
||||
def test_passes_through_estimate_and_uses_optimized_tokens() -> None:
|
||||
captured: dict = {}
|
||||
|
||||
def estimate(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return 0.0135
|
||||
|
||||
log = {
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"input_tokens_original": 1000,
|
||||
"input_tokens_optimized": 600,
|
||||
"output_tokens": 50,
|
||||
}
|
||||
assert request_cost_usd(_tracker(estimate), log) == 0.0135
|
||||
# Bills the post-compression input the provider actually saw, plus output.
|
||||
assert captured["input_tokens"] == 600
|
||||
assert captured["output_tokens"] == 50
|
||||
|
||||
|
||||
def test_prefers_provider_billed_input_over_local_count() -> None:
|
||||
# Local tokenizer counted 600, the provider billed 800: price the 800 the
|
||||
# cost tracker records, not the local estimate.
|
||||
captured: dict = {}
|
||||
|
||||
def estimate(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return 0.02
|
||||
|
||||
log = {
|
||||
"model": "claude-3-5-sonnet-20241022",
|
||||
"input_tokens_optimized": 600,
|
||||
"provider_input_tokens": 800,
|
||||
"output_tokens": 50,
|
||||
}
|
||||
assert request_cost_usd(_tracker(estimate), log) == 0.02
|
||||
assert captured["input_tokens"] == 800
|
||||
assert captured["output_tokens"] == 50
|
||||
|
||||
|
||||
def test_falls_back_to_optimized_tokens_when_provider_count_unreported() -> None:
|
||||
# 0 means the provider reported no input usage, never "zero tokens billed".
|
||||
captured: dict = {}
|
||||
|
||||
def estimate(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return 0.01
|
||||
|
||||
log = {
|
||||
"model": "m",
|
||||
"input_tokens_optimized": 600,
|
||||
"provider_input_tokens": 0,
|
||||
"output_tokens": 50,
|
||||
}
|
||||
assert request_cost_usd(_tracker(estimate), log) == 0.01
|
||||
assert captured["input_tokens"] == 600
|
||||
|
||||
|
||||
def test_none_when_output_tokens_unknown() -> None:
|
||||
# Pricing a missing output count as zero would publish an input-only figure
|
||||
# that looks like a complete request cost.
|
||||
calls: list[dict] = []
|
||||
|
||||
def estimate(**kwargs):
|
||||
calls.append(kwargs)
|
||||
return 0.002
|
||||
|
||||
log = {"model": "m", "input_tokens_optimized": 100, "output_tokens": None}
|
||||
assert request_cost_usd(_tracker(estimate), log) is None
|
||||
assert (
|
||||
request_cost_usd(_tracker(estimate), {"model": "m", "input_tokens_optimized": 100}) is None
|
||||
)
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_zero_output_tokens_is_a_real_count() -> None:
|
||||
captured: dict = {}
|
||||
|
||||
def estimate(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return 0.002
|
||||
|
||||
log = {"model": "m", "input_tokens_optimized": 100, "output_tokens": 0}
|
||||
assert request_cost_usd(_tracker(estimate), log) == 0.002
|
||||
assert captured["output_tokens"] == 0
|
||||
|
||||
|
||||
def test_none_when_estimate_returns_none() -> None:
|
||||
# Unknown model / LiteLLM unavailable → estimate_cost returns None.
|
||||
log = {"model": "totally-unknown-model", "input_tokens_optimized": 600, "output_tokens": 50}
|
||||
assert request_cost_usd(_tracker(lambda **_: None), log) is None
|
||||
|
||||
|
||||
def test_none_and_isolated_when_estimate_raises() -> None:
|
||||
def boom(**_):
|
||||
raise RuntimeError("pricing exploded")
|
||||
|
||||
log = {"model": "m", "input_tokens_optimized": 600, "output_tokens": 50}
|
||||
assert request_cost_usd(_tracker(boom), log) is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Chain: emit_request_outcome -> RequestLog -> request_cost_usd
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_logged_request_prices_the_input_the_cost_tracker_billed(monkeypatch) -> None:
|
||||
cost_tracker = CostTracker()
|
||||
billed_inputs: list[int] = []
|
||||
|
||||
def estimate(**kwargs):
|
||||
billed_inputs.append(kwargs["input_tokens"])
|
||||
return kwargs["input_tokens"] * 1e-6
|
||||
|
||||
monkeypatch.setattr(cost_tracker, "estimate_cost", estimate)
|
||||
handler = SimpleNamespace(
|
||||
metrics=PrometheusMetrics(cost_tracker=cost_tracker, stateless=True),
|
||||
cost_tracker=cost_tracker,
|
||||
logger=RequestLogger(),
|
||||
)
|
||||
outcome = RequestOutcome(
|
||||
request_id="hr_test_mismatch",
|
||||
provider="openai",
|
||||
model="gpt-4o-mini",
|
||||
original_tokens=1000,
|
||||
optimized_tokens=600,
|
||||
provider_input_tokens=800,
|
||||
output_tokens=50,
|
||||
tokens_saved=400,
|
||||
attempted_input_tokens=1000,
|
||||
)
|
||||
|
||||
asyncio.run(emit_request_outcome(handler, outcome))
|
||||
|
||||
(log,) = handler.logger.get_recent(1)
|
||||
assert log["input_tokens_optimized"] == 600
|
||||
assert log["provider_input_tokens"] == 800
|
||||
assert request_cost_usd(cost_tracker, log) == 800 * 1e-6
|
||||
# The cost tracker and the per-request field price the same input count.
|
||||
assert billed_inputs == [800, 800]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration: /stats exposes cost_usd in recent_requests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sample_log(request_id: str = "hr_test_000001", turn_id: str | None = None) -> RequestLog:
|
||||
return RequestLog(
|
||||
request_id=request_id,
|
||||
timestamp="2026-06-29T00:00:00Z",
|
||||
provider="anthropic",
|
||||
model="claude-3-5-sonnet-20241022",
|
||||
input_tokens_original=1000,
|
||||
input_tokens_optimized=600,
|
||||
output_tokens=50,
|
||||
tokens_saved=400,
|
||||
savings_percent=40.0,
|
||||
optimization_latency_ms=1.0,
|
||||
total_latency_ms=2.0,
|
||||
tags={},
|
||||
cache_hit=False,
|
||||
transforms_applied=[],
|
||||
turn_id=turn_id,
|
||||
)
|
||||
|
||||
|
||||
def _stats_config() -> ProxyConfig:
|
||||
return ProxyConfig(
|
||||
optimize=False,
|
||||
image_optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=True,
|
||||
log_requests=True,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
)
|
||||
|
||||
|
||||
def test_stats_recent_requests_includes_cost_usd(monkeypatch) -> None:
|
||||
app = create_app(_stats_config())
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
assert proxy.cost_tracker is not None
|
||||
# Deterministic price, independent of the live LiteLLM DB.
|
||||
monkeypatch.setattr(proxy.cost_tracker, "estimate_cost", lambda **_: 0.0135)
|
||||
# recent_requests is served only to loopback callers; TestClient's peer
|
||||
# host is "testclient", so force the loopback branch (not under test).
|
||||
monkeypatch.setattr("headroom.proxy.server._request_is_loopback", lambda request: True)
|
||||
proxy.logger._logs.append(_sample_log())
|
||||
|
||||
resp = client.get("/stats")
|
||||
assert resp.status_code == 200
|
||||
recent = resp.json()["recent_requests"]
|
||||
assert recent, "expected at least one recent request"
|
||||
entry = next(e for e in recent if e["request_id"] == "hr_test_000001")
|
||||
assert entry["cost_usd"] == 0.0135
|
||||
|
||||
|
||||
def test_stats_recent_requests_carry_turn_id_beyond_raw_log_tail(monkeypatch) -> None:
|
||||
# recent_requests serves up to 25 priced rows while request_logs keeps only
|
||||
# the newest 10, so every priced row must carry its own turn_id for a
|
||||
# consumer to group cost by turn.
|
||||
app = create_app(_stats_config())
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
monkeypatch.setattr(proxy.cost_tracker, "estimate_cost", lambda **_: 0.001)
|
||||
monkeypatch.setattr("headroom.proxy.server._request_is_loopback", lambda request: True)
|
||||
for i in range(20):
|
||||
proxy.logger._logs.append(_sample_log(f"hr_test_{i:06d}", turn_id=f"turn_{i // 4}"))
|
||||
|
||||
payload = client.get("/stats").json()
|
||||
assert len(payload["request_logs"]) == 10
|
||||
recent = [e for e in payload["recent_requests"] if e["request_id"].startswith("hr_test_")]
|
||||
assert len(recent) == 20
|
||||
|
||||
cost_by_turn: dict[str, float] = {}
|
||||
for entry in recent:
|
||||
assert entry["cost_usd"] == 0.001
|
||||
cost_by_turn[entry["turn_id"]] = (
|
||||
cost_by_turn.get(entry["turn_id"], 0.0) + entry["cost_usd"]
|
||||
)
|
||||
assert sorted(cost_by_turn) == [f"turn_{t}" for t in range(5)]
|
||||
assert all(abs(total - 0.004) < 1e-12 for total in cost_by_turn.values())
|
||||
Reference in New Issue
Block a user