mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(savings): per-agent attribution in history checkpoints and rollups (#2627)
## Description Savings rollups attribute by upstream provider only, which cannot separate clients that share an upstream: Claude Code and OpenCode both hit anthropic, Codex and OpenCode both hit openai. Downstream dashboards are forced to show blended or misattributed rows - concretely, a user running non-anthropic/non-openai models through OpenCode sees those savings folded into Claude Code / Codex rows. The proxy already classifies the agent on every request (`classify_client` from User-Agent / `X-Client`: `claude-code`, `codex`, `opencode`, `grok_build`, ...) and logs it in every PERF line - it just never persists it into the savings data. This PR does that, additively. Complements #2626 (provider `xai` for api.x.ai custom-base traffic): that PR fixes the provider dimension where a distinct vendor makes it possible; this one adds the agent dimension for the cases where providers are inherently shared. Closes #3995 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: - `_normalize_agent()` + `AGENT_UNKNOWN` sentinel (mirrors `_normalize_provider`; checkpoints persisted before this change collapse into `"unknown"` rather than dropping savings). - `record_compression_savings(..., agent=None)` and `record_request(..., agent=None)` write `"agent"` into history checkpoints. - `_build_rollup()` emits a `by_agent` map per bucket alongside `by_provider`, with the same single-owner delta attribution (each checkpoint comes from one request, so its delta is wholly owned by one agent). - `headroom/proxy/prometheus_metrics.py`: threads the already-classified `client` into `savings_tracker.record_request` as `agent`. - `tests/test_proxy_savings_history.py`: new rollup test proving the case `by_provider` cannot express (two agents on the same provider in one bucket separate under `by_agent`; legacy checkpoints collapse to `unknown`); existing exact-shape history assertions updated for the new field. Backward compatibility: purely additive. Existing `by_provider`/`by_model` consumers untouched; `by_agent` rides into `/stats-history` with the bucket dicts; old persisted state files load unchanged (missing `agent` normalizes to `unknown`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py ======================== 43 passed, 1 warning in 6.38s ========================= $ uv run --frozen --extra dev pytest tests/test_proxy_cache_ttl_metrics.py tests/test_request_outcome.py ======================== 52 passed, 1 warning in 5.41s ========================= $ uvx ruff check headroom/proxy/savings_tracker.py headroom/proxy/prometheus_metrics.py tests/test_proxy_savings_history.py All checks passed! $ uvx ruff format --check ... (same files) 3 files already formatted $ uv run --frozen --extra dev mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), headroom-ai 0.32.1 desktop-managed proxy, real traffic from Claude Code, Codex, OpenCode 1.18.6 and Grok Build 0.2.112 in the same session. - Exact command / steps: drove all four clients through one proxy instance; inspected PERF log lines and the `/stats-history` rollups. - Observed result: on unpatched 0.32.1, every PERF line carries correct agent classification (`client=claude-code` / `codex` / `opencode` / `grok_build`), yet the rollup buckets contain only `by_provider` - OpenCode's anthropic-model savings are indistinguishable from Claude Code's, and third-party-provider traffic routed by the OpenCode transport plugin has no honest row at all. With this patch, the new rollup test reproduces that shared-provider case and shows `by_agent` separating what `by_provider` blends. - Repro on current main (2026-10-06, see #3995): one Claude Code request (3,000 tokens saved) and one OpenCode request (1,000), both on anthropic, through `PrometheusMetrics.record_request`. On upstream/main `855390d6` the daily bucket has only `by_provider: {"anthropic": 4000}`; on this branch (`d5673b86`, merged with that main) it adds `by_agent: {"claude-code": 3000, "opencode": 1000}` with `by_provider` unchanged. 234 tests across 18 savings/history files pass, ruff, ruff format and mypy clean. The earlier red `npm audit` jobs were `source-map-js` 1.2.1 (GHSA-68fv-2mgg-jv7q) in two lockfiles this PR does not touch; main fixed them in #3985 and #3988, which this merge brings in. - Not tested: a live proxy run with the patch applied (unit tests cover the changed functions; the motivating traffic captures are real). ## Runtime Rollout Safety - Rollout-managed feature(s): None. Savings history accounting is not rollout-managed. - Minimum rollout channel: N/A - Stable/default behavior changed: Additive only. History checkpoints gain an `agent` field and rollups gain a `by_agent` breakdown beside the existing `by_provider` one. Existing fields, totals and per-provider figures are unchanged, and entries persisted before this change read back as the `unknown` agent instead of failing. - Kill switch / disable path: No runtime flag; nothing reads the new fields unless a consumer asks for `by_agent`. Revert to remove them. - Unsafe override required: No - Qualification impact: New tests in `tests/test_proxy_savings_history.py`; no existing test changed or relaxed. - Rollback path: Revert the PR. Older code ignores the extra `agent` key in persisted checkpoints, so no migration or cleanup is needed. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable - telemetry/attribution change. ## Additional Notes - Documentation checklist unchecked: no doc page describes the rollup schema today; the inline comments cover the new dimension. Happy to add one if there is a preferred location. - Downstream motivation: Headroom Desktop currently displays blended "Claude Code / OpenCode" rows because provider data cannot do better; once this lands it will key display rows by agent, giving every connector an honest row. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2627" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Fable 5
JD Davis
parent
463c1a43e3
commit
489de46825
@@ -1065,6 +1065,7 @@ class PrometheusMetrics:
|
||||
# saved next to "0 tokens saved".
|
||||
tool_search_saved=tool_search_saved,
|
||||
provider=provider,
|
||||
agent=client,
|
||||
project=project,
|
||||
cache_read_tokens=cache_read_tokens,
|
||||
cache_write_tokens=cache_write_tokens,
|
||||
|
||||
@@ -265,6 +265,23 @@ def _normalize_provider(value: Any) -> str:
|
||||
return cleaned or PROVIDER_UNKNOWN
|
||||
|
||||
|
||||
AGENT_UNKNOWN = "unknown"
|
||||
|
||||
|
||||
def _normalize_agent(value: Any) -> str:
|
||||
"""Normalize a client/agent label, falling back to a stable sentinel.
|
||||
|
||||
The agent is the harness classified from User-Agent / X-Client
|
||||
(claude-code, codex, opencode, grok_build, ...). Checkpoints persisted
|
||||
before per-agent attribution existed have no agent field and collapse
|
||||
into ``AGENT_UNKNOWN``.
|
||||
"""
|
||||
if not isinstance(value, str):
|
||||
return AGENT_UNKNOWN
|
||||
cleaned = value.strip()
|
||||
return cleaned or AGENT_UNKNOWN
|
||||
|
||||
|
||||
MODEL_UNKNOWN = "unknown"
|
||||
|
||||
|
||||
@@ -649,6 +666,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_output_cost_usd = 0.0
|
||||
provider = PROVIDER_UNKNOWN
|
||||
model = MODEL_UNKNOWN
|
||||
agent = AGENT_UNKNOWN
|
||||
|
||||
if isinstance(entry, dict):
|
||||
timestamp = _parse_timestamp(entry.get("timestamp"))
|
||||
@@ -668,6 +686,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
total_output_cost_usd = _coerce_float(entry.get("total_output_cost_usd"))
|
||||
provider = _normalize_provider(entry.get("provider"))
|
||||
model = _normalize_model(entry.get("model"))
|
||||
agent = _normalize_agent(entry.get("agent"))
|
||||
elif isinstance(entry, list | tuple) and len(entry) >= 2:
|
||||
timestamp = _parse_timestamp(entry[0])
|
||||
total_tokens_saved = _coerce_int(entry[1])
|
||||
@@ -687,6 +706,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
"timestamp": _to_utc_iso(timestamp),
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"agent": agent,
|
||||
"total_tokens_saved": total_tokens_saved,
|
||||
"compression_savings_usd": round(compression_savings_usd, 6),
|
||||
"cache_read_tokens": cache_read_tokens,
|
||||
@@ -971,6 +991,7 @@ class SavingsTracker:
|
||||
model: str,
|
||||
tokens_saved: int,
|
||||
provider: str | None = None,
|
||||
agent: str | None = None,
|
||||
total_input_tokens: int | None = None,
|
||||
total_input_cost_usd: float | None = None,
|
||||
timestamp: datetime | str | None = None,
|
||||
@@ -1023,6 +1044,7 @@ class SavingsTracker:
|
||||
{
|
||||
"timestamp": _to_utc_iso(timestamp_dt),
|
||||
"provider": _normalize_provider(provider),
|
||||
"agent": _normalize_agent(agent),
|
||||
"model": _normalize_model(model),
|
||||
"total_tokens_saved": lifetime["tokens_saved"],
|
||||
"compression_savings_usd": lifetime["compression_savings_usd"],
|
||||
@@ -1052,6 +1074,7 @@ class SavingsTracker:
|
||||
# into ``lifetime["total_output_cost_usd"]``; see the delta below.
|
||||
output_tokens: int = 0,
|
||||
provider: str | None = None,
|
||||
agent: str | None = None,
|
||||
project: str | None = None,
|
||||
cache_read_tokens: int = 0,
|
||||
cache_write_tokens: int = 0,
|
||||
@@ -1325,6 +1348,7 @@ class SavingsTracker:
|
||||
{
|
||||
"timestamp": _to_utc_iso(timestamp_dt),
|
||||
"provider": _normalize_provider(provider),
|
||||
"agent": _normalize_agent(agent),
|
||||
"model": _normalize_model(model),
|
||||
"total_tokens_saved": lifetime["tokens_saved"],
|
||||
"compression_savings_usd": lifetime["compression_savings_usd"],
|
||||
@@ -2294,6 +2318,7 @@ class SavingsTracker:
|
||||
"total_output_cost_usd": total_output_cost_usd,
|
||||
**_empty_cache_delta(),
|
||||
"by_provider": {},
|
||||
"by_agent": {},
|
||||
"by_model": {},
|
||||
},
|
||||
)
|
||||
@@ -2371,6 +2396,39 @@ class SavingsTracker:
|
||||
6,
|
||||
)
|
||||
|
||||
# Same single-owner attribution by agent. Providers cannot
|
||||
# separate clients that share an upstream (Claude Code vs
|
||||
# OpenCode both hit anthropic; Codex vs OpenCode both hit
|
||||
# openai), so dashboards need this dimension for honest
|
||||
# per-connector rows.
|
||||
agent = _normalize_agent(point.get("agent"))
|
||||
agent_entry = entry["by_agent"].setdefault(
|
||||
agent,
|
||||
{
|
||||
"tokens_saved": 0,
|
||||
"compression_savings_usd_delta": 0.0,
|
||||
"total_input_tokens_delta": 0,
|
||||
"total_input_cost_usd_delta": 0.0,
|
||||
**_empty_cache_delta(),
|
||||
},
|
||||
)
|
||||
_add_cache(
|
||||
agent_entry,
|
||||
delta_cache_read_tokens,
|
||||
delta_cache_savings_usd,
|
||||
delta_cache_read_cost_usd,
|
||||
)
|
||||
agent_entry["tokens_saved"] += delta_tokens
|
||||
agent_entry["compression_savings_usd_delta"] = round(
|
||||
agent_entry["compression_savings_usd_delta"] + delta_usd,
|
||||
6,
|
||||
)
|
||||
agent_entry["total_input_tokens_delta"] += delta_input_tokens
|
||||
agent_entry["total_input_cost_usd_delta"] = round(
|
||||
agent_entry["total_input_cost_usd_delta"] + delta_input_cost_usd,
|
||||
6,
|
||||
)
|
||||
|
||||
mod = entry["by_model"].setdefault(
|
||||
model,
|
||||
{
|
||||
|
||||
@@ -76,6 +76,7 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
|
||||
"timestamp": "2026-03-27T09:00:00Z",
|
||||
"provider": "unknown",
|
||||
"model": "unknown",
|
||||
"agent": "unknown",
|
||||
"total_tokens_saved": 12,
|
||||
"compression_savings_usd": 0.5,
|
||||
"cache_read_tokens": 0,
|
||||
@@ -157,6 +158,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
||||
"timestamp": "2026-03-27T09:00:00Z",
|
||||
"provider": "unknown",
|
||||
"model": "unknown",
|
||||
"agent": "unknown",
|
||||
"total_tokens_saved": 30,
|
||||
"compression_savings_usd": 0.03,
|
||||
"cache_read_tokens": 0,
|
||||
@@ -244,6 +246,7 @@ def test_record_compression_savings_skips_empty_updates_and_normalizes_timestamp
|
||||
{
|
||||
"timestamp": "2026-03-27T08:00:00Z",
|
||||
"provider": "unknown",
|
||||
"agent": "unknown",
|
||||
"model": "gpt-4o",
|
||||
"total_tokens_saved": 10,
|
||||
"compression_savings_usd": 0.01,
|
||||
@@ -253,6 +256,7 @@ def test_record_compression_savings_skips_empty_updates_and_normalizes_timestamp
|
||||
{
|
||||
"timestamp": "2026-03-27T12:34:00Z",
|
||||
"provider": "unknown",
|
||||
"agent": "unknown",
|
||||
"model": "gpt-4o",
|
||||
"total_tokens_saved": 15,
|
||||
"compression_savings_usd": 0.015,
|
||||
@@ -937,6 +941,111 @@ def test_savings_tracker_rollup_attributes_savings_per_provider(tmp_path, monkey
|
||||
assert third["by_provider"]["unknown"]["tokens_saved"] == 15
|
||||
|
||||
|
||||
def test_savings_tracker_rollup_attributes_savings_per_agent(tmp_path, monkeypatch):
|
||||
path = tmp_path / "proxy_savings.json"
|
||||
tracker = SavingsTracker(
|
||||
path=str(path),
|
||||
max_history_points=100,
|
||||
max_history_age_days=30,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"headroom.proxy.savings_tracker._estimate_compression_savings_usd",
|
||||
lambda model, tokens_saved: tokens_saved / 1000.0,
|
||||
)
|
||||
|
||||
# Two agents sharing the SAME upstream provider in one hour bucket - the
|
||||
# case by_provider fundamentally cannot separate (Claude Code and
|
||||
# OpenCode both talk to anthropic).
|
||||
tracker.record_compression_savings(
|
||||
model="claude-3-5-sonnet",
|
||||
tokens_saved=100,
|
||||
provider="anthropic",
|
||||
agent="claude-code",
|
||||
total_input_tokens=120,
|
||||
total_input_cost_usd=0.24,
|
||||
timestamp="2026-03-27T09:10:00Z",
|
||||
)
|
||||
tracker.record_compression_savings(
|
||||
model="claude-3-5-sonnet",
|
||||
tokens_saved=40,
|
||||
provider="anthropic",
|
||||
agent="opencode",
|
||||
total_input_tokens=200,
|
||||
total_input_cost_usd=0.40,
|
||||
timestamp="2026-03-27T09:40:00Z",
|
||||
)
|
||||
# A legacy-style record with no agent collapses into "unknown".
|
||||
tracker.record_compression_savings(
|
||||
model="gpt-4o",
|
||||
tokens_saved=15,
|
||||
provider="openai",
|
||||
total_input_tokens=260,
|
||||
total_input_cost_usd=0.52,
|
||||
timestamp="2026-03-27T10:05:00Z",
|
||||
)
|
||||
|
||||
hourly = tracker.history_response()["series"]["hourly"]
|
||||
|
||||
first = hourly[0]
|
||||
# by_provider blends the two agents...
|
||||
assert set(first["by_provider"]) == {"anthropic"}
|
||||
assert first["by_provider"]["anthropic"]["tokens_saved"] == 140
|
||||
# ...by_agent separates them.
|
||||
assert set(first["by_agent"]) == {"claude-code", "opencode"}
|
||||
assert first["by_agent"]["claude-code"]["tokens_saved"] == 100
|
||||
assert first["by_agent"]["claude-code"]["total_input_tokens_delta"] == 120
|
||||
assert first["by_agent"]["claude-code"]["compression_savings_usd_delta"] == pytest.approx(0.1)
|
||||
assert first["by_agent"]["opencode"]["tokens_saved"] == 40
|
||||
assert first["by_agent"]["opencode"]["total_input_tokens_delta"] == 80
|
||||
assert (
|
||||
first["by_agent"]["claude-code"]["tokens_saved"]
|
||||
+ first["by_agent"]["opencode"]["tokens_saved"]
|
||||
== first["tokens_saved"]
|
||||
)
|
||||
|
||||
second = hourly[1]
|
||||
assert set(second["by_agent"]) == {"unknown"}
|
||||
assert second["by_agent"]["unknown"]["tokens_saved"] == 15
|
||||
|
||||
|
||||
def test_savings_tracker_rollup_reloads_persisted_agent_field(tmp_path):
|
||||
"""Persisted checkpoints keep their agent across a reload.
|
||||
|
||||
``_normalize_history_entry`` runs on every load; if it drops the ``agent``
|
||||
key, a real ``agent: "claude-code"`` checkpoint collapses into "unknown"
|
||||
on the next restart even though in-memory attribution was correct.
|
||||
"""
|
||||
path = tmp_path / "proxy_savings.json"
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"history": [
|
||||
{
|
||||
"timestamp": "2026-03-27T09:10:00Z",
|
||||
"provider": "anthropic",
|
||||
"agent": "claude-code",
|
||||
"model": "claude-3-5-sonnet",
|
||||
"total_tokens_saved": 100,
|
||||
"compression_savings_usd": 0.1,
|
||||
"total_input_tokens": 120,
|
||||
"total_input_cost_usd": 0.24,
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
tracker = SavingsTracker(
|
||||
path=str(path),
|
||||
max_history_points=100,
|
||||
max_history_age_days=30,
|
||||
)
|
||||
|
||||
by_agent = tracker.history_response()["series"]["hourly"][0]["by_agent"]
|
||||
assert set(by_agent) == {"claude-code"}
|
||||
assert by_agent["claude-code"]["tokens_saved"] == 100
|
||||
|
||||
|
||||
def test_savings_tracker_rollup_carries_exact_cache_read_cost_per_provider(tmp_path, monkeypatch):
|
||||
# Reads on "steep-model" bill at 0.025x list (Fable-5.1-shaped pricing),
|
||||
# on "flat-model" at the usual 0.1x. A consumer that backs the read cost
|
||||
|
||||
Reference in New Issue
Block a user