feat(savings): per-agent attribution in history checkpoints and rollups (#2627)

## Description

Savings rollups attribute by upstream provider only, which cannot
separate clients that share an upstream: Claude Code and OpenCode both
hit anthropic, Codex and OpenCode both hit openai. Downstream dashboards
are forced to show blended or misattributed rows - concretely, a user
running non-anthropic/non-openai models through OpenCode sees those
savings folded into Claude Code / Codex rows.

The proxy already classifies the agent on every request
(`classify_client` from User-Agent / `X-Client`: `claude-code`, `codex`,
`opencode`, `grok_build`, ...) and logs it in every PERF line - it just
never persists it into the savings data. This PR does that, additively.

Complements #2626 (provider `xai` for api.x.ai custom-base traffic):
that PR fixes the provider dimension where a distinct vendor makes it
possible; this one adds the agent dimension for the cases where
providers are inherently shared.

Closes #3995

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/savings_tracker.py`:
- `_normalize_agent()` + `AGENT_UNKNOWN` sentinel (mirrors
`_normalize_provider`; checkpoints persisted before this change collapse
into `"unknown"` rather than dropping savings).
- `record_compression_savings(..., agent=None)` and `record_request(...,
agent=None)` write `"agent"` into history checkpoints.
- `_build_rollup()` emits a `by_agent` map per bucket alongside
`by_provider`, with the same single-owner delta attribution (each
checkpoint comes from one request, so its delta is wholly owned by one
agent).
- `headroom/proxy/prometheus_metrics.py`: threads the already-classified
`client` into `savings_tracker.record_request` as `agent`.
- `tests/test_proxy_savings_history.py`: new rollup test proving the
case `by_provider` cannot express (two agents on the same provider in
one bucket separate under `by_agent`; legacy checkpoints collapse to
`unknown`); existing exact-shape history assertions updated for the new
field.

Backward compatibility: purely additive. Existing
`by_provider`/`by_model` consumers untouched; `by_agent` rides into
`/stats-history` with the bucket dicts; old persisted state files load
unchanged (missing `agent` normalizes to `unknown`).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_proxy_savings_history.py
======================== 43 passed, 1 warning in 6.38s =========================
$ uv run --frozen --extra dev pytest tests/test_proxy_cache_ttl_metrics.py tests/test_request_outcome.py
======================== 52 passed, 1 warning in 5.41s =========================
$ uvx ruff check headroom/proxy/savings_tracker.py headroom/proxy/prometheus_metrics.py tests/test_proxy_savings_history.py
All checks passed!
$ uvx ruff format --check ... (same files)
3 files already formatted
$ uv run --frozen --extra dev mypy headroom/proxy/savings_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), headroom-ai 0.32.1 desktop-managed
proxy, real traffic from Claude Code, Codex, OpenCode 1.18.6 and Grok
Build 0.2.112 in the same session.
- Exact command / steps: drove all four clients through one proxy
instance; inspected PERF log lines and the `/stats-history` rollups.
- Observed result: on unpatched 0.32.1, every PERF line carries correct
agent classification (`client=claude-code` / `codex` / `opencode` /
`grok_build`), yet the rollup buckets contain only `by_provider` -
OpenCode's anthropic-model savings are indistinguishable from Claude
Code's, and third-party-provider traffic routed by the OpenCode
transport plugin has no honest row at all. With this patch, the new
rollup test reproduces that shared-provider case and shows `by_agent`
separating what `by_provider` blends.
- Repro on current main (2026-10-06, see #3995): one Claude Code request
(3,000 tokens saved) and one OpenCode request (1,000), both on
anthropic, through `PrometheusMetrics.record_request`. On upstream/main
`855390d6` the daily bucket has only `by_provider: {"anthropic": 4000}`;
on this branch (`d5673b86`, merged with that main) it adds `by_agent:
{"claude-code": 3000, "opencode": 1000}` with `by_provider` unchanged.
234 tests across 18 savings/history files pass, ruff, ruff format and
mypy clean. The earlier red `npm audit` jobs were `source-map-js` 1.2.1
(GHSA-68fv-2mgg-jv7q) in two lockfiles this PR does not touch; main
fixed them in #3985 and #3988, which this merge brings in.
- Not tested: a live proxy run with the patch applied (unit tests cover
the changed functions; the motivating traffic captures are real).

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Savings history accounting is not
rollout-managed.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Additive only. History checkpoints
gain an `agent` field and rollups gain a `by_agent` breakdown beside the
existing `by_provider` one. Existing fields, totals and per-provider
figures are unchanged, and entries persisted before this change read
back as the `unknown` agent instead of failing.
- Kill switch / disable path: No runtime flag; nothing reads the new
fields unless a consumer asks for `by_agent`. Revert to remove them.
- Unsafe override required: No
- Qualification impact: New tests in
`tests/test_proxy_savings_history.py`; no existing test changed or
relaxed.
- Rollback path: Revert the PR. Older code ignores the extra `agent` key
in persisted checkpoints, so no migration or cleanup is needed.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable - telemetry/attribution change.

## Additional Notes

- Documentation checklist unchecked: no doc page describes the rollup
schema today; the inline comments cover the new dimension. Happy to add
one if there is a preferred location.
- Downstream motivation: Headroom Desktop currently displays blended
"Claude Code / OpenCode" rows because provider data cannot do better;
once this lands it will key display rows by agent, giving every
connector an honest row.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2627"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
2026-10-08 10:55:46 -05:00
committed by GitHub
co-authored by Claude Fable 5 JD Davis
parent 463c1a43e3
commit 489de46825
3 changed files with 168 additions and 0 deletions
+1
View File
@@ -1065,6 +1065,7 @@ class PrometheusMetrics:
# saved next to "0 tokens saved".
tool_search_saved=tool_search_saved,
provider=provider,
agent=client,
project=project,
cache_read_tokens=cache_read_tokens,
cache_write_tokens=cache_write_tokens,
+58
View File
@@ -265,6 +265,23 @@ def _normalize_provider(value: Any) -> str:
return cleaned or PROVIDER_UNKNOWN
AGENT_UNKNOWN = "unknown"
def _normalize_agent(value: Any) -> str:
"""Normalize a client/agent label, falling back to a stable sentinel.
The agent is the harness classified from User-Agent / X-Client
(claude-code, codex, opencode, grok_build, ...). Checkpoints persisted
before per-agent attribution existed have no agent field and collapse
into ``AGENT_UNKNOWN``.
"""
if not isinstance(value, str):
return AGENT_UNKNOWN
cleaned = value.strip()
return cleaned or AGENT_UNKNOWN
MODEL_UNKNOWN = "unknown"
@@ -649,6 +666,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
total_output_cost_usd = 0.0
provider = PROVIDER_UNKNOWN
model = MODEL_UNKNOWN
agent = AGENT_UNKNOWN
if isinstance(entry, dict):
timestamp = _parse_timestamp(entry.get("timestamp"))
@@ -668,6 +686,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
total_output_cost_usd = _coerce_float(entry.get("total_output_cost_usd"))
provider = _normalize_provider(entry.get("provider"))
model = _normalize_model(entry.get("model"))
agent = _normalize_agent(entry.get("agent"))
elif isinstance(entry, list | tuple) and len(entry) >= 2:
timestamp = _parse_timestamp(entry[0])
total_tokens_saved = _coerce_int(entry[1])
@@ -687,6 +706,7 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
"timestamp": _to_utc_iso(timestamp),
"provider": provider,
"model": model,
"agent": agent,
"total_tokens_saved": total_tokens_saved,
"compression_savings_usd": round(compression_savings_usd, 6),
"cache_read_tokens": cache_read_tokens,
@@ -971,6 +991,7 @@ class SavingsTracker:
model: str,
tokens_saved: int,
provider: str | None = None,
agent: str | None = None,
total_input_tokens: int | None = None,
total_input_cost_usd: float | None = None,
timestamp: datetime | str | None = None,
@@ -1023,6 +1044,7 @@ class SavingsTracker:
{
"timestamp": _to_utc_iso(timestamp_dt),
"provider": _normalize_provider(provider),
"agent": _normalize_agent(agent),
"model": _normalize_model(model),
"total_tokens_saved": lifetime["tokens_saved"],
"compression_savings_usd": lifetime["compression_savings_usd"],
@@ -1052,6 +1074,7 @@ class SavingsTracker:
# into ``lifetime["total_output_cost_usd"]``; see the delta below.
output_tokens: int = 0,
provider: str | None = None,
agent: str | None = None,
project: str | None = None,
cache_read_tokens: int = 0,
cache_write_tokens: int = 0,
@@ -1325,6 +1348,7 @@ class SavingsTracker:
{
"timestamp": _to_utc_iso(timestamp_dt),
"provider": _normalize_provider(provider),
"agent": _normalize_agent(agent),
"model": _normalize_model(model),
"total_tokens_saved": lifetime["tokens_saved"],
"compression_savings_usd": lifetime["compression_savings_usd"],
@@ -2294,6 +2318,7 @@ class SavingsTracker:
"total_output_cost_usd": total_output_cost_usd,
**_empty_cache_delta(),
"by_provider": {},
"by_agent": {},
"by_model": {},
},
)
@@ -2371,6 +2396,39 @@ class SavingsTracker:
6,
)
# Same single-owner attribution by agent. Providers cannot
# separate clients that share an upstream (Claude Code vs
# OpenCode both hit anthropic; Codex vs OpenCode both hit
# openai), so dashboards need this dimension for honest
# per-connector rows.
agent = _normalize_agent(point.get("agent"))
agent_entry = entry["by_agent"].setdefault(
agent,
{
"tokens_saved": 0,
"compression_savings_usd_delta": 0.0,
"total_input_tokens_delta": 0,
"total_input_cost_usd_delta": 0.0,
**_empty_cache_delta(),
},
)
_add_cache(
agent_entry,
delta_cache_read_tokens,
delta_cache_savings_usd,
delta_cache_read_cost_usd,
)
agent_entry["tokens_saved"] += delta_tokens
agent_entry["compression_savings_usd_delta"] = round(
agent_entry["compression_savings_usd_delta"] + delta_usd,
6,
)
agent_entry["total_input_tokens_delta"] += delta_input_tokens
agent_entry["total_input_cost_usd_delta"] = round(
agent_entry["total_input_cost_usd_delta"] + delta_input_cost_usd,
6,
)
mod = entry["by_model"].setdefault(
model,
{
+109
View File
@@ -76,6 +76,7 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
"timestamp": "2026-03-27T09:00:00Z",
"provider": "unknown",
"model": "unknown",
"agent": "unknown",
"total_tokens_saved": 12,
"compression_savings_usd": 0.5,
"cache_read_tokens": 0,
@@ -157,6 +158,7 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
"timestamp": "2026-03-27T09:00:00Z",
"provider": "unknown",
"model": "unknown",
"agent": "unknown",
"total_tokens_saved": 30,
"compression_savings_usd": 0.03,
"cache_read_tokens": 0,
@@ -244,6 +246,7 @@ def test_record_compression_savings_skips_empty_updates_and_normalizes_timestamp
{
"timestamp": "2026-03-27T08:00:00Z",
"provider": "unknown",
"agent": "unknown",
"model": "gpt-4o",
"total_tokens_saved": 10,
"compression_savings_usd": 0.01,
@@ -253,6 +256,7 @@ def test_record_compression_savings_skips_empty_updates_and_normalizes_timestamp
{
"timestamp": "2026-03-27T12:34:00Z",
"provider": "unknown",
"agent": "unknown",
"model": "gpt-4o",
"total_tokens_saved": 15,
"compression_savings_usd": 0.015,
@@ -937,6 +941,111 @@ def test_savings_tracker_rollup_attributes_savings_per_provider(tmp_path, monkey
assert third["by_provider"]["unknown"]["tokens_saved"] == 15
def test_savings_tracker_rollup_attributes_savings_per_agent(tmp_path, monkeypatch):
path = tmp_path / "proxy_savings.json"
tracker = SavingsTracker(
path=str(path),
max_history_points=100,
max_history_age_days=30,
)
monkeypatch.setattr(
"headroom.proxy.savings_tracker._estimate_compression_savings_usd",
lambda model, tokens_saved: tokens_saved / 1000.0,
)
# Two agents sharing the SAME upstream provider in one hour bucket - the
# case by_provider fundamentally cannot separate (Claude Code and
# OpenCode both talk to anthropic).
tracker.record_compression_savings(
model="claude-3-5-sonnet",
tokens_saved=100,
provider="anthropic",
agent="claude-code",
total_input_tokens=120,
total_input_cost_usd=0.24,
timestamp="2026-03-27T09:10:00Z",
)
tracker.record_compression_savings(
model="claude-3-5-sonnet",
tokens_saved=40,
provider="anthropic",
agent="opencode",
total_input_tokens=200,
total_input_cost_usd=0.40,
timestamp="2026-03-27T09:40:00Z",
)
# A legacy-style record with no agent collapses into "unknown".
tracker.record_compression_savings(
model="gpt-4o",
tokens_saved=15,
provider="openai",
total_input_tokens=260,
total_input_cost_usd=0.52,
timestamp="2026-03-27T10:05:00Z",
)
hourly = tracker.history_response()["series"]["hourly"]
first = hourly[0]
# by_provider blends the two agents...
assert set(first["by_provider"]) == {"anthropic"}
assert first["by_provider"]["anthropic"]["tokens_saved"] == 140
# ...by_agent separates them.
assert set(first["by_agent"]) == {"claude-code", "opencode"}
assert first["by_agent"]["claude-code"]["tokens_saved"] == 100
assert first["by_agent"]["claude-code"]["total_input_tokens_delta"] == 120
assert first["by_agent"]["claude-code"]["compression_savings_usd_delta"] == pytest.approx(0.1)
assert first["by_agent"]["opencode"]["tokens_saved"] == 40
assert first["by_agent"]["opencode"]["total_input_tokens_delta"] == 80
assert (
first["by_agent"]["claude-code"]["tokens_saved"]
+ first["by_agent"]["opencode"]["tokens_saved"]
== first["tokens_saved"]
)
second = hourly[1]
assert set(second["by_agent"]) == {"unknown"}
assert second["by_agent"]["unknown"]["tokens_saved"] == 15
def test_savings_tracker_rollup_reloads_persisted_agent_field(tmp_path):
"""Persisted checkpoints keep their agent across a reload.
``_normalize_history_entry`` runs on every load; if it drops the ``agent``
key, a real ``agent: "claude-code"`` checkpoint collapses into "unknown"
on the next restart even though in-memory attribution was correct.
"""
path = tmp_path / "proxy_savings.json"
path.write_text(
json.dumps(
{
"history": [
{
"timestamp": "2026-03-27T09:10:00Z",
"provider": "anthropic",
"agent": "claude-code",
"model": "claude-3-5-sonnet",
"total_tokens_saved": 100,
"compression_savings_usd": 0.1,
"total_input_tokens": 120,
"total_input_cost_usd": 0.24,
}
]
}
)
)
tracker = SavingsTracker(
path=str(path),
max_history_points=100,
max_history_age_days=30,
)
by_agent = tracker.history_response()["series"]["hourly"][0]["by_agent"]
assert set(by_agent) == {"claude-code"}
assert by_agent["claude-code"]["tokens_saved"] == 100
def test_savings_tracker_rollup_carries_exact_cache_read_cost_per_provider(tmp_path, monkeypatch):
# Reads on "steep-model" bill at 0.025x list (Fable-5.1-shaped pricing),
# on "flat-model" at the usual 0.1x. A consumer that backs the read cost