mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(savings): show the proxy's model-routing savings in headroom savings (#4304)
## Problem
`headroom savings` reads only the compression ledger, so the money a
model router saves by serving a cheaper model never shows. In a workshop
sandbox test, routing saved $0.057 against compression's $0.015, and the
command showed only the $0.015.
## Spec
- **User story:** as an operator running Headroom with a model router, I
run `headroom savings` and see what routing saved beside what
compression saved, without opening the dashboard.
- **Source:** the routing-stats provider contract
(`headroom.proxy.routing_stats`), as served by the running proxy's
`/stats` → `routing`.
- **Moved:** `enforced` per pair, the requests whose model was
rewritten. Holdouts and shadow-mode decisions don't count.
- **Saved:** the sum of the pairs' signed `savings_usd`.
- **Scope:** the provider's `window`, or "all decisions on record" when
it gives none.
- **Failure and recovery:** with no proxy, no router, or an unreachable
`/stats` (2 s timeout), the section is omitted and the ledger output is
unchanged.
- A pair with no measured usage is shown as "not priced" rather than
counted. Example: Codex over WebSocket, where the router never sees the
response usage.
- **Security:** a GET of `http://127.0.0.1:<port>/stats` only, the same
probe `headroom doctor` makes. No credentials and no new endpoint.
- **Follow-up, not in this PR:** routing history while the proxy is
stopped. It lives in the plugin's own store, which core has no seam to
read.
## Change
```
Model routing (lifetime): 5 of 20 requests moved · $0.0256 saved
claude-sonnet-5-5 → claude-haiku-5-5 4 moved $0.0256
gpt-5.6-sol → gpt-5.6-terra 1 moved not priced (no usage reported)
Saved = each moved request's usage priced on the model asked for, minus its cost on the model served.
```
- **Port:** found as `headroom doctor` finds it: `HEADROOM_PORT`, then a
managed deployment's port, then the default.
- **JSON:** `--json` adds `"routing"`, the provider snapshot as served.
## Real behavior proof
- **Environment:**
- a Daytona workshop sandbox, Debian 12, Python 3.12;
- headroom-ai 0.41.0+main.fbf7bc32 with this `savings.py`, plus
headroom-enterprise 0.4.3 and routemegood 0.5.3 as the routing provider;
- real Claude Code 2.1 and Codex CLI 0.163 sessions through the proxy.
- **Commands:**
1. `headroom login`, `headroom route apply`, `headroom start`;
2. a Claude Code session on Opus 5.5, one on Sonnet 5.5 (routed to Haiku
5.5), and a Codex session on gpt-5.6-sol (routed to Terra);
3. then `headroom savings`. The output above is that run.
- **Cross-checked:**
- the section's totals and pairs against `curl localhost:8787/stats | jq
.routing` on the same proxy;
- a pair of 7 holdout requests, Haiku to Haiku from the Explore
subagent, is correctly left out of the moved count.
- **Not tested live:**
- a provider reporting `window: session`, or none (unit tests cover
both);
- a managed deployment on a non-default port;
- upgrade pairs with negative savings (handled by the signed sum, no
live data).
## Tests
- 4 tests in `tests/test_savings_ledger.py`:
- routing beside compression, including a holdout pair;
- routing with an empty ledger;
- each `window` label;
- no router or no proxy, giving no section.
- The routing tests fail on the first version of this PR, before
8128b56e.
- `test_proxy_record_request_appends_ledger_event` fails the same way on
main in my environment (`CancelledError`); it's unrelated.
- ruff, ruff format and mypy are clean on the changed file.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JerrettDavis
parent
a26faaeee3
commit
aaab338b64
+68
-1
@@ -4,11 +4,16 @@ Reads the append-only savings ledger (``~/.headroom/savings_events.jsonl``,
|
||||
written by both the MCP tool path and the proxy) and renders a cost-avoided
|
||||
summary with Today / Last 7 days / All time bars plus per-model and
|
||||
per-client breakdowns. Durable across restarts; aggregated on read.
|
||||
|
||||
When the local proxy runs a model router (a routing-stats provider, see
|
||||
``headroom.proxy.routing_stats``), its savings are shown beside: what each
|
||||
moved request cost on the cheaper model against the model it asked for.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import click
|
||||
@@ -82,6 +87,58 @@ _BASIS_NOTES = {
|
||||
}
|
||||
|
||||
|
||||
def _proxy_routing() -> dict[str, Any] | None:
|
||||
"""The running proxy's routing rollup (``/stats`` ``routing``), or None.
|
||||
|
||||
None when no proxy answers or it routes nothing: the ledger sections are
|
||||
unaffected either way, so a stopped proxy only hides this section.
|
||||
"""
|
||||
from headroom.install.health import probe_json
|
||||
from headroom.install.state import list_manifests
|
||||
|
||||
from .doctor import resolve_probe_port
|
||||
|
||||
env_port = os.environ.get("HEADROOM_PORT")
|
||||
requested = int(env_port) if env_port and env_port.isdigit() else None
|
||||
port = resolve_probe_port(requested, list_manifests())
|
||||
stats = probe_json(f"http://127.0.0.1:{port}/stats", timeout=2.0)
|
||||
routing = stats.get("routing") if isinstance(stats, dict) else None
|
||||
if not isinstance(routing, dict) or not routing.get("decisions"):
|
||||
return None
|
||||
return routing
|
||||
|
||||
|
||||
#: How the routing section names the provider's ``window`` (routing_stats contract).
|
||||
_ROUTING_WINDOWS = {"lifetime": "lifetime", "session": "this proxy session"}
|
||||
|
||||
|
||||
def _echo_routing(routing: dict[str, Any]) -> None:
|
||||
# Per the routing_stats contract: a pair's `enforced` requests are the ones whose
|
||||
# model was rewritten (holdouts and shadow decisions are not), and the saving is
|
||||
# the sum of the pairs' signed `savings_usd`.
|
||||
pairs = [p for p in routing.get("pairs") or [] if p.get("direction") != "same"]
|
||||
moved = sum(int(p.get("enforced") or 0) for p in pairs)
|
||||
saved = sum(float(p.get("savings_usd") or 0.0) for p in pairs)
|
||||
window = _ROUTING_WINDOWS.get(str(routing.get("window")), "all decisions on record")
|
||||
click.echo("")
|
||||
click.echo(
|
||||
f"Model routing ({window}): {moved:,} of {int(routing.get('decisions') or 0):,} "
|
||||
f"requests moved · {_money(saved)} saved"
|
||||
)
|
||||
for pair in pairs:
|
||||
pair_name = f"{pair.get('requested')} → {pair.get('served')}"
|
||||
priced = (
|
||||
_money(float(pair.get("savings_usd") or 0.0))
|
||||
if int(pair.get("measured") or 0)
|
||||
else "not priced (no usage reported)"
|
||||
)
|
||||
click.echo(f" {pair_name:<44} {int(pair.get('enforced') or 0):,} moved {priced}")
|
||||
click.echo(
|
||||
" Saved = each moved request's usage priced on the model asked for, minus its cost "
|
||||
"on the model served."
|
||||
)
|
||||
|
||||
|
||||
def _echo_basis_note(basis: Any) -> None:
|
||||
"""Print a caveat when the headline rests on something weaker than the catalog."""
|
||||
note = _BASIS_NOTES.get(str(basis or ""))
|
||||
@@ -113,13 +170,20 @@ def savings(as_json: bool, days: int, reset: bool) -> None:
|
||||
return
|
||||
|
||||
report = savings_ledger.aggregate_savings(retention_days=days)
|
||||
routing = _proxy_routing()
|
||||
|
||||
if as_json:
|
||||
click.echo(json.dumps(report.to_dict(), indent=2))
|
||||
data = report.to_dict()
|
||||
if routing is not None: # no router installed: not even an empty key
|
||||
data["routing"] = routing
|
||||
click.echo(json.dumps(data, indent=2))
|
||||
return
|
||||
|
||||
lifetime = report.lifetime
|
||||
calls = int(lifetime.get("calls", 0) or 0)
|
||||
if calls == 0 and routing is not None:
|
||||
_echo_routing(routing)
|
||||
return
|
||||
if calls == 0:
|
||||
click.echo("No savings recorded yet.")
|
||||
click.echo(
|
||||
@@ -156,3 +220,6 @@ def savings(as_json: bool, days: int, reset: bool) -> None:
|
||||
f" {str(row['client']):<24} {int(row['calls']):,} calls · "
|
||||
f"{_tokens(int(row['tokens_saved']))} tokens saved"
|
||||
)
|
||||
|
||||
if routing is not None:
|
||||
_echo_routing(routing)
|
||||
|
||||
@@ -666,3 +666,108 @@ def test_many_unwritable_ledgers_do_not_warn_on_every_cycle(
|
||||
assert len(warnings) == 65
|
||||
assert sum("append to" in w for w in warnings) == 64
|
||||
assert "More than 64 warnings about unwritable savings ledgers" in warnings[-1]
|
||||
|
||||
|
||||
_ROUTING = { # what routemegood publishes, trimmed to the routing_stats contract
|
||||
"decisions": 11,
|
||||
"window": "lifetime",
|
||||
"pairs": [
|
||||
{
|
||||
"requested": "claude-opus-5-5",
|
||||
"served": "claude-opus-5-5",
|
||||
"direction": "same",
|
||||
"count": 5,
|
||||
"enforced": 0,
|
||||
"holdout": 0,
|
||||
"measured": 0,
|
||||
"savings_usd": 0.0,
|
||||
},
|
||||
{
|
||||
"requested": "claude-sonnet-5-5",
|
||||
"served": "claude-haiku-5-5",
|
||||
"direction": "downgrade",
|
||||
"count": 5,
|
||||
"enforced": 4,
|
||||
"holdout": 1,
|
||||
"measured": 4,
|
||||
"savings_usd": 0.057101,
|
||||
},
|
||||
{
|
||||
"requested": "gpt-5.6-sol",
|
||||
"served": "gpt-5.6-terra",
|
||||
"direction": "downgrade",
|
||||
"count": 2,
|
||||
"enforced": 2,
|
||||
"holdout": 0,
|
||||
"measured": 0,
|
||||
"savings_usd": 0.0,
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def test_cli_shows_the_proxys_routing_savings_beside_compression(monkeypatch, tmp_path):
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli import savings as cli
|
||||
|
||||
_events_env(monkeypatch, tmp_path)
|
||||
L.record_savings_event(tokens_before=1000, tokens_after=300, model=None, client="claude-code")
|
||||
monkeypatch.setattr(cli, "_proxy_routing", lambda: _ROUTING)
|
||||
|
||||
out = CliRunner().invoke(cli.savings, []).output
|
||||
# The headline is the sum of the pairs; a holdout stayed on Sonnet and is not moved.
|
||||
assert "Model routing (lifetime): 6 of 11 requests moved · $0.0571 saved" in out
|
||||
assert "claude-sonnet-5-5 → claude-haiku-5-5" in out
|
||||
assert "not priced (no usage reported)" in out # Sol -> Terra: a saving we cannot claim
|
||||
assert "claude-opus-5-5 → claude-opus-5-5" not in out
|
||||
|
||||
data = json.loads(CliRunner().invoke(cli.savings, ["--json"]).output)
|
||||
assert data["routing"]["pairs"][1]["savings_usd"] == 0.057101
|
||||
|
||||
|
||||
def test_cli_shows_routing_savings_before_any_compression(monkeypatch, tmp_path):
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli import savings as cli
|
||||
|
||||
_events_env(monkeypatch, tmp_path)
|
||||
monkeypatch.setattr(cli, "_proxy_routing", lambda: _ROUTING)
|
||||
out = CliRunner().invoke(cli.savings, []).output
|
||||
assert "No savings recorded yet." not in out
|
||||
assert "$0.0571 saved" in out
|
||||
|
||||
|
||||
def test_proxy_routing_is_none_without_a_router(monkeypatch):
|
||||
from headroom.cli import savings as cli
|
||||
|
||||
monkeypatch.setattr(
|
||||
"headroom.install.health.probe_json", lambda url, timeout=2.0: {"routing": None}
|
||||
)
|
||||
assert cli._proxy_routing() is None
|
||||
monkeypatch.setattr("headroom.install.health.probe_json", lambda url, timeout=2.0: None)
|
||||
assert cli._proxy_routing() is None
|
||||
|
||||
|
||||
def test_cli_names_the_routing_window_the_provider_reports(monkeypatch, tmp_path):
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli import savings as cli
|
||||
|
||||
_events_env(monkeypatch, tmp_path)
|
||||
for window, label in (("session", "this proxy session"), (None, "all decisions on record")):
|
||||
monkeypatch.setattr(cli, "_proxy_routing", lambda w=window: {**_ROUTING, "window": w})
|
||||
out = CliRunner().invoke(cli.savings, []).output
|
||||
assert f"Model routing ({label}): 6 of 11 requests moved" in out
|
||||
|
||||
|
||||
def test_cli_without_a_router_never_mentions_routing(monkeypatch, tmp_path):
|
||||
from click.testing import CliRunner
|
||||
|
||||
from headroom.cli import savings as cli
|
||||
|
||||
_events_env(monkeypatch, tmp_path)
|
||||
L.record_savings_event(tokens_before=1000, tokens_after=300, model=None, client="claude-code")
|
||||
monkeypatch.setattr(cli, "_proxy_routing", lambda: None)
|
||||
assert "routing" not in CliRunner().invoke(cli.savings, []).output.lower()
|
||||
assert "routing" not in json.loads(CliRunner().invoke(cli.savings, ["--json"]).output)
|
||||
|
||||
Reference in New Issue
Block a user