fix(dashboard): separate rolling cache economics by owner (#3319)

## Description

The Session dashboard can show `$0.00` for current-process provider
cache savings while the persisted rolling display session contains real
cache and compression savings. The dashboard now presents those rolling
values as separate owner-labeled amounts and keeps the current-process
cache card explicitly scoped to the current process.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that could break existing behavior)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a standalone rolling display-session economics block with
separate provider cache discount and Headroom compression savings
values.
- Kept both rolling values sourced from normalized display-session data
and gated the block by its owner-provided activity timestamp.
- Scoped the existing Prefix Cache Impact amount to the current process
without changing its diagnostics, availability, or lifetime fallback.
- Made the rolling scope text and values readable in light and dark
themes, stacked the Prefix Cache header safely at narrow widths, and
verified full-page overflow at 768 and 400 pixels.
- Built the OpenAI-compatible reproduction through `PrometheusMetrics`,
`CostTracker`, `build_prefix_cache_stats()`, and `SavingsTracker` using
40 coherent 100,000-token records, 3.96M cache-read tokens, 40,000
uncached tokens, `openai`, `deepseek-v4-flash`, zero current-process net
savings, and a 99.0% token hit rate.

## Testing

- [x] Focused dashboard browser tests pass
- [x] Ruff check and format check pass
- [ ] Type checking pass
- [x] New regression tests cover the changed behavior
- [x] Manual screenshot inspection performed

### Test Output

```text
uv run --with playwright python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_ttl_playwright.py tests/test_dashboard_cache_lifetime_playwright.py -q
20 passed, 1 skipped, 1 warning in 22.06s

uv run ruff check tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_lifetime_playwright.py
All checks passed!

uv run ruff format tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_lifetime_playwright.py --check
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows, Python 3.12, headless Chromium, no external
provider request.
- Exact command / steps: Run the focused dashboard browser tests and the
responsive layout test; the route-fulfilled dashboard renders
production-shaped current-process and rolling producer output at 1280,
768, and 400 pixels in light and dark themes.
- Observed result: 20 focused tests passed, 1 was skipped, and one
existing Starlette warning was reported. The reproduction contains 40
coherent 100,000-token records, 3.96M cache-read tokens, 40,000 uncached
tokens, provider `openai`, model `deepseek-v4-flash`, zero
current-process net savings, 99.0% token hit rate, zero cache writes,
separate rolling values, readable scope copy, a non-compressing Prefix
Cache header, and no full-page horizontal overflow. The actual rendered
rolling values are not summed.
- Not tested: hosted `test-dashboard-ui`; live paid-provider request

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this is a dashboard presentation
change over existing statistics.
- Minimum rollout channel: Normal dashboard deployment.
- Stable/default behavior changed: The Session view adds a separate
rolling comparison and clarifies the current-process cache label;
statistics production and persistence are unchanged.
- Kill switch / disable path: Revert the dashboard template change or
deploy the previous version.
- Unsafe override required: No.
- Qualification impact: No migration, configuration change, or client
qualification is required.
- Rollback path: Restore the previous dashboard template; no data repair
is required.

## Screenshots

The light-theme 1280px capture shows the owner-labeled rolling
display-session economics block beside the current-process card:

![Rolling display-session economics at 1280px, light
theme](https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-2/rolling-light-1280.png)

Six light and dark captures cover 1280, 768, and 400 pixel viewports.
The light 1280px image is linked above; the other captures use the same
fixture and responsive assertions.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code where the behavior needs explanation
- [x] I have made corresponding changes to the code and tests
- [x] My changes generate no new warnings
- [x] I have added tests that prove the fix works
- [x] Focused tests pass locally with my changes
- [x] `CHANGELOG.md` is unchanged because this repository generates
release notes from the conventional PR title

## Additional Notes

Provider cache discount is provider-native economics. Headroom
compression savings are Headroom-attributable. The UI keeps them as
separate peer values and does not present an aggregate.

Refs #960
This commit is contained in:
Rod Boev
2026-09-24 23:56:12 -05:00
committed by GitHub
parent c4df2dde62
commit a29162baa6
3 changed files with 358 additions and 11 deletions
+42 -5
View File
@@ -111,6 +111,13 @@
/* Light mode: fix hardcoded dark card backgrounds */
.bg-card-alt { background: var(--card-alt-bg); }
.hover-bg-surface:hover { background: var(--color-surface); }
.rolling-provider-label, .rolling-provider-value { color: #0e7490; }
.rolling-compression-label, .rolling-compression-value { color: #047857; }
.rolling-scope-copy { color: #334155; }
.prefix-cache-current-label { color: #047857; }
html.dark .rolling-provider-label, html.dark .rolling-provider-value { color: #67e8f9; }
html.dark .rolling-compression-label, html.dark .rolling-compression-value { color: #6ee7b7; }
html.dark .rolling-scope-copy, html.dark .prefix-cache-current-label { color: #cbd5e1; }
</style>
</head>
<!-- No x-init: Alpine already calls the x-data object's own init(). Naming it
@@ -118,12 +125,12 @@
the R shortcut twice, doubling every request the dashboard makes. -->
<body class="text-gray-200 min-h-screen" x-data="dashboard()">
<!-- Header -->
<header class="border-b border-border px-6 py-4 flex flex-col gap-3 md:flex-row md:items-center md:justify-between">
<header class="border-b border-border px-6 py-4 flex flex-col gap-3 md:flex-row md:flex-wrap md:items-center md:justify-between">
<div class="flex items-center gap-4">
<h1 class="text-xl font-semibold tracking-tight">HEADROOM</h1>
<span class="text-xs text-gray-500 font-mono" x-text="formatVersion(version)"></span>
</div>
<div class="flex flex-col gap-3 md:flex-row md:items-center md:gap-6">
<div class="flex min-w-0 max-w-full flex-col gap-3 md:flex-row md:flex-wrap md:items-center md:gap-6">
<div class="inline-flex rounded-lg border border-border bg-surface p-1">
<button type="button" :aria-pressed="viewMode === 'session'" class="px-3 py-1.5 text-sm rounded-md transition-colors"
:class="viewMode === 'session' ? 'bg-accent text-black' : 'text-gray-400 hover:text-gray-200'"
@@ -524,14 +531,36 @@
</div>
</div>
<!-- Rolling display-session economics: provider and Headroom owners stay separate -->
<template x-if="rollingSessionActive">
<div data-testid="rolling-display-session-economics" class="bg-surface rounded-lg p-4 border border-border mb-6">
<div class="text-sm font-medium text-gray-300">Rolling display session economics</div>
<div class="rolling-scope-copy text-xs mt-1 mb-4">Values cover the active persisted display session.</div>
<div class="grid grid-cols-1 sm:grid-cols-2 gap-4">
<div data-testid="rolling-provider-cache" class="rounded-lg border border-cyan-500/20 bg-cyan-500/5 p-4">
<div class="rolling-provider-label text-xs uppercase tracking-wide">Provider cache discount</div>
<div class="rolling-provider-value text-2xl font-light tabular-nums mt-2"
x-text="'$' + formatCurrency(rollingSession.cache_savings_usd || 0)"></div>
<div class="rolling-scope-copy text-xs mt-1">provider-observed, rolling display session</div>
</div>
<div data-testid="rolling-compression-savings" class="rounded-lg border border-emerald-500/20 bg-emerald-500/5 p-4">
<div class="rolling-compression-label text-xs uppercase tracking-wide">Headroom compression savings</div>
<div class="rolling-compression-value text-2xl font-light tabular-nums mt-2"
x-text="'$' + formatCurrency(rollingSession.compression_savings_usd || 0)"></div>
<div class="rolling-scope-copy text-xs mt-1">Headroom-attributable, rolling display session</div>
</div>
</div>
</div>
</template>
<!-- Prefix Cache Impact: current process plus durable cache reads after restart -->
<template x-if="cacheCardAvailable">
<div class="bg-surface rounded-lg p-4 border border-border mb-6">
<div class="flex justify-between items-center mb-3">
<div class="flex flex-col items-start gap-2 mb-3 sm:flex-row sm:items-center sm:justify-between">
<div class="text-sm font-medium text-gray-300">Prefix Cache Impact</div>
<div class="text-xs text-emerald-400 font-mono"
<div class="prefix-cache-current-label text-xs font-mono"
x-show="cacheSessionActive"
x-text="'Net savings: $' + formatCurrency(stats.prefix_cache?.totals?.net_savings_usd || 0)"></div>
x-text="'Provider cache discount, current process: $' + formatCurrency(stats.prefix_cache?.totals?.net_savings_usd || 0)"></div>
<div class="text-xs text-gray-500 font-mono"
x-show="!cacheSessionActive">no activity since restart</div>
</div>
@@ -2709,6 +2738,14 @@
return (this.stats.prefix_cache?.totals?.requests || 0) > 0;
},
get rollingSession() {
return this.stats.persistent_savings?.display_session || {};
},
get rollingSessionActive() {
return Boolean(this.rollingSession.last_activity_at);
},
get lifetimeCacheReadTokens() {
return this.stats.persistent_savings?.lifetime?.cache_read_tokens || 0;
},
@@ -7,17 +7,16 @@ that split after a restart with zero session traffic.
from __future__ import annotations
import copy
import json
from urllib.parse import urlsplit
import pytest
from headroom.dashboard import get_dashboard_html
from headroom.proxy.savings_tracker import _empty_display_session
from tests.test_dashboard_cache_ttl_playwright import (
_fulfill_static_asset,
_sample_history,
_sample_stats,
)
playwright = pytest.importorskip("playwright.sync_api")
@@ -28,10 +27,30 @@ sync_playwright = playwright.sync_playwright
def _session_stats_no_cache() -> dict:
"""Post-restart session payload: zero current-process cache traffic."""
stats = copy.deepcopy(_sample_stats())
totals = stats.setdefault("prefix_cache", {}).setdefault("totals", {})
totals.update({"requests": 0, "cache_read_tokens": 0, "cache_write_tokens": 0})
return stats
return {
"cost": {},
"requests": {},
"tokens": {},
"overhead": {},
"ttfb": {},
"latency": {},
"waste_signals": {},
"savings_history": [],
"persistent_savings": {"display_session": {}, "lifetime": {}},
"pipeline_timing": {},
"compression_cache": {},
"prefix_cache": {
"by_provider": {},
"totals": {
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"requests": 0,
"hit_requests": 0,
"observed_ttl_buckets": {},
"observed_ttl_mix": {"active_buckets": []},
},
},
}
def _lifetime_cache_payload(
@@ -125,3 +144,24 @@ def test_card_hidden_when_no_session_and_no_lifetime_data() -> None:
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_have_count(0)
browser.close()
def test_expired_display_session_preserves_session_lifetime_fallback() -> None:
stats = _session_stats_no_cache()
stats["persistent_savings"]["display_session"] = _empty_display_session()
stats["persistent_savings"]["lifetime"] = {
"cache_read_tokens": 629_537_547,
"cache_savings_usd": 7.2,
}
lifetime = _lifetime_cache_payload()
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1600})
_open_dashboard(page, stats, lifetime)
expect(page.get_by_test_id("rolling-display-session-economics")).to_have_count(0)
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_be_visible()
expect(page.get_by_text("Cache Reads (lifetime)", exact=True)).to_be_visible()
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_text("$7.20", exact=True)).to_be_visible()
browser.close()
@@ -9,15 +9,20 @@ invalidated, plus the prefix-freeze net benefit.
from __future__ import annotations
import asyncio
import copy
import json
import os
from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.parse import urlsplit
import pytest
from headroom.dashboard import get_dashboard_html
from headroom.proxy.cost import CostTracker, build_prefix_cache_stats
from headroom.proxy.prometheus_metrics import PrometheusMetrics
from headroom.proxy.savings_tracker import SavingsTracker, _empty_display_session
from tests.test_dashboard_cache_ttl_playwright import (
_fulfill_static_asset,
_sample_history,
@@ -47,6 +52,97 @@ def _stats_with_compression_vs_cache() -> dict:
return stats
def _neutral_stats() -> dict:
return {
"cost": {},
"requests": {},
"tokens": {},
"overhead": {},
"ttfb": {},
"latency": {},
"waste_signals": {},
"savings_history": [],
"persistent_savings": {"display_session": {}, "lifetime": {}},
"pipeline_timing": {},
"compression_cache": {},
"prefix_cache": {"by_provider": {}, "totals": {}},
}
def _openai_fixture_stats(
*,
cache_read_tokens: int = 99_000,
tokens_saved: int = 25_000,
include_current_process: bool = True,
include_rolling: bool = True,
reload_tracker: bool = False,
) -> dict:
metrics = PrometheusMetrics(stateless=True)
cost_tracker = CostTracker()
if include_current_process:
for _ in range(40):
asyncio.run(
metrics.record_request(
"openai",
"deepseek-v4-flash",
100_000,
0,
0,
0,
cached=True,
cache_read_tokens=99_000,
uncached_input_tokens=1_000,
)
)
for _ in range(40):
cost_tracker.record_tokens(
model="deepseek-v4-flash",
tokens_saved=0,
tokens_sent=100_000,
cache_read_tokens=99_000,
uncached_tokens=1_000,
)
prefix_cache = build_prefix_cache_stats(metrics, cost_tracker)
assert set(prefix_cache["by_provider"]) == ({"openai"} if include_current_process else set())
assert prefix_cache["totals"]["requests"] == (40 if include_current_process else 0)
if include_current_process:
assert prefix_cache["totals"]["cache_read_tokens"] == 3_960_000
assert prefix_cache["totals"]["uncached_input_tokens"] == 40_000
assert prefix_cache["totals"]["hit_rate"] == 99.0
assert prefix_cache["totals"]["cache_write_tokens"] == 0
assert prefix_cache["totals"]["net_savings_usd"] == 0.0
with TemporaryDirectory(prefix="headroom-960-2-") as directory:
tracker_path = str(Path(directory) / "savings.json")
tracker = SavingsTracker(path=tracker_path, stateless=False)
if include_rolling:
for _ in range(40):
tracker.record_request(
model="deepseek-v4-flash",
provider="openai",
input_tokens=100_000,
tokens_saved=tokens_saved,
cache_read_tokens=cache_read_tokens,
uncached_input_tokens=100_000 - cache_read_tokens,
total_input_tokens=100_000,
)
if reload_tracker:
tracker = SavingsTracker(path=tracker_path, stateless=False)
display_session = tracker.stats_preview()["display_session"]
assert display_session["requests"] == (40 if include_rolling else 0)
assert display_session["cache_read_tokens"] == (
40 * cache_read_tokens if include_rolling else 0
)
assert display_session["total_input_tokens"] == (4_000_000 if include_rolling else 0)
if not include_rolling:
assert display_session == _empty_display_session()
stats = _neutral_stats()
stats["prefix_cache"] = prefix_cache
stats["persistent_savings"] = {"display_session": display_session, "lifetime": {}}
return stats
def _install_dashboard_routes(page: Page, stats: dict) -> None:
history = _sample_history()
health = {"status": "healthy", "version": "0.3.0"}
@@ -135,6 +231,26 @@ def test_dashboard_marks_negative_compression_vs_cache_net_in_red() -> None:
browser.close()
def test_dashboard_preserves_negative_prefix_freeze_diagnostic_in_red() -> None:
stats = _stats_with_compression_vs_cache()
stats["prefix_cache"]["prefix_freeze"] = {
"busts_avoided": 1,
"tokens_preserved": 12_000,
"compression_foregone_tokens": 48_000,
"net_benefit_tokens": -36_000,
}
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1600})
_open_dashboard(page, stats)
expect(page.get_by_test_id("freeze-net-value")).to_have_text("-36.0k")
expect(page.get_by_test_id("freeze-net-value")).to_have_class(
"mt-2 text-3xl font-light text-red-400"
)
browser.close()
def test_dashboard_hides_compression_vs_cache_section_without_data() -> None:
stats = copy.deepcopy(_sample_stats())
stats["prefix_cache"]["compression_vs_cache"] = {
@@ -158,3 +274,157 @@ def test_dashboard_hides_compression_vs_cache_section_without_data() -> None:
expect(page.get_by_test_id("cvc-net-headline")).to_have_count(0)
browser.close()
def test_dashboard_separates_openai_zero_current_process_from_rolling_economics() -> None:
stats = _openai_fixture_stats()
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1600})
_open_dashboard(page, stats)
expect(page.get_by_text("Rolling display session economics", exact=True)).to_be_visible()
session = stats["persistent_savings"]["display_session"]
expect(page.get_by_test_id("rolling-provider-cache")).to_contain_text(
f"${session['cache_savings_usd']:.2f}"
)
expect(page.get_by_test_id("rolling-compression-savings")).to_contain_text(
f"${session['compression_savings_usd']:.2f}"
)
expect(
page.get_by_text("Provider cache discount, current process: $0.00", exact=True)
).to_be_visible()
expect(page.get_by_text("openai", exact=True)).to_be_visible()
expect(page.get_by_text("Net savings", exact=True)).to_have_count(0)
rendered_values = page.locator(
".rolling-provider-value, .rolling-compression-value"
).all_text_contents()
numeric_values = [float(value.removeprefix("$")) for value in rendered_values]
rendered_sum = f"${sum(numeric_values):.2f}"
assert rendered_sum not in page.locator("body").inner_text()
browser.close()
def test_dashboard_shows_persisted_rolling_session_without_current_process_cache() -> None:
stats = _openai_fixture_stats(include_current_process=False, reload_tracker=True)
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1600})
_open_dashboard(page, stats)
expect(page.get_by_test_id("rolling-display-session-economics")).to_be_visible()
expect(page.get_by_test_id("rolling-provider-cache")).to_contain_text(
f"${stats['persistent_savings']['display_session']['cache_savings_usd']:.2f}"
)
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_have_count(0)
browser.close()
def test_dashboard_keeps_current_process_card_when_rolling_session_is_empty() -> None:
stats = _openai_fixture_stats(include_rolling=False)
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1600})
_open_dashboard(page, stats)
expect(page.get_by_test_id("rolling-display-session-economics")).to_have_count(0)
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_be_visible()
expect(
page.get_by_text("Provider cache discount, current process: $0.00", exact=True)
).to_be_visible()
browser.close()
@pytest.mark.parametrize(("cache_read_tokens", "tokens_saved"), [(0, 25_000), (99_000, 0)])
def test_dashboard_keeps_zero_rolling_peer_visible(
cache_read_tokens: int, tokens_saved: int
) -> None:
stats = _openai_fixture_stats(cache_read_tokens=cache_read_tokens, tokens_saved=tokens_saved)
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 768, "height": 1600})
_open_dashboard(page, stats)
rolling = page.get_by_test_id("rolling-display-session-economics")
expect(rolling).to_be_visible()
session = stats["persistent_savings"]["display_session"]
expect(rolling.get_by_test_id("rolling-provider-cache")).to_contain_text(
f"${session['cache_savings_usd']:.2f}"
)
expect(rolling.get_by_test_id("rolling-compression-savings")).to_contain_text(
f"${session['compression_savings_usd']:.2f}"
)
browser.close()
def test_dashboard_rolling_economics_layout_at_required_viewports(
monkeypatch: pytest.MonkeyPatch,
) -> None:
artifact_dir = os.environ.get("HEADROOM_PLAYWRIGHT_ARTIFACT_DIR")
if artifact_dir is None:
artifact_dir = str(Path.cwd().parent / ".claude/pr-sweep/proof/960-2-screenshots")
monkeypatch.setenv("HEADROOM_PLAYWRIGHT_ARTIFACT_DIR", artifact_dir)
stats = _openai_fixture_stats()
with sync_playwright() as p:
browser = p.chromium.launch()
for theme in ("light", "dark"):
for width in (1280, 768, 400):
page = browser.new_page(viewport={"width": width, "height": 1600})
page.emulate_media(color_scheme=theme)
_open_dashboard(page, stats)
rolling = page.get_by_test_id("rolling-display-session-economics")
provider = page.get_by_test_id("rolling-provider-cache")
compression = page.get_by_test_id("rolling-compression-savings")
expect(rolling).to_be_visible()
assert awaitable_width(page, rolling) <= width
assert awaitable_width(page, provider) <= width
assert awaitable_width(page, compression) <= width
for locator in (
provider.locator(".rolling-provider-label"),
provider.locator(".rolling-provider-value"),
provider.locator(".rolling-scope-copy"),
compression.locator(".rolling-compression-label"),
compression.locator(".rolling-compression-value"),
compression.locator(".rolling-scope-copy"),
):
assert locator.evaluate(
"""(element) => {
const rgb = getComputedStyle(element).color.match(/\\d+/g).map(Number);
const bg = getComputedStyle(document.documentElement).getPropertyValue('--color-surface').trim().match(/#[0-9a-f]{6}/i)[0];
const channels = bg.slice(1).match(/../g).map(value => parseInt(value, 16));
const luminance = values => values.map(value => value / 255).map(value => value <= 0.03928 ? value / 12.92 : ((value + 0.055) / 1.055) ** 2.4).reduce((sum, value, index) => sum + value * [0.2126, 0.7152, 0.0722][index], 0);
const ratio = (a, b) => (Math.max(a, b) + 0.05) / (Math.min(a, b) + 0.05);
return ratio(luminance(rgb), luminance(channels)) >= 4.5;
}""",
)
if width == 400:
assert provider.bounding_box()["y"] < compression.bounding_box()["y"]
else:
assert provider.bounding_box()["x"] < compression.bounding_box()["x"]
overflow = page.evaluate(
"""() => ({
bodyScrollWidth: document.body.scrollWidth,
bodyClientWidth: document.body.clientWidth,
viewportWidth: document.documentElement.clientWidth,
overflowing: [...document.querySelectorAll('*')]
.filter(element => element.getBoundingClientRect().right > window.innerWidth)
.slice(0, 5)
.map(element => [element.tagName, element.className, element.getBoundingClientRect().right])
})"""
)
assert (
overflow["bodyScrollWidth"] <= overflow["viewportWidth"]
and overflow["bodyScrollWidth"] <= overflow["bodyClientWidth"]
), overflow
if artifact_dir:
output = Path(artifact_dir)
output.mkdir(parents=True, exist_ok=True)
page.screenshot(
path=str(output / f"rolling-{theme}-{width}.png"), full_page=True
)
page.close()
browser.close()
def awaitable_width(page: Page, locator) -> float:
box = locator.bounding_box()
assert box is not None
return box["width"]