Files
headroom/tests/test_dashboard_cache_lifetime_playwright.py
Rod BoevandJD Davis 6af7efe0d1 feat(dashboard): give savings metrics one canonical home (#3320)
## Description

Repeated dashboard totals made the same metric occupy several visual
homes. This change gives session savings, observed TTL token totals, and
lifetime history totals one canonical home while retaining useful
decompositions, request counts, performance ranges, checkpoints, and
rollups.

Refs #960

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed repeated session and trend totals while retaining the
before/removed/after breakdown and the proxy share ratio beside Proxy
Removed.
- Made Bucket Mix the sole home for observed 1h and 5m token totals; the
adjacent cards show request counts.
- Decoupled TTL-section availability from positive bucket token totals,
so request-count cards render when a partial payload reports positive
request counts even when token totals are zero; zero-request payloads
remain hidden while Bucket Mix continues to read the canonical
token-total fields.
- Removed the duplicate latest total from Historical Summary; lifetime
tiles, checkpoints, and rollups remain.
- Added rendered assertions for visible values, retired duplicate
content, preserved performance ranges and empty states, and responsive
screenshots of the changed Session, TTL, and Historical Summary surfaces
in both themes at 1280, 768, and 400 pixels.

## Testing

- [x] Focused dashboard browser and metric regression tests ran on
Python 3.12.
- [x] Responsive coverage exercised the changed dashboard surfaces at
1280, 768, and 400 pixels in both themes.
- [x] Ruff and formatting checks ran on the refreshed branch.

```text
uv run --python 3.12 --with pytest --with pytest-asyncio --with playwright python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_ttl_playwright.py tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_token_savings.py -q
24 passed, 3 skipped in 38.96s
uv run ruff check .
All checks passed!
uv run ruff format . --check
1597 files already formatted
```

risk accepted: Render lint still flags two 400px `container-escape`
warnings inside the horizontally scrollable Recent Requests table. The
warnings mark that table's scroll boundary; the table is outside this
change, and nothing in it changed. CI reruns on the refreshed PR head.

## Real Behavior Proof

- Environment: Windows, Python 3.12.13, headless Chromium,
fixture-backed dashboard data.
- Exact command / steps: rendered Session, TTL, and History with
populated dashboard fixtures in light and dark themes; checked the
single total-wire headline, canonical TTL totals, request counts,
performance ranges, and history checkpoints at 1280, 768, and 400
pixels. The template regression also checks that the “Of new input” text
retains its positive-token visibility gate.
- Observed result: 24 tests passed and 3 tests were skipped. The
headline regression confirms the duplicate “Of total wire” caption is
absent; the template-level check confirms the “Of new input” text and
positive-token gate remain. The responsive case retains the `143.0k`
total, `91.0k` Proxy Removed, `23.5% of before-compression tokens`,
`235.0k` and `185.0k` Bucket Mix totals, `24` and `18` request counts,
`5 - 43ms` Overhead Range, and `0.42 - 2.90s` TTFB Range; `Latest total`
remains absent and checkpoint details remain visible.
- Screenshots: the gallery below compares the same fixture before and
after in both themes. The responsive test exercises all three widths.
- Not tested: live provider traffic

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: normal dashboard deployment
- Stable/default behavior changed: canonical metric placement and
duplicate-value presentation change; API and persistence contracts
remain unchanged
- Kill switch / disable path: revert the dashboard template change or
deploy the previous dashboard version
- Unsafe override required: no
- Qualification impact: no migration, configuration change, or client
qualification is required
- Rollback path: revert the template change; no data repair or migration
is required.

## Screenshots

The before/after images use the same fixture at 1200 × 600. The
responsive test also covers Session, TTL, and History at 1280, 768, and
400 pixels in both themes.

| Surface | Theme | Before | After |
| --- | --- | --- | --- |
| Session | Light | <img alt="Session before, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-light.png"
width="480"> | <img alt="Session after, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-light.png"
width="480"> |
| Session | Dark | <img alt="Session before, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-dark.png"
width="480"> | <img alt="Session after, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-dark.png"
width="480"> |
| TTL | Light | <img alt="TTL before, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-light.png"
width="480"> | <img alt="TTL after, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-light.png"
width="480"> |
| TTL | Dark | <img alt="TTL before, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-dark.png"
width="480"> | <img alt="TTL after, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-dark.png"
width="480"> |
| History | Light | <img alt="History before, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-light.png"
width="480"> | <img alt="History after, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-light.png"
width="480"> |
| History | Dark | <img alt="History before, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-dark.png"
width="480"> | <img alt="History after, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-dark.png"
width="480"> |

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Headroom generates changelog entries from conventional commits. Metric
sources and separate cache-attribution labels remain unchanged.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3320"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 15:55:01 -05:00

187 lines
6.8 KiB
Python

"""Playwright validation for persisted lifetime Prefix Cache metrics.
The Session dashboard now reports only current-process cache activity,
while durable cache totals live behind the Lifetime view. These tests pin
that split after a restart with zero session traffic.
"""
from __future__ import annotations
import json
from urllib.parse import urlsplit
import pytest
from headroom.dashboard import get_dashboard_html
from headroom.proxy.savings_tracker import _empty_display_session
from tests.test_dashboard_cache_ttl_playwright import (
_fulfill_static_asset,
_sample_history,
)
playwright = pytest.importorskip("playwright.sync_api")
Page = playwright.Page
expect = playwright.expect
sync_playwright = playwright.sync_playwright
def _session_stats_no_cache() -> dict:
"""Post-restart session payload: zero current-process cache traffic."""
return {
"cost": {},
"requests": {},
"tokens": {},
"overhead": {},
"ttfb": {},
"latency": {},
"waste_signals": {},
"savings_history": [],
"persistent_savings": {"display_session": {}, "lifetime": {}},
"pipeline_timing": {},
"compression_cache": {},
"prefix_cache": {
"by_provider": {},
"totals": {
"cache_read_tokens": 0,
"cache_write_tokens": 0,
"requests": 0,
"hit_requests": 0,
"observed_ttl_buckets": {},
"observed_ttl_mix": {"active_buckets": []},
},
},
}
def _lifetime_cache_payload(
*,
cache_read_tokens: int = 629_537_547,
cache_savings_usd: float = 7.2,
) -> dict:
return {
"requests": {"total": 6088, "failed": 0, "rate_limited": 0, "cached": 0},
"tokens": {"saved": 42_181},
"cost": {
"input_usd": 12.5,
"compression_savings_usd": 0.5,
"cache_savings_usd": cache_savings_usd,
},
"prefix_cache": {
"cache_read_tokens": cache_read_tokens,
"cache_write_tokens": 0,
"hit_requests": 0,
"requests": 0,
"bust_count": 0,
"bust_tokens": 0,
},
"projects": {},
"by_model": {},
"by_stack": {},
"waste_signals": {},
"persistence": {"healthy": True, "enabled": True, "pending_records": 0},
}
def _install_dashboard_routes(page: Page, stats: dict, lifetime: dict) -> None:
history = _sample_history()
health = {"status": "healthy", "version": "0.3.0"}
dashboard_html = get_dashboard_html()
def handler(route) -> None: # type: ignore[no-untyped-def]
path = urlsplit(route.request.url).path
if path in ("/dashboard", "/"):
route.fulfill(status=200, content_type="text/html", body=dashboard_html)
return
if _fulfill_static_asset(route, path):
return
if "/stats-history" in path:
route.fulfill(status=200, content_type="application/json", body=json.dumps(history))
return
if path.endswith("/stats-lifetime"):
route.fulfill(status=200, content_type="application/json", body=json.dumps(lifetime))
return
if path.endswith("/stats"):
route.fulfill(status=200, content_type="application/json", body=json.dumps(stats))
return
if path.endswith("/health"):
route.fulfill(status=200, content_type="application/json", body=json.dumps(health))
return
route.continue_()
page.route("**/*", handler)
def _open_dashboard(page: Page, stats: dict, lifetime: dict) -> None:
_install_dashboard_routes(page, stats, lifetime)
page.goto("http://headroom.local/dashboard")
page.wait_for_load_state("networkidle")
def test_card_renders_lifetime_cache_reads_after_zero_traffic_restart() -> None:
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1440, "height": 1600})
_open_dashboard(page, _session_stats_no_cache(), _lifetime_cache_payload())
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_have_count(0)
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_role("heading", name="Prefix Cache", exact=True)).to_be_visible()
expect(page.get_by_text("629.5M / 0", exact=True)).to_be_visible()
expect(page.get_by_text("$7.20", exact=True)).to_be_visible()
browser.close()
def test_card_hidden_when_no_session_and_no_lifetime_data() -> None:
stats = _session_stats_no_cache()
lifetime = _lifetime_cache_payload(cache_read_tokens=0, cache_savings_usd=0.0)
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1440, "height": 1600})
_open_dashboard(page, stats, lifetime)
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_have_count(0)
browser.close()
def test_expired_display_session_preserves_session_lifetime_fallback() -> None:
stats = _session_stats_no_cache()
stats["persistent_savings"]["display_session"] = _empty_display_session()
stats["persistent_savings"]["lifetime"] = {
"cache_read_tokens": 629_537_547,
"cache_savings_usd": 7.2,
}
lifetime = _lifetime_cache_payload()
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1600})
_open_dashboard(page, stats, lifetime)
expect(page.get_by_test_id("rolling-display-session-economics")).to_have_count(0)
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_be_visible()
expect(page.get_by_text("Cache Reads (lifetime)", exact=True)).to_be_visible()
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_text("$7.20", exact=True)).to_be_visible()
browser.close()
def test_history_keeps_one_lifetime_total_and_retains_checkpoints() -> None:
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1280, "height": 1400})
_open_dashboard(page, _session_stats_no_cache(), _lifetime_cache_payload())
page.get_by_role("button", name="Historical", exact=True).click()
expect(page.get_by_text("Lifetime Tokens Saved", exact=True)).to_be_visible()
expect(page.get_by_test_id("history-total-value")).to_have_text("143.0k")
expect(page.get_by_test_id("history-total-value")).to_have_count(1)
expect(page.get_by_text("Latest total", exact=True)).to_have_count(0)
expect(page.get_by_text("Recent Historical Checkpoints", exact=True)).to_be_visible()
expect(page.get_by_text("Cumulative proxy compression savings", exact=True)).to_have_count(
2
)
browser.close()