mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:06:14 +08:00
## Description Repeated dashboard totals made the same metric occupy several visual homes. This change gives session savings, observed TTL token totals, and lifetime history totals one canonical home while retaining useful decompositions, request counts, performance ranges, checkpoints, and rollups. Refs #960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed repeated session and trend totals while retaining the before/removed/after breakdown and the proxy share ratio beside Proxy Removed. - Made Bucket Mix the sole home for observed 1h and 5m token totals; the adjacent cards show request counts. - Decoupled TTL-section availability from positive bucket token totals, so request-count cards render when a partial payload reports positive request counts even when token totals are zero; zero-request payloads remain hidden while Bucket Mix continues to read the canonical token-total fields. - Removed the duplicate latest total from Historical Summary; lifetime tiles, checkpoints, and rollups remain. - Added rendered assertions for visible values, retired duplicate content, preserved performance ranges and empty states, and responsive screenshots of the changed Session, TTL, and Historical Summary surfaces in both themes at 1280, 768, and 400 pixels. ## Testing - [x] Focused dashboard browser and metric regression tests ran on Python 3.12. - [x] Responsive coverage exercised the changed dashboard surfaces at 1280, 768, and 400 pixels in both themes. - [x] Ruff and formatting checks ran on the refreshed branch. ```text uv run --python 3.12 --with pytest --with pytest-asyncio --with playwright python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_ttl_playwright.py tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_token_savings.py -q 24 passed, 3 skipped in 38.96s uv run ruff check . All checks passed! uv run ruff format . --check 1597 files already formatted ``` risk accepted: Render lint still flags two 400px `container-escape` warnings inside the horizontally scrollable Recent Requests table. The warnings mark that table's scroll boundary; the table is outside this change, and nothing in it changed. CI reruns on the refreshed PR head. ## Real Behavior Proof - Environment: Windows, Python 3.12.13, headless Chromium, fixture-backed dashboard data. - Exact command / steps: rendered Session, TTL, and History with populated dashboard fixtures in light and dark themes; checked the single total-wire headline, canonical TTL totals, request counts, performance ranges, and history checkpoints at 1280, 768, and 400 pixels. The template regression also checks that the “Of new input” text retains its positive-token visibility gate. - Observed result: 24 tests passed and 3 tests were skipped. The headline regression confirms the duplicate “Of total wire” caption is absent; the template-level check confirms the “Of new input” text and positive-token gate remain. The responsive case retains the `143.0k` total, `91.0k` Proxy Removed, `23.5% of before-compression tokens`, `235.0k` and `185.0k` Bucket Mix totals, `24` and `18` request counts, `5 - 43ms` Overhead Range, and `0.42 - 2.90s` TTFB Range; `Latest total` remains absent and checkpoint details remain visible. - Screenshots: the gallery below compares the same fixture before and after in both themes. The responsive test exercises all three widths. - Not tested: live provider traffic ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: normal dashboard deployment - Stable/default behavior changed: canonical metric placement and duplicate-value presentation change; API and persistence contracts remain unchanged - Kill switch / disable path: revert the dashboard template change or deploy the previous dashboard version - Unsafe override required: no - Qualification impact: no migration, configuration change, or client qualification is required - Rollback path: revert the template change; no data repair or migration is required. ## Screenshots The before/after images use the same fixture at 1200 × 600. The responsive test also covers Session, TTL, and History at 1280, 768, and 400 pixels in both themes. | Surface | Theme | Before | After | | --- | --- | --- | --- | | Session | Light | <img alt="Session before, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-light.png" width="480"> | <img alt="Session after, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-light.png" width="480"> | | Session | Dark | <img alt="Session before, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-dark.png" width="480"> | <img alt="Session after, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-dark.png" width="480"> | | TTL | Light | <img alt="TTL before, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-light.png" width="480"> | <img alt="TTL after, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-light.png" width="480"> | | TTL | Dark | <img alt="TTL before, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-dark.png" width="480"> | <img alt="TTL after, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-dark.png" width="480"> | | History | Light | <img alt="History before, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-light.png" width="480"> | <img alt="History after, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-light.png" width="480"> | | History | Dark | <img alt="History before, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-dark.png" width="480"> | <img alt="History after, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-dark.png" width="480"> | ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Headroom generates changelog entries from conventional commits. Metric sources and separate cache-attribution labels remain unchanged. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3320" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
187 lines
6.8 KiB
Python
187 lines
6.8 KiB
Python
"""Playwright validation for persisted lifetime Prefix Cache metrics.
|
|
|
|
The Session dashboard now reports only current-process cache activity,
|
|
while durable cache totals live behind the Lifetime view. These tests pin
|
|
that split after a restart with zero session traffic.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from urllib.parse import urlsplit
|
|
|
|
import pytest
|
|
|
|
from headroom.dashboard import get_dashboard_html
|
|
from headroom.proxy.savings_tracker import _empty_display_session
|
|
from tests.test_dashboard_cache_ttl_playwright import (
|
|
_fulfill_static_asset,
|
|
_sample_history,
|
|
)
|
|
|
|
playwright = pytest.importorskip("playwright.sync_api")
|
|
Page = playwright.Page
|
|
expect = playwright.expect
|
|
sync_playwright = playwright.sync_playwright
|
|
|
|
|
|
def _session_stats_no_cache() -> dict:
|
|
"""Post-restart session payload: zero current-process cache traffic."""
|
|
return {
|
|
"cost": {},
|
|
"requests": {},
|
|
"tokens": {},
|
|
"overhead": {},
|
|
"ttfb": {},
|
|
"latency": {},
|
|
"waste_signals": {},
|
|
"savings_history": [],
|
|
"persistent_savings": {"display_session": {}, "lifetime": {}},
|
|
"pipeline_timing": {},
|
|
"compression_cache": {},
|
|
"prefix_cache": {
|
|
"by_provider": {},
|
|
"totals": {
|
|
"cache_read_tokens": 0,
|
|
"cache_write_tokens": 0,
|
|
"requests": 0,
|
|
"hit_requests": 0,
|
|
"observed_ttl_buckets": {},
|
|
"observed_ttl_mix": {"active_buckets": []},
|
|
},
|
|
},
|
|
}
|
|
|
|
|
|
def _lifetime_cache_payload(
|
|
*,
|
|
cache_read_tokens: int = 629_537_547,
|
|
cache_savings_usd: float = 7.2,
|
|
) -> dict:
|
|
return {
|
|
"requests": {"total": 6088, "failed": 0, "rate_limited": 0, "cached": 0},
|
|
"tokens": {"saved": 42_181},
|
|
"cost": {
|
|
"input_usd": 12.5,
|
|
"compression_savings_usd": 0.5,
|
|
"cache_savings_usd": cache_savings_usd,
|
|
},
|
|
"prefix_cache": {
|
|
"cache_read_tokens": cache_read_tokens,
|
|
"cache_write_tokens": 0,
|
|
"hit_requests": 0,
|
|
"requests": 0,
|
|
"bust_count": 0,
|
|
"bust_tokens": 0,
|
|
},
|
|
"projects": {},
|
|
"by_model": {},
|
|
"by_stack": {},
|
|
"waste_signals": {},
|
|
"persistence": {"healthy": True, "enabled": True, "pending_records": 0},
|
|
}
|
|
|
|
|
|
def _install_dashboard_routes(page: Page, stats: dict, lifetime: dict) -> None:
|
|
history = _sample_history()
|
|
health = {"status": "healthy", "version": "0.3.0"}
|
|
dashboard_html = get_dashboard_html()
|
|
|
|
def handler(route) -> None: # type: ignore[no-untyped-def]
|
|
path = urlsplit(route.request.url).path
|
|
if path in ("/dashboard", "/"):
|
|
route.fulfill(status=200, content_type="text/html", body=dashboard_html)
|
|
return
|
|
if _fulfill_static_asset(route, path):
|
|
return
|
|
if "/stats-history" in path:
|
|
route.fulfill(status=200, content_type="application/json", body=json.dumps(history))
|
|
return
|
|
if path.endswith("/stats-lifetime"):
|
|
route.fulfill(status=200, content_type="application/json", body=json.dumps(lifetime))
|
|
return
|
|
if path.endswith("/stats"):
|
|
route.fulfill(status=200, content_type="application/json", body=json.dumps(stats))
|
|
return
|
|
if path.endswith("/health"):
|
|
route.fulfill(status=200, content_type="application/json", body=json.dumps(health))
|
|
return
|
|
route.continue_()
|
|
|
|
page.route("**/*", handler)
|
|
|
|
|
|
def _open_dashboard(page: Page, stats: dict, lifetime: dict) -> None:
|
|
_install_dashboard_routes(page, stats, lifetime)
|
|
page.goto("http://headroom.local/dashboard")
|
|
page.wait_for_load_state("networkidle")
|
|
|
|
|
|
def test_card_renders_lifetime_cache_reads_after_zero_traffic_restart() -> None:
|
|
with sync_playwright() as p:
|
|
browser = p.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1440, "height": 1600})
|
|
_open_dashboard(page, _session_stats_no_cache(), _lifetime_cache_payload())
|
|
|
|
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_have_count(0)
|
|
page.get_by_role("button", name="Lifetime", exact=True).click()
|
|
expect(page.get_by_role("heading", name="Prefix Cache", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("629.5M / 0", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("$7.20", exact=True)).to_be_visible()
|
|
|
|
browser.close()
|
|
|
|
|
|
def test_card_hidden_when_no_session_and_no_lifetime_data() -> None:
|
|
stats = _session_stats_no_cache()
|
|
lifetime = _lifetime_cache_payload(cache_read_tokens=0, cache_savings_usd=0.0)
|
|
|
|
with sync_playwright() as p:
|
|
browser = p.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1440, "height": 1600})
|
|
_open_dashboard(page, stats, lifetime)
|
|
|
|
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_have_count(0)
|
|
|
|
browser.close()
|
|
|
|
|
|
def test_expired_display_session_preserves_session_lifetime_fallback() -> None:
|
|
stats = _session_stats_no_cache()
|
|
stats["persistent_savings"]["display_session"] = _empty_display_session()
|
|
stats["persistent_savings"]["lifetime"] = {
|
|
"cache_read_tokens": 629_537_547,
|
|
"cache_savings_usd": 7.2,
|
|
}
|
|
lifetime = _lifetime_cache_payload()
|
|
|
|
with sync_playwright() as p:
|
|
browser = p.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1280, "height": 1600})
|
|
_open_dashboard(page, stats, lifetime)
|
|
expect(page.get_by_test_id("rolling-display-session-economics")).to_have_count(0)
|
|
expect(page.get_by_text("Prefix Cache Impact", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("Cache Reads (lifetime)", exact=True)).to_be_visible()
|
|
page.get_by_role("button", name="Lifetime", exact=True).click()
|
|
expect(page.get_by_text("$7.20", exact=True)).to_be_visible()
|
|
browser.close()
|
|
|
|
|
|
def test_history_keeps_one_lifetime_total_and_retains_checkpoints() -> None:
|
|
with sync_playwright() as p:
|
|
browser = p.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1280, "height": 1400})
|
|
_open_dashboard(page, _session_stats_no_cache(), _lifetime_cache_payload())
|
|
|
|
page.get_by_role("button", name="Historical", exact=True).click()
|
|
expect(page.get_by_text("Lifetime Tokens Saved", exact=True)).to_be_visible()
|
|
expect(page.get_by_test_id("history-total-value")).to_have_text("143.0k")
|
|
expect(page.get_by_test_id("history-total-value")).to_have_count(1)
|
|
expect(page.get_by_text("Latest total", exact=True)).to_have_count(0)
|
|
expect(page.get_by_text("Recent Historical Checkpoints", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("Cumulative proxy compression savings", exact=True)).to_have_count(
|
|
2
|
|
)
|
|
|
|
browser.close()
|