mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat(dashboard): give savings metrics one canonical home (#3320)
## Description Repeated dashboard totals made the same metric occupy several visual homes. This change gives session savings, observed TTL token totals, and lifetime history totals one canonical home while retaining useful decompositions, request counts, performance ranges, checkpoints, and rollups. Refs #960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed repeated session and trend totals while retaining the before/removed/after breakdown and the proxy share ratio beside Proxy Removed. - Made Bucket Mix the sole home for observed 1h and 5m token totals; the adjacent cards show request counts. - Decoupled TTL-section availability from positive bucket token totals, so request-count cards render when a partial payload reports positive request counts even when token totals are zero; zero-request payloads remain hidden while Bucket Mix continues to read the canonical token-total fields. - Removed the duplicate latest total from Historical Summary; lifetime tiles, checkpoints, and rollups remain. - Added rendered assertions for visible values, retired duplicate content, preserved performance ranges and empty states, and responsive screenshots of the changed Session, TTL, and Historical Summary surfaces in both themes at 1280, 768, and 400 pixels. ## Testing - [x] Focused dashboard browser and metric regression tests ran on Python 3.12. - [x] Responsive coverage exercised the changed dashboard surfaces at 1280, 768, and 400 pixels in both themes. - [x] Ruff and formatting checks ran on the refreshed branch. ```text uv run --python 3.12 --with pytest --with pytest-asyncio --with playwright python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_ttl_playwright.py tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_token_savings.py -q 24 passed, 3 skipped in 38.96s uv run ruff check . All checks passed! uv run ruff format . --check 1597 files already formatted ``` risk accepted: Render lint still flags two 400px `container-escape` warnings inside the horizontally scrollable Recent Requests table. The warnings mark that table's scroll boundary; the table is outside this change, and nothing in it changed. CI reruns on the refreshed PR head. ## Real Behavior Proof - Environment: Windows, Python 3.12.13, headless Chromium, fixture-backed dashboard data. - Exact command / steps: rendered Session, TTL, and History with populated dashboard fixtures in light and dark themes; checked the single total-wire headline, canonical TTL totals, request counts, performance ranges, and history checkpoints at 1280, 768, and 400 pixels. The template regression also checks that the “Of new input” text retains its positive-token visibility gate. - Observed result: 24 tests passed and 3 tests were skipped. The headline regression confirms the duplicate “Of total wire” caption is absent; the template-level check confirms the “Of new input” text and positive-token gate remain. The responsive case retains the `143.0k` total, `91.0k` Proxy Removed, `23.5% of before-compression tokens`, `235.0k` and `185.0k` Bucket Mix totals, `24` and `18` request counts, `5 - 43ms` Overhead Range, and `0.42 - 2.90s` TTFB Range; `Latest total` remains absent and checkpoint details remain visible. - Screenshots: the gallery below compares the same fixture before and after in both themes. The responsive test exercises all three widths. - Not tested: live provider traffic ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: normal dashboard deployment - Stable/default behavior changed: canonical metric placement and duplicate-value presentation change; API and persistence contracts remain unchanged - Kill switch / disable path: revert the dashboard template change or deploy the previous dashboard version - Unsafe override required: no - Qualification impact: no migration, configuration change, or client qualification is required - Rollback path: revert the template change; no data repair or migration is required. ## Screenshots The before/after images use the same fixture at 1200 × 600. The responsive test also covers Session, TTL, and History at 1280, 768, and 400 pixels in both themes. | Surface | Theme | Before | After | | --- | --- | --- | --- | | Session | Light | <img alt="Session before, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-light.png" width="480"> | <img alt="Session after, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-light.png" width="480"> | | Session | Dark | <img alt="Session before, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-dark.png" width="480"> | <img alt="Session after, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-dark.png" width="480"> | | TTL | Light | <img alt="TTL before, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-light.png" width="480"> | <img alt="TTL after, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-light.png" width="480"> | | TTL | Dark | <img alt="TTL before, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-dark.png" width="480"> | <img alt="TTL after, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-dark.png" width="480"> | | History | Light | <img alt="History before, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-light.png" width="480"> | <img alt="History after, light theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-light.png" width="480"> | | History | Dark | <img alt="History before, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-dark.png" width="480"> | <img alt="History after, dark theme" src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-dark.png" width="480"> | ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Headroom generates changelog entries from conventional commits. Metric sources and separate cache-attribution labels remain unchanged. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3320" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
@@ -210,7 +210,7 @@
|
||||
|
||||
<main class="p-6 max-w-7xl mx-auto">
|
||||
<template x-if="viewMode === 'session'">
|
||||
<div>
|
||||
<div data-testid="session-view">
|
||||
<p class="mb-4 text-xs text-gray-500">Current proxy process · runtime counters reset on restart</p>
|
||||
|
||||
<!-- Runtime Overview -->
|
||||
@@ -226,7 +226,7 @@
|
||||
</div>
|
||||
|
||||
<!-- Session Optimization -->
|
||||
<div class="grid grid-cols-1 gap-4 mb-6 md:grid-cols-2 lg:grid-cols-3">
|
||||
<div data-testid="session-optimization" class="grid grid-cols-1 gap-4 mb-6 md:grid-cols-2 lg:grid-cols-3">
|
||||
<!-- Token Savings (%) -->
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="text-xs text-gray-500 uppercase tracking-wide mb-1">Tokens Saved</div>
|
||||
@@ -242,9 +242,6 @@
|
||||
higher on long sessions and must not take this slot. -->
|
||||
<span class="text-sm text-accent" x-text="headlineSavingsPercent.toFixed(1) + '%'" :title="headlineSavingsTitle"></span>
|
||||
</div>
|
||||
<div class="mt-1 text-xs text-gray-500 leading-relaxed">
|
||||
<span x-text="'Proxy ' + formatNumber(stats.tokens?.proxy_compression_saved || 0) + ' (' + proxyShareOfTotal.toFixed(1) + '%)'"></span>
|
||||
</div>
|
||||
<div class="mt-1 text-xs text-gray-600 leading-relaxed" x-show="(stats.tokens?.new_input_tokens || 0) > 0">
|
||||
<span x-text="'Of new input: ' + (stats.tokens?.new_input_savings_percent || 0).toFixed(2) + '%'" title="Savings as a fraction of the tokens that newly entered context (uncached + cache write), which is what compression can touch. The headline above recounts the session's cached history on every turn, so it reads low on a long session even when compression is working."></span>
|
||||
</div>
|
||||
@@ -455,7 +452,7 @@
|
||||
<!-- Optimization Breakdown -->
|
||||
<div class="grid grid-cols-1 gap-4 mb-6 lg:grid-cols-3">
|
||||
<!-- Token Usage -->
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div data-testid="token-usage" class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="text-sm font-medium mb-4 text-gray-300">Token Usage</div>
|
||||
<div class="space-y-3">
|
||||
<div class="flex justify-between items-center">
|
||||
@@ -466,6 +463,8 @@
|
||||
<span class="text-sm text-gray-400">Proxy Removed</span>
|
||||
<span class="font-mono text-sm text-accent" x-text="formatNumber(stats.tokens?.proxy_compression_saved || 0)"></span>
|
||||
</div>
|
||||
<div class="text-right text-xs text-gray-500" data-testid="proxy-share-of-total"
|
||||
x-text="proxyShareOfTotal.toFixed(1) + '% of before-compression tokens'"></div>
|
||||
<div class="flex justify-between items-center">
|
||||
<span class="text-sm text-gray-400">After Compression (sent)</span>
|
||||
<span class="font-mono text-sm" x-text="formatNumber(stats.tokens?.input || 0)"></span>
|
||||
@@ -509,7 +508,6 @@
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="flex justify-between items-center mb-4">
|
||||
<span class="text-sm font-medium text-gray-300">Savings Over Time</span>
|
||||
<span class="text-xs text-gray-500 font-mono" x-text="formatNumber(stats.tokens?.saved || 0) + ' tokens total'"></span>
|
||||
</div>
|
||||
<template x-if="(stats.savings_history || []).length >= 2">
|
||||
<div class="h-32">
|
||||
@@ -632,7 +630,7 @@
|
||||
</div>
|
||||
</div>
|
||||
<template x-if="hasObservedTtlBuckets">
|
||||
<div class="mt-5 border-t border-border pt-4">
|
||||
<div data-testid="ttl-metrics" class="mt-5 border-t border-border pt-4">
|
||||
<div class="flex items-center justify-between mb-3">
|
||||
<div>
|
||||
<div class="text-xs text-gray-500 uppercase tracking-[0.22em]">Observed TTL Buckets</div>
|
||||
@@ -673,13 +671,13 @@
|
||||
<div class="grid grid-cols-1 sm:grid-cols-2 xl:grid-cols-1 gap-3">
|
||||
<div class="rounded-2xl border border-white/8 bg-black/20 p-4">
|
||||
<div class="text-[11px] uppercase tracking-[0.18em] text-cyan-300/75">1h Cache Writes</div>
|
||||
<div data-testid="ttl-bucket-1h-value" class="mt-2 text-3xl font-light text-cyan-50" x-text="formatNumber(stats.prefix_cache?.totals?.observed_ttl_buckets?.['1h']?.tokens || 0)"></div>
|
||||
<div class="mt-1 text-xs text-gray-500" x-text="formatNumber(stats.prefix_cache?.totals?.observed_ttl_buckets?.['1h']?.requests || 0) + ' requests observed'"></div>
|
||||
<div data-testid="ttl-bucket-1h-requests" class="mt-2 text-3xl font-light text-cyan-50" x-text="formatNumber(stats.prefix_cache?.totals?.observed_ttl_buckets?.['1h']?.requests || 0)"></div>
|
||||
<div class="mt-1 text-xs text-gray-500">requests observed</div>
|
||||
</div>
|
||||
<div class="rounded-2xl border border-white/8 bg-black/20 p-4">
|
||||
<div class="text-[11px] uppercase tracking-[0.18em] text-violet-300/75">5m Cache Writes</div>
|
||||
<div data-testid="ttl-bucket-5m-value" class="mt-2 text-3xl font-light text-violet-50" x-text="formatNumber(stats.prefix_cache?.totals?.observed_ttl_buckets?.['5m']?.tokens || 0)"></div>
|
||||
<div class="mt-1 text-xs text-gray-500" x-text="formatNumber(stats.prefix_cache?.totals?.observed_ttl_buckets?.['5m']?.requests || 0) + ' requests observed'"></div>
|
||||
<div data-testid="ttl-bucket-5m-requests" class="mt-2 text-3xl font-light text-violet-50" x-text="formatNumber(stats.prefix_cache?.totals?.observed_ttl_buckets?.['5m']?.requests || 0)"></div>
|
||||
<div class="mt-1 text-xs text-gray-500">requests observed</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1630,7 +1628,7 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="grid grid-cols-2 md:grid-cols-3 xl:grid-cols-5 gap-4 mb-6">
|
||||
<div data-testid="history-summary" class="grid grid-cols-2 md:grid-cols-3 xl:grid-cols-5 gap-4 mb-6">
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="text-xs text-gray-500 uppercase tracking-wide mb-1">Lifetime Compression Savings</div>
|
||||
<div class="flex items-baseline gap-2">
|
||||
@@ -1645,7 +1643,7 @@
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="text-xs text-gray-500 uppercase tracking-wide mb-1">Lifetime Tokens Saved</div>
|
||||
<div class="flex items-baseline gap-2">
|
||||
<span class="text-3xl font-light tabular-nums text-accent"
|
||||
<span data-testid="history-total-value" class="text-3xl font-light tabular-nums text-accent"
|
||||
x-text="formatNumber(historyStats.lifetime?.tokens_saved || 0)"></span>
|
||||
</div>
|
||||
<div class="mt-2 text-xs text-gray-500"
|
||||
@@ -1915,14 +1913,9 @@
|
||||
</template>
|
||||
</div>
|
||||
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div data-testid="historical-summary-card" class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="text-sm font-medium mb-4 text-gray-300">Historical Summary</div>
|
||||
<div class="space-y-3">
|
||||
<div class="flex justify-between items-center">
|
||||
<span class="text-sm text-gray-400">Latest total</span>
|
||||
<span class="font-mono text-sm"
|
||||
x-text="formatNumber(historyStats.lifetime?.tokens_saved || 0) + ' tokens'"></span>
|
||||
</div>
|
||||
<div class="flex justify-between items-center">
|
||||
<span class="text-sm text-gray-400">Selected series</span>
|
||||
<span class="font-mono text-sm"
|
||||
@@ -2797,7 +2790,7 @@
|
||||
|
||||
get hasObservedTtlBuckets() {
|
||||
const buckets = this.stats.prefix_cache?.totals?.observed_ttl_buckets || {};
|
||||
return ((buckets['5m']?.tokens || 0) + (buckets['1h']?.tokens || 0)) > 0;
|
||||
return ['5m', '1h'].some((ttl) => (buckets[ttl]?.requests || 0) > 0);
|
||||
},
|
||||
|
||||
get observedTtlHeadline() {
|
||||
|
||||
@@ -165,3 +165,22 @@ def test_expired_display_session_preserves_session_lifetime_fallback() -> None:
|
||||
page.get_by_role("button", name="Lifetime", exact=True).click()
|
||||
expect(page.get_by_text("$7.20", exact=True)).to_be_visible()
|
||||
browser.close()
|
||||
|
||||
|
||||
def test_history_keeps_one_lifetime_total_and_retains_checkpoints() -> None:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 1400})
|
||||
_open_dashboard(page, _session_stats_no_cache(), _lifetime_cache_payload())
|
||||
|
||||
page.get_by_role("button", name="Historical", exact=True).click()
|
||||
expect(page.get_by_text("Lifetime Tokens Saved", exact=True)).to_be_visible()
|
||||
expect(page.get_by_test_id("history-total-value")).to_have_text("143.0k")
|
||||
expect(page.get_by_test_id("history-total-value")).to_have_count(1)
|
||||
expect(page.get_by_text("Latest total", exact=True)).to_have_count(0)
|
||||
expect(page.get_by_text("Recent Historical Checkpoints", exact=True)).to_be_visible()
|
||||
expect(page.get_by_text("Cumulative proxy compression savings", exact=True)).to_have_count(
|
||||
2
|
||||
)
|
||||
|
||||
browser.close()
|
||||
|
||||
@@ -428,3 +428,73 @@ def awaitable_width(page: Page, locator) -> float:
|
||||
box = locator.bounding_box()
|
||||
assert box is not None
|
||||
return box["width"]
|
||||
|
||||
|
||||
def test_dashboard_canonical_metric_homes_at_responsive_viewports(tmp_path: Path) -> None:
|
||||
configured_artifact_dir = os.environ.get("HEADROOM_PLAYWRIGHT_ARTIFACT_DIR", "").strip()
|
||||
artifact_dir = (
|
||||
Path(configured_artifact_dir)
|
||||
if configured_artifact_dir
|
||||
else tmp_path / "responsive-artifacts"
|
||||
)
|
||||
|
||||
Path(artifact_dir).mkdir(parents=True, exist_ok=True)
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
for theme in ("light", "dark"):
|
||||
for width in (1280, 768, 400):
|
||||
page = browser.new_page(
|
||||
viewport={"width": width, "height": 1400}, color_scheme=theme
|
||||
)
|
||||
_open_dashboard(page, _stats_with_compression_vs_cache())
|
||||
session_metrics = page.get_by_test_id("session-optimization")
|
||||
expect(session_metrics.get_by_text("Tokens Saved", exact=True)).to_be_visible()
|
||||
expect(session_metrics.get_by_text("143.0k", exact=True)).to_be_visible()
|
||||
expect(page.get_by_test_id("proxy-share-of-total")).to_have_text(
|
||||
"23.5% of before-compression tokens"
|
||||
)
|
||||
expect(
|
||||
page.get_by_test_id("token-usage").get_by_text("91.0k", exact=True)
|
||||
).to_be_visible()
|
||||
performance = page.get_by_text("Performance", exact=True).locator("..")
|
||||
expect(performance.get_by_text("Overhead Range", exact=True)).to_be_visible()
|
||||
expect(performance.get_by_text("5 - 43ms", exact=True)).to_be_visible()
|
||||
expect(performance.get_by_text("TTFB Range", exact=True)).to_be_visible()
|
||||
expect(performance.get_by_text("0.42 - 2.90s", exact=True)).to_be_visible()
|
||||
page.get_by_test_id("session-view").screenshot(
|
||||
path=str(Path(artifact_dir) / f"dashboard-session-{theme}-{width}.png")
|
||||
)
|
||||
ttl_metrics = page.get_by_test_id("ttl-metrics")
|
||||
expect(ttl_metrics.get_by_text("Bucket Mix", exact=True)).to_be_visible()
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-mix-1h-total")).to_have_text("235.0k")
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-mix-5m-total")).to_have_text("185.0k")
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-1h-value")).to_have_count(0)
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-5m-value")).to_have_count(0)
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-1h-requests")).to_have_text("24")
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-5m-requests")).to_have_text("18")
|
||||
ttl_metrics.screenshot(
|
||||
path=str(Path(artifact_dir) / f"dashboard-ttl-{theme}-{width}.png")
|
||||
)
|
||||
page.get_by_role("button", name="Historical", exact=True).click()
|
||||
history_summary = page.get_by_test_id("history-summary")
|
||||
expect(
|
||||
history_summary.get_by_text("Lifetime Tokens Saved", exact=True)
|
||||
).to_be_visible()
|
||||
expect(page.get_by_test_id("history-total-value")).to_have_text("143.0k")
|
||||
expect(page.get_by_test_id("history-total-value")).to_have_count(1)
|
||||
expect(page.get_by_text("Latest total", exact=True)).to_have_count(0)
|
||||
expect(
|
||||
page.get_by_text("Recent Historical Checkpoints", exact=True)
|
||||
).to_be_visible()
|
||||
historical_card = page.get_by_test_id("historical-summary-card")
|
||||
expect(historical_card).to_be_visible()
|
||||
historical_checkpoints = page.get_by_text(
|
||||
"Recent Historical Checkpoints", exact=True
|
||||
).locator("..")
|
||||
expect(historical_checkpoints).to_be_visible()
|
||||
historical_checkpoints.screenshot(
|
||||
path=str(Path(artifact_dir) / f"dashboard-history-{theme}-{width}.png")
|
||||
)
|
||||
page.close()
|
||||
|
||||
browser.close()
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
@@ -36,6 +37,7 @@ def _sample_stats() -> dict:
|
||||
"input": 245_000,
|
||||
"output": 88_000,
|
||||
"saved": 143_000,
|
||||
"proxy_compression_saved": 91_000,
|
||||
"total_before_compression": 388_000,
|
||||
"savings_percent": 36.86,
|
||||
},
|
||||
@@ -172,8 +174,8 @@ def _fulfill_static_asset(route, path: str) -> bool: # type: ignore[no-untyped-
|
||||
return True
|
||||
|
||||
|
||||
def _install_dashboard_routes(page: Page) -> None:
|
||||
stats = _sample_stats()
|
||||
def _install_dashboard_routes(page: Page, stats: dict | None = None) -> None:
|
||||
stats = copy.deepcopy(stats) if stats is not None else _sample_stats()
|
||||
history = _sample_history()
|
||||
lifetime = {"projects": {}}
|
||||
health = {"status": "healthy", "version": "0.3.0"}
|
||||
@@ -245,8 +247,8 @@ def test_dashboard_per_project_setup_url_uses_current_origin() -> None:
|
||||
browser.close()
|
||||
|
||||
|
||||
def test_dashboard_renders_observed_ttl_metrics_and_can_capture_screenshot() -> None:
|
||||
artifact_dir = os.environ.get("HEADROOM_PLAYWRIGHT_ARTIFACT_DIR")
|
||||
def test_dashboard_renders_observed_ttl_metrics_and_can_capture_screenshot(tmp_path: Path) -> None:
|
||||
configured_artifact_dir = os.environ.get("HEADROOM_PLAYWRIGHT_ARTIFACT_DIR", "").strip()
|
||||
|
||||
with sync_playwright() as pw:
|
||||
browser = pw.chromium.launch()
|
||||
@@ -259,15 +261,56 @@ def test_dashboard_renders_observed_ttl_metrics_and_can_capture_screenshot() ->
|
||||
expect(page.get_by_test_id("ttl-bucket-headline")).to_have_text("1h leaning")
|
||||
expect(page.get_by_test_id("ttl-bucket-mix-1h-pct")).to_have_text("1h 56.0%")
|
||||
expect(page.get_by_test_id("ttl-bucket-mix-5m-pct")).to_have_text("5m 44.0%")
|
||||
expect(page.get_by_test_id("ttl-bucket-1h-value")).to_have_text("235.0k")
|
||||
expect(page.get_by_test_id("ttl-bucket-5m-value")).to_have_text("185.0k")
|
||||
expect(page.get_by_test_id("ttl-bucket-mix-1h-total")).to_have_text("235.0k")
|
||||
expect(page.get_by_test_id("ttl-bucket-mix-5m-total")).to_have_text("185.0k")
|
||||
expect(page.get_by_test_id("ttl-bucket-1h-requests")).to_have_text("24")
|
||||
expect(page.get_by_test_id("ttl-bucket-5m-requests")).to_have_text("18")
|
||||
expect(page.get_by_text("TTL 1h 56.0% / 5m 44.0%")).to_be_visible()
|
||||
|
||||
screenshot_path = (
|
||||
Path(artifact_dir) / "dashboard-cache-ttl-main.png"
|
||||
if artifact_dir
|
||||
else Path.cwd() / "dashboard-cache-ttl-main.png"
|
||||
Path(configured_artifact_dir) / "dashboard-cache-ttl-main.png"
|
||||
if configured_artifact_dir
|
||||
else tmp_path / "dashboard-cache-ttl-main.png"
|
||||
)
|
||||
screenshot_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
page.screenshot(path=str(screenshot_path), full_page=True)
|
||||
browser.close()
|
||||
|
||||
|
||||
def test_dashboard_renders_ttl_request_counts_when_token_totals_are_zero() -> None:
|
||||
stats = _sample_stats()
|
||||
buckets = stats["prefix_cache"]["totals"]["observed_ttl_buckets"]
|
||||
buckets["5m"]["tokens"] = 0
|
||||
buckets["1h"]["tokens"] = 0
|
||||
|
||||
with sync_playwright() as pw:
|
||||
browser = pw.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1440, "height": 1400})
|
||||
_install_dashboard_routes(page, stats)
|
||||
page.goto("http://headroom.local/dashboard", wait_until="load")
|
||||
|
||||
ttl_metrics = page.get_by_test_id("ttl-metrics")
|
||||
expect(ttl_metrics).to_be_visible()
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-mix-1h-total")).to_have_text("235.0k")
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-mix-5m-total")).to_have_text("185.0k")
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-1h-requests")).to_have_text("24")
|
||||
expect(ttl_metrics.get_by_test_id("ttl-bucket-5m-requests")).to_have_text("18")
|
||||
|
||||
browser.close()
|
||||
|
||||
|
||||
def test_dashboard_hides_ttl_metrics_when_request_counts_are_zero() -> None:
|
||||
stats = _sample_stats()
|
||||
buckets = stats["prefix_cache"]["totals"]["observed_ttl_buckets"]
|
||||
for bucket in buckets.values():
|
||||
bucket["requests"] = 0
|
||||
|
||||
with sync_playwright() as pw:
|
||||
browser = pw.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1440, "height": 1400})
|
||||
_install_dashboard_routes(page, stats)
|
||||
page.goto("http://headroom.local/dashboard", wait_until="load")
|
||||
|
||||
expect(page.get_by_test_id("ttl-metrics")).to_have_count(0)
|
||||
|
||||
browser.close()
|
||||
|
||||
@@ -6,7 +6,10 @@ import copy
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.test_dashboard_cache_lifetime_playwright import _open_dashboard
|
||||
from tests.test_dashboard_cache_lifetime_playwright import (
|
||||
_lifetime_cache_payload,
|
||||
_open_dashboard,
|
||||
)
|
||||
from tests.test_dashboard_cache_ttl_playwright import _sample_stats
|
||||
|
||||
playwright = pytest.importorskip("playwright.sync_api")
|
||||
@@ -25,7 +28,7 @@ def test_co2_card_renders_estimated_savings() -> None:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch()
|
||||
page = browser.new_page(viewport={"width": 1440, "height": 1000})
|
||||
_open_dashboard(page, stats)
|
||||
_open_dashboard(page, stats, _lifetime_cache_payload())
|
||||
|
||||
expect(page.get_by_text("CO2 Saved", exact=True)).to_be_visible()
|
||||
expect(page.get_by_text("120.00 g", exact=True)).to_be_visible()
|
||||
|
||||
@@ -205,6 +205,8 @@ def test_dashboard_session_metrics_do_not_repeat_proxy_tokens_without_new_contex
|
||||
html = get_dashboard_html()
|
||||
|
||||
assert "proxy tokens removed" not in html
|
||||
assert "'Proxy ' + formatNumber(stats.tokens?.proxy_compression_saved" not in html
|
||||
assert "formatNumber(stats.tokens?.saved || 0) + ' tokens total'" not in html
|
||||
assert '<span class="text-sm text-gray-400">Headroom Overhead</span>' not in html
|
||||
assert '<span class="text-sm text-gray-400">TTFB (upstream)</span>' not in html
|
||||
assert "Overhead Range" in html
|
||||
|
||||
Reference in New Issue
Block a user