Files
headroom/tests/test_dashboard_cache_ttl_playwright.py
eb9932a671 fix(dashboard): lifetime metrics and refresh (#2374)
## Description

Unifies Dashboard Lifetime and Historical refresh behavior behind one
active-view refresher and canonicalizes durable lifetime waste-signal
names so persisted `json_noise` data is normalized to the current
`json_bloat` key.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Set dashboard stats polling to 15 seconds and use one active-view
refresh path for Lifetime and Historical views.
- Keep Lifetime and Historical view refreshes at 30 seconds, with
immediate refresh on view switch or keyboard force refresh.
- Normalize persisted waste-signal aliases from `json_noise` to
`json_bloat` and bucket unknown waste signals as `unknown`.
- Bump savings history schema to 6 so existing persisted data is
rewritten with canonical waste-signal keys.
- Add and update tests for waste-signal normalization, schema migration,
dashboard polling markup, and schema-version responses.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q
23 passed, 1 skipped, 1 warning

uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q
1 passed, 1 warning

uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory"
41 passed, 1 deselected, 1 warning

uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
All checks passed!

uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
8 files already formatted

git diff --check headroomlabs/main...HEAD
passed
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.3, local review worktree.
- Exact command / steps: Ran the focused pytest and Ruff commands listed
above against the dashboard, persistent metrics, savings history, and
project savings surfaces touched by this PR.
- Observed result: Focused tests passed, Ruff passed, and whitespace
checks passed. Full `tests/test_proxy_savings_history.py` on Windows
still has the existing directory-fsync assertion issue in
`test_savings_tracker_save_fsyncs_parent_directory`, so the broader
savings-history run excluded only that known platform-specific
assertion.
- Not tested: Browser-driven dashboard interaction; full repository test
suite; full `mypy headroom`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation is not needed for this internal dashboard polling and
persistence-normalization fix.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main0ddaa56c integration; final diff remains seven
lifetime-metrics/dashboard source/test files. Fresh325tests
pass,12skip,1warning, including ALL16browsercases (zero browser skips)
using existing Playwright1.59.0 overlay and installed Chrome in offline
contexts with service workers blocked. No package/browser installation.
Six changed Python files pass Rufflint/format; candidate offline lock
and whitespace pass. Full native
trackedinputset/everyhash/artifactchecksum verified before3982DLL
reuse,notfreshbuild.

Actual temporary-file load/save/reload proof: prior-schema alias-bearing
file is rewritten at construction to canonical json_bloat30/unknown10
and survives reopening; current-schema file normalizes in memory without
immediate rewrite, then persists canonical data on next record;
stateless prior-schema file remains byte-identical. Do not claim every
current-schema alias file is rewritten immediately. Actual
browser-rendered lifetime counters also show canonical
json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict
synthetic request fulfillment blocks all other requests, context
offline, no page errors/unexpected requests. Browser interactions
demonstrate no lazy lifetime/history fetch before selecting a view,
active-view refresh after virtual deadline, keyboard forced refresh and
refresh on tab reentry; synthetic counters visibly update. Observed
configuration: statsPollMs5000 and viewPollMs30000 (5-second session
polling,30-second active lifetime/history refresh); earlier15-second
description is not the implemented interval. Virtual clock and synthetic
responses are not production timing/server/billing proof. Initial
missing default Playwright package/browser revision and wrong 'History'
selector were harness issues; corrected to existing overlay, installed
Chrome and actual 'Historical' button. No PR failure attributed to those
attempts.

Skip qualification:11optional-provider cases plus1stale throughput test.
Narrow rerun5pass/1skip reports missing server._throughput_cache export;
generic skip text incorrectly calls it unbuilt Rust core. Server bytes
and throughput test AST match current main, so this is an inherited
test-API limitation, not this PR's introduced defect or native-build
failure. No unrelated throughput behavior rewritten.

Read author/reviewer and outstanding Devin discussion; fresh
disk/browser evidence addresses prior proof gaps, and polling interval
is qualified accurately above. No new unreviewed external findings at
prepublication audit. Source/test delta has no
workflow/setup/dependency/native/CLI paths; publication security gate
passes. Exact published-head CI/Actions/rollup, current-main
ancestry/mergeability and fresh discussion remain before approval.
Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s.
Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are
byte-identical to the prior reviewed head; prior offline Chrome
interaction evidence remains historical, not a fresh browser run. The
one throughput test skip retains the inherited missing
server._throughput_cache API limitation, not an unbuilt-core claim.
Native artifact reuse verified against full recorded input set/every
hash and artifact checksum; no fresh native build claimed. Static
publication gate excludes inherited workflow-validation/native
Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-11-07 13:35:26 -06:00

319 lines
12 KiB
Python

"""Behavior-driven Playwright validation for dashboard TTL cache metrics."""
from __future__ import annotations
import copy
import json
import os
from pathlib import Path
from urllib.parse import urlsplit
import pytest
from headroom.dashboard import STATIC_DIR, get_dashboard_html
playwright = pytest.importorskip("playwright.sync_api")
Page = playwright.Page
expect = playwright.expect
sync_playwright = playwright.sync_playwright
def _sample_stats() -> dict:
return {
"cost": {
"savings_usd": 12.34,
"compression_savings_usd": 12.34,
"cache_savings_usd": 5.25,
},
"requests": {
"total": 128,
"cached": 96,
"rate_limited": 0,
"failed": 0,
"by_provider": {"anthropic": 128},
"by_model": {"claude-opus-4-6": 128},
},
"tokens": {
"input": 245_000,
"output": 88_000,
"saved": 143_000,
"proxy_compression_saved": 91_000,
"total_before_compression": 388_000,
"savings_percent": 36.86,
},
"overhead": {"average_ms": 14.2, "min_ms": 4.5, "max_ms": 42.7},
"ttfb": {"average_ms": 1320.0, "min_ms": 420.0, "max_ms": 2900.0},
"latency": {"average_ms": 1510.0, "min_ms": 520.0, "max_ms": 3300.0},
"waste_signals": {"json_bloat": 95_000, "repetition": 48_000},
"savings_history": [
["2026-04-01T00:00:00Z", 12_000],
["2026-04-02T00:00:00Z", 38_000],
["2026-04-03T00:00:00Z", 57_000],
["2026-04-04T00:00:00Z", 102_000],
["2026-04-05T00:00:00Z", 143_000],
],
"persistent_savings": {
"display_session": {},
"lifetime": {"tokens_saved": 143_000, "compression_savings_usd": 12.34},
},
"pipeline_timing": {},
"compression_cache": {"mode": "cache"},
"prefix_cache": {
"by_provider": {
"anthropic": {
"cache_read_tokens": 9_800_000,
"cache_write_tokens": 420_000,
"cache_write_5m_tokens": 185_000,
"cache_write_1h_tokens": 235_000,
"cache_write_5m_requests": 18,
"cache_write_1h_requests": 24,
"requests": 128,
"hit_requests": 96,
"hit_rate": 75.0,
"bust_count": 0,
"bust_write_tokens": 0,
"read_discount": "90%",
"write_premium": "25%",
"savings_usd": 5.67,
"write_premium_usd": 0.42,
"net_savings_usd": 5.25,
"label": "Explicit breakpoints, 5-min TTL",
"observed_ttl_buckets": {
"5m": {"tokens": 185_000, "requests": 18},
"1h": {"tokens": 235_000, "requests": 24},
},
"observed_ttl_mix": {
"5m_pct": 44.0,
"1h_pct": 56.0,
"active_buckets": ["5m", "1h"],
},
}
},
"totals": {
"cache_read_tokens": 9_800_000,
"cache_write_tokens": 420_000,
"cache_write_5m_tokens": 185_000,
"cache_write_1h_tokens": 235_000,
"cache_write_5m_requests": 18,
"cache_write_1h_requests": 24,
"requests": 128,
"hit_requests": 96,
"bust_count": 0,
"bust_write_tokens": 0,
"savings_usd": 5.67,
"write_premium_usd": 0.42,
"net_savings_usd": 5.25,
"hit_rate": 75.0,
"observed_ttl_buckets": {
"5m": {"tokens": 185_000, "requests": 18},
"1h": {"tokens": 235_000, "requests": 24},
},
"observed_ttl_mix": {
"5m_pct": 44.0,
"1h_pct": 56.0,
"active_buckets": ["5m", "1h"],
},
},
"prefix_freeze": {
"busts_avoided": 0,
"tokens_preserved": 0,
"compression_foregone_tokens": 0,
"net_benefit_tokens": 0,
},
"attribution": "Observed provider TTL buckets.",
},
}
def _sample_history() -> dict:
return {
"history": [
{
"timestamp": "2026-04-01T00:00:00Z",
"total_tokens_saved": 12_000,
"compression_savings_usd": 0.6,
},
{
"timestamp": "2026-04-05T00:00:00Z",
"total_tokens_saved": 143_000,
"compression_savings_usd": 12.34,
},
],
"series": {
"daily": [
{
"timestamp": "2026-04-05T00:00:00Z",
"tokens_saved": 20_000,
"total_tokens_saved": 143_000,
"compression_savings_usd_delta": 1.7,
}
],
"weekly": [],
"monthly": [],
},
"lifetime": {"tokens_saved": 143_000, "compression_savings_usd": 12.34},
}
def _fulfill_static_asset(route, path: str) -> bool: # type: ignore[no-untyped-def]
"""Serve the vendored dashboard JS from disk; True when it handled the route.
The harnesses in this file and its siblings intercept every request, so the
dashboard's relative asset URLs would otherwise fall through to a real fetch
against a fake origin with nothing listening. Alpine has to load: every
section of <main> lives inside a `<template x-if>`, which renders nothing at
all without it, so an empty <main> is the symptom to look for here.
"""
if not path.startswith("/dashboard/static/"):
return False
route.fulfill(
status=200,
content_type="text/javascript",
body=(STATIC_DIR / Path(path).name).read_bytes(),
)
return True
def _install_dashboard_routes(page: Page, stats: dict | None = None) -> None:
stats = copy.deepcopy(stats) if stats is not None else _sample_stats()
history = _sample_history()
lifetime = {"projects": {}}
health = {"status": "healthy", "version": "0.3.0"}
dashboard_html = get_dashboard_html()
def handler(route) -> None: # type: ignore[no-untyped-def]
# Match on the URL path only: the dashboard fetches /stats?cached=1,
# so suffix checks against the full URL miss it and the request
# escapes the harness to the real network.
path = urlsplit(route.request.url).path
if path in ("/dashboard", "/"):
route.fulfill(status=200, content_type="text/html", body=dashboard_html)
return
if _fulfill_static_asset(route, path):
return
if "/stats-history" in path:
route.fulfill(
status=200,
content_type="application/json",
body=json.dumps(history),
)
return
if path.endswith("/stats-lifetime"):
route.fulfill(status=200, content_type="application/json", body=json.dumps(lifetime))
return
if path.endswith("/stats"):
route.fulfill(status=200, content_type="application/json", body=json.dumps(stats))
return
if path.endswith("/health"):
route.fulfill(status=200, content_type="application/json", body=json.dumps(health))
return
route.continue_()
page.route("**/*", handler)
def test_dashboard_per_project_setup_url_uses_current_origin() -> None:
with sync_playwright() as pw:
browser = pw.chromium.launch()
page = browser.new_page(viewport={"width": 1720, "height": 1400}, color_scheme="dark")
_install_dashboard_routes(page)
page.goto("http://127.0.0.1:8788/dashboard", wait_until="load")
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_text("Per-Project Savings")).to_be_visible()
expect(
page.get_by_text(
"ANTHROPIC_BASE_URL: http://127.0.0.1:8788/p/<project-name>", exact=True
)
).to_be_visible()
expect(
page.get_by_text(
"ANTHROPIC_BASE_URL: http://127.0.0.1:8787/p/<project-name>", exact=True
)
).to_have_count(0)
page.goto("http://headroom.local:9393/dashboard", wait_until="load")
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_text("Per-Project Savings")).to_be_visible()
expect(
page.get_by_text(
"ANTHROPIC_BASE_URL: http://headroom.local:9393/p/<project-name>", exact=True
)
).to_be_visible()
expect(
page.get_by_text(
"ANTHROPIC_BASE_URL: http://127.0.0.1:8787/p/<project-name>", exact=True
)
).to_have_count(0)
browser.close()
def test_dashboard_renders_observed_ttl_metrics_and_can_capture_screenshot(tmp_path: Path) -> None:
configured_artifact_dir = os.environ.get("HEADROOM_PLAYWRIGHT_ARTIFACT_DIR", "").strip()
with sync_playwright() as pw:
browser = pw.chromium.launch()
page = browser.new_page(viewport={"width": 1720, "height": 1400}, color_scheme="dark")
_install_dashboard_routes(page)
page.goto("http://headroom.local/dashboard", wait_until="load")
expect(page.get_by_text("Observed TTL Buckets")).to_be_visible()
expect(page.get_by_text("Provider-reported cache write mix")).to_be_visible()
expect(page.get_by_test_id("ttl-bucket-headline")).to_have_text("1h leaning")
expect(page.get_by_test_id("ttl-bucket-mix-1h-pct")).to_have_text("1h 56.0%")
expect(page.get_by_test_id("ttl-bucket-mix-5m-pct")).to_have_text("5m 44.0%")
expect(page.get_by_test_id("ttl-bucket-mix-1h-total")).to_have_text("235.0k")
expect(page.get_by_test_id("ttl-bucket-mix-5m-total")).to_have_text("185.0k")
expect(page.get_by_test_id("ttl-bucket-1h-requests")).to_have_text("24")
expect(page.get_by_test_id("ttl-bucket-5m-requests")).to_have_text("18")
expect(page.get_by_text("TTL 1h 56.0% / 5m 44.0%")).to_be_visible()
screenshot_path = (
Path(configured_artifact_dir) / "dashboard-cache-ttl-main.png"
if configured_artifact_dir
else tmp_path / "dashboard-cache-ttl-main.png"
)
screenshot_path.parent.mkdir(parents=True, exist_ok=True)
page.screenshot(path=str(screenshot_path), full_page=True)
browser.close()
def test_dashboard_renders_ttl_request_counts_when_token_totals_are_zero() -> None:
stats = _sample_stats()
buckets = stats["prefix_cache"]["totals"]["observed_ttl_buckets"]
buckets["5m"]["tokens"] = 0
buckets["1h"]["tokens"] = 0
with sync_playwright() as pw:
browser = pw.chromium.launch()
page = browser.new_page(viewport={"width": 1440, "height": 1400})
_install_dashboard_routes(page, stats)
page.goto("http://headroom.local/dashboard", wait_until="load")
ttl_metrics = page.get_by_test_id("ttl-metrics")
expect(ttl_metrics).to_be_visible()
expect(ttl_metrics.get_by_test_id("ttl-bucket-mix-1h-total")).to_have_text("235.0k")
expect(ttl_metrics.get_by_test_id("ttl-bucket-mix-5m-total")).to_have_text("185.0k")
expect(ttl_metrics.get_by_test_id("ttl-bucket-1h-requests")).to_have_text("24")
expect(ttl_metrics.get_by_test_id("ttl-bucket-5m-requests")).to_have_text("18")
browser.close()
def test_dashboard_hides_ttl_metrics_when_request_counts_are_zero() -> None:
stats = _sample_stats()
buckets = stats["prefix_cache"]["totals"]["observed_ttl_buckets"]
for bucket in buckets.values():
bucket["requests"] = 0
with sync_playwright() as pw:
browser = pw.chromium.launch()
page = browser.new_page(viewport={"width": 1440, "height": 1400})
_install_dashboard_routes(page, stats)
page.goto("http://headroom.local/dashboard", wait_until="load")
expect(page.get_by_test_id("ttl-metrics")).to_have_count(0)
browser.close()