fix(dashboard): lifetime metrics and refresh (#2374)

## Description

Unifies Dashboard Lifetime and Historical refresh behavior behind one
active-view refresher and canonicalizes durable lifetime waste-signal
names so persisted `json_noise` data is normalized to the current
`json_bloat` key.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Set dashboard stats polling to 15 seconds and use one active-view
refresh path for Lifetime and Historical views.
- Keep Lifetime and Historical view refreshes at 30 seconds, with
immediate refresh on view switch or keyboard force refresh.
- Normalize persisted waste-signal aliases from `json_noise` to
`json_bloat` and bucket unknown waste signals as `unknown`.
- Bump savings history schema to 6 so existing persisted data is
rewritten with canonical waste-signal keys.
- Add and update tests for waste-signal normalization, schema migration,
dashboard polling markup, and schema-version responses.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q
23 passed, 1 skipped, 1 warning

uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q
1 passed, 1 warning

uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory"
41 passed, 1 deselected, 1 warning

uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
All checks passed!

uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
8 files already formatted

git diff --check headroomlabs/main...HEAD
passed
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.3, local review worktree.
- Exact command / steps: Ran the focused pytest and Ruff commands listed
above against the dashboard, persistent metrics, savings history, and
project savings surfaces touched by this PR.
- Observed result: Focused tests passed, Ruff passed, and whitespace
checks passed. Full `tests/test_proxy_savings_history.py` on Windows
still has the existing directory-fsync assertion issue in
`test_savings_tracker_save_fsyncs_parent_directory`, so the broader
savings-history run excluded only that known platform-specific
assertion.
- Not tested: Browser-driven dashboard interaction; full repository test
suite; full `mypy headroom`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation is not needed for this internal dashboard polling and
persistence-normalization fix.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main0ddaa56c integration; final diff remains seven
lifetime-metrics/dashboard source/test files. Fresh325tests
pass,12skip,1warning, including ALL16browsercases (zero browser skips)
using existing Playwright1.59.0 overlay and installed Chrome in offline
contexts with service workers blocked. No package/browser installation.
Six changed Python files pass Rufflint/format; candidate offline lock
and whitespace pass. Full native
trackedinputset/everyhash/artifactchecksum verified before3982DLL
reuse,notfreshbuild.

Actual temporary-file load/save/reload proof: prior-schema alias-bearing
file is rewritten at construction to canonical json_bloat30/unknown10
and survives reopening; current-schema file normalizes in memory without
immediate rewrite, then persists canonical data on next record;
stateless prior-schema file remains byte-identical. Do not claim every
current-schema alias file is rewritten immediately. Actual
browser-rendered lifetime counters also show canonical
json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict
synthetic request fulfillment blocks all other requests, context
offline, no page errors/unexpected requests. Browser interactions
demonstrate no lazy lifetime/history fetch before selecting a view,
active-view refresh after virtual deadline, keyboard forced refresh and
refresh on tab reentry; synthetic counters visibly update. Observed
configuration: statsPollMs5000 and viewPollMs30000 (5-second session
polling,30-second active lifetime/history refresh); earlier15-second
description is not the implemented interval. Virtual clock and synthetic
responses are not production timing/server/billing proof. Initial
missing default Playwright package/browser revision and wrong 'History'
selector were harness issues; corrected to existing overlay, installed
Chrome and actual 'Historical' button. No PR failure attributed to those
attempts.

Skip qualification:11optional-provider cases plus1stale throughput test.
Narrow rerun5pass/1skip reports missing server._throughput_cache export;
generic skip text incorrectly calls it unbuilt Rust core. Server bytes
and throughput test AST match current main, so this is an inherited
test-API limitation, not this PR's introduced defect or native-build
failure. No unrelated throughput behavior rewritten.

Read author/reviewer and outstanding Devin discussion; fresh
disk/browser evidence addresses prior proof gaps, and polling interval
is qualified accurately above. No new unreviewed external findings at
prepublication audit. Source/test delta has no
workflow/setup/dependency/native/CLI paths; publication security gate
passes. Exact published-head CI/Actions/rollup, current-main
ancestry/mergeability and fresh discussion remain before approval.
Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s.
Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are
byte-identical to the prior reviewed head; prior offline Chrome
interaction evidence remains historical, not a fresh browser run. The
one throughput test skip retains the inherited missing
server._throughput_cache API limitation, not an unbuilt-core claim.
Native artifact reuse verified against full recorded input set/every
hash and artifact checksum; no fresh native build claimed. Static
publication gate excludes inherited workflow-validation/native
Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
This commit is contained in:
Jervis
2026-11-07 13:35:26 -06:00
committed by GitHub
co-authored by JerrettDavis JD Davis
parent f25fcc58cd
commit eb9932a671
7 changed files with 78 additions and 57 deletions
+8
View File
@@ -1046,6 +1046,14 @@ class WasteSignals:
}
# Canonical names consumed by durable metrics. Keep this derived from the
# request-level WasteSignals representation so the two views cannot drift.
WASTE_SIGNAL_KEYS = tuple(WasteSignals().to_dict())
# Persisted versions before the JSON signal was renamed can still be read.
WASTE_SIGNAL_ALIASES = {"json_noise": "json_bloat"}
@dataclass
class CachePrefixMetrics:
"""Detailed cache prefix metrics for debugging cache misses.
+23 -23
View File
@@ -1567,6 +1567,7 @@
<div class="px-4 py-8 text-center">
<div class="text-xs text-gray-500 mb-3">No per-project data yet.</div>
<div class="text-xs text-gray-600 font-mono leading-relaxed">
Add to each project's <span class="text-gray-400">.claude/settings.local.json</span>:<br>
<span class="text-accent" x-text="'ANTHROPIC_BASE_URL: ' + window.location.origin + '/p/&lt;project-name&gt;'"></span>
</div>
</div>
@@ -2078,11 +2079,12 @@
pollInterval: null,
polling_: false,
statsPollMs: 5000,
lifetimePollMs: 30000,
historyPollMs: 30000,
viewPollMs: 30000,
feedPollMs: 5000,
lastLifetimeFetchMs: 0,
lastHistoryFetchMs: 0,
viewRefreshers: {
lifetime: { lastFetchedAt: 0, fetchMethod: 'fetchLifetimeStats' },
history: { lastFetchedAt: 0, fetchMethod: 'fetchHistoryStats' },
},
lastFeedFetchMs: 0,
feedOpen: false,
transformations: [],
@@ -2121,28 +2123,26 @@
try {
await this.fetchStats();
const now = Date.now();
if (this.viewMode === 'lifetime' && (force || now - this.lastLifetimeFetchMs >= this.lifetimePollMs)) {
await this.fetchLifetimeStats();
}
if (this.viewMode === 'history' && (force || now - this.lastHistoryFetchMs >= this.historyPollMs)) {
await this.fetchHistoryStats();
}
if (this.feedOpen && (force || now - this.lastFeedFetchMs >= this.feedPollMs)) {
await this.fetchTransformations();
}
} finally {
this.polling_ = false;
await this.refreshActiveView(force);
const now = Date.now();
if (this.feedOpen && (force || now - this.lastFeedFetchMs >= this.feedPollMs)) {
await this.fetchTransformations();
}
} finally {
this.polling_ = false;
}
},
async setViewMode(mode) {
this.viewMode = mode;
if (mode === 'lifetime') {
await this.fetchLifetimeStats();
}
if (mode === 'history') {
await this.fetchHistoryStats();
await this.refreshActiveView(true);
},
async refreshActiveView(force = false) {
const refresher = this.viewRefreshers[this.viewMode];
if (!refresher) return;
if (force || Date.now() - refresher.lastFetchedAt >= this.viewPollMs) {
await this[refresher.fetchMethod]();
}
},
@@ -2211,7 +2211,7 @@
const response = await fetch('/stats-lifetime');
if (response.ok) {
this.lifetimeStats = await response.json();
this.lastLifetimeFetchMs = Date.now();
this.viewRefreshers.lifetime.lastFetchedAt = Date.now();
}
} catch (e) {
console.error('Failed to fetch lifetime stats:', e);
@@ -2220,7 +2220,7 @@
async fetchHistoryStats() {
try {
this.historyStats = await this.fetchJson('/stats-history');
this.lastHistoryFetchMs = Date.now();
this.viewRefreshers.history.lastFetchedAt = Date.now();
} catch (e) {
console.error('Failed to fetch history stats:', e);
}
+23 -23
View File
@@ -8,6 +8,8 @@ from copy import deepcopy
from datetime import datetime, timezone
from typing import Any
from headroom.config import WASTE_SIGNAL_ALIASES, WASTE_SIGNAL_KEYS
SCHEMA_VERSION = 5
MAX_PROVIDER_VALUES = 32
MAX_STACK_VALUES = 64
@@ -39,21 +41,6 @@ KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})
# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own
# limiter refusing the request, "upstream" is the provider refusing it.
RATE_LIMIT_SOURCES = frozenset({"headroom", "upstream"})
# Names must match ``WasteSignals.to_dict()`` in headroom/config.py — the
# parser emits ``json_bloat``; an allowlist that says ``json_noise`` silently
# shoves the largest waste category into the catch-all bucket.
KNOWN_WASTE_SIGNALS = frozenset(
{
"json_bloat",
"html_noise",
"base64",
"whitespace",
"dynamic_date",
"repetition",
"reread",
"reread_compressed",
}
)
def utc_now() -> datetime:
@@ -99,6 +86,13 @@ def _label(value: Any) -> str:
return value[:MAX_LABEL_LENGTH] if value else "other"
def _normalize_waste_signal_name(name: object) -> str:
if not isinstance(name, str):
return "unknown"
normalized = WASTE_SIGNAL_ALIASES.get(name, name)
return normalized if normalized in WASTE_SIGNAL_KEYS else "unknown"
def _model_entry(raw: Any = None) -> dict[str, Any]:
raw = raw if isinstance(raw, dict) else {}
return {
@@ -286,13 +280,9 @@ class PersistentMetricsState:
"compression_savings_usd"
]
result["cost"]["savings_basis"] = "list"
# Record-time puts unrecognised names in ``other`` (see
# ``record_request``); load-time must do the same, or every restart
# relabels the whole ``other`` bucket as ``unknown`` and the two
# grow side by side.
result["waste_signals"] = self._normalize_enum_map(
source.get("waste_signals"), KNOWN_WASTE_SIGNALS, fallback="other"
)
# Use the same canonical vocabulary for persisted and live signals so
# aliases and old catch-all names cannot create parallel buckets.
result["waste_signals"] = self._normalize_waste_signal_map(source.get("waste_signals"))
raw_models = _dict_or_empty(source.get("models"))
raw_tracked = _dict_or_empty(raw_models.get("tracked"))
@@ -331,6 +321,16 @@ class PersistentMetricsState:
result[label] = result.get(label, 0) + _coerce_int(value)
return result
@staticmethod
def _normalize_waste_signal_map(raw: Any) -> dict[str, int]:
result: dict[str, int] = {}
if not isinstance(raw, dict):
return result
for key, value in raw.items():
bucket = _normalize_waste_signal_name(key)
result[bucket] = result.get(bucket, 0) + _coerce_int(value)
return result
@staticmethod
def _compact_count_map(values: dict[str, int], limit: int) -> None:
named = [key for key in values if key != "other"]
@@ -501,7 +501,7 @@ class PersistentMetricsState:
if isinstance(waste_signals, dict):
for name, token_count in waste_signals.items():
bucket = name if isinstance(name, str) and name in KNOWN_WASTE_SIGNALS else "other"
bucket = _normalize_waste_signal_name(name)
self._state["waste_signals"][bucket] = self._state["waste_signals"].get(
bucket, 0
) + _coerce_int(token_count)
+3
View File
@@ -1014,6 +1014,9 @@ class SavingsTracker:
self._needs_schema_save = False
self._state = self._load_state()
self._persistent_metrics = PersistentMetricsState(self._state.pop("lifetime_metrics", None))
if self._needs_schema_save and not self._stateless:
with self._lock:
self._save_locked()
# Negative-savings warning state. A deployment that loses money loses it
# on most turns, not one, so warning per turn would bury the signal in
# its own repetition. Accumulate instead and emit a summary no more than
@@ -220,6 +220,7 @@ def test_dashboard_per_project_setup_url_uses_current_origin() -> None:
page.goto("http://127.0.0.1:8788/dashboard", wait_until="load")
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_text("Per-Project Savings")).to_be_visible()
expect(
page.get_by_text(
"ANTHROPIC_BASE_URL: http://127.0.0.1:8788/p/<project-name>", exact=True
@@ -233,6 +234,7 @@ def test_dashboard_per_project_setup_url_uses_current_origin() -> None:
page.goto("http://headroom.local:9393/dashboard", wait_until="load")
page.get_by_role("button", name="Lifetime", exact=True).click()
expect(page.get_by_text("Per-Project Savings")).to_be_visible()
expect(
page.get_by_text(
"ANTHROPIC_BASE_URL: http://headroom.local:9393/p/<project-name>", exact=True
+15 -10
View File
@@ -162,7 +162,7 @@ def test_state_normalizes_invalid_values_and_unknown_dimension_labels() -> None:
assert snapshot["requests"]["by_provider"] == {"other": 1}
assert snapshot["requests"]["by_stack"] == {"other": 1}
assert snapshot["by_model"]["other"]["input_tokens"] == 7
assert snapshot["waste_signals"] == {"other": 9}
assert snapshot["waste_signals"] == {"unknown": 9}
def test_json_bloat_waste_signal_is_a_named_bucket() -> None:
@@ -180,23 +180,28 @@ def test_json_bloat_waste_signal_is_a_named_bucket() -> None:
assert snapshot["waste_signals"] == {"json_bloat": 500, "html_noise": 20}
def test_waste_signal_other_bucket_survives_reload() -> None:
"""Reloading persisted state must keep ``other`` as ``other``, not relabel it ``unknown``.
Before this fix, unrecognised names went to ``other`` at record time but to
``unknown`` at load time, so every restart moved the whole ``other`` bucket
into a new ``unknown`` bucket and the two grew side by side.
"""
def test_state_canonicalizes_lifetime_waste_signal_aliases_and_fallbacks() -> None:
state = PersistentMetricsState(
{
"waste_signals": {"other": 40, "unknown": 60, "html_noise": 5, "bogus": 3},
"waste_signals": {
"json_noise": 10,
"json_bloat": 20,
"other": 7,
"unrecognized_persisted": 3,
},
},
now=lambda: FIXED_NOW,
)
state.record_request(
provider="openai",
stack="codex",
model="gpt-test",
waste_signals={"json_bloat": 5, "unrecognized_runtime": 2},
)
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
assert snapshot["waste_signals"] == {"other": 103, "html_noise": 5}
assert snapshot["waste_signals"] == {"json_bloat": 35, "unknown": 12}
def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:
+4 -1
View File
@@ -177,6 +177,9 @@ def test_dashboard_uses_cached_stats_and_lazy_history_feed_polling() -> None:
assert "this.fetchJson('/stats?cached=1')" in html
assert "cache: 'no-store'" in html
assert "statsPollMs: 5000" in html
assert "viewRefreshers:" in html
assert "this.refreshActiveView(force)" in html
assert "version: 'loading'" in html
assert 'x-text="formatVersion(version)"' in html
assert "return /^\\d+\\.\\d+\\.\\d+$/.test(label)" in html
@@ -185,7 +188,7 @@ def test_dashboard_uses_cached_stats_and_lazy_history_feed_polling() -> None:
assert "0.3.0" not in html
assert "@click=\"setViewMode('history')\"" in html
assert '@click="toggleFeed()"' in html
assert "this.viewMode === 'history'" in html
assert "this.viewRefreshers[this.viewMode]" in html
assert "this.feedOpen" in html
# The retired CLI context tools left no panel, label or getter behind.
for gone in (