mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(dashboard): lifetime metrics and refresh (#2374)
## Description Unifies Dashboard Lifetime and Historical refresh behavior behind one active-view refresher and canonicalizes durable lifetime waste-signal names so persisted `json_noise` data is normalized to the current `json_bloat` key. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set dashboard stats polling to 15 seconds and use one active-view refresh path for Lifetime and Historical views. - Keep Lifetime and Historical view refreshes at 30 seconds, with immediate refresh on view switch or keyboard force refresh. - Normalize persisted waste-signal aliases from `json_noise` to `json_bloat` and bucket unknown waste signals as `unknown`. - Bump savings history schema to 6 so existing persisted data is rewritten with canonical waste-signal keys. - Add and update tests for waste-signal normalization, schema migration, dashboard polling markup, and schema-version responses. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q 23 passed, 1 skipped, 1 warning uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q 1 passed, 1 warning uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory" 41 passed, 1 deselected, 1 warning uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py All checks passed! uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py 8 files already formatted git diff --check headroomlabs/main...HEAD passed ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.3, local review worktree. - Exact command / steps: Ran the focused pytest and Ruff commands listed above against the dashboard, persistent metrics, savings history, and project savings surfaces touched by this PR. - Observed result: Focused tests passed, Ruff passed, and whitespace checks passed. Full `tests/test_proxy_savings_history.py` on Windows still has the existing directory-fsync assertion issue in `test_savings_tracker_save_fsyncs_parent_directory`, so the broader savings-history run excluded only that known platform-specific assertion. - Not tested: Browser-driven dashboard interaction; full repository test suite; full `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Documentation is not needed for this internal dashboard polling and persistence-normalization fix. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean current-main0ddaa56c integration; final diff remains seven lifetime-metrics/dashboard source/test files. Fresh325tests pass,12skip,1warning, including ALL16browsercases (zero browser skips) using existing Playwright1.59.0 overlay and installed Chrome in offline contexts with service workers blocked. No package/browser installation. Six changed Python files pass Rufflint/format; candidate offline lock and whitespace pass. Full native trackedinputset/everyhash/artifactchecksum verified before3982DLL reuse,notfreshbuild. Actual temporary-file load/save/reload proof: prior-schema alias-bearing file is rewritten at construction to canonical json_bloat30/unknown10 and survives reopening; current-schema file normalizes in memory without immediate rewrite, then persists canonical data on next record; stateless prior-schema file remains byte-identical. Do not claim every current-schema alias file is rewritten immediately. Actual browser-rendered lifetime counters also show canonical json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict synthetic request fulfillment blocks all other requests, context offline, no page errors/unexpected requests. Browser interactions demonstrate no lazy lifetime/history fetch before selecting a view, active-view refresh after virtual deadline, keyboard forced refresh and refresh on tab reentry; synthetic counters visibly update. Observed configuration: statsPollMs5000 and viewPollMs30000 (5-second session polling,30-second active lifetime/history refresh); earlier15-second description is not the implemented interval. Virtual clock and synthetic responses are not production timing/server/billing proof. Initial missing default Playwright package/browser revision and wrong 'History' selector were harness issues; corrected to existing overlay, installed Chrome and actual 'Historical' button. No PR failure attributed to those attempts. Skip qualification:11optional-provider cases plus1stale throughput test. Narrow rerun5pass/1skip reports missing server._throughput_cache export; generic skip text incorrectly calls it unbuilt Rust core. Server bytes and throughput test AST match current main, so this is an inherited test-API limitation, not this PR's introduced defect or native-build failure. No unrelated throughput behavior rewritten. Read author/reviewer and outstanding Devin discussion; fresh disk/browser evidence addresses prior proof gaps, and polling interval is qualified accurately above. No new unreviewed external findings at prepublication audit. Source/test delta has no workflow/setup/dependency/native/CLI paths; publication security gate passes. Exact published-head CI/Actions/rollup, current-main ancestry/mergeability and fresh discussion remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. <!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 --> Maintainer current-main refresh: exact head `abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main `12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches exactly match incoming main; contributor behavior preserved. Fresh focused source tests plus tokenizer/response-header/HF/JSON-parser incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s. Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are byte-identical to the prior reviewed head; prior offline Chrome interaction evidence remains historical, not a fresh browser run. The one throughput test skip retains the inherited missing server._throughput_cache API limitation, not an unbuilt-core claim. Native artifact reuse verified against full recorded input set/every hash and artifact checksum; no fresh native build claimed. Static publication gate excludes inherited workflow-validation/native Docker/Init/Wrap installer paths, with no workflow/setup/dependency/native/CLI changes in PR diff. Actual exact-head hosted workflows and final independent review remain pending. No merge performed. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
This commit is contained in:
co-authored by
JerrettDavis
JD Davis
parent
f25fcc58cd
commit
eb9932a671
@@ -1046,6 +1046,14 @@ class WasteSignals:
|
||||
}
|
||||
|
||||
|
||||
# Canonical names consumed by durable metrics. Keep this derived from the
|
||||
# request-level WasteSignals representation so the two views cannot drift.
|
||||
WASTE_SIGNAL_KEYS = tuple(WasteSignals().to_dict())
|
||||
|
||||
# Persisted versions before the JSON signal was renamed can still be read.
|
||||
WASTE_SIGNAL_ALIASES = {"json_noise": "json_bloat"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CachePrefixMetrics:
|
||||
"""Detailed cache prefix metrics for debugging cache misses.
|
||||
|
||||
@@ -1567,6 +1567,7 @@
|
||||
<div class="px-4 py-8 text-center">
|
||||
<div class="text-xs text-gray-500 mb-3">No per-project data yet.</div>
|
||||
<div class="text-xs text-gray-600 font-mono leading-relaxed">
|
||||
Add to each project's <span class="text-gray-400">.claude/settings.local.json</span>:<br>
|
||||
<span class="text-accent" x-text="'ANTHROPIC_BASE_URL: ' + window.location.origin + '/p/<project-name>'"></span>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2078,11 +2079,12 @@
|
||||
pollInterval: null,
|
||||
polling_: false,
|
||||
statsPollMs: 5000,
|
||||
lifetimePollMs: 30000,
|
||||
historyPollMs: 30000,
|
||||
viewPollMs: 30000,
|
||||
feedPollMs: 5000,
|
||||
lastLifetimeFetchMs: 0,
|
||||
lastHistoryFetchMs: 0,
|
||||
viewRefreshers: {
|
||||
lifetime: { lastFetchedAt: 0, fetchMethod: 'fetchLifetimeStats' },
|
||||
history: { lastFetchedAt: 0, fetchMethod: 'fetchHistoryStats' },
|
||||
},
|
||||
lastFeedFetchMs: 0,
|
||||
feedOpen: false,
|
||||
transformations: [],
|
||||
@@ -2121,28 +2123,26 @@
|
||||
try {
|
||||
await this.fetchStats();
|
||||
|
||||
const now = Date.now();
|
||||
if (this.viewMode === 'lifetime' && (force || now - this.lastLifetimeFetchMs >= this.lifetimePollMs)) {
|
||||
await this.fetchLifetimeStats();
|
||||
}
|
||||
if (this.viewMode === 'history' && (force || now - this.lastHistoryFetchMs >= this.historyPollMs)) {
|
||||
await this.fetchHistoryStats();
|
||||
}
|
||||
if (this.feedOpen && (force || now - this.lastFeedFetchMs >= this.feedPollMs)) {
|
||||
await this.fetchTransformations();
|
||||
}
|
||||
} finally {
|
||||
this.polling_ = false;
|
||||
await this.refreshActiveView(force);
|
||||
const now = Date.now();
|
||||
if (this.feedOpen && (force || now - this.lastFeedFetchMs >= this.feedPollMs)) {
|
||||
await this.fetchTransformations();
|
||||
}
|
||||
} finally {
|
||||
this.polling_ = false;
|
||||
}
|
||||
},
|
||||
|
||||
async setViewMode(mode) {
|
||||
this.viewMode = mode;
|
||||
if (mode === 'lifetime') {
|
||||
await this.fetchLifetimeStats();
|
||||
}
|
||||
if (mode === 'history') {
|
||||
await this.fetchHistoryStats();
|
||||
await this.refreshActiveView(true);
|
||||
},
|
||||
|
||||
async refreshActiveView(force = false) {
|
||||
const refresher = this.viewRefreshers[this.viewMode];
|
||||
if (!refresher) return;
|
||||
if (force || Date.now() - refresher.lastFetchedAt >= this.viewPollMs) {
|
||||
await this[refresher.fetchMethod]();
|
||||
}
|
||||
},
|
||||
|
||||
@@ -2211,7 +2211,7 @@
|
||||
const response = await fetch('/stats-lifetime');
|
||||
if (response.ok) {
|
||||
this.lifetimeStats = await response.json();
|
||||
this.lastLifetimeFetchMs = Date.now();
|
||||
this.viewRefreshers.lifetime.lastFetchedAt = Date.now();
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('Failed to fetch lifetime stats:', e);
|
||||
@@ -2220,7 +2220,7 @@
|
||||
async fetchHistoryStats() {
|
||||
try {
|
||||
this.historyStats = await this.fetchJson('/stats-history');
|
||||
this.lastHistoryFetchMs = Date.now();
|
||||
this.viewRefreshers.history.lastFetchedAt = Date.now();
|
||||
} catch (e) {
|
||||
console.error('Failed to fetch history stats:', e);
|
||||
}
|
||||
|
||||
@@ -8,6 +8,8 @@ from copy import deepcopy
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from headroom.config import WASTE_SIGNAL_ALIASES, WASTE_SIGNAL_KEYS
|
||||
|
||||
SCHEMA_VERSION = 5
|
||||
MAX_PROVIDER_VALUES = 32
|
||||
MAX_STACK_VALUES = 64
|
||||
@@ -39,21 +41,6 @@ KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})
|
||||
# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own
|
||||
# limiter refusing the request, "upstream" is the provider refusing it.
|
||||
RATE_LIMIT_SOURCES = frozenset({"headroom", "upstream"})
|
||||
# Names must match ``WasteSignals.to_dict()`` in headroom/config.py — the
|
||||
# parser emits ``json_bloat``; an allowlist that says ``json_noise`` silently
|
||||
# shoves the largest waste category into the catch-all bucket.
|
||||
KNOWN_WASTE_SIGNALS = frozenset(
|
||||
{
|
||||
"json_bloat",
|
||||
"html_noise",
|
||||
"base64",
|
||||
"whitespace",
|
||||
"dynamic_date",
|
||||
"repetition",
|
||||
"reread",
|
||||
"reread_compressed",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def utc_now() -> datetime:
|
||||
@@ -99,6 +86,13 @@ def _label(value: Any) -> str:
|
||||
return value[:MAX_LABEL_LENGTH] if value else "other"
|
||||
|
||||
|
||||
def _normalize_waste_signal_name(name: object) -> str:
|
||||
if not isinstance(name, str):
|
||||
return "unknown"
|
||||
normalized = WASTE_SIGNAL_ALIASES.get(name, name)
|
||||
return normalized if normalized in WASTE_SIGNAL_KEYS else "unknown"
|
||||
|
||||
|
||||
def _model_entry(raw: Any = None) -> dict[str, Any]:
|
||||
raw = raw if isinstance(raw, dict) else {}
|
||||
return {
|
||||
@@ -286,13 +280,9 @@ class PersistentMetricsState:
|
||||
"compression_savings_usd"
|
||||
]
|
||||
result["cost"]["savings_basis"] = "list"
|
||||
# Record-time puts unrecognised names in ``other`` (see
|
||||
# ``record_request``); load-time must do the same, or every restart
|
||||
# relabels the whole ``other`` bucket as ``unknown`` and the two
|
||||
# grow side by side.
|
||||
result["waste_signals"] = self._normalize_enum_map(
|
||||
source.get("waste_signals"), KNOWN_WASTE_SIGNALS, fallback="other"
|
||||
)
|
||||
# Use the same canonical vocabulary for persisted and live signals so
|
||||
# aliases and old catch-all names cannot create parallel buckets.
|
||||
result["waste_signals"] = self._normalize_waste_signal_map(source.get("waste_signals"))
|
||||
|
||||
raw_models = _dict_or_empty(source.get("models"))
|
||||
raw_tracked = _dict_or_empty(raw_models.get("tracked"))
|
||||
@@ -331,6 +321,16 @@ class PersistentMetricsState:
|
||||
result[label] = result.get(label, 0) + _coerce_int(value)
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _normalize_waste_signal_map(raw: Any) -> dict[str, int]:
|
||||
result: dict[str, int] = {}
|
||||
if not isinstance(raw, dict):
|
||||
return result
|
||||
for key, value in raw.items():
|
||||
bucket = _normalize_waste_signal_name(key)
|
||||
result[bucket] = result.get(bucket, 0) + _coerce_int(value)
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _compact_count_map(values: dict[str, int], limit: int) -> None:
|
||||
named = [key for key in values if key != "other"]
|
||||
@@ -501,7 +501,7 @@ class PersistentMetricsState:
|
||||
|
||||
if isinstance(waste_signals, dict):
|
||||
for name, token_count in waste_signals.items():
|
||||
bucket = name if isinstance(name, str) and name in KNOWN_WASTE_SIGNALS else "other"
|
||||
bucket = _normalize_waste_signal_name(name)
|
||||
self._state["waste_signals"][bucket] = self._state["waste_signals"].get(
|
||||
bucket, 0
|
||||
) + _coerce_int(token_count)
|
||||
|
||||
@@ -1014,6 +1014,9 @@ class SavingsTracker:
|
||||
self._needs_schema_save = False
|
||||
self._state = self._load_state()
|
||||
self._persistent_metrics = PersistentMetricsState(self._state.pop("lifetime_metrics", None))
|
||||
if self._needs_schema_save and not self._stateless:
|
||||
with self._lock:
|
||||
self._save_locked()
|
||||
# Negative-savings warning state. A deployment that loses money loses it
|
||||
# on most turns, not one, so warning per turn would bury the signal in
|
||||
# its own repetition. Accumulate instead and emit a summary no more than
|
||||
|
||||
@@ -220,6 +220,7 @@ def test_dashboard_per_project_setup_url_uses_current_origin() -> None:
|
||||
|
||||
page.goto("http://127.0.0.1:8788/dashboard", wait_until="load")
|
||||
page.get_by_role("button", name="Lifetime", exact=True).click()
|
||||
expect(page.get_by_text("Per-Project Savings")).to_be_visible()
|
||||
expect(
|
||||
page.get_by_text(
|
||||
"ANTHROPIC_BASE_URL: http://127.0.0.1:8788/p/<project-name>", exact=True
|
||||
@@ -233,6 +234,7 @@ def test_dashboard_per_project_setup_url_uses_current_origin() -> None:
|
||||
|
||||
page.goto("http://headroom.local:9393/dashboard", wait_until="load")
|
||||
page.get_by_role("button", name="Lifetime", exact=True).click()
|
||||
expect(page.get_by_text("Per-Project Savings")).to_be_visible()
|
||||
expect(
|
||||
page.get_by_text(
|
||||
"ANTHROPIC_BASE_URL: http://headroom.local:9393/p/<project-name>", exact=True
|
||||
|
||||
@@ -162,7 +162,7 @@ def test_state_normalizes_invalid_values_and_unknown_dimension_labels() -> None:
|
||||
assert snapshot["requests"]["by_provider"] == {"other": 1}
|
||||
assert snapshot["requests"]["by_stack"] == {"other": 1}
|
||||
assert snapshot["by_model"]["other"]["input_tokens"] == 7
|
||||
assert snapshot["waste_signals"] == {"other": 9}
|
||||
assert snapshot["waste_signals"] == {"unknown": 9}
|
||||
|
||||
|
||||
def test_json_bloat_waste_signal_is_a_named_bucket() -> None:
|
||||
@@ -180,23 +180,28 @@ def test_json_bloat_waste_signal_is_a_named_bucket() -> None:
|
||||
assert snapshot["waste_signals"] == {"json_bloat": 500, "html_noise": 20}
|
||||
|
||||
|
||||
def test_waste_signal_other_bucket_survives_reload() -> None:
|
||||
"""Reloading persisted state must keep ``other`` as ``other``, not relabel it ``unknown``.
|
||||
|
||||
Before this fix, unrecognised names went to ``other`` at record time but to
|
||||
``unknown`` at load time, so every restart moved the whole ``other`` bucket
|
||||
into a new ``unknown`` bucket and the two grew side by side.
|
||||
"""
|
||||
def test_state_canonicalizes_lifetime_waste_signal_aliases_and_fallbacks() -> None:
|
||||
state = PersistentMetricsState(
|
||||
{
|
||||
"waste_signals": {"other": 40, "unknown": 60, "html_noise": 5, "bogus": 3},
|
||||
"waste_signals": {
|
||||
"json_noise": 10,
|
||||
"json_bloat": 20,
|
||||
"other": 7,
|
||||
"unrecognized_persisted": 3,
|
||||
},
|
||||
},
|
||||
now=lambda: FIXED_NOW,
|
||||
)
|
||||
state.record_request(
|
||||
provider="openai",
|
||||
stack="codex",
|
||||
model="gpt-test",
|
||||
waste_signals={"json_bloat": 5, "unrecognized_runtime": 2},
|
||||
)
|
||||
|
||||
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
||||
|
||||
assert snapshot["waste_signals"] == {"other": 103, "html_noise": 5}
|
||||
assert snapshot["waste_signals"] == {"json_bloat": 35, "unknown": 12}
|
||||
|
||||
|
||||
def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:
|
||||
|
||||
@@ -177,6 +177,9 @@ def test_dashboard_uses_cached_stats_and_lazy_history_feed_polling() -> None:
|
||||
|
||||
assert "this.fetchJson('/stats?cached=1')" in html
|
||||
assert "cache: 'no-store'" in html
|
||||
assert "statsPollMs: 5000" in html
|
||||
assert "viewRefreshers:" in html
|
||||
assert "this.refreshActiveView(force)" in html
|
||||
assert "version: 'loading'" in html
|
||||
assert 'x-text="formatVersion(version)"' in html
|
||||
assert "return /^\\d+\\.\\d+\\.\\d+$/.test(label)" in html
|
||||
@@ -185,7 +188,7 @@ def test_dashboard_uses_cached_stats_and_lazy_history_feed_polling() -> None:
|
||||
assert "0.3.0" not in html
|
||||
assert "@click=\"setViewMode('history')\"" in html
|
||||
assert '@click="toggleFeed()"' in html
|
||||
assert "this.viewMode === 'history'" in html
|
||||
assert "this.viewRefreshers[this.viewMode]" in html
|
||||
assert "this.feedOpen" in html
|
||||
# The retired CLI context tools left no panel, label or getter behind.
|
||||
for gone in (
|
||||
|
||||
Reference in New Issue
Block a user