mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Unifies Dashboard Lifetime and Historical refresh behavior behind one active-view refresher and canonicalizes durable lifetime waste-signal names so persisted `json_noise` data is normalized to the current `json_bloat` key. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set dashboard stats polling to 15 seconds and use one active-view refresh path for Lifetime and Historical views. - Keep Lifetime and Historical view refreshes at 30 seconds, with immediate refresh on view switch or keyboard force refresh. - Normalize persisted waste-signal aliases from `json_noise` to `json_bloat` and bucket unknown waste signals as `unknown`. - Bump savings history schema to 6 so existing persisted data is rewritten with canonical waste-signal keys. - Add and update tests for waste-signal normalization, schema migration, dashboard polling markup, and schema-version responses. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q 23 passed, 1 skipped, 1 warning uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q 1 passed, 1 warning uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory" 41 passed, 1 deselected, 1 warning uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py All checks passed! uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py 8 files already formatted git diff --check headroomlabs/main...HEAD passed ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.3, local review worktree. - Exact command / steps: Ran the focused pytest and Ruff commands listed above against the dashboard, persistent metrics, savings history, and project savings surfaces touched by this PR. - Observed result: Focused tests passed, Ruff passed, and whitespace checks passed. Full `tests/test_proxy_savings_history.py` on Windows still has the existing directory-fsync assertion issue in `test_savings_tracker_save_fsyncs_parent_directory`, so the broader savings-history run excluded only that known platform-specific assertion. - Not tested: Browser-driven dashboard interaction; full repository test suite; full `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Documentation is not needed for this internal dashboard polling and persistence-normalization fix. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean current-main0ddaa56c integration; final diff remains seven lifetime-metrics/dashboard source/test files. Fresh325tests pass,12skip,1warning, including ALL16browsercases (zero browser skips) using existing Playwright1.59.0 overlay and installed Chrome in offline contexts with service workers blocked. No package/browser installation. Six changed Python files pass Rufflint/format; candidate offline lock and whitespace pass. Full native trackedinputset/everyhash/artifactchecksum verified before3982DLL reuse,notfreshbuild. Actual temporary-file load/save/reload proof: prior-schema alias-bearing file is rewritten at construction to canonical json_bloat30/unknown10 and survives reopening; current-schema file normalizes in memory without immediate rewrite, then persists canonical data on next record; stateless prior-schema file remains byte-identical. Do not claim every current-schema alias file is rewritten immediately. Actual browser-rendered lifetime counters also show canonical json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict synthetic request fulfillment blocks all other requests, context offline, no page errors/unexpected requests. Browser interactions demonstrate no lazy lifetime/history fetch before selecting a view, active-view refresh after virtual deadline, keyboard forced refresh and refresh on tab reentry; synthetic counters visibly update. Observed configuration: statsPollMs5000 and viewPollMs30000 (5-second session polling,30-second active lifetime/history refresh); earlier15-second description is not the implemented interval. Virtual clock and synthetic responses are not production timing/server/billing proof. Initial missing default Playwright package/browser revision and wrong 'History' selector were harness issues; corrected to existing overlay, installed Chrome and actual 'Historical' button. No PR failure attributed to those attempts. Skip qualification:11optional-provider cases plus1stale throughput test. Narrow rerun5pass/1skip reports missing server._throughput_cache export; generic skip text incorrectly calls it unbuilt Rust core. Server bytes and throughput test AST match current main, so this is an inherited test-API limitation, not this PR's introduced defect or native-build failure. No unrelated throughput behavior rewritten. Read author/reviewer and outstanding Devin discussion; fresh disk/browser evidence addresses prior proof gaps, and polling interval is qualified accurately above. No new unreviewed external findings at prepublication audit. Source/test delta has no workflow/setup/dependency/native/CLI paths; publication security gate passes. Exact published-head CI/Actions/rollup, current-main ancestry/mergeability and fresh discussion remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. <!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 --> Maintainer current-main refresh: exact head `abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main `12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches exactly match incoming main; contributor behavior preserved. Fresh focused source tests plus tokenizer/response-header/HF/JSON-parser incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s. Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are byte-identical to the prior reviewed head; prior offline Chrome interaction evidence remains historical, not a fresh browser run. The one throughput test skip retains the inherited missing server._throughput_cache API limitation, not an unbuilt-core claim. Native artifact reuse verified against full recorded input set/every hash and artifact checksum; no fresh native build claimed. Static publication gate excludes inherited workflow-validation/native Docker/Init/Wrap installer paths, with no workflow/setup/dependency/native/CLI changes in PR diff. Actual exact-head hosted workflows and final independent review remain pending. No merge performed. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
283 lines
9.1 KiB
Python
283 lines
9.1 KiB
Python
from __future__ import annotations
|
|
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
from headroom.dashboard import get_dashboard_html
|
|
|
|
|
|
class _StatsStub:
|
|
def __init__(self, calls: dict[str, int], key: str, payload: dict):
|
|
self._calls = calls
|
|
self._key = key
|
|
self._payload = payload
|
|
|
|
def get_stats(self) -> dict:
|
|
self._calls[self._key] += 1
|
|
return dict(self._payload)
|
|
|
|
|
|
class _ToinStub:
|
|
def get_stats(self) -> dict:
|
|
return {"patterns": 0}
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
|
|
|
|
|
|
def test_stats_cached_query_reuses_short_ttl_snapshot(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.testclient import TestClient
|
|
|
|
import headroom.proxy.server as server
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
calls = {"store": 0, "telemetry": 0, "feedback": 0}
|
|
now = {"value": 100.0}
|
|
|
|
monkeypatch.setattr(server.time, "monotonic", lambda: now["value"])
|
|
monkeypatch.setattr(
|
|
server,
|
|
"get_compression_store",
|
|
lambda: _StatsStub(calls, "store", {"entry_count": 1, "max_entries": 100}),
|
|
)
|
|
monkeypatch.setattr(
|
|
server,
|
|
"get_telemetry_collector",
|
|
lambda: _StatsStub(calls, "telemetry", {"enabled": True}),
|
|
)
|
|
monkeypatch.setattr(
|
|
server,
|
|
"get_compression_feedback",
|
|
lambda: _StatsStub(calls, "feedback", {}),
|
|
)
|
|
|
|
monkeypatch.setattr(server, "get_toin", lambda: _ToinStub())
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
|
|
with TestClient(app) as client:
|
|
first = client.get("/stats?cached=1")
|
|
second = client.get("/stats?cached=1")
|
|
now["value"] += 5.1
|
|
third = client.get("/stats?cached=1")
|
|
uncached = client.get("/stats")
|
|
|
|
assert first.status_code == 200
|
|
assert second.status_code == 200
|
|
assert third.status_code == 200
|
|
assert uncached.status_code == 200
|
|
|
|
assert calls == {"store": 3, "telemetry": 3, "feedback": 3}
|
|
assert first.json()["tokens"]["proxy_compression_saved"] == 0
|
|
# The retired CLI context tools must leave no trace in the payload.
|
|
payload = first.json()
|
|
assert "context_tool" not in payload
|
|
assert "cli_filtering" not in payload
|
|
assert not any("rtk" in key or "lean_ctx" in key for key in payload["tokens"])
|
|
|
|
|
|
def test_session_summary_surfaces_codex_ws_counters() -> None:
|
|
from headroom.proxy.cost import build_session_summary
|
|
|
|
proxy = SimpleNamespace(
|
|
config=SimpleNamespace(mode="token"),
|
|
logger=SimpleNamespace(_logs=[]),
|
|
cost_tracker=SimpleNamespace(stats=lambda: {}),
|
|
)
|
|
metrics = SimpleNamespace(
|
|
requests_by_model={},
|
|
tokens_saved_total=0,
|
|
codex_ws_units_total=12,
|
|
codex_ws_units_modified_total=9,
|
|
codex_ws_unit_tokens_saved_sum=4321,
|
|
)
|
|
|
|
payload = build_session_summary(proxy, metrics, {}, total_tokens_before=0)
|
|
|
|
assert payload["codex_ws"] == {
|
|
"units_total": 12,
|
|
"units_modified": 9,
|
|
"tokens_saved": 4321,
|
|
}
|
|
|
|
|
|
def test_stats_reset_clears_runtime_proxy_counters(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.testclient import TestClient
|
|
|
|
import headroom.proxy.server as server
|
|
from headroom.proxy.loopback_guard import require_loopback
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
monkeypatch.setattr(
|
|
server,
|
|
"get_compression_store",
|
|
lambda: _StatsStub({"store": 0}, "store", {}),
|
|
)
|
|
monkeypatch.setattr(
|
|
server,
|
|
"get_telemetry_collector",
|
|
lambda: _StatsStub({"telemetry": 0}, "telemetry", {}),
|
|
)
|
|
monkeypatch.setattr(
|
|
server,
|
|
"get_compression_feedback",
|
|
lambda: _StatsStub({"feedback": 0}, "feedback", {}),
|
|
)
|
|
monkeypatch.setattr(server, "get_toin", lambda: _ToinStub())
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
app.dependency_overrides[require_loopback] = lambda: None
|
|
|
|
with TestClient(app) as client:
|
|
proxy = client.app.state.proxy
|
|
proxy.metrics.tokens_saved_total = 123
|
|
proxy.metrics.tokens_input_total = 456
|
|
proxy.metrics.requests_total = 2
|
|
|
|
before = client.get("/stats").json()
|
|
reset = client.post("/stats/reset")
|
|
after = client.get("/stats").json()
|
|
|
|
assert before["tokens"]["proxy_compression_saved"] == 123
|
|
assert reset.status_code == 200
|
|
assert after["tokens"]["proxy_compression_saved"] == 0
|
|
assert after["tokens"]["input"] == 0
|
|
assert after["requests"]["total"] == 0
|
|
|
|
|
|
def test_dashboard_uses_cached_stats_and_lazy_history_feed_polling() -> None:
|
|
html = get_dashboard_html()
|
|
|
|
assert "this.fetchJson('/stats?cached=1')" in html
|
|
assert "cache: 'no-store'" in html
|
|
assert "statsPollMs: 5000" in html
|
|
assert "viewRefreshers:" in html
|
|
assert "this.refreshActiveView(force)" in html
|
|
assert "version: 'loading'" in html
|
|
assert 'x-text="formatVersion(version)"' in html
|
|
assert "return /^\\d+\\.\\d+\\.\\d+$/.test(label)" in html
|
|
assert "return /^\\d/.test(value)" not in html
|
|
assert "this.version = health.version || 'unknown'" in html
|
|
assert "0.3.0" not in html
|
|
assert "@click=\"setViewMode('history')\"" in html
|
|
assert '@click="toggleFeed()"' in html
|
|
assert "this.viewRefreshers[this.viewMode]" in html
|
|
assert "this.feedOpen" in html
|
|
# The retired CLI context tools left no panel, label or getter behind.
|
|
for gone in (
|
|
"CLI Filtering (rtk)",
|
|
"RTK Filtered",
|
|
"|| 'RTK'",
|
|
"rtkShareOfTotal",
|
|
"Lean-ctx",
|
|
"Context Tool",
|
|
"cliFiltering",
|
|
"cli_filtering",
|
|
):
|
|
assert gone not in html, f"dashboard still references {gone!r}"
|
|
|
|
|
|
def test_dashboard_session_metrics_do_not_repeat_proxy_tokens_without_new_context() -> None:
|
|
html = get_dashboard_html()
|
|
|
|
assert "proxy tokens removed" not in html
|
|
assert "'Proxy ' + formatNumber(stats.tokens?.proxy_compression_saved" not in html
|
|
assert "formatNumber(stats.tokens?.saved || 0) + ' tokens total'" not in html
|
|
assert '<span class="text-sm text-gray-400">Headroom Overhead</span>' not in html
|
|
assert '<span class="text-sm text-gray-400">TTFB (upstream)</span>' not in html
|
|
assert "Overhead Range" in html
|
|
assert "TTFB Range" in html
|
|
assert "Proxy Removed" in html
|
|
|
|
|
|
def test_proxy_throughput_in_stats_endpoint(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Verify that the /stats endpoint includes a 'throughput' key in the response.
|
|
|
|
The server's _compute_throughput closure does a fresh
|
|
`from headroom.perf.analyzer import ...` on every call, so we patch the
|
|
names directly on the `headroom.perf.analyzer` module so the local import
|
|
inside the closure picks up our fakes.
|
|
|
|
Skipped locally when headroom._core (Rust extension) is not compiled.
|
|
"""
|
|
pytest.importorskip("fastapi")
|
|
from fastapi.testclient import TestClient
|
|
|
|
import headroom.perf.analyzer as _analyzer_mod
|
|
|
|
try:
|
|
from headroom.proxy.server import (
|
|
_throughput_cache,
|
|
create_app,
|
|
require_loopback,
|
|
)
|
|
except (ImportError, ModuleNotFoundError) as exc:
|
|
pytest.skip(f"headroom._core not available (Rust extension not compiled): {exc}")
|
|
|
|
from headroom.config import ProxyConfig
|
|
|
|
# Reset the module-level cache so CI doesn't reuse a stale value
|
|
_throughput_cache.update({"expires_at": 0.0, "value": None})
|
|
|
|
# Patch at the module level so the local import inside _compute_throughput
|
|
# picks up our stubs instead of the real implementations.
|
|
monkeypatch.setattr(
|
|
_analyzer_mod,
|
|
"parse_log_files",
|
|
lambda last_n_hours=1.0: _analyzer_mod.PerfReport(),
|
|
)
|
|
monkeypatch.setattr(
|
|
_analyzer_mod,
|
|
"build_perf_summary",
|
|
lambda report: {"throughput": {"input_wall_clock": 99.0}},
|
|
)
|
|
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
)
|
|
)
|
|
app.dependency_overrides[require_loopback] = lambda: None
|
|
|
|
with TestClient(app) as client:
|
|
response = client.get("/stats")
|
|
|
|
assert response.status_code == 200
|
|
payload = response.json()
|
|
assert "throughput" in payload
|
|
assert payload["throughput"] == {"input_wall_clock": 99.0}
|