Files
headroom/tests/test_proxy_dashboard_stats_cache.py
eb9932a671 fix(dashboard): lifetime metrics and refresh (#2374)
## Description

Unifies Dashboard Lifetime and Historical refresh behavior behind one
active-view refresher and canonicalizes durable lifetime waste-signal
names so persisted `json_noise` data is normalized to the current
`json_bloat` key.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Set dashboard stats polling to 15 seconds and use one active-view
refresh path for Lifetime and Historical views.
- Keep Lifetime and Historical view refreshes at 30 seconds, with
immediate refresh on view switch or keyboard force refresh.
- Normalize persisted waste-signal aliases from `json_noise` to
`json_bloat` and bucket unknown waste signals as `unknown`.
- Bump savings history schema to 6 so existing persisted data is
rewritten with canonical waste-signal keys.
- Add and update tests for waste-signal normalization, schema migration,
dashboard polling markup, and schema-version responses.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q
23 passed, 1 skipped, 1 warning

uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q
1 passed, 1 warning

uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory"
41 passed, 1 deselected, 1 warning

uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
All checks passed!

uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
8 files already formatted

git diff --check headroomlabs/main...HEAD
passed
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.3, local review worktree.
- Exact command / steps: Ran the focused pytest and Ruff commands listed
above against the dashboard, persistent metrics, savings history, and
project savings surfaces touched by this PR.
- Observed result: Focused tests passed, Ruff passed, and whitespace
checks passed. Full `tests/test_proxy_savings_history.py` on Windows
still has the existing directory-fsync assertion issue in
`test_savings_tracker_save_fsyncs_parent_directory`, so the broader
savings-history run excluded only that known platform-specific
assertion.
- Not tested: Browser-driven dashboard interaction; full repository test
suite; full `mypy headroom`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation is not needed for this internal dashboard polling and
persistence-normalization fix.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main0ddaa56c integration; final diff remains seven
lifetime-metrics/dashboard source/test files. Fresh325tests
pass,12skip,1warning, including ALL16browsercases (zero browser skips)
using existing Playwright1.59.0 overlay and installed Chrome in offline
contexts with service workers blocked. No package/browser installation.
Six changed Python files pass Rufflint/format; candidate offline lock
and whitespace pass. Full native
trackedinputset/everyhash/artifactchecksum verified before3982DLL
reuse,notfreshbuild.

Actual temporary-file load/save/reload proof: prior-schema alias-bearing
file is rewritten at construction to canonical json_bloat30/unknown10
and survives reopening; current-schema file normalizes in memory without
immediate rewrite, then persists canonical data on next record;
stateless prior-schema file remains byte-identical. Do not claim every
current-schema alias file is rewritten immediately. Actual
browser-rendered lifetime counters also show canonical
json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict
synthetic request fulfillment blocks all other requests, context
offline, no page errors/unexpected requests. Browser interactions
demonstrate no lazy lifetime/history fetch before selecting a view,
active-view refresh after virtual deadline, keyboard forced refresh and
refresh on tab reentry; synthetic counters visibly update. Observed
configuration: statsPollMs5000 and viewPollMs30000 (5-second session
polling,30-second active lifetime/history refresh); earlier15-second
description is not the implemented interval. Virtual clock and synthetic
responses are not production timing/server/billing proof. Initial
missing default Playwright package/browser revision and wrong 'History'
selector were harness issues; corrected to existing overlay, installed
Chrome and actual 'Historical' button. No PR failure attributed to those
attempts.

Skip qualification:11optional-provider cases plus1stale throughput test.
Narrow rerun5pass/1skip reports missing server._throughput_cache export;
generic skip text incorrectly calls it unbuilt Rust core. Server bytes
and throughput test AST match current main, so this is an inherited
test-API limitation, not this PR's introduced defect or native-build
failure. No unrelated throughput behavior rewritten.

Read author/reviewer and outstanding Devin discussion; fresh
disk/browser evidence addresses prior proof gaps, and polling interval
is qualified accurately above. No new unreviewed external findings at
prepublication audit. Source/test delta has no
workflow/setup/dependency/native/CLI paths; publication security gate
passes. Exact published-head CI/Actions/rollup, current-main
ancestry/mergeability and fresh discussion remain before approval.
Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s.
Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are
byte-identical to the prior reviewed head; prior offline Chrome
interaction evidence remains historical, not a fresh browser run. The
one throughput test skip retains the inherited missing
server._throughput_cache API limitation, not an unbuilt-core claim.
Native artifact reuse verified against full recorded input set/every
hash and artifact checksum; no fresh native build claimed. Static
publication gate excludes inherited workflow-validation/native
Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-11-07 13:35:26 -06:00

283 lines
9.1 KiB
Python

from __future__ import annotations
from types import SimpleNamespace
import pytest
from headroom.dashboard import get_dashboard_html
class _StatsStub:
def __init__(self, calls: dict[str, int], key: str, payload: dict):
self._calls = calls
self._key = key
self._payload = payload
def get_stats(self) -> dict:
self._calls[self._key] += 1
return dict(self._payload)
class _ToinStub:
def get_stats(self) -> dict:
return {"patterns": 0}
@pytest.fixture(autouse=True)
def _stub_env(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
def test_stats_cached_query_reuses_short_ttl_snapshot(monkeypatch: pytest.MonkeyPatch) -> None:
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
import headroom.proxy.server as server
from headroom.proxy.server import ProxyConfig, create_app
calls = {"store": 0, "telemetry": 0, "feedback": 0}
now = {"value": 100.0}
monkeypatch.setattr(server.time, "monotonic", lambda: now["value"])
monkeypatch.setattr(
server,
"get_compression_store",
lambda: _StatsStub(calls, "store", {"entry_count": 1, "max_entries": 100}),
)
monkeypatch.setattr(
server,
"get_telemetry_collector",
lambda: _StatsStub(calls, "telemetry", {"enabled": True}),
)
monkeypatch.setattr(
server,
"get_compression_feedback",
lambda: _StatsStub(calls, "feedback", {}),
)
monkeypatch.setattr(server, "get_toin", lambda: _ToinStub())
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
)
with TestClient(app) as client:
first = client.get("/stats?cached=1")
second = client.get("/stats?cached=1")
now["value"] += 5.1
third = client.get("/stats?cached=1")
uncached = client.get("/stats")
assert first.status_code == 200
assert second.status_code == 200
assert third.status_code == 200
assert uncached.status_code == 200
assert calls == {"store": 3, "telemetry": 3, "feedback": 3}
assert first.json()["tokens"]["proxy_compression_saved"] == 0
# The retired CLI context tools must leave no trace in the payload.
payload = first.json()
assert "context_tool" not in payload
assert "cli_filtering" not in payload
assert not any("rtk" in key or "lean_ctx" in key for key in payload["tokens"])
def test_session_summary_surfaces_codex_ws_counters() -> None:
from headroom.proxy.cost import build_session_summary
proxy = SimpleNamespace(
config=SimpleNamespace(mode="token"),
logger=SimpleNamespace(_logs=[]),
cost_tracker=SimpleNamespace(stats=lambda: {}),
)
metrics = SimpleNamespace(
requests_by_model={},
tokens_saved_total=0,
codex_ws_units_total=12,
codex_ws_units_modified_total=9,
codex_ws_unit_tokens_saved_sum=4321,
)
payload = build_session_summary(proxy, metrics, {}, total_tokens_before=0)
assert payload["codex_ws"] == {
"units_total": 12,
"units_modified": 9,
"tokens_saved": 4321,
}
def test_stats_reset_clears_runtime_proxy_counters(monkeypatch: pytest.MonkeyPatch) -> None:
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
import headroom.proxy.server as server
from headroom.proxy.loopback_guard import require_loopback
from headroom.proxy.server import ProxyConfig, create_app
monkeypatch.setattr(
server,
"get_compression_store",
lambda: _StatsStub({"store": 0}, "store", {}),
)
monkeypatch.setattr(
server,
"get_telemetry_collector",
lambda: _StatsStub({"telemetry": 0}, "telemetry", {}),
)
monkeypatch.setattr(
server,
"get_compression_feedback",
lambda: _StatsStub({"feedback": 0}, "feedback", {}),
)
monkeypatch.setattr(server, "get_toin", lambda: _ToinStub())
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
with TestClient(app) as client:
proxy = client.app.state.proxy
proxy.metrics.tokens_saved_total = 123
proxy.metrics.tokens_input_total = 456
proxy.metrics.requests_total = 2
before = client.get("/stats").json()
reset = client.post("/stats/reset")
after = client.get("/stats").json()
assert before["tokens"]["proxy_compression_saved"] == 123
assert reset.status_code == 200
assert after["tokens"]["proxy_compression_saved"] == 0
assert after["tokens"]["input"] == 0
assert after["requests"]["total"] == 0
def test_dashboard_uses_cached_stats_and_lazy_history_feed_polling() -> None:
html = get_dashboard_html()
assert "this.fetchJson('/stats?cached=1')" in html
assert "cache: 'no-store'" in html
assert "statsPollMs: 5000" in html
assert "viewRefreshers:" in html
assert "this.refreshActiveView(force)" in html
assert "version: 'loading'" in html
assert 'x-text="formatVersion(version)"' in html
assert "return /^\\d+\\.\\d+\\.\\d+$/.test(label)" in html
assert "return /^\\d/.test(value)" not in html
assert "this.version = health.version || 'unknown'" in html
assert "0.3.0" not in html
assert "@click=\"setViewMode('history')\"" in html
assert '@click="toggleFeed()"' in html
assert "this.viewRefreshers[this.viewMode]" in html
assert "this.feedOpen" in html
# The retired CLI context tools left no panel, label or getter behind.
for gone in (
"CLI Filtering (rtk)",
"RTK Filtered",
"|| 'RTK'",
"rtkShareOfTotal",
"Lean-ctx",
"Context Tool",
"cliFiltering",
"cli_filtering",
):
assert gone not in html, f"dashboard still references {gone!r}"
def test_dashboard_session_metrics_do_not_repeat_proxy_tokens_without_new_context() -> None:
html = get_dashboard_html()
assert "proxy tokens removed" not in html
assert "'Proxy ' + formatNumber(stats.tokens?.proxy_compression_saved" not in html
assert "formatNumber(stats.tokens?.saved || 0) + ' tokens total'" not in html
assert '<span class="text-sm text-gray-400">Headroom Overhead</span>' not in html
assert '<span class="text-sm text-gray-400">TTFB (upstream)</span>' not in html
assert "Overhead Range" in html
assert "TTFB Range" in html
assert "Proxy Removed" in html
def test_proxy_throughput_in_stats_endpoint(monkeypatch: pytest.MonkeyPatch) -> None:
"""Verify that the /stats endpoint includes a 'throughput' key in the response.
The server's _compute_throughput closure does a fresh
`from headroom.perf.analyzer import ...` on every call, so we patch the
names directly on the `headroom.perf.analyzer` module so the local import
inside the closure picks up our fakes.
Skipped locally when headroom._core (Rust extension) is not compiled.
"""
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
import headroom.perf.analyzer as _analyzer_mod
try:
from headroom.proxy.server import (
_throughput_cache,
create_app,
require_loopback,
)
except (ImportError, ModuleNotFoundError) as exc:
pytest.skip(f"headroom._core not available (Rust extension not compiled): {exc}")
from headroom.config import ProxyConfig
# Reset the module-level cache so CI doesn't reuse a stale value
_throughput_cache.update({"expires_at": 0.0, "value": None})
# Patch at the module level so the local import inside _compute_throughput
# picks up our stubs instead of the real implementations.
monkeypatch.setattr(
_analyzer_mod,
"parse_log_files",
lambda last_n_hours=1.0: _analyzer_mod.PerfReport(),
)
monkeypatch.setattr(
_analyzer_mod,
"build_perf_summary",
lambda report: {"throughput": {"input_wall_clock": 99.0}},
)
app = create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
)
app.dependency_overrides[require_loopback] = lambda: None
with TestClient(app) as client:
response = client.get("/stats")
assert response.status_code == 200
payload = response.json()
assert "throughput" in payload
assert payload["throughput"] == {"input_wall_clock": 99.0}