mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description Unifies Dashboard Lifetime and Historical refresh behavior behind one active-view refresher and canonicalizes durable lifetime waste-signal names so persisted `json_noise` data is normalized to the current `json_bloat` key. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set dashboard stats polling to 15 seconds and use one active-view refresh path for Lifetime and Historical views. - Keep Lifetime and Historical view refreshes at 30 seconds, with immediate refresh on view switch or keyboard force refresh. - Normalize persisted waste-signal aliases from `json_noise` to `json_bloat` and bucket unknown waste signals as `unknown`. - Bump savings history schema to 6 so existing persisted data is rewritten with canonical waste-signal keys. - Add and update tests for waste-signal normalization, schema migration, dashboard polling markup, and schema-version responses. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q 23 passed, 1 skipped, 1 warning uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q 1 passed, 1 warning uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory" 41 passed, 1 deselected, 1 warning uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py All checks passed! uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py 8 files already formatted git diff --check headroomlabs/main...HEAD passed ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.3, local review worktree. - Exact command / steps: Ran the focused pytest and Ruff commands listed above against the dashboard, persistent metrics, savings history, and project savings surfaces touched by this PR. - Observed result: Focused tests passed, Ruff passed, and whitespace checks passed. Full `tests/test_proxy_savings_history.py` on Windows still has the existing directory-fsync assertion issue in `test_savings_tracker_save_fsyncs_parent_directory`, so the broader savings-history run excluded only that known platform-specific assertion. - Not tested: Browser-driven dashboard interaction; full repository test suite; full `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Documentation is not needed for this internal dashboard polling and persistence-normalization fix. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main `0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`. Clean current-main0ddaa56c integration; final diff remains seven lifetime-metrics/dashboard source/test files. Fresh325tests pass,12skip,1warning, including ALL16browsercases (zero browser skips) using existing Playwright1.59.0 overlay and installed Chrome in offline contexts with service workers blocked. No package/browser installation. Six changed Python files pass Rufflint/format; candidate offline lock and whitespace pass. Full native trackedinputset/everyhash/artifactchecksum verified before3982DLL reuse,notfreshbuild. Actual temporary-file load/save/reload proof: prior-schema alias-bearing file is rewritten at construction to canonical json_bloat30/unknown10 and survives reopening; current-schema file normalizes in memory without immediate rewrite, then persists canonical data on next record; stateless prior-schema file remains byte-identical. Do not claim every current-schema alias file is rewritten immediately. Actual browser-rendered lifetime counters also show canonical json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict synthetic request fulfillment blocks all other requests, context offline, no page errors/unexpected requests. Browser interactions demonstrate no lazy lifetime/history fetch before selecting a view, active-view refresh after virtual deadline, keyboard forced refresh and refresh on tab reentry; synthetic counters visibly update. Observed configuration: statsPollMs5000 and viewPollMs30000 (5-second session polling,30-second active lifetime/history refresh); earlier15-second description is not the implemented interval. Virtual clock and synthetic responses are not production timing/server/billing proof. Initial missing default Playwright package/browser revision and wrong 'History' selector were harness issues; corrected to existing overlay, installed Chrome and actual 'Historical' button. No PR failure attributed to those attempts. Skip qualification:11optional-provider cases plus1stale throughput test. Narrow rerun5pass/1skip reports missing server._throughput_cache export; generic skip text incorrectly calls it unbuilt Rust core. Server bytes and throughput test AST match current main, so this is an inherited test-API limitation, not this PR's introduced defect or native-build failure. No unrelated throughput behavior rewritten. Read author/reviewer and outstanding Devin discussion; fresh disk/browser evidence addresses prior proof gaps, and polling interval is qualified accurately above. No new unreviewed external findings at prepublication audit. Source/test delta has no workflow/setup/dependency/native/CLI paths; publication security gate passes. Exact published-head CI/Actions/rollup, current-main ancestry/mergeability and fresh discussion remain before approval. Maintainers handle CI; no PR merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. <!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 --> Maintainer current-main refresh: exact head `abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main `12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches exactly match incoming main; contributor behavior preserved. Fresh focused source tests plus tokenizer/response-header/HF/JSON-parser incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s. Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are byte-identical to the prior reviewed head; prior offline Chrome interaction evidence remains historical, not a fresh browser run. The one throughput test skip retains the inherited missing server._throughput_cache API limitation, not an unbuilt-core claim. Native artifact reuse verified against full recorded input set/every hash and artifact checksum; no fresh native build claimed. Static publication gate excludes inherited workflow-validation/native Docker/Init/Wrap installer paths, with no workflow/setup/dependency/native/CLI changes in PR diff. Actual exact-head hosted workflows and final independent review remain pending. No merge performed. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
269 lines
9.6 KiB
Python
269 lines
9.6 KiB
Python
"""Tests for durable, aggregate-only proxy Lifetime metrics."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from datetime import datetime, timezone
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.persistent_metrics import PersistentMetricsState
|
|
|
|
FIXED_NOW = datetime(2026, 7, 14, 8, 30, tzinfo=timezone.utc)
|
|
|
|
|
|
def _new_state() -> PersistentMetricsState:
|
|
return PersistentMetricsState(now=lambda: FIXED_NOW)
|
|
|
|
|
|
def test_snapshot_accumulates_request_token_cache_cost_and_waste_metrics() -> None:
|
|
state = _new_state()
|
|
|
|
state.record_request(
|
|
provider="anthropic",
|
|
stack="codex",
|
|
model="claude-test",
|
|
input_tokens=100,
|
|
output_tokens=20,
|
|
attempted_input_tokens=150,
|
|
tokens_saved=50,
|
|
cached=True,
|
|
cache_read_tokens=80,
|
|
cache_write_tokens=40,
|
|
cache_write_5m_tokens=10,
|
|
cache_write_1h_tokens=30,
|
|
uncached_input_tokens=20,
|
|
input_usd=0.4,
|
|
compression_savings_usd=0.2,
|
|
cache_savings_usd=0.1,
|
|
waste_signals={"repetition": 7},
|
|
)
|
|
state.record_failed(provider="anthropic", model="claude-test")
|
|
state.record_rate_limited(provider="anthropic", model="claude-test")
|
|
state.record_cache_bust(tokens_lost=9)
|
|
state.record_cache_miss(provider="anthropic", reason="prefix_change")
|
|
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert snapshot["scope"] == "lifetime"
|
|
assert snapshot["requests"] == {
|
|
"total": 1,
|
|
"cached": 1,
|
|
"failed": 1,
|
|
"rate_limited": 1,
|
|
# Lifetime carries the same splits the Prometheus labels do.
|
|
"failed_by_provider": {"anthropic": 1},
|
|
"rate_limited_by_source": {"headroom": 1},
|
|
"by_provider": {"anthropic": 1},
|
|
"by_stack": {"codex": 1},
|
|
}
|
|
assert snapshot["tokens"] == {
|
|
"input": 100,
|
|
"output": 20,
|
|
"attempted_input": 150,
|
|
"saved": 50,
|
|
"token_savings_percent": pytest.approx(50 / 150 * 100),
|
|
}
|
|
assert snapshot["prefix_cache"]["requests"] == 1
|
|
assert snapshot["prefix_cache"]["hit_requests"] == 1
|
|
assert snapshot["prefix_cache"]["cache_read_tokens"] == 80
|
|
assert snapshot["prefix_cache"]["cache_write_tokens"] == 40
|
|
assert snapshot["prefix_cache"]["cache_hit_rate"] == 100.0
|
|
assert snapshot["prefix_cache"]["ttl_1h_percent"] == 75.0
|
|
assert snapshot["prefix_cache"]["ttl_5m_percent"] == 25.0
|
|
assert snapshot["prefix_cache"]["bust_count"] == 1
|
|
assert snapshot["prefix_cache"]["bust_tokens"] == 9
|
|
assert snapshot["prefix_cache"]["misses_by_reason"] == {"prefix_change": 1}
|
|
assert snapshot["cost"] == {
|
|
"input_usd": 0.4,
|
|
"compression_savings_usd": 0.2,
|
|
# This call records no list ceiling, so the ceiling defaults to the
|
|
# cache-aware figure -- the truthful reading of "these are the same
|
|
# number" for a request with no cache mix to price against. The basis
|
|
# stays `unknown` because the caller named none: a fresh aggregate must
|
|
# not claim its untouched total was list-priced.
|
|
"compression_savings_list_usd": 0.2,
|
|
"savings_basis": "unknown",
|
|
"cache_savings_usd": 0.1,
|
|
}
|
|
assert snapshot["waste_signals"] == {"repetition": 7}
|
|
assert snapshot["by_model"]["claude-test"]["input_tokens"] == 100
|
|
|
|
|
|
def test_snapshot_uses_null_for_ratios_without_a_denominator() -> None:
|
|
snapshot = _new_state().snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert snapshot["tokens"]["token_savings_percent"] is None
|
|
assert snapshot["prefix_cache"]["cache_hit_rate"] is None
|
|
assert snapshot["prefix_cache"]["ttl_1h_percent"] is None
|
|
assert snapshot["prefix_cache"]["ttl_5m_percent"] is None
|
|
|
|
|
|
def test_candidate_models_remain_available_until_the_two_hundred_and_first_model() -> None:
|
|
state = _new_state()
|
|
for index in range(200):
|
|
state.record_request(
|
|
provider="provider",
|
|
stack="stack",
|
|
model=f"model-{index:03}",
|
|
input_tokens=index + 1,
|
|
)
|
|
|
|
persisted = state.to_dict()
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert len(persisted["models"]["tracked"]) == 200
|
|
assert "model-000" not in snapshot["by_model"]
|
|
assert snapshot["by_model"]["other"]["input_tokens"] == sum(range(1, 101))
|
|
|
|
|
|
def test_two_hundred_and_first_model_permanently_compacts_non_top_candidates() -> None:
|
|
state = _new_state()
|
|
for index in range(201):
|
|
state.record_request(
|
|
provider="provider",
|
|
stack="stack",
|
|
model=f"model-{index:03}",
|
|
input_tokens=index + 1,
|
|
)
|
|
|
|
persisted = state.to_dict()
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert len(persisted["models"]["tracked"]) == 100
|
|
assert set(snapshot["by_model"]) == {
|
|
*(f"model-{index:03}" for index in range(101, 201)),
|
|
"other",
|
|
}
|
|
assert snapshot["by_model"]["other"]["input_tokens"] == sum(range(1, 102))
|
|
|
|
|
|
def test_state_normalizes_invalid_values_and_unknown_dimension_labels() -> None:
|
|
state = PersistentMetricsState(
|
|
{
|
|
"requests": {"total": "not-a-number"},
|
|
"tokens": {"input": float("nan"), "output": -3},
|
|
"models": {"tracked": {"unknown": {"input_tokens": "7"}}},
|
|
},
|
|
now=lambda: FIXED_NOW,
|
|
)
|
|
state.record_request(
|
|
provider=" ",
|
|
stack=None,
|
|
model=" ",
|
|
input_tokens=-1,
|
|
output_tokens=float("inf"),
|
|
waste_signals={"unrecognized": 9},
|
|
)
|
|
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert snapshot["tokens"]["input"] == 0
|
|
assert snapshot["tokens"]["output"] == 0
|
|
assert snapshot["requests"]["by_provider"] == {"other": 1}
|
|
assert snapshot["requests"]["by_stack"] == {"other": 1}
|
|
assert snapshot["by_model"]["other"]["input_tokens"] == 7
|
|
assert snapshot["waste_signals"] == {"unknown": 9}
|
|
|
|
|
|
def test_json_bloat_waste_signal_is_a_named_bucket() -> None:
|
|
"""The parser emits ``json_bloat`` (WasteSignals.to_dict); it must not fall to ``other``."""
|
|
state = _new_state()
|
|
state.record_request(
|
|
provider="anthropic",
|
|
stack="claude",
|
|
model="claude-sonnet-5",
|
|
waste_signals={"json_bloat": 500, "html_noise": 20},
|
|
)
|
|
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert snapshot["waste_signals"] == {"json_bloat": 500, "html_noise": 20}
|
|
|
|
|
|
def test_state_canonicalizes_lifetime_waste_signal_aliases_and_fallbacks() -> None:
|
|
state = PersistentMetricsState(
|
|
{
|
|
"waste_signals": {
|
|
"json_noise": 10,
|
|
"json_bloat": 20,
|
|
"other": 7,
|
|
"unrecognized_persisted": 3,
|
|
},
|
|
},
|
|
now=lambda: FIXED_NOW,
|
|
)
|
|
state.record_request(
|
|
provider="openai",
|
|
stack="codex",
|
|
model="gpt-test",
|
|
waste_signals={"json_bloat": 5, "unrecognized_runtime": 2},
|
|
)
|
|
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert snapshot["waste_signals"] == {"json_bloat": 35, "unknown": 12}
|
|
|
|
|
|
def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:
|
|
state = PersistentMetricsState(
|
|
{"prefix_cache": {"misses_by_reason": {"ttl_expiry": 2, "bogus": 1}}},
|
|
now=lambda: FIXED_NOW,
|
|
)
|
|
|
|
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
|
|
|
|
assert snapshot["prefix_cache"]["misses_by_reason"] == {"ttl_expiry": 2, "unknown": 1}
|
|
|
|
|
|
def test_rate_limited_is_split_by_source_and_failures_by_provider() -> None:
|
|
"""Headroom's own limiter and an upstream 429 must stay distinguishable.
|
|
|
|
Both land in the same ``rate_limited`` total (unchanged), but an operator
|
|
acts on them differently: ours firing means raise the cap, the provider's
|
|
means back off. See issue #3696.
|
|
"""
|
|
state = _new_state()
|
|
|
|
state.record_rate_limited(provider="anthropic", source="headroom")
|
|
state.record_rate_limited(provider="anthropic", source="upstream")
|
|
state.record_rate_limited(provider="openai", source="upstream")
|
|
state.record_failed(provider="anthropic")
|
|
state.record_failed(provider="openai")
|
|
state.record_failed(provider="openai")
|
|
|
|
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
|
|
|
|
assert requests["rate_limited"] == 3
|
|
assert requests["rate_limited_by_source"] == {"headroom": 1, "upstream": 2}
|
|
assert requests["failed"] == 3
|
|
assert requests["failed_by_provider"] == {"anthropic": 1, "openai": 2}
|
|
|
|
|
|
def test_rate_limit_source_defaults_to_headroom_and_clamps_unknown_values() -> None:
|
|
state = _new_state()
|
|
|
|
state.record_rate_limited(provider="anthropic")
|
|
state.record_rate_limited(provider="anthropic", source="bogus")
|
|
|
|
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
|
|
assert requests["rate_limited_by_source"] == {"headroom": 2}
|
|
|
|
|
|
def test_pre_3696_state_loads_without_backfilling_invented_history() -> None:
|
|
"""A state file written before the split keeps its totals and starts the maps empty.
|
|
|
|
Back-filling the legacy ``rate_limited`` total into one bucket would claim we
|
|
observed a split we never recorded.
|
|
"""
|
|
state = PersistentMetricsState(
|
|
{"requests": {"total": 10, "failed": 4, "rate_limited": 7}},
|
|
now=lambda: FIXED_NOW,
|
|
)
|
|
|
|
requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
|
|
|
|
assert requests["rate_limited"] == 7
|
|
assert requests["failed"] == 4
|
|
assert requests["rate_limited_by_source"] == {}
|
|
assert requests["failed_by_provider"] == {}
|