Commit Graph
17 Commits
Author SHA1 Message Date
eb9932a671 fix(dashboard): lifetime metrics and refresh (#2374)
## Description

Unifies Dashboard Lifetime and Historical refresh behavior behind one
active-view refresher and canonicalizes durable lifetime waste-signal
names so persisted `json_noise` data is normalized to the current
`json_bloat` key.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Set dashboard stats polling to 15 seconds and use one active-view
refresh path for Lifetime and Historical views.
- Keep Lifetime and Historical view refreshes at 30 seconds, with
immediate refresh on view switch or keyboard force refresh.
- Normalize persisted waste-signal aliases from `json_noise` to
`json_bloat` and bucket unknown waste signals as `unknown`.
- Bump savings history schema to 6 so existing persisted data is
rewritten with canonical waste-signal keys.
- Add and update tests for waste-signal normalization, schema migration,
dashboard polling markup, and schema-version responses.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run --extra dev pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_dashboard_stats_cache.py -q
23 passed, 1 skipped, 1 warning

uv run --extra dev pytest tests/test_proxy_project_savings.py::test_funnel_attributes_savings_from_context_and_stats_exposes_them -q
1 passed, 1 warning

uv run --extra dev pytest tests/test_proxy_savings_history.py -q -k "not save_fsyncs_parent_directory"
41 passed, 1 deselected, 1 warning

uv run --extra dev ruff check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
All checks passed!

uv run --extra dev ruff format --check headroom/config.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py tests/test_persistent_metrics.py tests/test_persistent_metrics_persistence.py tests/test_proxy_savings_history.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_project_savings.py
8 files already formatted

git diff --check headroomlabs/main...HEAD
passed
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.3, local review worktree.
- Exact command / steps: Ran the focused pytest and Ruff commands listed
above against the dashboard, persistent metrics, savings history, and
project savings surfaces touched by this PR.
- Observed result: Focused tests passed, Ruff passed, and whitespace
checks passed. Full `tests/test_proxy_savings_history.py` on Windows
still has the existing directory-fsync assertion issue in
`test_savings_tracker_save_fsyncs_parent_directory`, so the broader
savings-history run excluded only that known platform-specific
assertion.
- Not tested: Browser-driven dashboard interaction; full repository test
suite; full `mypy headroom`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation is not needed for this internal dashboard polling and
persistence-normalization fix.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2374"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `2aaeeeb96e8aabd45a7cbd7128227c0714c47df1`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean current-main0ddaa56c integration; final diff remains seven
lifetime-metrics/dashboard source/test files. Fresh325tests
pass,12skip,1warning, including ALL16browsercases (zero browser skips)
using existing Playwright1.59.0 overlay and installed Chrome in offline
contexts with service workers blocked. No package/browser installation.
Six changed Python files pass Rufflint/format; candidate offline lock
and whitespace pass. Full native
trackedinputset/everyhash/artifactchecksum verified before3982DLL
reuse,notfreshbuild.

Actual temporary-file load/save/reload proof: prior-schema alias-bearing
file is rewritten at construction to canonical json_bloat30/unknown10
and survives reopening; current-schema file normalizes in memory without
immediate rewrite, then persists canonical data on next record;
stateless prior-schema file remains byte-identical. Do not claim every
current-schema alias file is rewritten immediately. Actual
browser-rendered lifetime counters also show canonical
json_bloat/unknown. Fresh Chrome screenshots visually inspected. Strict
synthetic request fulfillment blocks all other requests, context
offline, no page errors/unexpected requests. Browser interactions
demonstrate no lazy lifetime/history fetch before selecting a view,
active-view refresh after virtual deadline, keyboard forced refresh and
refresh on tab reentry; synthetic counters visibly update. Observed
configuration: statsPollMs5000 and viewPollMs30000 (5-second session
polling,30-second active lifetime/history refresh); earlier15-second
description is not the implemented interval. Virtual clock and synthetic
responses are not production timing/server/billing proof. Initial
missing default Playwright package/browser revision and wrong 'History'
selector were harness issues; corrected to existing overlay, installed
Chrome and actual 'Historical' button. No PR failure attributed to those
attempts.

Skip qualification:11optional-provider cases plus1stale throughput test.
Narrow rerun5pass/1skip reports missing server._throughput_cache export;
generic skip text incorrectly calls it unbuilt Rust core. Server bytes
and throughput test AST match current main, so this is an inherited
test-API limitation, not this PR's introduced defect or native-build
failure. No unrelated throughput behavior rewritten.

Read author/reviewer and outstanding Devin discussion; fresh
disk/browser evidence addresses prior proof gaps, and polling interval
is qualified accurately above. No new unreviewed external findings at
prepublication audit. Source/test delta has no
workflow/setup/dependency/native/CLI paths; publication security gate
passes. Exact published-head CI/Actions/rollup, current-main
ancestry/mergeability and fresh discussion remain before approval.
Maintainers handle CI; no PR merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


<!-- maintainer-refresh-12864fc8f01a1ca6c58440eccef0dabdd3e595b8 -->

Maintainer current-main refresh: exact head
`abbf9f0744ce7faeb5f46513b6a535b9f5f4c5ee`, authoritative main
`12864fc8f01a1ca6c58440eccef0dabdd3e595b8`. Clean integration patches
exactly match incoming main; contributor behavior preserved. Fresh
focused source tests plus tokenizer/response-header/HF/JSON-parser
incoming-main controls: 109 passed, 1 skipped, 1 warning in 12.17s.
Ruff/format/whitespace pass. Dashboard/template/browser-test inputs are
byte-identical to the prior reviewed head; prior offline Chrome
interaction evidence remains historical, not a fresh browser run. The
one throughput test skip retains the inherited missing
server._throughput_cache API limitation, not an unbuilt-core claim.
Native artifact reuse verified against full recorded input set/every
hash and artifact checksum; no fresh native build claimed. Static
publication gate excludes inherited workflow-validation/native
Docker/Init/Wrap installer paths, with no
workflow/setup/dependency/native/CLI changes in PR diff. Actual
exact-head hosted workflows and final independent review remain pending.
No merge performed.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
2026-11-07 13:35:26 -06:00
Rod BoevandJD Davis 6af7efe0d1 feat(dashboard): give savings metrics one canonical home (#3320)
## Description

Repeated dashboard totals made the same metric occupy several visual
homes. This change gives session savings, observed TTL token totals, and
lifetime history totals one canonical home while retaining useful
decompositions, request counts, performance ranges, checkpoints, and
rollups.

Refs #960

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed repeated session and trend totals while retaining the
before/removed/after breakdown and the proxy share ratio beside Proxy
Removed.
- Made Bucket Mix the sole home for observed 1h and 5m token totals; the
adjacent cards show request counts.
- Decoupled TTL-section availability from positive bucket token totals,
so request-count cards render when a partial payload reports positive
request counts even when token totals are zero; zero-request payloads
remain hidden while Bucket Mix continues to read the canonical
token-total fields.
- Removed the duplicate latest total from Historical Summary; lifetime
tiles, checkpoints, and rollups remain.
- Added rendered assertions for visible values, retired duplicate
content, preserved performance ranges and empty states, and responsive
screenshots of the changed Session, TTL, and Historical Summary surfaces
in both themes at 1280, 768, and 400 pixels.

## Testing

- [x] Focused dashboard browser and metric regression tests ran on
Python 3.12.
- [x] Responsive coverage exercised the changed dashboard surfaces at
1280, 768, and 400 pixels in both themes.
- [x] Ruff and formatting checks ran on the refreshed branch.

```text
uv run --python 3.12 --with pytest --with pytest-asyncio --with playwright python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_dashboard_cache_net_playwright.py tests/test_dashboard_cache_ttl_playwright.py tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_token_savings.py -q
24 passed, 3 skipped in 38.96s
uv run ruff check .
All checks passed!
uv run ruff format . --check
1597 files already formatted
```

risk accepted: Render lint still flags two 400px `container-escape`
warnings inside the horizontally scrollable Recent Requests table. The
warnings mark that table's scroll boundary; the table is outside this
change, and nothing in it changed. CI reruns on the refreshed PR head.

## Real Behavior Proof

- Environment: Windows, Python 3.12.13, headless Chromium,
fixture-backed dashboard data.
- Exact command / steps: rendered Session, TTL, and History with
populated dashboard fixtures in light and dark themes; checked the
single total-wire headline, canonical TTL totals, request counts,
performance ranges, and history checkpoints at 1280, 768, and 400
pixels. The template regression also checks that the “Of new input” text
retains its positive-token visibility gate.
- Observed result: 24 tests passed and 3 tests were skipped. The
headline regression confirms the duplicate “Of total wire” caption is
absent; the template-level check confirms the “Of new input” text and
positive-token gate remain. The responsive case retains the `143.0k`
total, `91.0k` Proxy Removed, `23.5% of before-compression tokens`,
`235.0k` and `185.0k` Bucket Mix totals, `24` and `18` request counts,
`5 - 43ms` Overhead Range, and `0.42 - 2.90s` TTFB Range; `Latest total`
remains absent and checkpoint details remain visible.
- Screenshots: the gallery below compares the same fixture before and
after in both themes. The responsive test exercises all three widths.
- Not tested: live provider traffic

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: normal dashboard deployment
- Stable/default behavior changed: canonical metric placement and
duplicate-value presentation change; API and persistence contracts
remain unchanged
- Kill switch / disable path: revert the dashboard template change or
deploy the previous dashboard version
- Unsafe override required: no
- Qualification impact: no migration, configuration change, or client
qualification is required
- Rollback path: revert the template change; no data repair or migration
is required.

## Screenshots

The before/after images use the same fixture at 1200 × 600. The
responsive test also covers Session, TTL, and History at 1280, 768, and
400 pixels in both themes.

| Surface | Theme | Before | After |
| --- | --- | --- | --- |
| Session | Light | <img alt="Session before, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-light.png"
width="480"> | <img alt="Session after, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-light.png"
width="480"> |
| Session | Dark | <img alt="Session before, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-before-dark.png"
width="480"> | <img alt="Session after, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-session-after-dark.png"
width="480"> |
| TTL | Light | <img alt="TTL before, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-light.png"
width="480"> | <img alt="TTL after, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-light.png"
width="480"> |
| TTL | Dark | <img alt="TTL before, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-before-dark.png"
width="480"> | <img alt="TTL after, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-ttl-after-dark.png"
width="480"> |
| History | Light | <img alt="History before, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-light.png"
width="480"> | <img alt="History after, light theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-light.png"
width="480"> |
| History | Dark | <img alt="History before, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-before-dark.png"
width="480"> | <img alt="History after, dark theme"
src="https://raw.githubusercontent.com/rodboev/headroom/pr-screenshots-headroom-20260827/screenshots/960-3/dashboard-history-after-dark.png"
width="480"> |

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Headroom generates changelog entries from conventional commits. Metric
sources and separate cache-attribution labels remain unchanged.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3320"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 15:55:01 -05:00
ecc496709b fix: surface Codex responses traffic in dashboard (#399)
## Description

Fixes Codex `/v1/responses` traffic not showing up correctly in
Headroom’s dashboard-visible telemetry surfaces.

This branch restores Python-side fallback handling for OpenAI/Codex
Responses API traffic so that when the Python proxy handles
`/v1/responses` directly, request compression + telemetry are still
recorded instead of appearing as pass-through /
 zero-savings traffic.

## Problem

Issue: #310

Codex traffic over `/v1/responses` was reaching Headroom, but
dashboard-visible request surfaces could stay stale or misleading
because:

- Python fallback handling for `/v1/responses` did not properly compress
Responses-shaped input
- WebSocket `response.create` traffic was not consistently turned into
request log entries comparable to other paths
- Codex tool-output item types such as `local_shell_call_output` and
`apply_patch_call_output` were not treated as compressible tool content
in the Python fallback path

Result:
- real Codex traffic could flow through Headroom
- compression savings could remain `0`
- recent request telemetry could be incomplete or misleading for
`/v1/responses`

## Changes Made

### Proxy behavior
- Re-enabled Python fallback compression for `/v1/responses`
- Convert Responses API item input into chat-style messages before
compression
- Reconstruct Responses API items after compression before forwarding
upstream
- Compress first WebSocket `response.create` frames for Python-handled
`/v1/responses`
- Record request telemetry for these Responses API paths so
dashboard-visible request surfaces reflect Codex traffic

### Responses item handling
- Added `headroom/proxy/responses_converter.py`
- Supports conversion/reconstruction for Responses API payloads
- Treats these output item types as compressible tool content:
  - `function_call_output`
  - `local_shell_call_output`
  - `apply_patch_call_output`

### Tests
Added/updated regression coverage for:
- HTTP `/v1/responses` compression path
- WebSocket `/v1/responses` lifecycle + telemetry path
- Responses item conversion/reconstruction behavior

## Files

- `headroom/proxy/handlers/openai.py`
- `headroom/proxy/responses_converter.py`
- `tests/test_openai_codex_routing.py`
- `tests/test_openai_codex_ws_lifecycle.py`
- `tests/test_responses_converter.py`

## Testing

- [x] Focused Responses HTTP/WebSocket tests pass
- [x] Current-main dashboard and compression regressions pass

### Test Output

Ran:

```bash
HEADROOM_REQUIRE_RUST_CORE=false .venv/bin/python -m pytest \
  tests/test_responses_converter.py \
  tests/test_openai_codex_ws_lifecycle.py \
  tests/test_openai_codex_routing.py -q
```
Result:

 ```text
21 passed
 ```

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Real Behavior Proof

- Environment: current-main reconciled OpenAI Responses proxy and
dashboard test environment.
- Exact command / steps: ran focused Responses routing/WebSocket tests
and current compression-unit, dashboard-cache, and savings-history
regressions; rendered the dashboard screenshot artifact.
- Observed result: Responses traffic contributes compression and request
telemetry, historical items remain compressible while the current user
turn is protected, and dashboard session data refreshes correctly.
- Not tested: a long-running production Codex session under sustained
WebSocket traffic.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Kayzo <kayzo@users.noreply.github.com>
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 22:05:58 -05:00
Tejas Chopra e0ce4b1d48 fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description

Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.

The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:

| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |

Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.

**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.

**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.

**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.

**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!

$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted

$ mypy headroom/
Success: no issues found in 508 source files

$ pytest tests/test_context_tool_cleanup.py -q
11 passed

$ pytest tests/test_fsutil.py -q
12 passed

$ pytest tests/test_cli/test_wrap_codex.py -q            # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```

Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.

Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.

## Real Behavior Proof

- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**

```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.

$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...

# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999']   rc=1  ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk']           rc=1  ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh   # syntax OK

# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
  removed ~/.headroom/bin/lean-ctx        (51 MB)
  removed ~/.headroom/bin/rtk             (7.7 MB)
  removed ~/.local/bin/rtk                (symlink into ~/.headroom/bin)
  removed ~/.claude/hooks/rtk-rewrite.sh
  removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched

# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}}     # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk

# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help   → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT

# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90;  projects 19 -> 19;  LOST keys: none
all content outside mcpServers byte-identical: True
```

Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.

- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.

**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.

**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.

**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
2026-07-30 22:59:41 -07:00
Tejas Chopra 1d79e70f95 fix(tests): repair three main-branch test failures (#2306)
## Description

`main` CI is red on three independent test failures. All three are
**test-side** bugs (stale cache, semantic merge conflict, stale mock) —
no product code regressed. Each test passed in isolation but failed on
`main`, and each also blocks the `chore: release main` PR (#1923).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **`test_l2_appends_transform_label`** — `tool_desc_max_chars()`
memoises into a module global. An earlier test in shard 1 reads it with
the env unset, pinning the cache to `0`, so this test's
`setenv("HEADROOM_TOOL_DESC_MAX_CHARS=20")` was swallowed (`assert 0 ==
20`). Reset the cache before reading and after, mirroring the sibling
`test_l2_skips_label_when_disabled`.
- **`test_dashboard_uses_cached_stats_and_lazy_history_feed_polling`** —
semantic merge conflict: #2198 (persist lifetime metrics) intentionally
retired the session-card `Filtered (lifetime)` row and moved
CLI-filtering lifetime into the history tab as `Lifetime Saved`, while
the assertion from #1433 still checked the old string. Assert the
current `Lifetime Saved` label.
- **`test_smart_crusher_log_fallback_runs_for_valid_json`** — stale
mock: #1857 made token counting whitespace-aware, so the router now
rates the JSON above the naive `len(content.split())==8` the no-op
kompress mock reported, making it look like a saving and
short-circuiting before the Log fallback. Mock now reports
`_estimate_tokens(content)` to match the router.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)

### Test Output

```text
$ pytest tests/test_anthropic_compaction_transforms.py \
         tests/test_proxy_dashboard_stats_cache.py \
         tests/test_transforms_content_router.py -q
78 passed, 1 skipped in 12.14s

$ ruff check <the three files>
All checks passed!
$ ruff format --check <the three files>
3 files already formatted
```

## Real Behavior Proof

- Environment: local `.venv`, Python 3.12.6, pytest 9.0.2 (same three
tests that fail on the `main` CI shards 1/3/4).
- Exact command / steps: ran the three previously-failing tests by node
id — all pass. Reproduced the shard-isolation failure for #1 by calling
`tool_desc_max_chars()` with the env unset (cache → 0) before the test,
confirmed the reset makes it pass.
- Observed result: 3/3 target tests pass; 78 passed / 1 skipped across
the three full files.
- Not tested: full suite (unchanged product code); CI shards will re-run
on this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

`mypy headroom` (the CI-enforced scope) is unaffected — these edits
touch only `tests/`, which CI does not type-check. Once this lands on
`main`, the `chore: release main` PR (#1923) drops to just the
`test_root_server_json_matches_builder` failure, which is the release
version-bump `server.json` regen (not a code bug).
2026-07-16 09:21:41 -07:00
Rod Boev 361adcd1a0 fix(dashboard): distinguish unavailable RTK from zero stats in Docker (#1901)
## Description

Dockerized Headroom shows `0` for RTK/context-tool dashboard figures
whenever the `rtk` binary isn't reachable inside the proxy's runtime —
indistinguishable from "genuinely nothing saved yet." The backend
already computes this distinction (an `installed`/`available` flag on
the context-tool stats payload) but it never reaches two of the JSON
surfaces the dashboard reads from, and the dashboard template never
checks the one surface that already has it. This PR threads that
existing availability flag through to both surfaces and updates the
dashboard to show a distinct "not installed" message instead of a bare
`0`, plus a short Docker note so operators know `rtk` needs to be
installed inside the container for those figures to populate at all.

Closes #1831

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/server.py`: reuse the existing context-tool
`installed` flag as one `available` boolean, add it to
`savings.by_layer.cli_filtering` in `/stats`, and add it to the curated
`cli_filtering` block in `/stats-history`; corrected that endpoint's
stale docstring claim that `cli_filtering` is `None` whenever RTK is
absent.
- `headroom/dashboard/templates/dashboard.html`: added
`cliFilteringAvailable`/`historyCliFilteringAvailable` getters and used
them to show a "not installed" message instead of `0` in the session
view's Token Usage panel and Token Savings breakdown, and to keep the
Historical tab's lifetime card hidden (its existing behavior) instead of
showing a stale zero.
- `docker-compose.yml` and `docker/docker-compose.native.yml`: added a
one-line comment noting that `rtk` needs to be installed inside the
container for CLI-filtering dashboard figures to populate.
- `docs/content/docs/docker-install.mdx`: added a note to the existing
Notes section about the same requirement.
- Added focused pytest coverage for the new JSON field on both endpoints
(installed, not-installed, and hard-failure cases) and a new Playwright
spec covering the rendered not-installed / genuine-zero / Historical-tab
states.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_dashboard_stats_cache.py
tests/test_proxy_savings_history.py -q`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`) or explain N/A
truthfully
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed

### Test Output

```text
uv run pytest tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_savings_history.py -q
51 passed, 1 skipped, 1 failed

uv run ruff check headroom/proxy/server.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_savings_history.py tests/test_dashboard_context_tool_availability_playwright.py
All checks passed!
```

The one failure (`test_savings_tracker_save_fsyncs_parent_directory`) is
pre-existing and unrelated to this change; it reproduces identically on
a clean `origin/main` checkout with this diff removed (Windows
filesystem fsync behavior).

## Real Behavior Proof

- Environment: Windows sandbox, Python (uv-managed), no live Docker
container
- Exact command / steps: `GET /stats` and `GET /stats-history` against a
`TestClient` app with the context-tool stats source monkeypatched to a
not-installed payload (mirrors the exact shape
`_context_tool_zero_payload` produces when `rtk` is absent), then the
same with an installed-but-zero payload
- Observed result: `savings.by_layer.cli_filtering.available` and
`/stats-history`'s `cli_filtering.available` are `False` for the
not-installed payload and `True` for the installed-but-zero payload,
matching the pre-existing `context_tool.available` field; the new
Playwright spec exercises the corresponding dashboard rendering states
and runs in CI's "Dashboard Playwright" check
- Not tested: real rendering in a live browser against a live Docker
container (this sandbox cannot run the CI-only Dashboard Playwright job
locally); the fix is proved locally at the JSON-contract level and the
rendering claim is proved by the contributed CI-executed Playwright spec

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

CHANGELOG.md was intentionally left unchanged — release automation
derives changelog entries from conventional commits per this repo's
convention, and this is a dashboard/docs clarity fix rather than a new
user-facing command or config option. Type checking was not re-run in
isolation for this change; it's covered by the repo's CI lint job.
2026-07-09 09:39:16 -04:00
Vinay Gupta 38074888ac fix(docker): report source build version (#1862)
## Description

Closes #1858

Docker/Compose source builds could report stale or misleading version
information: the dashboard initially rendered a hardcoded `v0.3.0`, then
`/health` replaced it with installed package metadata, which can be
stale when building locally from `main` without release metadata in the
image.

This change makes source Docker Compose builds report an explicit
source-build identity, removes the stale dashboard fallback, and keeps
CLI/doctor version checks from treating source-build labels as
release-version drift.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `HEADROOM_VERSION` / `HEADROOM_BUILD_VERSION` runtime version
overrides and optional packaged `_build_info.py` metadata.
- Teach Docker Compose source builds to pass a `source-build` sentinel
that the Dockerfile expands to `source-build+g<sha>` when git metadata
is available, or `source-build+sha256.<digest>` otherwise.
- Keep release/published image builds on normal package metadata when
`HEADROOM_BUILD_VERSION` is unset.
- Include only minimal `.git` metadata in the Docker build context so
the source-build label can identify the checkout without copying git
objects.
- Treat source-build labels and raw hashes as non-release labels in
`wrap` and `doctor`, avoiding false stale-proxy restarts and drift
warnings.
- Replace the dashboard hardcoded `0.3.0` fallback with `loading` /
`unknown` and format non-release build labels without a `v` prefix.
- Include the runtime version in proxy startup logs, `/health`,
`/livez`, and OTEL service version reporting.

## Testing

- [x] Unit tests pass (`pytest` in GitHub CI)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
GitHub CI: all checks passing
- CI: build, build-wheel, lint, test shards, test-extras, test-agno, test-dashboard-ui
- Docker: docker-native-e2e, docker-wrap-e2e, docker-init-e2e
- Native wrappers: macOS, Windows, Ubuntu
- Security: CodeQL, gitleaks, pip-audit
- Governance: template, label, merge-conflicts, commitlint

$ HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1858-version-mismatch pytest tests/test_package_init_lazy.py::test_version_prefers_explicit_build_env tests/test_package_init_lazy.py::test_version_label_helpers_only_prefix_release_versions tests/test_package_init_lazy.py::test_version_uses_packaged_build_metadata tests/test_package_init_lazy.py::test_observability_version_uses_runtime_version tests/test_docker_compose_persistence.py tests/test_cli_doctor.py::TestProxyLiveness::test_up_leaves_source_label_unprefixed tests/test_cli_doctor.py::TestVersionDrift::test_non_release_version_labels_skip_drift_comparison tests/test_cli/test_wrap_persistent.py::test_proxy_version_restart_ignores_non_release_source_labels tests/test_proxy_dashboard_stats_cache.py::test_dashboard_uses_cached_stats_and_lazy_history_feed_polling -q
13 passed, 1 warning

$ uvx ruff==0.15.17 check .
All checks passed!

$ uvx ruff==0.15.17 format --check .
1058 files already formatted

$ uvx mypy==1.20.2 headroom --ignore-missing-imports
Success: no issues found in 407 source files

$ git diff --check
# no output

$ docker compose config
# resolved headroom-proxy build args include HEADROOM_BUILD_VERSION: source-build

$ HEADROOM_BUILD_VERSION=6266a1d docker compose config
# explicit override is preserved as HEADROOM_BUILD_VERSION: 6266a1d

$ docker build --check --build-arg HEADROOM_BUILD_VERSION=source-build .
Check complete, no warnings found.
```

## Real Behavior Proof

- Environment: macOS local checkout, Python 3.13.5, Docker Desktop
builder `desktop-linux`, plus GitHub Actions CI.
- Exact command / steps: `docker compose config`,
`HEADROOM_BUILD_VERSION=6266a1d docker compose config`, and `docker
build --check --build-arg HEADROOM_BUILD_VERSION=source-build .`.
- Observed result: Compose defaults the top-level `headroom-proxy` build
arg to the `source-build` sentinel, preserves explicit overrides, and
Dockerfile syntax/check validation passes for the source-build path.
- Not tested: Full end-to-end release publishing flow; this PR only
changes local/source-build reporting.
- CI proof: GitHub Actions completed successfully across Docker E2E, CI
test shards, lint/type checks, native wrapper checks, security checks,
and PR governance.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally/CI with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Docs and changelog are N/A for this runtime-reporting bug fix. The PR is
open and ready for review with all GitHub checks passing.
2026-07-08 13:32:04 -05:00
Rod Boev 88f935a1eb fix(dashboard): deduplicate repeated savings metrics (#1804)
## Description

The session dashboard repeats the same savings and performance numbers
in adjacent places. `proxy_compression_saved` appears in several
captions and detail rows, and average overhead and TTFB appear both in
the hero area and again in Performance without adding new context.

This narrows the non-hero dashboard presentation so repeated session
metrics have one visible home plus decomposition where it adds
information. It leaves `/stats`, savings math, cache attribution, and
the hero proxy savings card unchanged.

Refs #960

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed redundant non-hero session-view captions that restated
proxy-compression token counts without adding a new dimension.
- Kept canonical homes for proxy compression and token usage details.
- Preserved Performance range context while avoiding adjacent
restatement of hero averages.
- Added a static dashboard regression for repeated session metrics.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_dashboard_stats_cache.py -q`)
- [x] Linting passes (`uv run ruff check
tests/test_proxy_dashboard_stats_cache.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_proxy_dashboard_stats_cache.py -q
12 passed, 1 skipped, 1 warning in 19.24s

$ uv run ruff check tests/test_proxy_dashboard_stats_cache.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows, Python environment from `uv sync --extra dev`,
browserless dashboard HTML inspection.
- Exact command / steps: load `get_dashboard_html()` in the focused
dashboard stats test and assert removed duplicate captions stay removed
while canonical metric owners remain present.
- Observed result: session-view repeated savings and performance labels
no longer duplicate the same numbers without context.
- Not tested: full browser screenshot and history-view de-duplication.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

No `CHANGELOG.md` edit: this repo generates changelog entries from
conventional commits. This intentionally avoids the hero proxy savings
card already covered by #927 and #1649, and it does not fold provider
cache discount into Headroom-value savings.
2026-07-05 16:00:25 -07:00
gglucassandClaude Fable 5 2fe19c39e4 feat(stats): surface Codex WS compression counters in /stats summary (#1680)
## Description

Codex rides a long-lived WebSocket `/responses` connection. WS units are
compressed and counted into the `codex_ws_*` metrics immediately, but
turn-level records — the ones that feed `tokens_saved_total` and
therefore the `/stats` `summary` block — only land when a
`response.completed` frame carries usage tokens. A user watching
`summary.api_requests` / `summary.compression` during an active Codex WS
session sees frozen counters and concludes Headroom isn't working, even
though the `codex_ws` stats section is advancing. (Reported by a
Headroom Desktop user who cross-checked `/stats` against a healthy proxy
and confirmed-correct Codex routing.)

This PR surfaces the live per-unit counters inside `summary` so WS-only
sessions are visible at a glance:

```json
"codex_ws": {"units_total": 12, "units_modified": 9, "tokens_saved": 4321}
```

The block is deliberately **not** summed into
`compression.total_tokens_removed`: turns that did record already
contributed the same savings to `tokens_saved_total`, and the
recorded-vs-unrecorded split is not tracked globally, so folding the
unit sums into the totals would double-count. Additive visibility, not a
second ledger.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/cost.py`: `build_session_summary` emits a
`summary.codex_ws` block (`units_total`, `units_modified`,
`tokens_saved`) sourced from the live per-unit metrics; only present
when `codex_ws_units_total > 0`, so non-Codex sessions keep the existing
summary shape. `getattr` defaults keep older/partial metrics objects
working.
- `tests/test_proxy_dashboard_stats_cache.py`: new
`test_session_summary_surfaces_codex_ws_counters`; extended
`test_session_summary_uses_generic_cli_filtering_keys` to assert the
block is absent when counters are missing.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev pytest tests/test_proxy_dashboard_stats_cache.py
=================== 11 passed, 1 skipped, 1 warning in 3.47s ===================

$ uv run --extra dev pytest tests/test_compression_observability.py tests/test_proxy_healthchecks.py tests/test_pr208_changes.py
======================== 72 passed, 1 warning in 32.18s ========================

$ uv run --extra dev mypy headroom/proxy/cost.py
Success: no issues found in 1 source file

$ ruff check headroom/proxy/cost.py tests/test_proxy_dashboard_stats_cache.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS 15 (Darwin 24.6.0), Python 3.10 venv via `uv`,
branch `fix/stats-summary-codex-ws` @ upstream main
- Exact command / steps: called `build_session_summary` with metrics
carrying `codex_ws_units_total=12`, `codex_ws_units_modified_total=9`,
`codex_ws_unit_tokens_saved_sum=4321` (same shape `create_app` passes at
`/stats`), printed `summary["codex_ws"]`
- Observed result: `{"units_total": 12, "units_modified": 9,
"tokens_saved": 4321}`; with counters absent, `"codex_ws" not in
summary`
- Not tested: end-to-end `/stats` against a live Codex WS session on
this build (the installed desktop bundle runs 0.28.0, which predates
this branch); unit path is identical since `/stats` calls
`build_session_summary` with the live metrics object

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Documentation / CHANGELOG unchecked: `/stats` response fields aren't
documented per-key, and CHANGELOG did not appear to track additive stats
fields — happy to add either if maintainers want it.
- Follow-up candidate (out of scope here): fold WS savings into the
compression *totals* correctly by tracking a
`codex_ws_tokens_saved_recorded_total` at turn-record time, so the
unrecorded remainder could be added without double-counting.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 14:25:24 -07:00
Tejas Chopra 93627471b7 fix(perf): surface RTK/CLI context-tool savings in perf and the session card (#1433)
## Description

`headroom perf` read only `proxy.log` compression records, so RTK's
savings — which live in RTK's own lifetime counter and never land in
`proxy.log` — were **invisible**: perf reported "token savings" while
silently dropping the entire CLI-filtering layer. The dashboard
**Session** card likewise showed only the session-delta (≈0 right after
a proxy restart), with no scope label and no lifetime figure.

This surfaces RTK lifetime savings in `headroom perf` (text + JSON) and
clarifies the dashboard Session card. It complements #1324 (which added
RTK to the Historical tab) by covering the two surfaces #1324 didn't:
`perf` and the live Session card.

Closes # N/A — complements #1324; no standalone issue.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/perf/analyzer.py`: `format_report` and `build_perf_summary`
now attach RTK/CLI context-tool **lifetime** savings, sourced
best-effort from `_get_context_tool_stats().lifetime` (the same source
`/stats` and #1324 use). Lifetime — not session — is the right scope for
a one-shot CLI, since the proxy-session baseline `/stats` subtracts is
meaningless out of process. Omitted entirely when no tool is installed
or its stats can't be read, so the report degrades to proxy-only rather
than erroring.
- `headroom/dashboard/templates/dashboard.html`: the Session card now
labels the RTK number **"this session"**, uses the real
`session_savings_pct` (via a new `cliFilteringSessionPctDisplay` getter)
instead of an ad-hoc share, and shows **lifetime** alongside it (new
`cliFilteringLifetime` getter + row, hidden when 0).
- `tests/test_perf_cli_filtering.py` (new): perf surfaces RTK in text +
JSON; omits cleanly when the tool is absent.
- `tests/test_rtk_session_savings.py` (new): exercises the real
`_get_context_tool_stats()` plumbing to pin that session RTK savings are
the **delta from the startup baseline**, and session `savings_pct` is
derived from that delta — not RTK's lifetime-diluted average.
- `tests/test_proxy_dashboard_stats_cache.py`: updated the Session-card
label assertion and added one for the new lifetime row.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/perf/analyzer.py tests/test_perf_cli_filtering.py tests/test_rtk_session_savings.py tests/test_proxy_dashboard_stats_cache.py
All checks passed!

$ mypy headroom/perf/analyzer.py
mypy: No issues found

$ python -m pytest tests/test_perf_cli_filtering.py tests/test_rtk_session_savings.py tests/test_proxy_dashboard_stats_cache.py tests/test_owned_asset_encoding.py -q
17 passed, 1 skipped in 15.66s
```

## Real Behavior Proof

- Environment: macOS (darwin), Python 3.12, branch
`fix/rtk-savings-perf-dashboard`, RTK v0.28.2.
- Exact command / steps: `headroom perf` and `headroom perf --format
json`.
- Observed result: the text report now includes a section
`RTK CLI Filtering (lifetime, all-time) — Tokens saved: 26,867,610
(68.8%), Commands: 8,023`,
and the JSON output carries `"cli_filtering":
{"tool":"rtk","label":"RTK","tokens_saved":26867610,"commands":8023,"savings_pct":68.8}`.
Before this change, both omitted RTK entirely (perf's "Total saved" was
proxy-compression only). The dashboard template renders the new "this
session" / "lifetime" RTK rows (verified via `get_dashboard_html()` +
substring test).
- Not tested: live dashboard browser click-through (template loads and
the new strings are asserted by the substring test); CSV output of
`perf` (per-model table only, by design).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG: left to Release Please (the conventional `fix(perf):`
commit generates the entry on merge), matching how the existing "Bug
Fixes" entries are produced.
- Follow-up: #1403 (`fix/rtk-savings-scope-regression`) bundles
unrelated kompress must-keep work (overlaps #1400/#1419) and only
documents the scope `%` invariant in the abstract. The real,
code-exercising session-delta regression now lives here
(`test_rtk_session_savings.py`), so #1403 can be split — route the
kompress bits to #1400/#1419 and drop the rest.
2026-06-25 21:13:36 -07:00
Shlok TiwariandAntigravity Agent 0d89c674cd feat: measure and surface token throughput (tokens/sec) through the proxy (#983)
## Description

This PR implements measuring and surfacing token throughput
(tokens/second) through the proxy in the `headroom perf` CLI/analyzer
and the dashboard UI. It tracks multiple throughput metrics—Input
(wall-clock/active), Compression, Forward, and Generation
throughput—supporting both rolling percentiles (p50/p95) and current
(last 5 minutes) metrics.

Closes #959

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Proxy Instrumentation (`headroom/proxy/outcome.py`)**: Added
`total_ms`, `tok_out`, and `ttfb_ms` to the structured `PERF` logging
payload.
- **Log Parsing & Computations (`headroom/perf/analyzer.py`)**: Updated
log parsing to read `STAGE_TIMINGS` and correlation fields from `PERF`,
computing active/wall-clock throughputs for input, compression, forward,
and generation stages.
- **API Exposing (`headroom/proxy/server.py`)**: Exposes calculated
rolling throughput percentiles and last-5-minute averages under the
`throughput` field in `/stats`.
- **Dashboard UI Layout
(`headroom/dashboard/templates/dashboard.html`)**: Refactored the
dashboard grid layout from 3 columns to 4 columns to house the new
throughput hero card showing real-time token performance.
- **Verification Tests (`tests/test_cli_perf_format.py`)**: Added test
coverage specifically targeting token throughput log parser extraction,
stage correlation, math correctness, and edge-case handling (empty
fields, division by zero).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss"; .venv\Scripts\pytest tests/test_cli_perf_format.py
============================= test session starts =============================
platform win32 -- Python 3.11.15, pytest-9.1.0, pluggy-1.6.0 -- C:\Users\hp\Desktop\Headroom_oss\.venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\hp\Desktop\Headroom_oss
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 14 items

tests/test_cli_perf_format.py::test_build_perf_summary_totals_and_pct PASSED [  7%]
tests/test_cli_perf_format.py::test_build_perf_summary_by_model_and_transform PASSED [ 14%]
tests/test_cli_perf_format.py::test_build_perf_summary_empty_report_no_zero_division PASSED [ 21%]
tests/test_cli_perf_format.py::test_perf_records_as_dicts_roundtrips_fields PASSED [ 28%]
tests/test_cli_perf_format.py::test_perf_json_format PASSED              [ 35%]
tests/test_cli_perf_format.py::test_perf_json_raw_is_array PASSED        [ 42%]
tests/test_cli_perf_format.py::test_perf_json_raw_preserves_client_field PASSED [ 50%]
tests/test_cli_perf_format.py::test_parse_perf_line_preserves_client_field PASSED [ 57%]
tests/test_cli_perf_format.py::test_perf_csv_by_model PASSED             [ 64%]
tests/test_cli_perf_format.py::test_perf_csv_raw_per_record PASSED       [ 71%]
tests/test_cli_perf_format.py::test_perf_text_default_unchanged PASSED   [ 78%]
tests/test_cli_perf_format.py::test_perf_rejects_unknown_format PASSED   [ 85%]
tests/test_cli_perf_format.py::test_parse_perf_line_preserves_blank_client_field PASSED [ 92%]
tests/test_cli_perf_format.py::test_throughput_parsing_and_calculations PASSED [100%]

============================== warnings summary ===============================
.venv\Lib\site-packages\_pytest\config\__init__.py:1464
  C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\_pytest\config\__init__.py:1464: PytestConfigWarning: Unknown config option: asyncio_mode
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32
  C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
    return EntryPoints(ep for group_eps in eps.values() for ep in group_eps)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================= 14 passed, 2 warnings in 4.77s ========================
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.15
- Exact command / steps: Run the pytest suite against the newly created
token throughput parsing routines:
`$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss";
.venv\Scripts\pytest tests/test_cli_perf_format.py`
- Observed result: The suite executes 14 tests successfully, including
the newly added `test_throughput_parsing_and_calculations` verification
test verifying mathematical precision and fallback logic.
- Not tested: None (all metrics are fully covered by unit tests in
`test_cli_perf_format.py`)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Backwards compatibility: Older log outputs lacking `tok_out` or
`ttfb_ms` parse cleanly and fallback defaults prevent parser crashes.

---------

Co-authored-by: Antigravity Agent <agent@antigravity.local>
2026-06-17 09:42:38 -05:00
skblue b70fccbe17 fix(proxy): read RTK gain stats globally by default (#957)
## Description

Closes #900.

The proxy now reads RTK lifetime savings with global scope by default.
This matches shared daemon deployments where the proxy process cwd is
often `$HOME` or a service directory, while RTK savings are accumulated
across the operator's projects.

`HEADROOM_RTK_GAIN_SCOPE=project` keeps the old `rtk gain --project`
behavior for operators who explicitly want the proxy process working
directory as the scope.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Default RTK stats subprocess command to `rtk gain --format json`
- Add `HEADROOM_RTK_GAIN_SCOPE=project` to opt into `rtk gain --project
--format json`
- Keep fallback/synthetic-zero payload `scope` aligned with the queried
scope
- Deduplicate context-tool zero payload construction
- Document the new RTK gain scope environment variable

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --extra dev python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py -q
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0
rootdir: /Users/joshuasiu/vibe/temp/headroom
configfile: pyproject.toml
plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 26 items

tests/test_proxy_dashboard_stats_cache.py ..........                     [ 38%]
tests/test_subscription_tracker_rtk_wired.py ................            [100%]

============================== 26 passed in 0.40s ==============================

uv run --extra dev python -m pytest tests/test_proxy_stats_recent_requests.py tests/test_proxy_healthchecks.py -q
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0
rootdir: /Users/joshuasiu/vibe/temp/headroom
configfile: pyproject.toml
plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 15 items

tests/test_proxy_stats_recent_requests.py ...                            [ 20%]
tests/test_proxy_healthchecks.py ............                            [100%]

============================= 15 passed in 10.41s ==============================

uv run --extra dev ruff check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py
All checks passed!

uv run --extra dev ruff format --check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py
3 files already formatted

uv run --extra dev mypy headroom
headroom/proxy/server.py:1141: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
headroom/proxy/server.py:1211: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
headroom/proxy/server.py:1215: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
Success: no issues found in 358 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.14.5 via `uv run --extra dev`
- Exact command / steps: mocked RTK subprocess argv in unit tests
- Observed result: default command is `rtk gain --format json`; project
scope command is `rtk gain --project --format json`; invalid scope logs
`event=rtk_gain_scope_invalid` and falls back to global
- Not tested: full repository pytest suite

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The unchecked comment and changelog items are not applicable for this
scoped proxy stats fix.
2026-06-13 21:21:38 -07:00
Tejas Chopra e9cae0131b fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress.
2026-05-12 13:34:08 -07:00
Gili Tzabari 4b061792b2 feat: add lean-ctx context tool support 2026-05-11 17:54:17 -04:00
Tejas Chopra eaf5980b4a fix: stabilize codex compression, stats, and proxy lifecycle 2026-05-09 13:47:53 -07:00
Tejas Chopra 6eec38767b Combine rtk and compression savings in dashboard 2026-05-08 14:46:54 -07:00
Kayzo 2b1ab269ca fix: cache dashboard stats snapshots 2026-04-22 09:30:19 +00:00