Commit Graph
16 Commits
Author SHA1 Message Date
Tejas Chopra f30d330682 refactor(install,subscription): share manifest parsing and warn on corrupt manifests (#4125)
## Description

This PR shares the manifest parsing that was written twice, logs corrupt
manifests instead of skipping them silently, and pins the owner-only
mode of two state files with tests.

- `load_manifest` and `list_manifests` had the same payload-to-manifest
code, including the retired-image migration. They now share
`_manifest_from_payload`.
- `list_manifests` used to skip a corrupt manifest with no log. It now
logs `Skipping corrupt deployment manifest <path>: <error>. Fix or
delete the file to restore that profile.` at warning level. The returned
list is the same as before.
- A manifest that cannot be read (permissions, non-UTF-8 bytes) is no
longer called corrupt. `list_manifests` logs `Skipping unreadable
deployment manifest <path>: <error>. Check the file's permissions and
that it is UTF-8.`, and `load_manifest` raises
`ManifestError("deployment profile '<name>' could not be read (<path>):
<error>. Check the file's permissions and that it is UTF-8.")`. Only a
JSON or schema failure keeps the "is corrupt" text.
- `subscription/client.py` `_credentials_path` reuses
`session_tracking._claude_config_dir()`. Both resolve
`CLAUDE_CONFIG_DIR`, falling back to `~/.claude`, with identical code.
- New tests pin the deployment manifest and the subscription state file
at 0600, both when the file is created and when a 0644 file is
overwritten. The current writers already do this, because the temp file
they rename into place is created 0600.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/install/state.py`: add `_manifest_from_payload`, use it in
`load_manifest` and `list_manifests`, warn on corrupt manifests in
`list_manifests`, and report read failures separately from malformed
JSON in both.
- `headroom/subscription/client.py`: `_credentials_path` uses
`_claude_config_dir()`.
- `tests/test_install/test_state.py`:
`test_list_manifests_ignores_invalid_payloads` now asserts that the
warning names the broken file and does not name the valid one. Add
`test_save_manifest_keeps_manifest_owner_only` and
`test_unreadable_and_malformed_manifests_get_distinct_messages` (a
PermissionError on read and a malformed file get different messages in
`list_manifests` and `load_manifest`).
- `tests/test_subscription_tracker.py`: add
`test_persist_state_keeps_file_owner_only`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_install/test_state.py tests/test_subscription_tracker.py tests/test_subscription_client.py tests/test_subscription_session_tracking.py tests/test_cli/test_install_cli.py tests/test_quota_registry.py tests/test_mcp_registry tests/test_stateless_no_disk.py tests/test_telemetry_context.py tests/test_cli_doctor.py tests/test_fsutil.py
# on origin/main (a9815bbb):
502 passed, 1 warning in 18.30s
# on this branch (2 new mode tests + 1 read-vs-malformed test):
505 passed, 1 warning in 18.29s
# the read-vs-malformed test against the previous commit (828b107a): 1 failed (both called corrupt)
# the 2 mode tests run against a variant that writes both files with fsutil.write_text:
2 failed in 0.76s
$ uvx ruff@0.16.8 check <4 touched files>
All checks passed!
$ mypy headroom/install/state.py headroom/subscription/client.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree.
Temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`.
- Exact command / steps:
1. Shared smoke harness: a real proxy from this checkout against a local
fake upstream (`smoke.sh <worktree>`).
2. Real CLI against two broken profiles in the temp workspace:
`malformed` holds `{not json`, and `unreadable` is a valid manifest with
mode 000. `headroom install status` with each `--profile`, then without
one, which falls back to `list_manifests()`.
3. File modes were measured directly, as before: the manifest and
tracker writers give 0600 on create and 0600 on overwrite of a 0644
file.
- Observed result:

```text
branch: cleanup/cli-integrations-fsutil-state-writes @ 71e669eb
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-50695.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ headroom install status --profile malformed
Error: deployment profile 'malformed' is corrupt ($TMP/ws/deploy/malformed/manifest.json): Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
exit=1
$ headroom install status --profile unreadable
Error: deployment profile 'unreadable' could not be read ($TMP/ws/deploy/unreadable/manifest.json): [Errno 13] Permission denied: '$TMP/ws/deploy/unreadable/manifest.json'. Check the file's permissions and that it is UTF-8.
exit=1
$ headroom install status   # no --profile: falls back to list_manifests()
Skipping corrupt deployment manifest $TMP/ws/deploy/malformed/manifest.json: Expecting property name enclosed in double quotes: line 1 column 2 (char 1). Fix or delete the file to restore that profile.
Skipping unreadable deployment manifest $TMP/ws/deploy/unreadable/manifest.json: [Errno 13] Permission denied: '$TMP/ws/deploy/unreadable/manifest.json'. Check the file's permissions and that it is UTF-8.
Error: No deployment profile named 'default' is installed. No deployments are installed; run `headroom init` or `headroom install apply`.
exit=1
```

- Not tested: Windows, where mode bits do not decide access; the mode
tests skip there.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; internal refactor plus one warning log.
- Stable/default behavior changed: No. The only visible change is a new
warning when a corrupt manifest is skipped.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Not done, on purpose: the cleanup plan proposed replacing
`install/state.py` `_atomic_write_text` and the tracker's
`_persist_state` writer with `fsutil.write_text`. That would weaken file
permissions. `fsutil.write_text` copies an existing file's mode onto the
replacement, so a manifest or subscription state file that is 0644 today
would stay readable by other users. The current writers reset it to 0600
on every write. Both writers stay, and the new tests pin that behaviour.
Two follow-ups for the maintainer:
- `_persist_state` has no fsync and leaves its `.tmp` file behind if
`os.replace` fails.
- Either an owner-only mode option on `fsutil.write_text`, or an atomic
writer in `headroom/fileperms.py`, would let these writers share code.

The new `list_manifests` warning fires once per corrupt file per call.
The callers are CLI commands, doctor, port discovery, and the telemetry
install-mode check. None of them is on the per-request path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-12 18:58:51 -07:00
Tejas ChopraandClaude Opus 5.5 c191beef59 refactor: remove write-only and test-only code in subscription and install (#4123)
## Description

This PR removes private code in `subscription/` and `install/` that
nothing reads.

- `subscription/tracker.py`: `_PERSIST_FILE_ENV`, `_DEFAULT_PERSIST_DIR`
and `_DEFAULT_PERSIST_FILE` are left over from before the path moved to
`paths.subscription_state_path()`. `_full_tokens` was a counter that was
written on every request and never read. It also kept an 8-character
OAuth token prefix in memory.
- `install/runtime.py`: `_process_matches_runtime` was a test-only
wrapper for `_process_identity(...) is True`. Product code calls
`_process_identity` directly.

Evidence: `rg -n '<name>' headroom tests crates plugins sdk examples
benchmarks scripts e2e docs wiki .github pyproject.toml` and the same
search over the consumer repos find only the definitions, plus the tests
changed here. No `getattr` string spelling exists for any of them.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/subscription/tracker.py`: remove the three persist constants
and `_full_tokens`.
- `headroom/install/runtime.py`: remove `_process_matches_runtime`.
- `tests/test_install/test_runtime.py`: the six asserts now check
`_process_identity(...) is True` or `is False` directly.
- `tests/test_subscription_tracker.py`: drop the one assert on
`_full_tokens`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_subscription_tracker.py tests/test_install/test_runtime.py tests/test_network_diff_capture.py tests/test_subscription_window_render.py tests/test_subscription_poll_credential_policy.py tests/test_stateless_no_disk.py tests/test_cli_learn.py
# on origin/main (a9815bbb):
176 passed, 1 warning in 9.92s
# on this branch:
176 passed, 1 warning in 9.52s
$ uvx ruff@0.16.8 check <4 touched files>
All checks passed!
$ mypy headroom/subscription/tracker.py headroom/install/runtime.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree.
Temp `HOME`, `HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched
the real home.
- Exact command / steps:
1. Shared smoke harness: a real proxy from this checkout against a local
fake upstream, with one Anthropic and one OpenAI request carrying a
compressible tool result (`smoke.sh <worktree>`).
2. Lifecycle path that the removed wrapper used to front: saved a
`persistent-task` manifest whose `runner.pid` holds the PID of an
unrelated live process (`sleep 300`), then ran the real CLI `headroom
install status --profile rt`. `runtime_status` goes through
`_process_identity`, which rejects the PID.
3. Tracker path: `SubscriptionTracker().notify_active("Bearer
oauth-token-123", from_local_operator=True)`.
- Observed result:

```text
branch: cleanup/cli-integrations-dead-subscription-install-capture @ 06377018
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-50694.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
# runner.pid holds the PID of an unrelated live process (sleep 300)
$ headroom install status --profile rt
Profile:    rt
Preset:     persistent-task
Runtime:    python
Supervisor: none
Scope:      user
Port:       18999
Status:     stopped
Healthy:    no
exit=0
sleep still alive (status never signalled it)

$ python -c "...notify_active('Bearer oauth-token-123', from_local_operator=True)..."
active: True | token prefix counter present: False
```

- Not tested: a live persistent install under launchd/systemd; this PR
deletes only private names, and `_process_identity` itself is unchanged.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A; dead-code removal.
- Stable/default behavior changed: No.
- Kill switch / disable path: N/A.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Left for the maintainer: public dataclass field.
`CapturedExchange.request_body_preview` in
`headroom/capture/network_diff.py` is filled for every captured exchange
(by `_preview_body`, capped at `MAX_BODY_PREVIEW_CHARS`) and never read;
`CaptureDiff.to_dict` does not serialize exchanges. `CapturedExchange`
is in `headroom.capture.__all__`, so the field and the code that fills
it stay unchanged here.

`SubscriptionTracker.latest_snapshot` is used only by tests, but it
stays: it is a public property, and removing it was optional.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:01:22 -07:00
Tejas ChopraandClaude Opus 5.5 7d9dd55e7c fix(subscription): log a failed OTEL window update instead of passing silently (#4131)
## Description

`SubscriptionTracker._maybe_poll` exports the subscription window to
OTEL after each poll. That export ended in `except Exception: pass`, so
a broken or misconfigured exporter left no trace at all. It now logs
`Subscription OTEL metrics update failed: <error>` at DEBUG. Polling is
unaffected.

This was split out of the mcp-registry-atomic-writes branch, where it
was off theme.

Refs #4136
## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/subscription/tracker.py`: `except Exception: pass` becomes
`except Exception as exc: logger.debug(...)`. The catch stays broad on
purpose: OTEL is optional observability, and no exporter failure may
break subscription polling. It is now logged, not swallowed.
- Tests: new `test_maybe_poll_logs_a_failed_otel_update` in
`tests/test_subscription_tracker.py`. Its exporter raises; the test
asserts the snapshot is kept, `last_error` stays `None`, and the debug
record is written.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest -q -p no:cacheprovider tests/test_subscription_tracker.py tests/test_subscription_contribution.py
16 passed
# New test against origin/main tracker.py:
1 failed (no log record)
$ uvx ruff@0.16.8 check headroom/subscription/tracker.py tests/test_subscription_tracker.py
All checks passed!
$ mypy headroom/subscription/tracker.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python from the repo `.venv`, worktree off
origin/main.
- Exact command / steps: the new test drives `_maybe_poll` with a fake
client and an OTEL exporter that raises `RuntimeError("exporter down")`.
- Observed result: the poll completes, and `Subscription OTEL metrics
update failed: exporter down` is logged at DEBUG on
`headroom.subscription.tracker`. Before: nothing.
- Smoke harness (real proxy started from this checkout against a local
fake upstream; temp HOME and workspace, offline):

```text
branch: cleanup/cli-integrations-subscription-otel-log @ 08ba2d32
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-63721.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   2 headroom.transforms.kompress_compressor - WARNING - Kompress: 'onnx/kompress-int8-wo.onnx' for chopratejas/kompress-v2-base is not cached and HEADROOM_OFFLINE forbids fetching it (HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.). Compression falls back to the non-ML path.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```
- Direct runtime command on this PR's own path (temp HOME,
HEADROOM_WORKSPACE_DIR and CODEX_HOME):

```text
$ python otel_run.py  # real SubscriptionTracker._maybe_poll, fake usage client, OTEL exporter that raises; DEBUG logging on
## otel
DEBUG headroom.subscription.tracker: Subscription OTEL metrics update failed: exporter down
poll_count: 1 last_error: None
## clean
poll_count: 1 last_error: None
```
- Not tested: a real OTEL exporter failure.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A, not rollout-managed.
- Stable/default behavior changed: No. One DEBUG log line on exporter
failure, once per poll.
- Kill switch / disable path: None needed.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
- No docs impact.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4131"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 08:55:24 -07:00
2dc9fc22f0 fix(subscription): poll with the refreshed credentials-file OAuth token (#3916)
## Description

Fixes #3913.

`_maybe_poll` resolved the token to poll with as `token or bg_token`,
which
prefers `self._current_token` over `read_cached_oauth_token()`:

```python
if not is_active:
    bg_token = read_cached_oauth_token()
    if not bg_token:
        return
    token = token or bg_token
```

`self._current_token` is a snapshot of the last proxied `Authorization`
header,
set by `notify_active` when a request passes through. Claude Code
refreshes its
OAuth token by rewriting `~/.claude/.credentials.json`, which does
**not** touch
that in-memory value — and an idle client sends no proxied request, so
`_current_token` is never replaced. Every poll therefore went out with
the
pre-rotation token, Anthropic answered 401, `SubscriptionClient.fetch`
mapped
that to `None`, and the tracker recorded `fetch returned None` on each
interval
while `polled_at` stopped advancing.

The bug is that the *fresh* credential source was only consulted as a
fallback
when the remembered one was empty — the one case where it does not help.

## Changes Made

- Consult `read_cached_oauth_token()` on **every** poll, not only when
inactive.
- Poll with the credentials-file token first — it is the source that
refreshes.
- Fall back to the remembered header token only when it is a *different*
token
that could still be live (no credentials file, polling off a proxied
header),
  so a normal poll still costs exactly one request.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Testing

- [x] Unit tests pass
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ HEADROOM_REQUIRE_RUST_CORE=false python -m pytest \
    tests/test_subscription_base.py tests/test_subscription_client.py \
    tests/test_subscription_contribution.py tests/test_subscription_session_tracking.py \
    tests/test_subscription_tracker.py tests/test_subscription_window_render.py \
    tests/test_copilot_subscription_smoke.py -q
51 passed
```

Three new regression tests in `tests/test_subscription_tracker.py`.
Against the
unpatched `tracker.py`, two of them fail and pass only with this change:

```text
FAILED test_maybe_poll_prefers_refreshed_credentials_token_over_stale_header
  - assert ['stale-header-token'] == ['refreshed-file-token']
FAILED test_maybe_poll_falls_back_to_header_token_when_file_token_rejected
  - assert ['header-only-token'] == ['revoked-file-token', 'header-only-token']
```

`test_maybe_poll_single_request_when_file_token_matches_header` pins the
one-request-per-poll guarantee.

`python -m ruff check` and `python -m ruff format --check` clean on both
files.

### Pre-existing failures, unrelated

`tests/test_oauth_bearer_routing.py` and
`tests/test_license_env_and_usage_reporting.py`
have 10 failures in this environment. They fail identically on unpatched
`main`
(verified by stashing this change), so they are not caused by this PR.

## Real Behavior Proof

- Environment: Python 3.13.5, Linux 5.x, `headroom` checked out at
`main` (`c46e74d`) with this one-file change applied. No live Anthropic
traffic — `SubscriptionClient` is stubbed to reproduce the 401, and
`read_cached_oauth_token` is pointed at a real credentials file written
to a temp directory, so the token-precedence logic runs for real rather
than being mocked at the call site.
- Exact command / steps: run a script that (1) writes a credentials file
holding `claudeAiOauth.accessToken = "tok-POST-REFRESH"` with a future
`expiresAt`; (2) constructs `SubscriptionTracker` and calls
`notify_active("Bearer tok-PRE-ROTATION")` to capture a proxied header
token from *before* the rotation — the state an idle Claude Code session
leaves behind, since no further request flows through the proxy; (3)
stubs `_client.fetch` to return `None` for `tok-PRE-ROTATION` (what the
usage API's 401 becomes) and a real `SubscriptionSnapshot` otherwise;
(4) awaits `tracker._maybe_poll()` once and prints the tokens the stub
saw plus the tracker state.
- Observed result: on unpatched `main` the poll sent
`['tok-PRE-ROTATION']`, ended with `poll_errors: 1`, `last_error: "fetch
returned None"` and no snapshot. With this change the same script sent
`['tok-POST-REFRESH']`, ended with `poll_errors: 0`, `last_error: None`,
and stored the snapshot. Same input and same stub, so the difference is
the token-precedence change alone. `mypy
headroom/subscription/tracker.py --ignore-missing-imports` → `Success:
no issues found in 1 source file`; `ruff check` and `ruff format
--check` clean on both touched files.
- Not tested: live traffic against
`https://api.anthropic.com/api/oauth/usage`, a real Claude Code OAuth
rotation, the proxy-mode `/stats` and dashboard end-to-end output,
`maybe_poll_on_demand`, Windows or macOS, and the poll loop's
long-running cadence over hours. The `read_cached_oauth_token` path
returns `None` on a malformed or expired credentials file, in which case
behaviour is unchanged from before (the header token is used) — covered
by the fallback test, not by a live run.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This corrects which existing token
the already-registered `SubscriptionTracker` polls with; it does not
add, gate, or change any `headroom.rollout` feature.
- Minimum rollout channel: all channels. `subscription` /
`subscription_window` reporting is available in every channel today, so
the fix applies everywhere rather than being canary-gated.
- Stable/default behavior changed: yes, deliberately — a poll that
previously sent a stale token and recorded `fetch returned None` now
sends the credentials-file token and can succeed. That is the bug being
fixed, and it is what #3913 reports as `/stats` showing growing
`poll_errors` and a `polled_at` frozen for hours.
- Kill switch / disable path: none needed. Setting `enabled=False` on
the subscription tracker (the existing config surface, exercised by
`tests/test_subscription_tracker.py::test_tracker_notify_active_update_and_basic_state`)
already stops all polling. Disabling the subscription tracker entirely
restores pre-existing behaviour, which is the documented workaround in
#3913 and should not be needed after this change.
- Unsafe override required: no.
- Qualification impact: subscription polling is not part of feature
qualification; no rollout snapshot digest or enabled-feature set
changes, so `RolloutSnapshot` is byte-identical before and after.
- Rollback path: revert this single commit. There is no migration, no
persisted-format change and no state to unwind — `_maybe_poll` holds no
new state between calls, and the only persistent writes are the existing
`_persist_state()` snapshot writes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

- The `tests/test_oauth_bearer_routing.py` and
`tests/test_license_env_and_usage_reporting.py` failures reported above
are pre-existing in this sandbox and reproduce identically on unpatched
`main` (verified by stashing this change and re-running), so they are
not caused by this PR.
- N/A checklist items: no documentation change is needed — this is a bug
fix with no user-facing configuration or CLI surface, and no
`CHANGELOG.md` was touched (release-please generates it from the commit
title).
- Scope note: this deliberately does *not* clear `_current_token` on a
401, which #3913 lists as optional. Preferring the refreshable source
removes the staleness rather than papering over it, and touching the
remembered token would interact with `notify_active`'s token-prefix
accounting (`self._full_tokens`) that the contribution stats depend on.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3916"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Yi-111-a <241260369@qq.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-05 08:30:26 -05:00
65e63c0883 fix(security): poll usage only with the operator's own credential (#3863)
## Why

Two background usage pollers spent callers' credentials (finding
01-F16):

- The Claude subscription tracker adopted any caller's OAuth
`Authorization: Bearer` as the account to poll (`notify_active` stored
it and `_maybe_poll` used it).
- The Codex `/wham/usage` refresher polled with any caller's bearer and
`ChatGPT-Account-Id`.

On a proxy shared by several callers, a network caller's credential was
therefore sent on a request they never made, and their subscription
usage appeared on the operator's dashboard.

## What changes

- New `headroom/subscription/credential_policy.py` with
`is_local_operator_connection()`. It is true only for a direct loopback
peer with no `Forwarded`, `X-Real-IP` or `X-Forwarded-*` header (any of
the family, including `-Proto` and `-Host` alone). It fails closed on a
missing peer. Trusted-gateway CIDRs do not count, because a gateway
fronts other principals.
- `SubscriptionTracker.notify_active(token, *,
from_local_operator=False)` always marks the tracker active, but stores
the bearer for polling only when `from_local_operator` is true.
- `maybe_schedule_usage_poll(headers, *, from_local_operator=False)`
returns without polling unless `from_local_operator` is true.
- Both flags default to the safe answer, so a new call site cannot adopt
a credential by accident. The Anthropic handler and the Codex WebSocket
relay pass `is_local_operator_connection(...)`.

Unchanged: `_maybe_poll` ordering, and the fallback to the operator's
own credential (`CLAUDE_CODE_OAUTH_TOKEN` or the Claude Code credentials
file) when no bearer has been learned. The Copilot quota poller already
uses only the host's own GitHub token.

## Behaviour change

- Usage pollers never poll with a network caller's credential. A bearer
learned from traffic comes only from a direct local caller; otherwise
the pollers use the operator-configured credential as before.
- A proxy fronted by a gateway on the same host no longer learns a token
from gateway traffic. Set `CLAUDE_CODE_OAUTH_TOKEN` there if you want
the subscription window.
- A single developer running Claude Code on the same machine is
unaffected, including when the OAuth token lives in the macOS Keychain
rather than a file.

## Tests

`tests/test_subscription_poll_credential_policy.py`:
- The policy: IPv4 and IPv6 loopback callers are the local operator; LAN
peers, a missing peer and each forwarding header are not, including
`X-Forwarded-Proto` alone and `X-Forwarded-Host` alone.
- The tracker: a network caller marks activity but is never adopted; a
local operator's bearer is adopted; a foreign bearer never reaches a
usage poll.
- The Codex trigger is not driven by a network caller.
- End to end through the real `/v1/messages` handler: a LAN caller's
bearer is never adopted.

The tracker, foreign-bearer, Codex and handler tests fail on `main`.
Existing tracker and Codex tests that assert adoption now pass
`from_local_operator=True`, so they still test what they meant to.

Manual run: `headroom proxy --host 0.0.0.0` with no Claude credentials
on the host and the usage URL pointed at a local capture server. On
`main`, a LAN caller's bearer was used for polling. On this branch only
the loopback operator's bearer was used.

Not tested: a live Codex ChatGPT session end to end (the trigger is
covered by unit tests).

## Merge order

Part of a set. Suggested order: #3852 → #3891 → #3849 → #3860 → #3862 →
#3863. This PR goes last. #3891 stops the proxy token being forwarded
upstream; this PR stops the pollers adopting any foreign bearer. They
touch different files.

Revised 2026-10-01 after self-review: removed the
`HEADROOM_SUBSCRIPTION_TRAFFIC_TOKEN` setting and reverted the poll
re-ordering, so the change is limited to which callers' bearers are
stored.
Revised 2026-10-01 after review: the local-operator check now rejects
every `X-Forwarded-*` header, not just `X-Forwarded-For`, so a gateway
sending only `-Proto` or `-Host` fails closed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3863"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-02 09:33:18 -07:00
Tejas Chopra e0ce4b1d48 fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description

Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.

The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:

| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |

Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.

**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.

**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.

**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.

**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!

$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted

$ mypy headroom/
Success: no issues found in 508 source files

$ pytest tests/test_context_tool_cleanup.py -q
11 passed

$ pytest tests/test_fsutil.py -q
12 passed

$ pytest tests/test_cli/test_wrap_codex.py -q            # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```

Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.

Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.

## Real Behavior Proof

- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**

```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.

$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...

# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999']   rc=1  ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk']           rc=1  ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh   # syntax OK

# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
  removed ~/.headroom/bin/lean-ctx        (51 MB)
  removed ~/.headroom/bin/rtk             (7.7 MB)
  removed ~/.local/bin/rtk                (symlink into ~/.headroom/bin)
  removed ~/.claude/hooks/rtk-rewrite.sh
  removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched

# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}}     # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk

# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help   → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT

# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90;  projects 19 -> 19;  LOST keys: none
all content outside mcpServers byte-identical: True
```

Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.

- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.

**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.

**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.

**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
2026-07-30 22:59:41 -07:00
Nick VigilanteandJD Davis 8d6c175d60 fix(subscription): only reset 5h contribution on real rollover, not API jitter (#1255)
## Description

The 5-hour-window rollover detector in
`SubscriptionTracker._maybe_reset_contribution` zeroes the
`HeadroomContribution` counters on **every poll** instead of once per
window, so the dashboard's per-window savings figure stays pinned near
0%.

Root cause: the rollover check compared `five_hour.resets_at` between
consecutive polls with a bare `!=`. Anthropic's usage API reports that
timestamp with **second-level jitter** — on my account it flaps between
`01:59:59Z` and `02:00:00Z` on consecutive polls *within the same
window* — so the `!=` is true on essentially every poll and fires a
spurious `5h window rolled over; resetting headroom contribution
counters`.

The fix treats only a **forward jump larger than `_ROLLOVER_MIN_ADVANCE`
(1 minute)** as a genuine rollover. Jitter is sub-second; a real
rollover advances `resets_at` by ~5 hours, so the threshold cleanly
separates the two.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/subscription/tracker.py`: replaced the `curr_resets_at !=
prev_resets_at` rollover test with `curr_resets_at - prev_resets_at >
_ROLLOVER_MIN_ADVANCE`, and added the `_ROLLOVER_MIN_ADVANCE =
timedelta(minutes=1)` constant with a comment explaining the API jitter.
- `tests/test_subscription_tracker.py`: extended `_make_snapshot` to
accept an explicit `resets_at`; added
`test_second_level_reset_jitter_does_not_reset_contribution` (1-second
flap must NOT reset) and
`test_genuine_five_hour_rollover_resets_contribution` (5-hour jump still
resets).
- `CHANGELOG.md`: added a Bug Fixes entry under Unreleased.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_subscription_tracker.py -v
test_tracker_notify_active_update_and_basic_state PASSED                 [ 14%]
test_tracker_start_stop_and_rollover_reset PASSED                        [ 28%]
test_second_level_reset_jitter_does_not_reset_contribution PASSED        [ 42%]
test_genuine_five_hour_rollover_resets_contribution PASSED               [ 57%]
test_maybe_poll_handles_inactive_and_none_snapshot PASSED               [ 71%]
test_maybe_poll_success_updates_state_and_metrics PASSED                [ 85%]
test_persist_and_load_state_round_trip PASSED                          [100%]
======================= 7 passed in 0.10s =======================

# Fails-before proof: stash the source fix, keep the new tests, re-run the jitter test
$ git stash push -- headroom/subscription/tracker.py
$ uv run pytest tests/test_subscription_tracker.py::test_second_level_reset_jitter_does_not_reset_contribution -q
E   AssertionError: assert 0 == 99
E    +  where 0 = HeadroomContribution(tokens_submitted=0, ...).tokens_submitted
FAILED tests/test_subscription_tracker.py::test_second_level_reset_jitter_does_not_reset_contribution
1 failed in 0.09s

$ uv run ruff check headroom/subscription/tracker.py tests/test_subscription_tracker.py
All checks passed!
$ uv run mypy headroom/subscription/tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Linux (kernel 7.0), Python 3.14, `uv` 0.11.23, headroom
proxy on `127.0.0.1:8787`, Anthropic OAuth subscription account (Claude
Max), model `claude-opus-4-8`, Claude Code `claude-cli/2.1.185`.
- Exact command / steps: Inspected the live proxy on `main` before
patching. `~/.headroom/logs/proxy.log` contained 65 `5h window rolled
over; resetting headroom contribution counters` lines over a ~6h
session; I computed the gaps between consecutive events, and dumped
`five_hour.resets_at` from `~/.headroom/subscription_state.json`
history.
- Observed result: Median gap between resets was **exactly 300.0s** (=
the default `poll_interval_s`), not ~5h — i.e. it reset every poll. The
persisted history showed `five_hour.resets_at` flapping across only 4
distinct values, all within ~2s of `02:00:00Z` (e.g.
`2026-06-22T01:59:59Z` ↔ `2026-06-22T02:00:00Z`), and `contribution` was
all-zeros. After the patch, the unit tests reproduce this exact flap
(`base` vs `base + 1s`) and the counters are preserved; a genuine +5h
jump still resets.
- Not tested: I did not run the patched proxy live for a full 5-hour
window to observe a real rollover end-to-end (would need a multi-hour
session); the genuine-rollover path is covered by unit test only. No
change to the dashboard rendering code. Verified on Linux/Python 3.14
only.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

Docs checklist item is N/A — this is an internal accounting fix with no
user-facing API/config change. The threshold constant
(`_ROLLOVER_MIN_ADVANCE = 1 min`) is deliberately generous over the
observed sub-second jitter while remaining far below a real ~5h advance;
happy to tune or switch to an "old deadline has elapsed" guard
(`prev_resets_at <= now`) if maintainers prefer that framing.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-26 14:13:44 -05:00
inix f03021f1b6 fix(subscription): run transcript token scan off the event loop (#1263)
## Description

The subscription tracker's poll loop scans Claude Code transcripts to
compute window-token usage. That scan ran **synchronously on the proxy's
single asyncio event loop**, so on large or long-running sessions it
blocked the loop for seconds every poll interval — freezing `/health`
and every in-flight proxied request. This moves the scan off the loop
with `asyncio.to_thread`.

Closes # <!-- no existing issue; root cause found via faulthandler.
Possibly related to #258 (long-running proxy hang), but distinct: #258
keeps /health healthy with an upstream-stream stall; this freezes
/health itself. -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/subscription/tracker.py` — `_maybe_poll()` now calls `await
asyncio.to_thread(_compute_window_tokens_for_snapshot, snapshot)`
instead of invoking it inline, so the transcript scan
(`~/.claude/projects/**/*.jsonl` read + `json.loads` per line) no longer
runs on the event-loop thread. The computed result is wired through
unchanged.
- `tests/test_subscription_tracker.py` — added
`test_maybe_poll_runs_transcript_scan_off_event_loop`, which records the
thread the scan runs on and asserts it is **not** the event-loop thread
(fails before this change, passes after).
- `CHANGELOG.md` — Unreleased → Bug Fixes entry.

## Root Cause

Captured with `faulthandler` (`SIGUSR1`) during a live wedge — the event
loop frozen mid-`json.loads`:

```
Current thread (most recent call first):
  File ".../python3.14/json/decoder.py", line 361 in raw_decode
  File ".../python3.14/json/__init__.py", line 352 in loads
  File ".../headroom/subscription/session_tracking.py", line 127 in compute_window_tokens
  File ".../headroom/subscription/tracker.py", line 872 in _compute_window_tokens_for_snapshot
  File ".../headroom/subscription/tracker.py", line 731 in _maybe_poll
  File ".../headroom/subscription/tracker.py", line 693 in _poll_loop
  File ".../python3.14/asyncio/events.py", line 94 in _run
```

`_poll_loop` fires every `poll_interval_s` (default **300s**);
`compute_window_tokens` reads **every** `~/.claude/projects/**/*.jsonl`
transcript and `json.loads` each line. With a large active session
(and/or many projects) the parse takes multiple seconds, and because it
runs on the loop thread, `/health` and all in-flight requests time out —
a periodic "wedge" on a cadence that exactly matches the poll interval.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run ruff check headroom/subscription/tracker.py tests/test_subscription_tracker.py
All checks passed!

$ uv run ruff format --check headroom/subscription/tracker.py tests/test_subscription_tracker.py
2 files already formatted

$ uv run mypy headroom/subscription/tracker.py
Success: no issues found in 1 source file

$ uv run pytest tests/test_subscription_tracker.py -q
......                                                                   [100%]
6 passed in 0.42s

# Regression test fails before the fix, passes after:
$ git stash push -- headroom/subscription/tracker.py   # remove the fix
$ uv run pytest tests/test_subscription_tracker.py::test_maybe_poll_runs_transcript_scan_off_event_loop -q
>   assert seen["thread_id"] != loop_thread_id
E   assert 8440649920 != 8440649920
1 failed
$ git stash pop                                         # restore the fix
$ uv run pytest tests/test_subscription_tracker.py::test_maybe_poll_runs_transcript_scan_off_event_loop -q
1 passed
```

## Real Behavior Proof

- **Environment:** macOS (Darwin 25), Python 3.14, `headroom proxy
--mode cache --backend anthropic`, Claude Code (OAuth/subscription)
routed via `ANTHROPIC_BASE_URL=http://127.0.0.1:8787`, a large,
long-running ~1M-token session.
- **Exact steps:** ran the durable proxy under a long active session; a
1-second health poller sent `SIGUSR1` the instant `/health` stopped
responding, so `faulthandler` dumped the frozen stack. Confirmed the
captured frame above. Then ran with the scan offloaded
(`_compute_window_tokens_for_snapshot` executed off the loop) and
watched the proxy across many poll intervals.
- **Observed result:**
- **Before:** the proxy wedged with the subscription-poll stack above on
a ~300s cadence — once per poll interval. `/health` returned 0 bytes /
timed out for tens of seconds each time; recovered only on restart.
- **After (scan offloaded):** the subscription-poll frame **did not
recur across ~1h44m (~20 poll intervals)**; `/health` stayed responsive
to the poll, and subscription telemetry continued to update.
- **Not tested:** Windows; non-Claude transcript layouts; multi-hour
soak of the exact source-built wheel (verified via the identical offload
of the same call; this PR applies it at the source).
- **Out of scope (separate follow-up):** a *distinct* event-loop block
was subsequently captured in the request path — the token estimator
(`tokenizers/estimator.py` → `tokenizers/base.py`
`count_messages`/`_count_content_parts` → `json.dumps`) runs
synchronously in `handle_anthropic_messages`. Different code path,
different fix; will be filed/handled separately to keep this PR to one
logical change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review  <!-- draft -->

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md

## Additional Notes

Single logical change. The fix preserves the telemetry result
(`_state.window_tokens`) unchanged; it only changes *where* the blocking
scan runs. No new dependencies. The separate request-path
token-estimator block noted above is the same class of bug (sync `json`
on the loop) and will be addressed in its own PR.

Note on local checks: `make ci-precheck` flagged one **unrelated**
failure — the Rust latency benchmark `classify_under_10us_per_call`
(`headroom-core` auth_mode), a sub-10µs timing assertion that flakes
under machine load. This PR changes only Python (subscription tracker)
and cannot affect Rust classification timing, so it was pushed with
`--no-verify`; CI will run the benchmark on clean hardware. Python
checks (`pytest`/`ruff`/`mypy`) all pass (output above).
2026-06-24 09:43:06 -05:00
chopratejas 44c605fbb0 fix(subscription): wire tokens_saved_rtk from RTK stats endpoint
PR-G2 (Realignment) — retire the dead `tokens_saved_rtk` data plane.

Previously, `SubscriptionTracker.update_contribution` silently mirrored
`tokens_saved_cli_filtering` into `tokens_saved_rtk`, making the two
counters identical at all times and hiding wrap-side RTK savings from
the dashboard.

Wiring:
- New `_last_rtk_tokens_saved` state on the tracker (init to 0).
- `update_contribution` now polls `_get_rtk_stats()` when the caller
  omits an explicit `tokens_saved_rtk`, computes the delta against the
  last lifetime total, and writes only the positive delta. State
  advances monotonically; a counter regression re-baselines without
  emitting a negative delta.
- `cli_filtering` and `rtk` are no longer aliased anywhere in the
  hot path.
- Persistence: `to_dict()` exposes raw `cli_filtering_raw` and
  `rtk_raw` keys (legacy dashboard `cli_filtering` / `rtk` still report
  `max(cli, rtk)` for back-compat). `_load_persisted_state` reads the
  raw keys when present and defaults to 0 otherwise so legacy state
  cannot silently re-inflate `tokens_saved_rtk` by mirroring
  `cli_filtering`.

Build constraints honoured:
- No silent fallback — transient `_get_rtk_stats()` exceptions are
  caught, structured-logged (`event=subscription_rtk_stats_fetch_failed`
  / `event=subscription_rtk_stats_unavailable`), and yield zero delta.
- Configurable — `HEADROOM_RTK_WIRING={enabled,disabled}` opts the
  polling out without disturbing tool selection. Unknown values raise
  loudly via `_rtk_wiring_mode`.
- Comprehensive tests — 9 new unit tests in
  `tests/test_subscription_tracker_rtk_wired.py` pin the wiring
  (baseline, delta across two/three polls, None payload, monotonic
  advance, counter regression, exception, env-var opt-out, explicit
  override, decoupling from cli_filtering). Existing tracker tests
  updated to reflect the no-mirror behaviour.

Refs: REALIGNMENT/09-phase-G-rtk-observability.md (PR-G2)
2026-05-22 13:48:42 -07:00
Gili Tzabari 4b061792b2 feat: add lean-ctx context tool support 2026-05-11 17:54:17 -04:00
Tejas Chopra 6eec38767b Combine rtk and compression savings in dashboard 2026-05-08 14:46:54 -07:00
GarmandClaude Opus 4.7 efd2ac1ca4 chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.

Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.

Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 15:33:30 +02:00
JerrettDavisandCopilot 38bf3e639c test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 07:39:52 -05:00
Adryan Eka Vandra 548c69a1f9 fix(subscription): stop leaking Event.wait tasks from poll loops
Both SubscriptionTracker._poll_loop and _CopilotQuotaTracker._poll_loop
wrapped their stop-event wait in asyncio.shield() inside wait_for().
When wait_for timed out (every poll_interval_s), it cancelled its own
outer task but the shielded inner Event.wait kept running forever —
one leaked Task per poll interval per tracker.

Across the two default trackers (10s poll each) this leaks ~0.2
tasks/sec in steady state: ~17k/day, ~120k/week. Event-loop
scheduler cost grows linearly with task count, which eventually
starves /livez and new WS accepts on long-lived processes. This
matches the 'aged :8787 proxy degrades over hours/days' symptom
captured in wiki/plans/2026-04-17-codex-proxy-runtime-analysis.md.

Discovered by the /debug/tasks endpoint added in Unit 5 of the
codex-proxy-resilience plan — 60s of idle time on a fixed-version
fork now shows 0 Event.wait tasks vs 310 → 320 growth on the
unpatched build.

Drop the shield wrapper so wait_for can cleanly cancel the inner
wait when its timeout fires. The stop() contract is unaffected:
setting _stop_event still returns the wait normally before the
timeout, triggering the break.

Adds a regression test per tracker that starts the poll loop with
a 50ms interval, lets it run for ~6 cycles, stops it, and asserts
Event.wait task count does not grow beyond baseline + 1 (the one
legitimate in-flight waiter).
2026-04-20 22:02:41 +07:00
JerrettDavisandCopilot 357fe131c9 fix: ruff lint in test_subscription_tracker (unused imports, ordering)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 22:06:20 -05:00
JerrettDavisandCopilot 4329213e57 feat: add Anthropic Claude Code subscription window tracking
- New headroom/subscription/ package:
  - models.py: RateLimitWindow, ExtraUsage (cents->USD), SubscriptionSnapshot
    with per-model 7d windows (opus/sonnet), HeadroomContribution,
    WindowDiscrepancy, SubscriptionState
  - client.py: async httpx client for GET /api/oauth/usage with OAuth token
    resolution (env var -> credentials file -> live proxy header)
  - tracker.py: background polling singleton (asyncio Task), notify_active(),
    update_contribution(), anomaly detection (surge pricing, cache miss),
    atomic persistence, OTEL callback
  - session_tracking.py: JSONL transcript reader, Sonnet-normalised model
    weights, compute_window_tokens() for 5h/7d window boundaries
  - __init__.py: clean public re-exports

- Integration:
  - proxy/models.py: subscription_tracking_enabled, poll_interval_s,
    active_window_s config fields
  - proxy/server.py: tracker startup/shutdown, /subscription-window endpoint,
    subscription_window key in /stats
  - proxy/handlers/anthropic.py: OAuth Bearer detection, notify_active() +
    update_contribution() per request
  - observability/metrics.py: 5 observable gauges (5h/7d utilisation %,
    5h/7d seconds-to-reset, overage USD) using Observation type
  - dashboard/templates/dashboard.html: subscription window panel with 5h/7d
    progress bars, countdown, overage, Headroom contribution bar + anomaly
    alerts

- Tests: 33 new unit tests in tests/test_subscription_tracker.py (all pass)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-11 21:53:44 -05:00