mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(subscription): poll with the refreshed credentials-file OAuth token (#3916)
## Description Fixes #3913. `_maybe_poll` resolved the token to poll with as `token or bg_token`, which prefers `self._current_token` over `read_cached_oauth_token()`: ```python if not is_active: bg_token = read_cached_oauth_token() if not bg_token: return token = token or bg_token ``` `self._current_token` is a snapshot of the last proxied `Authorization` header, set by `notify_active` when a request passes through. Claude Code refreshes its OAuth token by rewriting `~/.claude/.credentials.json`, which does **not** touch that in-memory value — and an idle client sends no proxied request, so `_current_token` is never replaced. Every poll therefore went out with the pre-rotation token, Anthropic answered 401, `SubscriptionClient.fetch` mapped that to `None`, and the tracker recorded `fetch returned None` on each interval while `polled_at` stopped advancing. The bug is that the *fresh* credential source was only consulted as a fallback when the remembered one was empty — the one case where it does not help. ## Changes Made - Consult `read_cached_oauth_token()` on **every** poll, not only when inactive. - Poll with the credentials-file token first — it is the source that refreshes. - Fall back to the remembered header token only when it is a *different* token that could still be live (no credentials file, polling off a proxied header), so a normal poll still costs exactly one request. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HEADROOM_REQUIRE_RUST_CORE=false python -m pytest \ tests/test_subscription_base.py tests/test_subscription_client.py \ tests/test_subscription_contribution.py tests/test_subscription_session_tracking.py \ tests/test_subscription_tracker.py tests/test_subscription_window_render.py \ tests/test_copilot_subscription_smoke.py -q 51 passed ``` Three new regression tests in `tests/test_subscription_tracker.py`. Against the unpatched `tracker.py`, two of them fail and pass only with this change: ```text FAILED test_maybe_poll_prefers_refreshed_credentials_token_over_stale_header - assert ['stale-header-token'] == ['refreshed-file-token'] FAILED test_maybe_poll_falls_back_to_header_token_when_file_token_rejected - assert ['header-only-token'] == ['revoked-file-token', 'header-only-token'] ``` `test_maybe_poll_single_request_when_file_token_matches_header` pins the one-request-per-poll guarantee. `python -m ruff check` and `python -m ruff format --check` clean on both files. ### Pre-existing failures, unrelated `tests/test_oauth_bearer_routing.py` and `tests/test_license_env_and_usage_reporting.py` have 10 failures in this environment. They fail identically on unpatched `main` (verified by stashing this change), so they are not caused by this PR. ## Real Behavior Proof - Environment: Python 3.13.5, Linux 5.x, `headroom` checked out at `main` (`c46e74d`) with this one-file change applied. No live Anthropic traffic — `SubscriptionClient` is stubbed to reproduce the 401, and `read_cached_oauth_token` is pointed at a real credentials file written to a temp directory, so the token-precedence logic runs for real rather than being mocked at the call site. - Exact command / steps: run a script that (1) writes a credentials file holding `claudeAiOauth.accessToken = "tok-POST-REFRESH"` with a future `expiresAt`; (2) constructs `SubscriptionTracker` and calls `notify_active("Bearer tok-PRE-ROTATION")` to capture a proxied header token from *before* the rotation — the state an idle Claude Code session leaves behind, since no further request flows through the proxy; (3) stubs `_client.fetch` to return `None` for `tok-PRE-ROTATION` (what the usage API's 401 becomes) and a real `SubscriptionSnapshot` otherwise; (4) awaits `tracker._maybe_poll()` once and prints the tokens the stub saw plus the tracker state. - Observed result: on unpatched `main` the poll sent `['tok-PRE-ROTATION']`, ended with `poll_errors: 1`, `last_error: "fetch returned None"` and no snapshot. With this change the same script sent `['tok-POST-REFRESH']`, ended with `poll_errors: 0`, `last_error: None`, and stored the snapshot. Same input and same stub, so the difference is the token-precedence change alone. `mypy headroom/subscription/tracker.py --ignore-missing-imports` → `Success: no issues found in 1 source file`; `ruff check` and `ruff format --check` clean on both touched files. - Not tested: live traffic against `https://api.anthropic.com/api/oauth/usage`, a real Claude Code OAuth rotation, the proxy-mode `/stats` and dashboard end-to-end output, `maybe_poll_on_demand`, Windows or macOS, and the poll loop's long-running cadence over hours. The `read_cached_oauth_token` path returns `None` on a malformed or expired credentials file, in which case behaviour is unchanged from before (the header token is used) — covered by the fallback test, not by a live run. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This corrects which existing token the already-registered `SubscriptionTracker` polls with; it does not add, gate, or change any `headroom.rollout` feature. - Minimum rollout channel: all channels. `subscription` / `subscription_window` reporting is available in every channel today, so the fix applies everywhere rather than being canary-gated. - Stable/default behavior changed: yes, deliberately — a poll that previously sent a stale token and recorded `fetch returned None` now sends the credentials-file token and can succeed. That is the bug being fixed, and it is what #3913 reports as `/stats` showing growing `poll_errors` and a `polled_at` frozen for hours. - Kill switch / disable path: none needed. Setting `enabled=False` on the subscription tracker (the existing config surface, exercised by `tests/test_subscription_tracker.py::test_tracker_notify_active_update_and_basic_state`) already stops all polling. Disabling the subscription tracker entirely restores pre-existing behaviour, which is the documented workaround in #3913 and should not be needed after this change. - Unsafe override required: no. - Qualification impact: subscription polling is not part of feature qualification; no rollout snapshot digest or enabled-feature set changes, so `RolloutSnapshot` is byte-identical before and after. - Rollback path: revert this single commit. There is no migration, no persisted-format change and no state to unwind — `_maybe_poll` holds no new state between calls, and the only persistent writes are the existing `_persist_state()` snapshot writes. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - The `tests/test_oauth_bearer_routing.py` and `tests/test_license_env_and_usage_reporting.py` failures reported above are pre-existing in this sandbox and reproduce identically on unpatched `main` (verified by stashing this change and re-running), so they are not caused by this PR. - N/A checklist items: no documentation change is needed — this is a bug fix with no user-facing configuration or CLI surface, and no `CHANGELOG.md` was touched (release-please generates it from the commit title). - Scope note: this deliberately does *not* clear `_current_token` on a 401, which #3913 lists as optional. Preferring the refreshable source removes the staleness rather than papering over it, and touching the remembered token would interact with `notify_active`'s token-prefix accounting (`self._full_tokens`) that the contribution stats depend on. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3916" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Yi-111-a <241260369@qq.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Yi-111-a
JD Davis
parent
2de582875b
commit
2dc9fc22f0
@@ -399,16 +399,31 @@ class SubscriptionTracker(QuotaTracker):
|
||||
is_active = self._state.is_active(active_window_s=self._active_window_s)
|
||||
token = self._current_token
|
||||
|
||||
if not is_active:
|
||||
# Try background poll using credentials file token
|
||||
from headroom.subscription.client import read_cached_oauth_token
|
||||
# Always consult the credentials file: it is the source Claude Code
|
||||
# refreshes in place, so it holds the *current* token. `self._current_token`
|
||||
# is a snapshot of the last proxied Authorization header and goes stale
|
||||
# as soon as Claude Code rotates its OAuth token — it is only replaced
|
||||
# when a request flows through the proxy, which an idle client never
|
||||
# does. Preferring it (`token or bg_token`) therefore pinned polling to
|
||||
# an expired token and surfaced as permanent `poll_errors` /
|
||||
# "fetch returned None" with a stale `polled_at` (issue #3913).
|
||||
from headroom.subscription.client import read_cached_oauth_token
|
||||
|
||||
bg_token = read_cached_oauth_token()
|
||||
if not bg_token:
|
||||
return
|
||||
token = token or bg_token
|
||||
bg_token = read_cached_oauth_token()
|
||||
if not is_active and not bg_token:
|
||||
return
|
||||
|
||||
snapshot = None
|
||||
# Fresh credentials-file token first, so a normal poll costs one request.
|
||||
if bg_token:
|
||||
snapshot = await self._client.fetch(bg_token)
|
||||
# Only spend a second request on the remembered header token when it is
|
||||
# a *different* token that could still be the live one (no credentials
|
||||
# file, e.g. CLAUDE_CODE_OAUTH_TOKEN absent and polling off a proxied
|
||||
# header).
|
||||
if snapshot is None and token and token != bg_token:
|
||||
snapshot = await self._client.fetch(token)
|
||||
|
||||
snapshot = await self._client.fetch(token)
|
||||
if snapshot is None:
|
||||
with self._lock:
|
||||
self._state.mark_error("fetch returned None")
|
||||
|
||||
@@ -103,9 +103,11 @@ def test_foreign_bearer_never_reaches_the_usage_poll(tmp_path, monkeypatch) -> N
|
||||
|
||||
tracker._client.fetch = fetch # type: ignore[method-assign]
|
||||
asyncio.run(tracker._maybe_poll())
|
||||
# The poll ran (the caller marked activity) but with no adopted token, so
|
||||
# the client falls back to the operator's own credential.
|
||||
assert polled == [None]
|
||||
# Activity was marked, but with no adopted local-operator token and no
|
||||
# operator-configured credential there is nothing to poll — and specifically
|
||||
# the foreign bearer must never be spent. Skip the unnecessary fetch(None).
|
||||
assert polled == []
|
||||
assert "tenant-b-oauth-token" not in {t for t in polled if t}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -174,6 +174,106 @@ async def test_maybe_poll_handles_inactive_and_none_snapshot(
|
||||
assert tracker._state.poll_errors == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_maybe_poll_prefers_refreshed_credentials_token_over_stale_header(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Regression (#3913): the credentials file is where Claude Code writes the
|
||||
*refreshed* OAuth token. Polling the remembered Authorization-header token
|
||||
first pinned every poll to an expired credential, so `/stats` reported
|
||||
permanent `poll_errors` and a `polled_at` that never advanced."""
|
||||
monkeypatch.setattr(SubscriptionTracker, "_load_persisted_state", lambda self: None)
|
||||
tracker = SubscriptionTracker()
|
||||
# Token captured from a proxied local-operator request, before Claude Code rotated it.
|
||||
tracker.notify_active("Bearer stale-header-token", from_local_operator=True)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"headroom.subscription.client.read_cached_oauth_token", lambda: "refreshed-file-token"
|
||||
)
|
||||
|
||||
snapshot = _make_snapshot()
|
||||
attempted: list[str | None] = []
|
||||
|
||||
async def fetch_snapshot(token: str | None):
|
||||
attempted.append(token)
|
||||
# The stale header token is what Anthropic rejects with 401 -> None.
|
||||
if token == "stale-header-token":
|
||||
return None
|
||||
return snapshot
|
||||
|
||||
tracker._client = SimpleNamespace(fetch=fetch_snapshot)
|
||||
monkeypatch.setattr(tracker, "_persist_state", lambda: None)
|
||||
|
||||
await tracker._maybe_poll()
|
||||
|
||||
assert attempted == ["refreshed-file-token"]
|
||||
assert tracker.latest_snapshot is snapshot
|
||||
assert tracker._state.poll_errors == 0
|
||||
assert tracker._state.last_error is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_maybe_poll_falls_back_to_header_token_when_file_token_rejected(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The remembered header token is still the only credential when no
|
||||
credentials file exists, so a rejection of the file token must fall back to
|
||||
it rather than mark the tracker errored."""
|
||||
monkeypatch.setattr(SubscriptionTracker, "_load_persisted_state", lambda self: None)
|
||||
tracker = SubscriptionTracker()
|
||||
tracker.notify_active("Bearer header-only-token", from_local_operator=True)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"headroom.subscription.client.read_cached_oauth_token", lambda: "revoked-file-token"
|
||||
)
|
||||
|
||||
snapshot = _make_snapshot()
|
||||
attempted: list[str | None] = []
|
||||
|
||||
async def fetch_snapshot(token: str | None):
|
||||
attempted.append(token)
|
||||
if token == "revoked-file-token":
|
||||
return None
|
||||
return snapshot
|
||||
|
||||
tracker._client = SimpleNamespace(fetch=fetch_snapshot)
|
||||
monkeypatch.setattr(tracker, "_persist_state", lambda: None)
|
||||
|
||||
await tracker._maybe_poll()
|
||||
|
||||
assert attempted == ["revoked-file-token", "header-only-token"]
|
||||
assert tracker.latest_snapshot is snapshot
|
||||
assert tracker._state.poll_errors == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_maybe_poll_single_request_when_file_token_matches_header(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Identical tokens must not be spent twice — the common case for a proxy
|
||||
session whose credentials file holds the same token as the header."""
|
||||
monkeypatch.setattr(SubscriptionTracker, "_load_persisted_state", lambda self: None)
|
||||
tracker = SubscriptionTracker()
|
||||
tracker.notify_active("Bearer shared-token", from_local_operator=True)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"headroom.subscription.client.read_cached_oauth_token", lambda: "shared-token"
|
||||
)
|
||||
|
||||
attempted: list[str | None] = []
|
||||
|
||||
async def fetch_snapshot(token: str | None):
|
||||
attempted.append(token)
|
||||
return _make_snapshot()
|
||||
|
||||
tracker._client = SimpleNamespace(fetch=fetch_snapshot)
|
||||
monkeypatch.setattr(tracker, "_persist_state", lambda: None)
|
||||
|
||||
await tracker._maybe_poll()
|
||||
|
||||
assert attempted == ["shared-token"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_maybe_poll_success_updates_state_and_metrics(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
@@ -181,6 +281,9 @@ async def test_maybe_poll_success_updates_state_and_metrics(
|
||||
monkeypatch.setattr(SubscriptionTracker, "_load_persisted_state", lambda self: None)
|
||||
tracker = SubscriptionTracker()
|
||||
tracker.notify_active("Bearer live-oauth-token", from_local_operator=True)
|
||||
# Pin the credentials-file/env source so this exercises the adopted
|
||||
# local-operator header path, not whatever token the host happens to hold.
|
||||
monkeypatch.setattr("headroom.subscription.client.read_cached_oauth_token", lambda: None)
|
||||
|
||||
snapshot = _make_snapshot()
|
||||
discrepancies = [WindowDiscrepancy(kind="cache_miss", description="miss", severity="warning")]
|
||||
@@ -227,6 +330,7 @@ async def test_maybe_poll_runs_transcript_scan_off_event_loop(
|
||||
monkeypatch.setattr(SubscriptionTracker, "_load_persisted_state", lambda self: None)
|
||||
tracker = SubscriptionTracker()
|
||||
tracker.notify_active("Bearer live-oauth-token", from_local_operator=True)
|
||||
monkeypatch.setattr("headroom.subscription.client.read_cached_oauth_token", lambda: None)
|
||||
|
||||
snapshot = _make_snapshot()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user