mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): forward Copilot accounting headers on streaming path (#4431)
## Description The streaming path only forwards upstream headers on a short list (`*ratelimit*`, `x-codex*`, `x-litellm-key-*`, request-id family). GitHub Copilot uses `x-github-*` / `*copilot*` headers for AI-credit accounting, and those are not on the list, so they get dropped. When they're missing, the Copilot CLI stamps streamed usage records `aiCreditsStatus: "unavailable"`, leaves out `serviceRequestId`/`providerCallId` and the `copilotUsage` block (which carries `totalNanoAiu`), and stops advancing `session.usage_checkpoint.totalNanoAiu`. Token counts arrive in the SSE body and look correct, so nothing errors. Credits are just silently under-reported by about 4–5x. The buffered path already forwards every upstream header. Real coding agents stream, so the streaming path is the one that matters for Copilot. In a 9-task agentic A/B on `claude-opus-4.8` (one arm direct, one through Headroom): | | direct | via Headroom | |---|---|---| | `aiCreditsStatus: complete` | 990 (100%) | 247 (34.9%) | | `aiCreditsStatus: unavailable` | 0 | 460 (65.1%) | | reported AI credits | 8,015.9 | 1,471.8 (−82%) | | actual cacheRead tokens | 98.5M | 68.8M (−30%) | A −82% drop in credits against a −30% drop in tokens is a measurement gap, not a saving. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/streaming.py`: forward headers that start with `x-github` or contain `copilot`, and update the comment to explain why. - `tests/test_proxy_streaming_ratelimit_headers.py`: add `test_copilot_accounting_headers_forwarded_in_streaming`, which checks that `x-github-request-id` and `x-copilot-usage` are forwarded on a streamed response and an unrelated header (`cf-ray`) is still dropped. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_proxy_streaming_ratelimit_headers.py 22 passed in 30.59s # The new test without the fix: $ pytest -q tests/test_proxy_streaming_ratelimit_headers.py -k copilot FAILED ...::test_copilot_accounting_headers_forwarded_in_streaming 1 failed, 21 deselected $ ruff check headroom/proxy/handlers/streaming.py tests/test_proxy_streaming_ratelimit_headers.py All checks passed! $ ruff format --check <same files> 2 files already formatted ``` ## Real Behavior Proof ## Real Behavior Proof - Environment: - macOS 26.6.2 (arm64) - Python 3.12.12 - headroom-ai 0.40.0 from PyPI in a venv (`~/.venvs/headroom-040`), with this PR's two-line change applied to `headroom/proxy/handlers/streaming.py` in site-packages (same diff as this branch) - GitHub Copilot CLI 1.0.103, model `claude-opus-4.8`, reasoning effort `medium` - Upstream: `https://api.githubcopilot.com` - Exact command / steps: 1. Start the proxy, with telemetry off and routed to Copilot: ```bash HEADROOM_COMPRESS_WORKERS=8 HEADROOM_COMPRESSION_DEADLINE_MS=60000 \ ANTHROPIC_TARGET_API_URL=https://api.githubcopilot.com \ OPENAI_TARGET_API_URL=https://api.githubcopilot.com \ GITHUB_COPILOT_API_URL=https://api.githubcopilot.com \ headroom proxy --port 8787 > /tmp/hr-proxy-hdrfix.log 2>&1 ``` 2. Send one streamed Copilot CLI request through it: ```bash cd /tmp && COPILOT_API_URL=http://127.0.0.1:8787 \ copilot --no-experimental --model claude-opus-4.8 --allow-all-tools \ -p "Reply with exactly: OK" ``` 3. Confirm the streaming path was used: ```bash grep "forwarder=streaming" /tmp/hr-proxy-hdrfix.log ``` 4. Check the CLI session transcript (`~/.copilot/session-state/<session-id>/events.jsonl`): ```bash grep -o '"aiCreditsStatus":"[a-z]*"' events.jsonl | sort | uniq -c grep -o '"serviceRequestId":"[^"]*"' events.jsonl | head -1 grep -o '"totalNanoAiu":[0-9]*' events.jsonl | tail -1 ``` - Observed result: Verified end-to-end. The proxy log shows `forwarder=streaming` (`keeping the streaming path (#3071)`). In the resulting transcript, all 9 records are `aiCreditsStatus: complete` and `serviceRequestId` is present. The last checkpoint is `totalNanoAiu = 30,213,500,000` → 30.21 AIC, exactly matching the CLI's displayed `AI Credits 30.21`. - Before the patch: in the A/B eval through unpatched 0.40.0 with the same proxy settings, streamed records were stamped `aiCreditsStatus: "unavailable"` with no `serviceRequestId` and no `copilotUsage`. For example, one task had 70 of 113 records unavailable, and the checkpoint covered only 22% of cacheRead tokens. - Not tested: a live run built from this branch (the identical patch was verified on the 0.40.0 wheel instead); non-Copilot providers end-to-end (unit tests only); `mypy`. ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: N/A - Stable/default behavior changed: Yes. The streaming path now also forwards `x-github*` and `*copilot*` response headers, which the buffered path already forwards. - Kill switch / disable path: None; revert the commit. - Unsafe override required: No - Qualification impact: None. Both are response metadata headers, never hop-by-hop. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4431" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: dmanepal <dmanepal@linkedin.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
dmanepal
JD Davis
parent
6cc0801219
commit
66023a0187
@@ -1616,7 +1616,11 @@ class StreamingMixin:
|
||||
# message id + request id. ``x-litellm-key-*`` carries the per-key
|
||||
# spend/budget envelope a LiteLLM virtual key reports per request;
|
||||
# downstream status widgets read it from the streaming response the
|
||||
# same way they already do from the buffered path. The buffered
|
||||
# same way they already do from the buffered path. GitHub Copilot keys
|
||||
# its AI-credit accounting off ``x-github-*`` / ``*copilot*`` headers;
|
||||
# without them the Copilot CLI marks streamed usage records
|
||||
# ``aiCreditsStatus: unavailable`` and silently under-reports credits.
|
||||
# The buffered
|
||||
# (non-streaming) path already forwards every upstream header, so
|
||||
# this keeps the two paths symmetric.
|
||||
forwarded_headers = {
|
||||
@@ -1625,6 +1629,8 @@ class StreamingMixin:
|
||||
if "ratelimit" in k.lower()
|
||||
or k.lower().startswith("x-codex")
|
||||
or k.lower().startswith("x-litellm-key-")
|
||||
or k.lower().startswith("x-github")
|
||||
or "copilot" in k.lower()
|
||||
or k.lower() in ("request-id", "anthropic-request-id", "x-request-id")
|
||||
}
|
||||
# Headroom's own compression metrics, as the buffered path stamps them.
|
||||
|
||||
@@ -349,6 +349,59 @@ class TestStreamingRatelimitHeaderForwarding:
|
||||
assert result.headers.get("x-litellm-key-budget-remaining") == "87.153"
|
||||
assert result.headers.get("cf-ray") is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_copilot_accounting_headers_forwarded_in_streaming(self, monkeypatch):
|
||||
"""Copilot's ``x-github-*`` / ``*copilot*`` headers survive the streaming path.
|
||||
|
||||
The Copilot CLI derives AI-credit accounting from these headers; when
|
||||
they are dropped it records ``aiCreditsStatus: unavailable`` and
|
||||
under-reports credits while token counts still look correct.
|
||||
"""
|
||||
|
||||
# A Copilot URL makes the proxy mint a Copilot OAuth token, which needs a
|
||||
# real login; stub it so the test runs without one.
|
||||
async def fake_copilot_auth(headers: dict[str, str], *, url: str) -> dict[str, str]:
|
||||
return {**headers, "authorization": "Bearer copilot-test-token"}
|
||||
|
||||
monkeypatch.setattr(streaming_module, "apply_copilot_api_auth", fake_copilot_auth)
|
||||
proxy = self._create_mock_proxy()
|
||||
mock_response = self._create_mock_upstream_response()
|
||||
mock_response.headers = httpx.Headers(
|
||||
{
|
||||
"content-type": "text/event-stream",
|
||||
"x-github-request-id": "ABCD:1234:5678",
|
||||
"x-copilot-usage": "nano-aiu=30213500000",
|
||||
"cf-ray": "ray-789",
|
||||
}
|
||||
)
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(return_value=mock_response)
|
||||
|
||||
result = await proxy._stream_response(
|
||||
url="https://api.githubcopilot.com/v1/messages",
|
||||
headers={"authorization": "Bearer test"},
|
||||
body={
|
||||
"model": "claude-opus-4.8",
|
||||
"max_tokens": 100,
|
||||
"stream": True,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
},
|
||||
provider="anthropic",
|
||||
model="claude-opus-4.8",
|
||||
request_id="test-copilot-accounting",
|
||||
original_tokens=10,
|
||||
optimized_tokens=10,
|
||||
tokens_saved=0,
|
||||
transforms_applied=[],
|
||||
tags={},
|
||||
optimization_latency=0.0,
|
||||
)
|
||||
|
||||
assert result.media_type == "text/event-stream"
|
||||
assert result.headers.get("x-github-request-id") == "ABCD:1234:5678"
|
||||
assert result.headers.get("x-copilot-usage") == "nano-aiu=30213500000"
|
||||
assert result.headers.get("cf-ray") is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_ratelimit_headers_still_works(self):
|
||||
"""When upstream has no ratelimit headers, streaming should still work."""
|
||||
|
||||
Reference in New Issue
Block a user