fix(proxy): scope OpenAI rate limits by api-key (#3365)

## Description

Scope OpenAI rate-limit buckets by either `Authorization` or `api-key`.
OpenAI-compatible clients that authenticate only through `api-key`
previously all consumed the shared `default` bucket and could
incorrectly rate-limit one another.

Closes #3364

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add one OpenAI rate-limit identity helper with bearer-token
precedence, `api-key` fallback, and an opaque HMAC-SHA256 identity
derived from the complete credential with a process-local random key.
- Use the helper in Chat Completions and Responses handlers.
- Add handler-level regressions proving distinct `api-key` credentials
receive independent buckets on both endpoints.

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest -q tests/test_proxy_openai_rate_limit_key.py tests/test_proxy_openai.py tests/test_proxy_openai_responses_bypass.py
27 passed, 46 warnings in 12.87s (minimal proxy test environment)

$ .venv/bin/ruff check headroom/proxy/handlers/openai.py tests/test_proxy_openai_rate_limit_key.py
All checks passed!

$ .venv/bin/ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_openai_rate_limit_key.py
2 files already formatted

$ .venv/bin/mypy --follow-imports=skip headroom/proxy/handlers/openai.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Linux, Python 3.14, current `main` plus this patch, real
FastAPI proxy app and real local OpenAI-compatible HTTP upstream, RPM
set to 1.
- Exact command / steps: started a local HTTP upstream with a call
counter, then sent the same `/v1/chat/completions` body twice using
`api-key: azure-key-A` and `api-key: azure-key-B`.
- Observed result: both credentials received HTTP 200 and independently
reached the upstream.

```text
HTTP Request: POST http://127.0.0.1:38689/v1/chat/completions "HTTP/1.0 200 OK"
HTTP Request: POST http://127.0.0.1:38689/v1/chat/completions "HTTP/1.0 200 OK"
results=[('azure-key-A', 200, None), ('azure-key-B', 200, None)]
upstream_calls=2
```

- Not tested: external/cloud gateways; full repository test suite
locally (hosted CI shards passed). A full-import mypy invocation
currently reaches three pre-existing `no-any-return` errors in
`headroom/perf/analyzer.py`; the changed handler passes isolated type
checking.

## Runtime Rollout Safety

- Rollout-managed feature(s): built-in OpenAI proxy rate limiting.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: only requests with `api-key` and no
`Authorization` move from the shared `default` bucket to a
credential-specific bucket.
- Kill switch / disable path: disable rate limiting with
`--no-rate-limit` or `rate_limit_enabled=False`.
- Unsafe override required: No.
- Qualification impact: OpenAI-compatible `api-key` clients no longer
consume one another's RPM allowance.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A.

## Additional Notes

Documentation is unchanged because this makes the existing per-key
rate-limit contract work for the already-forwarded `api-key`
authentication form. The warning above is the existing Starlette
`TestClient` deprecation warning.


The requested credential-isolation change was addressed in `6bac361` and
approved on 2026-08-31. The PR is ready for human review.


Review follow-up `c31e82f7` replaces request-path scrypt with
process-keyed HMAC-SHA256. API credentials are identifiers, not
passwords; the password-hashing CodeQL finding on HMAC is a false
positive for this use. A 100,000-call local microbenchmark measured 2.69
µs per derivation. The 27 targeted proxy tests pass locally. Hosted
CodeQL analysis itself passed, but the separate CodeQL alert check fails
on `py/weak-sensitive-data-hashing` at
`headroom/proxy/handlers/openai.py:147`. It treats an API credential
used as an HMAC-SHA256 rate-limit identifier as a stored password. This
is the false positive noted in review; an upstream maintainer with
code-scanning write access must dismiss this specific alert as a false
positive. The query must remain enabled for the rest of the repository.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
C0sm0cats
2026-09-25 10:23:35 -05:00
committed by GitHub
co-authored by JerrettDavis
parent ca02f28c67
commit b9e8462a3a
2 changed files with 125 additions and 2 deletions
+28 -2
View File
@@ -9,10 +9,12 @@ import asyncio
import contextlib
import copy
import hashlib
import hmac
import json
import logging
import os
import re
import secrets
import threading
import time
import uuid
@@ -121,6 +123,30 @@ _CCR_HASH_RE = re.compile(
r"(?:Retrieve (?:more|original): hash=|<<ccr:)([a-fA-F0-9]{12,24})(?=[^a-fA-F0-9]|$)"
)
_BARE_CCR_HASH_RE = re.compile(r"[a-fA-F0-9]{12,24}")
_OPENAI_RATE_KEY_SECRET = secrets.token_bytes(32)
def _openai_rate_limit_key(headers: dict[str, str]) -> str:
"""Return the credential identity used by the OpenAI rate limiter.
OpenAI-compatible gateways may authenticate with either a bearer token or
an ``api-key`` header. Deriving an identity from the complete value keeps common
prefixes distinct without retaining recoverable credential material in
bucket keys. The secret is process-local, like the limiter state.
Requests without either retain the existing shared fallback bucket.
"""
authorization = headers.get("authorization")
api_key = headers.get("api-key")
if authorization:
kind, credential = "authorization", authorization
elif api_key:
kind, credential = "api-key", api_key
else:
return "default"
# API credentials are identifiers, not passwords to verify. A process-keyed
# HMAC keeps bucket keys opaque without a password KDF on the request path.
digest = hmac.digest(_OPENAI_RATE_KEY_SECRET, f"{kind}:{credential}".encode(), "sha256").hex()
return f"{kind}:{digest}"
def _response_ccr_hashes(messages: list[dict[str, Any]], markers: list[str]) -> list[str]:
@@ -3566,7 +3592,7 @@ class OpenAIHandlerMixin:
# Rate limiting
if self.rate_limiter:
rate_key = headers.get("authorization", "default")[:20]
rate_key = _openai_rate_limit_key(headers)
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(
@@ -5728,7 +5754,7 @@ class OpenAIHandlerMixin:
# Rate limiting
if self.rate_limiter:
rate_key = headers.get("authorization", "default")[:20]
rate_key = _openai_rate_limit_key(headers)
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
await self.metrics.record_rate_limited(provider="openai", source="headroom")
+97
View File
@@ -0,0 +1,97 @@
"""Regression coverage for OpenAI credential-scoped rate limiting (#3364)."""
from __future__ import annotations
from unittest.mock import AsyncMock
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.handlers.openai import _openai_rate_limit_key # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
def test_openai_rate_limit_key_accepts_api_key_header() -> None:
assert _openai_rate_limit_key({"api-key": "gateway-key-A"}) != _openai_rate_limit_key(
{"api-key": "gateway-key-B"}
)
def test_openai_rate_limit_key_uses_complete_credential() -> None:
shared_prefix = "a" * 20
assert _openai_rate_limit_key({"api-key": f"{shared_prefix}-A"}) != _openai_rate_limit_key(
{"api-key": f"{shared_prefix}-B"}
)
def test_openai_rate_limit_key_prefers_authorization() -> None:
headers = {"authorization": "Bearer primary", "api-key": "secondary"}
assert _openai_rate_limit_key(headers) == _openai_rate_limit_key(
{"authorization": "Bearer primary"}
)
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
def test_distinct_api_keys_do_not_share_openai_rate_limit_bucket(monkeypatch, endpoint) -> None:
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=True,
rate_limit_requests_per_minute=1,
rate_limit_tokens_per_minute=100_000,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
retry_enabled=False,
)
if endpoint == "chat":
path = "/v1/chat/completions"
body = {
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "hello"}],
"stream": False,
}
upstream_body = {
"id": "chatcmpl-test",
"object": "chat.completion",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "ok"},
"finish_reason": "stop",
}
],
"usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4},
}
else:
path = "/v1/responses"
body = {"model": "gpt-4o-mini", "input": "hello", "stream": False}
upstream_body = {
"id": "resp-test",
"object": "response",
"status": "completed",
"output": [],
"usage": {"input_tokens": 3, "output_tokens": 1, "total_tokens": 4},
}
with TestClient(create_app(config)) as client:
proxy = client.app.state.proxy
proxy._retry_request = AsyncMock(
side_effect=lambda *args, **kwargs: httpx.Response(200, json=upstream_body)
)
first = client.post(path, headers={"api-key": "gateway-key-A"}, json=body)
second = client.post(path, headers={"api-key": "gateway-key-B"}, json=body)
assert first.status_code == 200
assert second.status_code == 200
assert proxy._retry_request.await_count == 2