mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): scope OpenAI rate limits by api-key (#3365)
## Description Scope OpenAI rate-limit buckets by either `Authorization` or `api-key`. OpenAI-compatible clients that authenticate only through `api-key` previously all consumed the shared `default` bucket and could incorrectly rate-limit one another. Closes #3364 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add one OpenAI rate-limit identity helper with bearer-token precedence, `api-key` fallback, and an opaque HMAC-SHA256 identity derived from the complete credential with a process-local random key. - Use the helper in Chat Completions and Responses handlers. - Add handler-level regressions proving distinct `api-key` credentials receive independent buckets on both endpoints. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest -q tests/test_proxy_openai_rate_limit_key.py tests/test_proxy_openai.py tests/test_proxy_openai_responses_bypass.py 27 passed, 46 warnings in 12.87s (minimal proxy test environment) $ .venv/bin/ruff check headroom/proxy/handlers/openai.py tests/test_proxy_openai_rate_limit_key.py All checks passed! $ .venv/bin/ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_openai_rate_limit_key.py 2 files already formatted $ .venv/bin/mypy --follow-imports=skip headroom/proxy/handlers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Linux, Python 3.14, current `main` plus this patch, real FastAPI proxy app and real local OpenAI-compatible HTTP upstream, RPM set to 1. - Exact command / steps: started a local HTTP upstream with a call counter, then sent the same `/v1/chat/completions` body twice using `api-key: azure-key-A` and `api-key: azure-key-B`. - Observed result: both credentials received HTTP 200 and independently reached the upstream. ```text HTTP Request: POST http://127.0.0.1:38689/v1/chat/completions "HTTP/1.0 200 OK" HTTP Request: POST http://127.0.0.1:38689/v1/chat/completions "HTTP/1.0 200 OK" results=[('azure-key-A', 200, None), ('azure-key-B', 200, None)] upstream_calls=2 ``` - Not tested: external/cloud gateways; full repository test suite locally (hosted CI shards passed). A full-import mypy invocation currently reaches three pre-existing `no-any-return` errors in `headroom/perf/analyzer.py`; the changed handler passes isolated type checking. ## Runtime Rollout Safety - Rollout-managed feature(s): built-in OpenAI proxy rate limiting. - Minimum rollout channel: N/A. - Stable/default behavior changed: only requests with `api-key` and no `Authorization` move from the shared `default` bucket to a credential-specific bucket. - Kill switch / disable path: disable rate limiting with `--no-rate-limit` or `rate_limit_enabled=False`. - Unsafe override required: No. - Qualification impact: OpenAI-compatible `api-key` clients no longer consume one another's RPM allowance. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes Documentation is unchanged because this makes the existing per-key rate-limit contract work for the already-forwarded `api-key` authentication form. The warning above is the existing Starlette `TestClient` deprecation warning. The requested credential-isolation change was addressed in `6bac361` and approved on 2026-08-31. The PR is ready for human review. Review follow-up `c31e82f7` replaces request-path scrypt with process-keyed HMAC-SHA256. API credentials are identifiers, not passwords; the password-hashing CodeQL finding on HMAC is a false positive for this use. A 100,000-call local microbenchmark measured 2.69 µs per derivation. The 27 targeted proxy tests pass locally. Hosted CodeQL analysis itself passed, but the separate CodeQL alert check fails on `py/weak-sensitive-data-hashing` at `headroom/proxy/handlers/openai.py:147`. It treats an API credential used as an HMAC-SHA256 rate-limit identifier as a stored password. This is the false positive noted in review; an upstream maintainer with code-scanning write access must dismiss this specific alert as a false positive. The query must remain enabled for the rest of the repository. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
ca02f28c67
commit
b9e8462a3a
@@ -9,10 +9,12 @@ import asyncio
|
||||
import contextlib
|
||||
import copy
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
@@ -121,6 +123,30 @@ _CCR_HASH_RE = re.compile(
|
||||
r"(?:Retrieve (?:more|original): hash=|<<ccr:)([a-fA-F0-9]{12,24})(?=[^a-fA-F0-9]|$)"
|
||||
)
|
||||
_BARE_CCR_HASH_RE = re.compile(r"[a-fA-F0-9]{12,24}")
|
||||
_OPENAI_RATE_KEY_SECRET = secrets.token_bytes(32)
|
||||
|
||||
|
||||
def _openai_rate_limit_key(headers: dict[str, str]) -> str:
|
||||
"""Return the credential identity used by the OpenAI rate limiter.
|
||||
|
||||
OpenAI-compatible gateways may authenticate with either a bearer token or
|
||||
an ``api-key`` header. Deriving an identity from the complete value keeps common
|
||||
prefixes distinct without retaining recoverable credential material in
|
||||
bucket keys. The secret is process-local, like the limiter state.
|
||||
Requests without either retain the existing shared fallback bucket.
|
||||
"""
|
||||
authorization = headers.get("authorization")
|
||||
api_key = headers.get("api-key")
|
||||
if authorization:
|
||||
kind, credential = "authorization", authorization
|
||||
elif api_key:
|
||||
kind, credential = "api-key", api_key
|
||||
else:
|
||||
return "default"
|
||||
# API credentials are identifiers, not passwords to verify. A process-keyed
|
||||
# HMAC keeps bucket keys opaque without a password KDF on the request path.
|
||||
digest = hmac.digest(_OPENAI_RATE_KEY_SECRET, f"{kind}:{credential}".encode(), "sha256").hex()
|
||||
return f"{kind}:{digest}"
|
||||
|
||||
|
||||
def _response_ccr_hashes(messages: list[dict[str, Any]], markers: list[str]) -> list[str]:
|
||||
@@ -3566,7 +3592,7 @@ class OpenAIHandlerMixin:
|
||||
|
||||
# Rate limiting
|
||||
if self.rate_limiter:
|
||||
rate_key = headers.get("authorization", "default")[:20]
|
||||
rate_key = _openai_rate_limit_key(headers)
|
||||
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
|
||||
if not allowed:
|
||||
await self.metrics.record_rate_limited(
|
||||
@@ -5728,7 +5754,7 @@ class OpenAIHandlerMixin:
|
||||
|
||||
# Rate limiting
|
||||
if self.rate_limiter:
|
||||
rate_key = headers.get("authorization", "default")[:20]
|
||||
rate_key = _openai_rate_limit_key(headers)
|
||||
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
|
||||
if not allowed:
|
||||
await self.metrics.record_rate_limited(provider="openai", source="headroom")
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Regression coverage for OpenAI credential-scoped rate limiting (#3364)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.proxy.handlers.openai import _openai_rate_limit_key # noqa: E402
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
|
||||
def test_openai_rate_limit_key_accepts_api_key_header() -> None:
|
||||
assert _openai_rate_limit_key({"api-key": "gateway-key-A"}) != _openai_rate_limit_key(
|
||||
{"api-key": "gateway-key-B"}
|
||||
)
|
||||
|
||||
|
||||
def test_openai_rate_limit_key_uses_complete_credential() -> None:
|
||||
shared_prefix = "a" * 20
|
||||
assert _openai_rate_limit_key({"api-key": f"{shared_prefix}-A"}) != _openai_rate_limit_key(
|
||||
{"api-key": f"{shared_prefix}-B"}
|
||||
)
|
||||
|
||||
|
||||
def test_openai_rate_limit_key_prefers_authorization() -> None:
|
||||
headers = {"authorization": "Bearer primary", "api-key": "secondary"}
|
||||
assert _openai_rate_limit_key(headers) == _openai_rate_limit_key(
|
||||
{"authorization": "Bearer primary"}
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("endpoint", ["chat", "responses"])
|
||||
def test_distinct_api_keys_do_not_share_openai_rate_limit_bucket(monkeypatch, endpoint) -> None:
|
||||
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=True,
|
||||
rate_limit_requests_per_minute=1,
|
||||
rate_limit_tokens_per_minute=100_000,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
image_optimize=False,
|
||||
retry_enabled=False,
|
||||
)
|
||||
|
||||
if endpoint == "chat":
|
||||
path = "/v1/chat/completions"
|
||||
body = {
|
||||
"model": "gpt-4o-mini",
|
||||
"messages": [{"role": "user", "content": "hello"}],
|
||||
"stream": False,
|
||||
}
|
||||
upstream_body = {
|
||||
"id": "chatcmpl-test",
|
||||
"object": "chat.completion",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": "ok"},
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
],
|
||||
"usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4},
|
||||
}
|
||||
else:
|
||||
path = "/v1/responses"
|
||||
body = {"model": "gpt-4o-mini", "input": "hello", "stream": False}
|
||||
upstream_body = {
|
||||
"id": "resp-test",
|
||||
"object": "response",
|
||||
"status": "completed",
|
||||
"output": [],
|
||||
"usage": {"input_tokens": 3, "output_tokens": 1, "total_tokens": 4},
|
||||
}
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
proxy = client.app.state.proxy
|
||||
proxy._retry_request = AsyncMock(
|
||||
side_effect=lambda *args, **kwargs: httpx.Response(200, json=upstream_body)
|
||||
)
|
||||
|
||||
first = client.post(path, headers={"api-key": "gateway-key-A"}, json=body)
|
||||
second = client.post(path, headers={"api-key": "gateway-key-B"}, json=body)
|
||||
|
||||
assert first.status_code == 200
|
||||
assert second.status_code == 200
|
||||
assert proxy._retry_request.await_count == 2
|
||||
Reference in New Issue
Block a user