mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): strip upstream transport framing headers from Gemini response replays (#4605)
## Description The Gemini `generateContent` and `countTokens` response replays (both the optimized fast path, the main path and the continuation-error replay) copied the upstream headers by hand and only removed `content-encoding`/`content-length`. A chunked upstream therefore leaked `transfer-encoding: chunked` to a client that receives an already-decoded, re-framed JSON body — the RFC 9112 wrong-framing class this codebase already avoids elsewhere — along with `connection`, `keep-alive` and the upstream `server` banner. Other paths in the same file and other providers already use the shared `sanitize_forwarded_response_headers`; these five sites were missed. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (non-breaking change that only restructures code) - [ ] Tests (no production code change) - [ ] Documentation (no production code change) - [ ] Chore (dependencies, CI, tooling, etc.) ## Changes Made - Import and use `sanitize_forwarded_response_headers` in the five Gemini response-replay sites instead of hand-copying headers. - Add regressions for `generateContent` and `countTokens` that assert `server`, `keep-alive`, `connection` and `transfer-encoding` from the upstream do not reach the client while `content-type`, body and `x-headroom-*` stay intact. ## Testing - [x] Unit tests pass (`tests/test_proxy` 491 passed) - [x] New tests added - [x] Manual testing performed (real uvicorn proxy + real httpx against a local fake upstream) ### Test Output ```text pytest tests/test_proxy/test_gemini_response_framing_headers.py -q # parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client # this branch: 2 passed pytest tests/test_proxy -q # 491 passed ruff check / ruff format --check / mypy <changed file> # pass ``` ## Real Behavior Proof <details> <summary>real behavior proof details</summary> - Exact command / steps: python -m pytest tests/test_proxy/test_gemini_response_framing_headers.py -q with a local uvicorn proxy, a local http.server fake upstream and a real httpx client comparing the before/after response headers. - Environment: macOS aarch64, Python 3.14.5 worktree venv, local uvicorn proxy with a Python `http.server` fake upstream, real httpx client, no provider keys. - Observed result: before the change the client received `server: uvicorn, BaseHTTP/0.6 ..., Upstream-Marker/9.9`, `keep-alive`, `connection: keep-alive` and `transfer-encoding: chunked`; after the change only uvicorn's own `server` header remains, with `content-type`, body and `x-headroom-tokens-*` intact for `generateContent` and `countTokens`. - Not tested: a real Gemini API call, the optimized/compression branch as a separate e2e, the `_GeminiContinuationError` branch as its own case, Windows/Linux, and the other providers that still hand-copy headers (left as a follow-up to keep this PR scoped). </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: Gemini response replays no longer forward upstream transport framing or server headers (intended). - Kill switch / disable path: revert this commit. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] No unrelated changes - [x] Tests added/pass - [ ] CHANGELOG updated — no changelog entry for this internal header fix; say so if you want one - [x] No secrets or local paths ## Notes No linked issue (source-derived, with a reproduction); other replay paths use the same hand-copy pattern and are left as a follow-up. Duplicate `date` headers (upstream Date plus uvicorn Date) are the shared sanitizer's existing behavior and out of scope. ## Paste relevant command output or artifact links here <details> <summary>Commands and output</summary> ```text pytest tests/test_proxy/test_gemini_response_framing_headers.py -q # parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client # this branch: 2 passed pytest tests/test_proxy -q # 491 passed ruff check / ruff format --check / mypy <changed file> # pass ``` </details> ## Screenshots (if applicable) N/A - not user-visible; the evidence is the header comparison in the Real Behavior Proof section (real uvicorn proxy + httpx client + local fake upstream). ## Additional Notes Other providers/buffered replay paths still hand-copy headers; left as a follow-up to keep this PR scoped. Duplicate `date` headers are the shared sanitizer's existing behavior and out of scope. No linked issue (source-derived bug with a reproduction). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4605" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JD Davis
parent
ee2252a315
commit
dfcac78119
@@ -27,6 +27,7 @@ from headroom.proxy.helpers import (
|
||||
COMPRESSION_TIMEOUT_SECONDS,
|
||||
extract_tags,
|
||||
invalid_request_body_message,
|
||||
sanitize_forwarded_response_headers,
|
||||
)
|
||||
from headroom.proxy.identity import resolve_memory_identity
|
||||
from headroom.proxy.outcome import RequestOutcome
|
||||
@@ -586,9 +587,7 @@ class GeminiHandlerMixin:
|
||||
client=client,
|
||||
)
|
||||
)
|
||||
response_headers = dict(response.headers)
|
||||
response_headers.pop("content-encoding", None)
|
||||
response_headers.pop("content-length", None)
|
||||
response_headers = sanitize_forwarded_response_headers(response.headers)
|
||||
response_headers["x-headroom-tokens-before"] = str(total_input_tokens)
|
||||
response_headers["x-headroom-tokens-after"] = str(total_input_tokens)
|
||||
response_headers["x-headroom-tokens-saved"] = "0"
|
||||
@@ -1090,12 +1089,9 @@ class GeminiHandlerMixin:
|
||||
else:
|
||||
logger.info(f"[{request_id}] Gemini {model}: {original_tokens:,} tokens")
|
||||
|
||||
# Remove compression headers
|
||||
response_headers = dict(response.headers)
|
||||
response_headers.pop("content-encoding", None)
|
||||
response_headers.pop("content-length", None)
|
||||
|
||||
# Inject Headroom compression metrics (for SaaS metering)
|
||||
# Remove transport framing headers (the body is re-framed here)
|
||||
# and inject Headroom compression metrics (for SaaS metering).
|
||||
response_headers = sanitize_forwarded_response_headers(response.headers)
|
||||
response_headers["x-headroom-tokens-before"] = str(original_tokens)
|
||||
response_headers["x-headroom-tokens-after"] = str(optimized_tokens)
|
||||
response_headers["x-headroom-tokens-saved"] = str(tokens_saved)
|
||||
@@ -1118,9 +1114,7 @@ class GeminiHandlerMixin:
|
||||
)
|
||||
except _GeminiContinuationError as e:
|
||||
await self.metrics.record_failed(provider=provider_name)
|
||||
response_headers = dict(e.headers)
|
||||
response_headers.pop("content-encoding", None)
|
||||
response_headers.pop("content-length", None)
|
||||
response_headers = sanitize_forwarded_response_headers(e.headers)
|
||||
return Response(content=e.content, status_code=e.status_code, headers=response_headers)
|
||||
except Exception as e:
|
||||
await self.metrics.record_failed(provider=provider_name)
|
||||
@@ -1434,9 +1428,7 @@ class GeminiHandlerMixin:
|
||||
url += f"?key={query_params['key']}"
|
||||
|
||||
response = await self._retry_request("POST", url, headers, body)
|
||||
response_headers = dict(response.headers)
|
||||
response_headers.pop("content-encoding", None)
|
||||
response_headers.pop("content-length", None)
|
||||
response_headers = sanitize_forwarded_response_headers(response.headers)
|
||||
return Response(
|
||||
content=response.content,
|
||||
status_code=response.status_code,
|
||||
@@ -1585,10 +1577,8 @@ class GeminiHandlerMixin:
|
||||
f"[{request_id}] Gemini countTokens {model}: {compressed_tokens:,} tokens"
|
||||
)
|
||||
|
||||
# Remove compression headers
|
||||
response_headers = dict(response.headers)
|
||||
response_headers.pop("content-encoding", None)
|
||||
response_headers.pop("content-length", None)
|
||||
# Remove transport framing headers (the body is re-framed here).
|
||||
response_headers = sanitize_forwarded_response_headers(response.headers)
|
||||
|
||||
return Response(
|
||||
content=response.content,
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Gemini response replay must strip upstream transport framing headers.
|
||||
|
||||
The Gemini generateContent / countTokens replay paths copied upstream headers
|
||||
by hand and popped only content-encoding / content-length. They therefore
|
||||
replayed headers the shared sanitize_forwarded_response_headers exists to drop:
|
||||
transfer-encoding from a chunked upstream, connection-local connection /
|
||||
keep-alive, and the upstream server banner.
|
||||
|
||||
A stale transfer-encoding on an already-decoded, re-framed body is the #3019
|
||||
class of bug: RFC 9112 makes Transfer-Encoding override Content-Length, so the
|
||||
client parses plain JSON as chunked frames.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("fastapi")
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
_UPSTREAM_HEADERS = {
|
||||
"content-type": "application/json",
|
||||
"server": "upstream-marker/9.9",
|
||||
"keep-alive": "timeout=5, max=100",
|
||||
"connection": "keep-alive",
|
||||
"transfer-encoding": "chunked",
|
||||
}
|
||||
|
||||
_FRAMING_HEADERS = ("server", "keep-alive", "connection", "transfer-encoding")
|
||||
|
||||
_GENERATE_BODY = json.dumps(
|
||||
{
|
||||
"candidates": [
|
||||
{
|
||||
"content": {"parts": [{"text": "hi"}], "role": "model"},
|
||||
"finishReason": "STOP",
|
||||
}
|
||||
],
|
||||
"usageMetadata": {
|
||||
"promptTokenCount": 4,
|
||||
"candidatesTokenCount": 1,
|
||||
"totalTokenCount": 5,
|
||||
},
|
||||
}
|
||||
).encode()
|
||||
|
||||
_COUNT_TOKENS_BODY = json.dumps({"totalTokens": 4}).encode()
|
||||
|
||||
|
||||
def _client(body: bytes) -> TestClient:
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
image_optimize=False,
|
||||
)
|
||||
app = create_app(config)
|
||||
proxy = app.state.proxy
|
||||
|
||||
async def _fake_retry(*_args: object, **_kwargs: object) -> httpx.Response:
|
||||
return httpx.Response(200, headers=_UPSTREAM_HEADERS, content=body)
|
||||
|
||||
proxy._retry_request = _fake_retry # type: ignore[method-assign]
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _assert_upstream_framing_is_stripped(response: httpx.Response) -> None:
|
||||
lower = {name.lower() for name in response.headers}
|
||||
for header in _FRAMING_HEADERS:
|
||||
assert header not in lower, (
|
||||
f"upstream {header!r} reached the client: {dict(response.headers)}"
|
||||
)
|
||||
assert response.headers["content-type"].startswith("application/json")
|
||||
|
||||
|
||||
def test_generate_content_replay_strips_upstream_framing_headers() -> None:
|
||||
client = _client(_GENERATE_BODY)
|
||||
|
||||
response = client.post(
|
||||
"/v1beta/models/gemini-pro:generateContent?key=test-key",
|
||||
json={"contents": [{"role": "user", "parts": [{"text": "hi"}]}]},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["candidates"][0]["content"]["parts"][0]["text"] == "hi"
|
||||
_assert_upstream_framing_is_stripped(response)
|
||||
|
||||
|
||||
def test_count_tokens_replay_strips_upstream_framing_headers() -> None:
|
||||
client = _client(_COUNT_TOKENS_BODY)
|
||||
|
||||
response = client.post(
|
||||
"/v1beta/models/gemini-pro:countTokens?key=test-key",
|
||||
json={"contents": [{"role": "user", "parts": [{"text": "hi"}]}]},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["totalTokens"] == 4
|
||||
_assert_upstream_framing_is_stripped(response)
|
||||
Reference in New Issue
Block a user