mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description The Gemini `generateContent` and `countTokens` response replays (both the optimized fast path, the main path and the continuation-error replay) copied the upstream headers by hand and only removed `content-encoding`/`content-length`. A chunked upstream therefore leaked `transfer-encoding: chunked` to a client that receives an already-decoded, re-framed JSON body — the RFC 9112 wrong-framing class this codebase already avoids elsewhere — along with `connection`, `keep-alive` and the upstream `server` banner. Other paths in the same file and other providers already use the shared `sanitize_forwarded_response_headers`; these five sites were missed. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (non-breaking change that only restructures code) - [ ] Tests (no production code change) - [ ] Documentation (no production code change) - [ ] Chore (dependencies, CI, tooling, etc.) ## Changes Made - Import and use `sanitize_forwarded_response_headers` in the five Gemini response-replay sites instead of hand-copying headers. - Add regressions for `generateContent` and `countTokens` that assert `server`, `keep-alive`, `connection` and `transfer-encoding` from the upstream do not reach the client while `content-type`, body and `x-headroom-*` stay intact. ## Testing - [x] Unit tests pass (`tests/test_proxy` 491 passed) - [x] New tests added - [x] Manual testing performed (real uvicorn proxy + real httpx against a local fake upstream) ### Test Output ```text pytest tests/test_proxy/test_gemini_response_framing_headers.py -q # parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client # this branch: 2 passed pytest tests/test_proxy -q # 491 passed ruff check / ruff format --check / mypy <changed file> # pass ``` ## Real Behavior Proof <details> <summary>real behavior proof details</summary> - Exact command / steps: python -m pytest tests/test_proxy/test_gemini_response_framing_headers.py -q with a local uvicorn proxy, a local http.server fake upstream and a real httpx client comparing the before/after response headers. - Environment: macOS aarch64, Python 3.14.5 worktree venv, local uvicorn proxy with a Python `http.server` fake upstream, real httpx client, no provider keys. - Observed result: before the change the client received `server: uvicorn, BaseHTTP/0.6 ..., Upstream-Marker/9.9`, `keep-alive`, `connection: keep-alive` and `transfer-encoding: chunked`; after the change only uvicorn's own `server` header remains, with `content-type`, body and `x-headroom-tokens-*` intact for `generateContent` and `countTokens`. - Not tested: a real Gemini API call, the optimized/compression branch as a separate e2e, the `_GeminiContinuationError` branch as its own case, Windows/Linux, and the other providers that still hand-copy headers (left as a follow-up to keep this PR scoped). </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: Gemini response replays no longer forward upstream transport framing or server headers (intended). - Kill switch / disable path: revert this commit. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] No unrelated changes - [x] Tests added/pass - [ ] CHANGELOG updated — no changelog entry for this internal header fix; say so if you want one - [x] No secrets or local paths ## Notes No linked issue (source-derived, with a reproduction); other replay paths use the same hand-copy pattern and are left as a follow-up. Duplicate `date` headers (upstream Date plus uvicorn Date) are the shared sanitizer's existing behavior and out of scope. ## Paste relevant command output or artifact links here <details> <summary>Commands and output</summary> ```text pytest tests/test_proxy/test_gemini_response_framing_headers.py -q # parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client # this branch: 2 passed pytest tests/test_proxy -q # 491 passed ruff check / ruff format --check / mypy <changed file> # pass ``` </details> ## Screenshots (if applicable) N/A - not user-visible; the evidence is the header comparison in the Real Behavior Proof section (real uvicorn proxy + httpx client + local fake upstream). ## Additional Notes Other providers/buffered replay paths still hand-copy headers; left as a follow-up to keep this PR scoped. Duplicate `date` headers are the shared sanitizer's existing behavior and out of scope. No linked issue (source-derived bug with a reproduction). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4605" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com>
108 lines
3.3 KiB
Python
108 lines
3.3 KiB
Python
"""Gemini response replay must strip upstream transport framing headers.
|
|
|
|
The Gemini generateContent / countTokens replay paths copied upstream headers
|
|
by hand and popped only content-encoding / content-length. They therefore
|
|
replayed headers the shared sanitize_forwarded_response_headers exists to drop:
|
|
transfer-encoding from a chunked upstream, connection-local connection /
|
|
keep-alive, and the upstream server banner.
|
|
|
|
A stale transfer-encoding on an already-decoded, re-framed body is the #3019
|
|
class of bug: RFC 9112 makes Transfer-Encoding override Content-Length, so the
|
|
client parses plain JSON as chunked frames.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
_UPSTREAM_HEADERS = {
|
|
"content-type": "application/json",
|
|
"server": "upstream-marker/9.9",
|
|
"keep-alive": "timeout=5, max=100",
|
|
"connection": "keep-alive",
|
|
"transfer-encoding": "chunked",
|
|
}
|
|
|
|
_FRAMING_HEADERS = ("server", "keep-alive", "connection", "transfer-encoding")
|
|
|
|
_GENERATE_BODY = json.dumps(
|
|
{
|
|
"candidates": [
|
|
{
|
|
"content": {"parts": [{"text": "hi"}], "role": "model"},
|
|
"finishReason": "STOP",
|
|
}
|
|
],
|
|
"usageMetadata": {
|
|
"promptTokenCount": 4,
|
|
"candidatesTokenCount": 1,
|
|
"totalTokenCount": 5,
|
|
},
|
|
}
|
|
).encode()
|
|
|
|
_COUNT_TOKENS_BODY = json.dumps({"totalTokens": 4}).encode()
|
|
|
|
|
|
def _client(body: bytes) -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
image_optimize=False,
|
|
)
|
|
app = create_app(config)
|
|
proxy = app.state.proxy
|
|
|
|
async def _fake_retry(*_args: object, **_kwargs: object) -> httpx.Response:
|
|
return httpx.Response(200, headers=_UPSTREAM_HEADERS, content=body)
|
|
|
|
proxy._retry_request = _fake_retry # type: ignore[method-assign]
|
|
return TestClient(app)
|
|
|
|
|
|
def _assert_upstream_framing_is_stripped(response: httpx.Response) -> None:
|
|
lower = {name.lower() for name in response.headers}
|
|
for header in _FRAMING_HEADERS:
|
|
assert header not in lower, (
|
|
f"upstream {header!r} reached the client: {dict(response.headers)}"
|
|
)
|
|
assert response.headers["content-type"].startswith("application/json")
|
|
|
|
|
|
def test_generate_content_replay_strips_upstream_framing_headers() -> None:
|
|
client = _client(_GENERATE_BODY)
|
|
|
|
response = client.post(
|
|
"/v1beta/models/gemini-pro:generateContent?key=test-key",
|
|
json={"contents": [{"role": "user", "parts": [{"text": "hi"}]}]},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert response.json()["candidates"][0]["content"]["parts"][0]["text"] == "hi"
|
|
_assert_upstream_framing_is_stripped(response)
|
|
|
|
|
|
def test_count_tokens_replay_strips_upstream_framing_headers() -> None:
|
|
client = _client(_COUNT_TOKENS_BODY)
|
|
|
|
response = client.post(
|
|
"/v1beta/models/gemini-pro:countTokens?key=test-key",
|
|
json={"contents": [{"role": "user", "parts": [{"text": "hi"}]}]},
|
|
)
|
|
|
|
assert response.status_code == 200
|
|
assert response.json()["totalTokens"] == 4
|
|
_assert_upstream_framing_is_stripped(response)
|