fix(proxy): strip upstream transport framing headers from Gemini response replays (#4605)

## Description

The Gemini `generateContent` and `countTokens` response replays (both
the optimized fast path, the main path and the continuation-error
replay) copied the upstream headers by hand and only removed
`content-encoding`/`content-length`. A chunked upstream therefore leaked
`transfer-encoding: chunked` to a client that receives an
already-decoded, re-framed JSON body — the RFC 9112 wrong-framing class
this codebase already avoids elsewhere — along with `connection`,
`keep-alive` and the upstream `server` banner. Other paths in the same
file and other providers already use the shared
`sanitize_forwarded_response_headers`; these five sites were missed.

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (non-breaking change that only restructures code)
- [ ] Tests (no production code change)
- [ ] Documentation (no production code change)
- [ ] Chore (dependencies, CI, tooling, etc.)

## Changes Made

- Import and use `sanitize_forwarded_response_headers` in the five
Gemini response-replay sites instead of hand-copying headers.
- Add regressions for `generateContent` and `countTokens` that assert
`server`, `keep-alive`, `connection` and `transfer-encoding` from the
upstream do not reach the client while `content-type`, body and
`x-headroom-*` stay intact.

## Testing

- [x] Unit tests pass (`tests/test_proxy` 491 passed)
- [x] New tests added
- [x] Manual testing performed (real uvicorn proxy + real httpx against
a local fake upstream)

### Test Output

```text
pytest tests/test_proxy/test_gemini_response_framing_headers.py -q
# parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client
# this branch:     2 passed
pytest tests/test_proxy -q                                     # 491 passed
ruff check / ruff format --check / mypy <changed file>         # pass
```

## Real Behavior Proof

<details>
<summary>real behavior proof details</summary>

- Exact command / steps: python -m pytest
tests/test_proxy/test_gemini_response_framing_headers.py -q with a local
uvicorn proxy, a local http.server fake upstream and a real httpx client
comparing the before/after response headers.
- Environment: macOS aarch64, Python 3.14.5 worktree venv, local uvicorn
proxy with a Python `http.server` fake upstream, real httpx client, no
provider keys.
- Observed result: before the change the client received `server:
uvicorn, BaseHTTP/0.6 ..., Upstream-Marker/9.9`, `keep-alive`,
`connection: keep-alive` and `transfer-encoding: chunked`; after the
change only uvicorn's own `server` header remains, with `content-type`,
body and `x-headroom-tokens-*` intact for `generateContent` and
`countTokens`.
- Not tested: a real Gemini API call, the optimized/compression branch
as a separate e2e, the `_GeminiContinuationError` branch as its own
case, Windows/Linux, and the other providers that still hand-copy
headers (left as a follow-up to keep this PR scoped).
</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Gemini response replays no longer
forward upstream transport framing or server headers (intended).
- Kill switch / disable path: revert this commit.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] No unrelated changes
- [x] Tests added/pass
- [ ] CHANGELOG updated — no changelog entry for this internal header
fix; say so if you want one
- [x] No secrets or local paths

## Notes

No linked issue (source-derived, with a reproduction); other replay
paths use the same hand-copy pattern and are left as a follow-up.
Duplicate `date` headers (upstream Date plus uvicorn Date) are the
shared sanitizer's existing behavior and out of scope.

## Paste relevant command output or artifact links here

<details>
<summary>Commands and output</summary>

```text
pytest tests/test_proxy/test_gemini_response_framing_headers.py -q
# parent revision: 2 failed - upstream server/keep-alive/connection/transfer-encoding reached the client
# this branch:    2 passed
pytest tests/test_proxy -q                                  # 491 passed
ruff check / ruff format --check / mypy <changed file>      # pass
```
</details>

## Screenshots (if applicable)

N/A - not user-visible; the evidence is the header comparison in the
Real Behavior Proof section (real uvicorn proxy + httpx client + local
fake upstream).

## Additional Notes

Other providers/buffered replay paths still hand-copy headers; left as a
follow-up to keep this PR scoped. Duplicate `date` headers are the
shared sanitizer's existing behavior and out of scope. No linked issue
(source-derived bug with a reproduction).

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4605"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Chen Yufeiyang
2026-11-04 09:13:56 -06:00
committed by GitHub
co-authored by JD Davis
parent ee2252a315
commit dfcac78119
2 changed files with 116 additions and 19 deletions
+9 -19
View File
@@ -27,6 +27,7 @@ from headroom.proxy.helpers import (
COMPRESSION_TIMEOUT_SECONDS,
extract_tags,
invalid_request_body_message,
sanitize_forwarded_response_headers,
)
from headroom.proxy.identity import resolve_memory_identity
from headroom.proxy.outcome import RequestOutcome
@@ -586,9 +587,7 @@ class GeminiHandlerMixin:
client=client,
)
)
response_headers = dict(response.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
response_headers = sanitize_forwarded_response_headers(response.headers)
response_headers["x-headroom-tokens-before"] = str(total_input_tokens)
response_headers["x-headroom-tokens-after"] = str(total_input_tokens)
response_headers["x-headroom-tokens-saved"] = "0"
@@ -1090,12 +1089,9 @@ class GeminiHandlerMixin:
else:
logger.info(f"[{request_id}] Gemini {model}: {original_tokens:,} tokens")
# Remove compression headers
response_headers = dict(response.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
# Inject Headroom compression metrics (for SaaS metering)
# Remove transport framing headers (the body is re-framed here)
# and inject Headroom compression metrics (for SaaS metering).
response_headers = sanitize_forwarded_response_headers(response.headers)
response_headers["x-headroom-tokens-before"] = str(original_tokens)
response_headers["x-headroom-tokens-after"] = str(optimized_tokens)
response_headers["x-headroom-tokens-saved"] = str(tokens_saved)
@@ -1118,9 +1114,7 @@ class GeminiHandlerMixin:
)
except _GeminiContinuationError as e:
await self.metrics.record_failed(provider=provider_name)
response_headers = dict(e.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
response_headers = sanitize_forwarded_response_headers(e.headers)
return Response(content=e.content, status_code=e.status_code, headers=response_headers)
except Exception as e:
await self.metrics.record_failed(provider=provider_name)
@@ -1434,9 +1428,7 @@ class GeminiHandlerMixin:
url += f"?key={query_params['key']}"
response = await self._retry_request("POST", url, headers, body)
response_headers = dict(response.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
response_headers = sanitize_forwarded_response_headers(response.headers)
return Response(
content=response.content,
status_code=response.status_code,
@@ -1585,10 +1577,8 @@ class GeminiHandlerMixin:
f"[{request_id}] Gemini countTokens {model}: {compressed_tokens:,} tokens"
)
# Remove compression headers
response_headers = dict(response.headers)
response_headers.pop("content-encoding", None)
response_headers.pop("content-length", None)
# Remove transport framing headers (the body is re-framed here).
response_headers = sanitize_forwarded_response_headers(response.headers)
return Response(
content=response.content,
@@ -0,0 +1,107 @@
"""Gemini response replay must strip upstream transport framing headers.
The Gemini generateContent / countTokens replay paths copied upstream headers
by hand and popped only content-encoding / content-length. They therefore
replayed headers the shared sanitize_forwarded_response_headers exists to drop:
transfer-encoding from a chunked upstream, connection-local connection /
keep-alive, and the upstream server banner.
A stale transfer-encoding on an already-decoded, re-framed body is the #3019
class of bug: RFC 9112 makes Transfer-Encoding override Content-Length, so the
client parses plain JSON as chunked frames.
"""
from __future__ import annotations
import json
import httpx
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
_UPSTREAM_HEADERS = {
"content-type": "application/json",
"server": "upstream-marker/9.9",
"keep-alive": "timeout=5, max=100",
"connection": "keep-alive",
"transfer-encoding": "chunked",
}
_FRAMING_HEADERS = ("server", "keep-alive", "connection", "transfer-encoding")
_GENERATE_BODY = json.dumps(
{
"candidates": [
{
"content": {"parts": [{"text": "hi"}], "role": "model"},
"finishReason": "STOP",
}
],
"usageMetadata": {
"promptTokenCount": 4,
"candidatesTokenCount": 1,
"totalTokenCount": 5,
},
}
).encode()
_COUNT_TOKENS_BODY = json.dumps({"totalTokens": 4}).encode()
def _client(body: bytes) -> TestClient:
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
image_optimize=False,
)
app = create_app(config)
proxy = app.state.proxy
async def _fake_retry(*_args: object, **_kwargs: object) -> httpx.Response:
return httpx.Response(200, headers=_UPSTREAM_HEADERS, content=body)
proxy._retry_request = _fake_retry # type: ignore[method-assign]
return TestClient(app)
def _assert_upstream_framing_is_stripped(response: httpx.Response) -> None:
lower = {name.lower() for name in response.headers}
for header in _FRAMING_HEADERS:
assert header not in lower, (
f"upstream {header!r} reached the client: {dict(response.headers)}"
)
assert response.headers["content-type"].startswith("application/json")
def test_generate_content_replay_strips_upstream_framing_headers() -> None:
client = _client(_GENERATE_BODY)
response = client.post(
"/v1beta/models/gemini-pro:generateContent?key=test-key",
json={"contents": [{"role": "user", "parts": [{"text": "hi"}]}]},
)
assert response.status_code == 200
assert response.json()["candidates"][0]["content"]["parts"][0]["text"] == "hi"
_assert_upstream_framing_is_stripped(response)
def test_count_tokens_replay_strips_upstream_framing_headers() -> None:
client = _client(_COUNT_TOKENS_BODY)
response = client.post(
"/v1beta/models/gemini-pro:countTokens?key=test-key",
json={"contents": [{"role": "user", "parts": [{"text": "hi"}]}]},
)
assert response.status_code == 200
assert response.json()["totalTokens"] == 4
_assert_upstream_framing_is_stripped(response)