fix(gemini): keep snake_case inline_data and function parts when compressing (#4290)

## Description

Gemini's REST API accepts part fields under both their JSON (camelCase)
names and their proto (snake_case) names, and Google's own `curl`
examples use snake_case: `inline_data` / `mime_type`, `file_data`,
`function_call`, `function_response`. Clients that build requests that
way (hand-rolled `curl`/`requests` code, copies of Google's docs
samples) send exactly those keys.

`_has_non_text_parts()` in `headroom/proxy/handlers/gemini.py` only
checks the camelCase keys. A snake_case image or tool part is not marked
as preserved, so whenever compression rewrites the request
(`optimized_messages != messages`), the content is rebuilt from text
alone:

- an entry with text plus `inline_data` loses the image,
- `function_call` / `function_response` entries have no text at all, so
they are dropped outright,

and Gemini then receives a conversation with the image and tool turns
missing. Through the real proxy, a 4-entry request reached the upstream
as 2 entries (see the proof below). The same helper feeds the
`cloudcode`, `countTokens` and batch paths.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/gemini.py`: `_has_non_text_parts()` also
recognises `inline_data`, `file_data`, `function_call`,
`function_response`, `executable_code`, `code_execution_result` and
`thought_signature`, the snake_case forms of the keys it already checks;
docstring notes both spellings.
- `headroom/proxy/handlers/gemini.py`: the telemetry-only
`include_function_responses` conversion (#819) reads `function_response`
as well as `functionResponse`, so snake_case tool output also reaches
waste-signal detection (added after review).
- `tests/test_google_multimodal.py`:
- the existing `test_each_non_text_key_detected` parametrization gains
the seven snake_case keys;
- `test_snake_case_parts_survive_compression_round_trip` runs the
handler's own `_gemini_contents_to_messages` → modify text →
`_messages_to_gemini_contents` → `_rebuild_gemini_contents` sequence on
a snake_case image / `function_call` / `function_response` conversation
and checks those three entries come back unchanged.
- `tests/test_gemini_function_response_waste.py`:
`test_flag_emits_tool_message_for_snake_case_function_response`.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_google_multimodal.py`,
`tests/test_proxy_handlers_batch.py`, `tests/test_gemini_*.py`,
`tests/test_proxy_gemini_*.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `main` (fix reverted, tests kept):

```text
$ python -m pytest tests/test_google_multimodal.py -q
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[inline_data]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[file_data]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[function_call]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[function_response]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[executable_code]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[code_execution_result]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[thought_signature]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_snake_case_parts_survive_compression_round_trip
========================= 8 failed, 61 passed in 8.25s =========================
```

With the fix:

```text
$ python -m pytest tests/test_google_multimodal.py tests/test_proxy_handlers_batch.py tests/test_gemini_*.py tests/test_proxy_gemini_*.py -q
======================= 148 passed, 27 skipped in 16.41s =======================
# (the skips are the live-API integration tests, which need a real GEMINI_API_KEY)

$ ruff check headroom/proxy/handlers/gemini.py tests/test_google_multimodal.py tests/test_gemini_function_response_waste.py
All checks passed!
$ ruff format --check (same files)
3 files already formatted

# waste-telemetry test on the previous commit (snake_case not read):
E   AssertionError: assert ['user'] == ['user', 'tool']
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel), real `headroom
proxy` in front of a local stub Gemini upstream (a small
`ThreadingHTTPServer` that logs every request body and returns a canned
`generateContent` reply), `HEADROOM_OFFLINE=1`
- Exact command / steps: started the stub on :18931 and
`HEADROOM_OFFLINE=1 python -m headroom.cli proxy --port 18932 --mode
token --gemini-api-url http://127.0.0.1:18931 --no-rate-limit`, then
`POST /v1beta/models/gemini-2.5-flash:generateContent` with 4 snake_case
entries: user text + `inline_data` PNG, model `function_call`, user
`function_response`, and a user turn with a 300-row JSON log (33 KB) so
compression rewrites the request. Read back what the stub received. Same
steps on `main` and this branch.
- Observed result: both runs compress the log turn (33 KB → 10.8 KB;
proxy log: `Tokens saved: 4,482`). On `main` the upstream gets 2 of 4
entries, with the image, `function_call` and `function_response` gone;
on this branch it gets all 4, with only the log text compressed.

  ```text
  == main
  proxy status: 200
  upstream received 2 of 4 contents entries
     user ['text'] len=60
     user ['text'] len=10822
  == this branch
  proxy status: 200
  upstream received 4 of 4 contents entries
     user ['text', 'inline_data'] len=141
     model ['function_call'] len=97
     user ['function_response'] len=99
     user ['text'] len=10822
  ```
- Not tested: a real Gemini API key / live model (stub upstream only);
the `cloudcode`, `countTokens` and batch endpoints, which call the same
helper; streaming `streamGenerateContent`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for requests that use snake_case
part names: those entries are now kept verbatim, like their camelCase
equivalents already were. camelCase requests are unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- A request where **every** entry has a non-text part is forwarded
unchanged by the existing early exit (`len(preserved_indices) ==
len(contents)`), before memory injection. That is already how camelCase
`inlineData` requests behave on `main`; this PR only makes snake_case
match it.
- `headroom/image/compressor.py` also reads only `inlineData`. A
snake_case image there is just not compressed, which is safe, so it
isn't part of this PR.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Rudy Mizrahi Celekli
2026-10-24 08:38:10 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 JerrettDavis
parent 2480100915
commit 4bf1bf55d1
3 changed files with 70 additions and 2 deletions
+14 -2
View File
@@ -156,6 +156,10 @@ class GeminiHandlerMixin:
- thought / thoughtSignature: thinking parts, which must round-trip intact
- non-string text, which the text round trip cannot rebuild
The REST API accepts both the camelCase JSON names and the snake_case
proto field names (``inline_data``, ``function_call``, ...), so both
spellings are checked.
Args:
content: A single Gemini content entry with 'parts' list.
@@ -174,6 +178,13 @@ class GeminiHandlerMixin:
"codeExecutionResult",
"thought",
"thoughtSignature",
"inline_data",
"file_data",
"function_call",
"function_response",
"executable_code",
"code_execution_result",
"thought_signature",
)
):
return True
@@ -269,9 +280,10 @@ class GeminiHandlerMixin:
if include_function_responses:
for part in parts:
if "functionResponse" not in part:
function_response = part.get("functionResponse", part.get("function_response"))
if function_response is None:
continue
payload = self._function_response_text(part["functionResponse"])
payload = self._function_response_text(function_response)
if payload:
messages.append({"role": "tool", "content": payload})
@@ -89,6 +89,23 @@ class TestFunctionResponseConversion:
# verbatim on the wire regardless of the telemetry conversion.
assert preserved == {1}
def test_flag_emits_tool_message_for_snake_case_function_response(self, proxy):
"""The REST API also accepts ``function_response``; its payload counts too."""
payload = _big_payload()
contents = [
{"role": "user", "parts": [{"text": "fetch the data"}]},
{
"role": "user",
"parts": [{"function_response": {"name": "fetch_data", "response": payload}}],
},
]
messages, preserved = proxy._gemini_contents_to_messages(
contents, include_function_responses=True
)
assert [m["role"] for m in messages] == ["user", "tool"]
assert json.loads(messages[1]["content"]) == payload
assert preserved == {1}
def test_flag_passes_string_response_through(self, proxy):
contents = [_function_response_content("plain text tool output")]
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
+39
View File
@@ -182,6 +182,15 @@ class TestHasNonTextParts:
# turns; previously not detected, so they were dropped on round-trip.
"executableCode",
"codeExecutionResult",
# The REST API also accepts the proto field names, which Google's
# own curl examples use; these were dropped on round-trip too.
"inline_data",
"file_data",
"function_call",
"function_response",
"executable_code",
"code_execution_result",
"thought_signature",
],
)
def test_each_non_text_key_detected(self, proxy, non_text_key):
@@ -194,6 +203,36 @@ class TestHasNonTextParts:
content = {"role": "user"}
assert proxy._has_non_text_parts(content) is False
def test_snake_case_parts_survive_compression_round_trip(self, proxy):
"""snake_case image and tool parts are restored after compression rewrites text."""
contents = [
{
"role": "user",
"parts": [
{"text": "Describe this image"},
{"inline_data": {"mime_type": "image/png", "data": "iVBORw0KGgo="}},
],
},
{"role": "model", "parts": [{"function_call": {"name": "lookup", "args": {}}}]},
{
"role": "user",
"parts": [{"function_response": {"name": "lookup", "response": {"r": "ok"}}}],
},
{"role": "user", "parts": [{"text": "thanks " * 100}]},
]
messages, preserved = proxy._gemini_contents_to_messages(contents)
assert preserved == {0, 1, 2}
optimized = [dict(m) for m in messages]
optimized[-1]["content"] = "thanks (compressed)"
optimized_contents, _ = proxy._messages_to_gemini_contents(optimized)
rebuilt = proxy._rebuild_gemini_contents(
contents, preserved, {i: contents[i] for i in preserved}, optimized_contents
)
assert rebuilt[:3] == contents[:3]
assert rebuilt[3] == {"role": "user", "parts": [{"text": "thanks (compressed)"}]}
# =============================================================================
# Tests for _gemini_contents_to_messages preserved indices