Files
headroom/tests/test_gemini_function_response_waste.py
4bf1bf55d1 fix(gemini): keep snake_case inline_data and function parts when compressing (#4290)
## Description

Gemini's REST API accepts part fields under both their JSON (camelCase)
names and their proto (snake_case) names, and Google's own `curl`
examples use snake_case: `inline_data` / `mime_type`, `file_data`,
`function_call`, `function_response`. Clients that build requests that
way (hand-rolled `curl`/`requests` code, copies of Google's docs
samples) send exactly those keys.

`_has_non_text_parts()` in `headroom/proxy/handlers/gemini.py` only
checks the camelCase keys. A snake_case image or tool part is not marked
as preserved, so whenever compression rewrites the request
(`optimized_messages != messages`), the content is rebuilt from text
alone:

- an entry with text plus `inline_data` loses the image,
- `function_call` / `function_response` entries have no text at all, so
they are dropped outright,

and Gemini then receives a conversation with the image and tool turns
missing. Through the real proxy, a 4-entry request reached the upstream
as 2 entries (see the proof below). The same helper feeds the
`cloudcode`, `countTokens` and batch paths.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/gemini.py`: `_has_non_text_parts()` also
recognises `inline_data`, `file_data`, `function_call`,
`function_response`, `executable_code`, `code_execution_result` and
`thought_signature`, the snake_case forms of the keys it already checks;
docstring notes both spellings.
- `headroom/proxy/handlers/gemini.py`: the telemetry-only
`include_function_responses` conversion (#819) reads `function_response`
as well as `functionResponse`, so snake_case tool output also reaches
waste-signal detection (added after review).
- `tests/test_google_multimodal.py`:
- the existing `test_each_non_text_key_detected` parametrization gains
the seven snake_case keys;
- `test_snake_case_parts_survive_compression_round_trip` runs the
handler's own `_gemini_contents_to_messages` → modify text →
`_messages_to_gemini_contents` → `_rebuild_gemini_contents` sequence on
a snake_case image / `function_call` / `function_response` conversation
and checks those three entries come back unchanged.
- `tests/test_gemini_function_response_waste.py`:
`test_flag_emits_tool_message_for_snake_case_function_response`.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_google_multimodal.py`,
`tests/test_proxy_handlers_batch.py`, `tests/test_gemini_*.py`,
`tests/test_proxy_gemini_*.py`, not the full suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `main` (fix reverted, tests kept):

```text
$ python -m pytest tests/test_google_multimodal.py -q
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[inline_data]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[file_data]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[function_call]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[function_response]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[executable_code]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[code_execution_result]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[thought_signature]
FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_snake_case_parts_survive_compression_round_trip
========================= 8 failed, 61 passed in 8.25s =========================
```

With the fix:

```text
$ python -m pytest tests/test_google_multimodal.py tests/test_proxy_handlers_batch.py tests/test_gemini_*.py tests/test_proxy_gemini_*.py -q
======================= 148 passed, 27 skipped in 16.41s =======================
# (the skips are the live-API integration tests, which need a real GEMINI_API_KEY)

$ ruff check headroom/proxy/handlers/gemini.py tests/test_google_multimodal.py tests/test_gemini_function_response_waste.py
All checks passed!
$ ruff format --check (same files)
3 files already formatted

# waste-telemetry test on the previous commit (snake_case not read):
E   AssertionError: assert ['user'] == ['user', 'tool']
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel), real `headroom
proxy` in front of a local stub Gemini upstream (a small
`ThreadingHTTPServer` that logs every request body and returns a canned
`generateContent` reply), `HEADROOM_OFFLINE=1`
- Exact command / steps: started the stub on :18931 and
`HEADROOM_OFFLINE=1 python -m headroom.cli proxy --port 18932 --mode
token --gemini-api-url http://127.0.0.1:18931 --no-rate-limit`, then
`POST /v1beta/models/gemini-2.5-flash:generateContent` with 4 snake_case
entries: user text + `inline_data` PNG, model `function_call`, user
`function_response`, and a user turn with a 300-row JSON log (33 KB) so
compression rewrites the request. Read back what the stub received. Same
steps on `main` and this branch.
- Observed result: both runs compress the log turn (33 KB → 10.8 KB;
proxy log: `Tokens saved: 4,482`). On `main` the upstream gets 2 of 4
entries, with the image, `function_call` and `function_response` gone;
on this branch it gets all 4, with only the log text compressed.

  ```text
  == main
  proxy status: 200
  upstream received 2 of 4 contents entries
     user ['text'] len=60
     user ['text'] len=10822
  == this branch
  proxy status: 200
  upstream received 4 of 4 contents entries
     user ['text', 'inline_data'] len=141
     model ['function_call'] len=97
     user ['function_response'] len=99
     user ['text'] len=10822
  ```
- Not tested: a real Gemini API key / live model (stub upstream only);
the `cloudcode`, `countTokens` and batch endpoints, which call the same
helper; streaming `streamGenerateContent`.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for requests that use snake_case
part names: those entries are now kept verbatim, like their camelCase
equivalents already were. camelCase requests are unchanged.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- A request where **every** entry has a non-text part is forwarded
unchanged by the existing early exit (`len(preserved_indices) ==
len(contents)`), before memory injection. That is already how camelCase
`inlineData` requests behave on `main`; this PR only makes snake_case
match it.
- `headroom/image/compressor.py` also reads only `inlineData`. A
snake_case image there is just not compressed, which is safe, so it
isn't part of this PR.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-24 08:38:10 -07:00

260 lines
10 KiB
Python

"""Gemini functionResponse waste-signal visibility (issue #819).
Gemini ``functionResponse`` parts are preserved verbatim on the wire (never
compressed), but their payloads previously never reached ``parse_messages``,
so tool output — where most waste lives — contributed nothing to waste
detection on the Gemini paths.
The fix is telemetry-only:
1. ``_gemini_contents_to_messages(..., include_function_responses=True)``
additionally emits each functionResponse payload as a ``role="tool"``
message.
2. ``TransformPipeline.apply(..., waste_messages=...)`` parses that richer
list for waste signals instead of the transform input. The transform path
and token accounting are untouched.
"""
from __future__ import annotations
import json
import pytest
pytest.importorskip("fastapi")
pytest.importorskip("httpx")
from headroom import OpenAIProvider, Tokenizer
from headroom.config import HeadroomConfig
from headroom.parser import parse_messages
from headroom.proxy.server import HeadroomProxy, ProxyConfig
from headroom.transforms.pipeline import TransformPipeline
_provider = OpenAIProvider()
@pytest.fixture
def proxy() -> HeadroomProxy:
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
return HeadroomProxy(config)
@pytest.fixture
def tokenizer() -> Tokenizer:
return Tokenizer(_provider.get_token_counter("gpt-4o"), "gpt-4o")
def _big_payload(rows: int = 200) -> dict:
return {
"result": [
{"id": i, "name": f"item_{i}", "status": "ok", "score": i * 3.14} for i in range(rows)
]
}
def _function_response_content(payload: object, name: str = "fetch_data") -> dict:
return {
"role": "user",
"parts": [{"functionResponse": {"name": name, "response": payload}}],
}
class TestFunctionResponseConversion:
def test_default_conversion_emits_no_tool_messages(self, proxy):
contents = [
{"role": "user", "parts": [{"text": "fetch the data"}]},
_function_response_content(_big_payload()),
]
messages, preserved = proxy._gemini_contents_to_messages(contents)
assert [m["role"] for m in messages] == ["user"]
assert preserved == {1}
def test_flag_emits_tool_message_for_dict_response(self, proxy):
payload = _big_payload()
contents = [
{"role": "user", "parts": [{"text": "fetch the data"}]},
_function_response_content(payload),
]
messages, preserved = proxy._gemini_contents_to_messages(
contents, include_function_responses=True
)
assert [m["role"] for m in messages] == ["user", "tool"]
assert json.loads(messages[1]["content"]) == payload
# preserved_indices semantics unchanged: the entry is still restored
# verbatim on the wire regardless of the telemetry conversion.
assert preserved == {1}
def test_flag_emits_tool_message_for_snake_case_function_response(self, proxy):
"""The REST API also accepts ``function_response``; its payload counts too."""
payload = _big_payload()
contents = [
{"role": "user", "parts": [{"text": "fetch the data"}]},
{
"role": "user",
"parts": [{"function_response": {"name": "fetch_data", "response": payload}}],
},
]
messages, preserved = proxy._gemini_contents_to_messages(
contents, include_function_responses=True
)
assert [m["role"] for m in messages] == ["user", "tool"]
assert json.loads(messages[1]["content"]) == payload
assert preserved == {1}
def test_flag_passes_string_response_through(self, proxy):
contents = [_function_response_content("plain text tool output")]
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
assert messages == [{"role": "tool", "content": "plain text tool output"}]
def test_flag_skips_missing_response(self, proxy):
contents = [
{"role": "user", "parts": [{"functionResponse": {"name": "noop"}}]},
{"role": "user", "parts": [{"functionResponse": {"name": "none", "response": None}}]},
]
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
assert messages == []
def test_flag_emits_text_before_tool_within_entry(self, proxy):
contents = [
{
"role": "user",
"parts": [
{"text": "tool said:"},
{"functionResponse": {"name": "f", "response": "output"}},
],
}
]
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
assert [m["role"] for m in messages] == ["user", "tool"]
assert messages[0]["content"] == "tool said:"
assert messages[1]["content"] == "output"
def test_unserializable_response_falls_back_to_str(self, proxy):
circular: dict = {"name": "loop"}
circular["self"] = circular
text = proxy._function_response_text({"response": circular})
assert "loop" in text
class TestMalformedPartsToleration:
"""A request-controlled `parts` that is null or carries non-dict elements
must not crash the compression-path conversion helpers."""
def test_null_parts_does_not_crash(self, proxy):
contents = [
{"role": "user", "parts": None},
{"role": "user", "parts": [{"text": "real"}]},
]
messages, preserved = proxy._gemini_contents_to_messages(contents)
assert messages == [{"role": "user", "content": "real"}]
assert preserved == set()
def test_string_part_elements_do_not_crash(self, proxy):
# A client that treats `parts` as a string array sends bare strings;
# they carry no `text` key, so they contribute nothing but must not
# crash `.get`.
contents = [{"role": "user", "parts": ["bare string", {"text": "kept"}]}]
messages, _ = proxy._gemini_contents_to_messages(contents)
assert messages == [{"role": "user", "content": "kept"}]
def test_null_part_element_is_skipped(self, proxy):
contents = [{"role": "user", "parts": [None, {"text": "kept"}]}]
messages, _ = proxy._gemini_contents_to_messages(contents)
assert messages == [{"role": "user", "content": "kept"}]
def test_has_non_text_parts_tolerates_null_parts(self, proxy):
assert proxy._has_non_text_parts({"role": "user", "parts": None}) is False
assert proxy._has_non_text_parts({"role": "user", "parts": ["str"]}) is False
assert proxy._has_non_text_parts({"parts": [{"inlineData": {"data": "x"}}]}) is True
def test_non_dict_content_entry_is_tolerated(self, proxy):
# A non-dict entry in contents[] is treated as an empty user turn rather
# than crashing content.get / the parts iteration.
contents = ["not a dict", {"role": "user", "parts": [{"text": "kept"}]}]
messages, _ = proxy._gemini_contents_to_messages(contents)
assert messages == [{"role": "user", "content": "kept"}]
class TestFunctionResponseWasteParsing:
def test_function_response_payload_reaches_waste_signals(self, proxy, tokenizer):
contents = [
{"role": "user", "parts": [{"text": "fetch the data"}]},
_function_response_content(_big_payload()),
]
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
blocks, _, waste = parse_messages(messages, tokenizer)
assert any(b.kind == "tool_result" for b in blocks)
assert waste.json_bloat_tokens > 0
def test_repeated_function_response_counts_as_reread(self, proxy, tokenizer):
payload = _big_payload()
filler = [{"role": "user", "parts": [{"text": f"working on step {i}"}]} for i in range(5)]
contents = [
_function_response_content(payload),
*filler,
_function_response_content(payload),
]
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
_, _, waste = parse_messages(messages, tokenizer)
assert waste.reread_tokens > 0
class TestPipelineWasteMessages:
@staticmethod
def _base_messages() -> list[dict]:
# Compressible enough that the pipeline clears the >100 saved-token
# gate that guards waste-signal detection.
return [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Inspect the data set."},
{"role": "tool", "content": json.dumps(_big_payload(400)["result"])},
]
def test_waste_messages_override_waste_source(self, tokenizer):
messages = self._base_messages()
extra_tool = {"role": "tool", "content": json.dumps(_big_payload(300))}
baseline = TransformPipeline(HeadroomConfig()).apply(
[dict(m) for m in messages], model="gpt-4o", model_limit=128000
)
enriched = TransformPipeline(HeadroomConfig()).apply(
[dict(m) for m in messages],
model="gpt-4o",
model_limit=128000,
waste_messages=[*messages, extra_tool],
)
assert baseline.waste_signals is not None
assert enriched.waste_signals is not None
assert enriched.waste_signals.json_bloat_tokens > baseline.waste_signals.json_bloat_tokens
def test_waste_messages_do_not_affect_transform_output(self, tokenizer):
messages = self._base_messages()
extra_tool = {"role": "tool", "content": json.dumps(_big_payload(300))}
baseline = TransformPipeline(HeadroomConfig()).apply(
[dict(m) for m in messages], model="gpt-4o", model_limit=128000
)
enriched = TransformPipeline(HeadroomConfig()).apply(
[dict(m) for m in messages],
model="gpt-4o",
model_limit=128000,
waste_messages=[*messages, extra_tool],
)
assert enriched.messages == baseline.messages
assert enriched.tokens_before == baseline.tokens_before
assert enriched.tokens_after == baseline.tokens_after
def test_no_waste_messages_falls_back_to_transform_input(self, tokenizer):
result = TransformPipeline(HeadroomConfig()).apply(
[dict(m) for m in self._base_messages()], model="gpt-4o", model_limit=128000
)
assert result.waste_signals is not None
assert result.waste_signals.json_bloat_tokens > 0