mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description Gemini's REST API accepts part fields under both their JSON (camelCase) names and their proto (snake_case) names, and Google's own `curl` examples use snake_case: `inline_data` / `mime_type`, `file_data`, `function_call`, `function_response`. Clients that build requests that way (hand-rolled `curl`/`requests` code, copies of Google's docs samples) send exactly those keys. `_has_non_text_parts()` in `headroom/proxy/handlers/gemini.py` only checks the camelCase keys. A snake_case image or tool part is not marked as preserved, so whenever compression rewrites the request (`optimized_messages != messages`), the content is rebuilt from text alone: - an entry with text plus `inline_data` loses the image, - `function_call` / `function_response` entries have no text at all, so they are dropped outright, and Gemini then receives a conversation with the image and tool turns missing. Through the real proxy, a 4-entry request reached the upstream as 2 entries (see the proof below). The same helper feeds the `cloudcode`, `countTokens` and batch paths. Closes: N/A (no matching issue found) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/gemini.py`: `_has_non_text_parts()` also recognises `inline_data`, `file_data`, `function_call`, `function_response`, `executable_code`, `code_execution_result` and `thought_signature`, the snake_case forms of the keys it already checks; docstring notes both spellings. - `headroom/proxy/handlers/gemini.py`: the telemetry-only `include_function_responses` conversion (#819) reads `function_response` as well as `functionResponse`, so snake_case tool output also reaches waste-signal detection (added after review). - `tests/test_google_multimodal.py`: - the existing `test_each_non_text_key_detected` parametrization gains the seven snake_case keys; - `test_snake_case_parts_survive_compression_round_trip` runs the handler's own `_gemini_contents_to_messages` → modify text → `_messages_to_gemini_contents` → `_rebuild_gemini_contents` sequence on a snake_case image / `function_call` / `function_response` conversation and checks those three entries come back unchanged. - `tests/test_gemini_function_response_waste.py`: `test_flag_emits_tool_message_for_snake_case_function_response`. ## Testing - [x] Unit tests pass (`pytest`): `tests/test_google_multimodal.py`, `tests/test_proxy_handlers_batch.py`, `tests/test_gemini_*.py`, `tests/test_proxy_gemini_*.py`, not the full suite - [x] Linting passes (`ruff check .`): on the changed files - [ ] Type checking passes (`mypy headroom`): not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New tests on `main` (fix reverted, tests kept): ```text $ python -m pytest tests/test_google_multimodal.py -q FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[inline_data] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[file_data] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[function_call] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[function_response] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[executable_code] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[code_execution_result] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_each_non_text_key_detected[thought_signature] FAILED tests/test_google_multimodal.py::TestHasNonTextParts::test_snake_case_parts_survive_compression_round_trip ========================= 8 failed, 61 passed in 8.25s ========================= ``` With the fix: ```text $ python -m pytest tests/test_google_multimodal.py tests/test_proxy_handlers_batch.py tests/test_gemini_*.py tests/test_proxy_gemini_*.py -q ======================= 148 passed, 27 skipped in 16.41s ======================= # (the skips are the live-API integration tests, which need a real GEMINI_API_KEY) $ ruff check headroom/proxy/handlers/gemini.py tests/test_google_multimodal.py tests/test_gemini_function_response_waste.py All checks passed! $ ruff format --check (same files) 3 files already formatted # waste-telemetry test on the previous commit (snake_case not read): E AssertionError: assert ['user'] == ['user', 'tool'] ``` ## Real Behavior Proof - Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this branch (native `headroom._core` from the 0.41.0 wheel), real `headroom proxy` in front of a local stub Gemini upstream (a small `ThreadingHTTPServer` that logs every request body and returns a canned `generateContent` reply), `HEADROOM_OFFLINE=1` - Exact command / steps: started the stub on :18931 and `HEADROOM_OFFLINE=1 python -m headroom.cli proxy --port 18932 --mode token --gemini-api-url http://127.0.0.1:18931 --no-rate-limit`, then `POST /v1beta/models/gemini-2.5-flash:generateContent` with 4 snake_case entries: user text + `inline_data` PNG, model `function_call`, user `function_response`, and a user turn with a 300-row JSON log (33 KB) so compression rewrites the request. Read back what the stub received. Same steps on `main` and this branch. - Observed result: both runs compress the log turn (33 KB → 10.8 KB; proxy log: `Tokens saved: 4,482`). On `main` the upstream gets 2 of 4 entries, with the image, `function_call` and `function_response` gone; on this branch it gets all 4, with only the log text compressed. ```text == main proxy status: 200 upstream received 2 of 4 contents entries user ['text'] len=60 user ['text'] len=10822 == this branch proxy status: 200 upstream received 4 of 4 contents entries user ['text', 'inline_data'] len=141 model ['function_call'] len=97 user ['function_response'] len=99 user ['text'] len=10822 ``` - Not tested: a real Gemini API key / live model (stub upstream only); the `cloudcode`, `countTokens` and batch endpoints, which call the same helper; streaming `streamGenerateContent`. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: only for requests that use snake_case part names: those entries are now kept verbatim, like their camelCase equivalents already were. camelCase requests are unchanged. - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - A request where **every** entry has a non-text part is forwarded unchanged by the existing early exit (`len(preserved_indices) == len(contents)`), before memory injection. That is already how camelCase `inlineData` requests behave on `main`; this PR only makes snake_case match it. - `headroom/image/compressor.py` also reads only `inlineData`. A snake_case image there is just not compressed, which is safe, so it isn't part of this PR. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
260 lines
10 KiB
Python
260 lines
10 KiB
Python
"""Gemini functionResponse waste-signal visibility (issue #819).
|
|
|
|
Gemini ``functionResponse`` parts are preserved verbatim on the wire (never
|
|
compressed), but their payloads previously never reached ``parse_messages``,
|
|
so tool output — where most waste lives — contributed nothing to waste
|
|
detection on the Gemini paths.
|
|
|
|
The fix is telemetry-only:
|
|
|
|
1. ``_gemini_contents_to_messages(..., include_function_responses=True)``
|
|
additionally emits each functionResponse payload as a ``role="tool"``
|
|
message.
|
|
2. ``TransformPipeline.apply(..., waste_messages=...)`` parses that richer
|
|
list for waste signals instead of the transform input. The transform path
|
|
and token accounting are untouched.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
pytest.importorskip("httpx")
|
|
|
|
from headroom import OpenAIProvider, Tokenizer
|
|
from headroom.config import HeadroomConfig
|
|
from headroom.parser import parse_messages
|
|
from headroom.proxy.server import HeadroomProxy, ProxyConfig
|
|
from headroom.transforms.pipeline import TransformPipeline
|
|
|
|
_provider = OpenAIProvider()
|
|
|
|
|
|
@pytest.fixture
|
|
def proxy() -> HeadroomProxy:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
)
|
|
return HeadroomProxy(config)
|
|
|
|
|
|
@pytest.fixture
|
|
def tokenizer() -> Tokenizer:
|
|
return Tokenizer(_provider.get_token_counter("gpt-4o"), "gpt-4o")
|
|
|
|
|
|
def _big_payload(rows: int = 200) -> dict:
|
|
return {
|
|
"result": [
|
|
{"id": i, "name": f"item_{i}", "status": "ok", "score": i * 3.14} for i in range(rows)
|
|
]
|
|
}
|
|
|
|
|
|
def _function_response_content(payload: object, name: str = "fetch_data") -> dict:
|
|
return {
|
|
"role": "user",
|
|
"parts": [{"functionResponse": {"name": name, "response": payload}}],
|
|
}
|
|
|
|
|
|
class TestFunctionResponseConversion:
|
|
def test_default_conversion_emits_no_tool_messages(self, proxy):
|
|
contents = [
|
|
{"role": "user", "parts": [{"text": "fetch the data"}]},
|
|
_function_response_content(_big_payload()),
|
|
]
|
|
messages, preserved = proxy._gemini_contents_to_messages(contents)
|
|
assert [m["role"] for m in messages] == ["user"]
|
|
assert preserved == {1}
|
|
|
|
def test_flag_emits_tool_message_for_dict_response(self, proxy):
|
|
payload = _big_payload()
|
|
contents = [
|
|
{"role": "user", "parts": [{"text": "fetch the data"}]},
|
|
_function_response_content(payload),
|
|
]
|
|
messages, preserved = proxy._gemini_contents_to_messages(
|
|
contents, include_function_responses=True
|
|
)
|
|
assert [m["role"] for m in messages] == ["user", "tool"]
|
|
assert json.loads(messages[1]["content"]) == payload
|
|
# preserved_indices semantics unchanged: the entry is still restored
|
|
# verbatim on the wire regardless of the telemetry conversion.
|
|
assert preserved == {1}
|
|
|
|
def test_flag_emits_tool_message_for_snake_case_function_response(self, proxy):
|
|
"""The REST API also accepts ``function_response``; its payload counts too."""
|
|
payload = _big_payload()
|
|
contents = [
|
|
{"role": "user", "parts": [{"text": "fetch the data"}]},
|
|
{
|
|
"role": "user",
|
|
"parts": [{"function_response": {"name": "fetch_data", "response": payload}}],
|
|
},
|
|
]
|
|
messages, preserved = proxy._gemini_contents_to_messages(
|
|
contents, include_function_responses=True
|
|
)
|
|
assert [m["role"] for m in messages] == ["user", "tool"]
|
|
assert json.loads(messages[1]["content"]) == payload
|
|
assert preserved == {1}
|
|
|
|
def test_flag_passes_string_response_through(self, proxy):
|
|
contents = [_function_response_content("plain text tool output")]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
|
|
assert messages == [{"role": "tool", "content": "plain text tool output"}]
|
|
|
|
def test_flag_skips_missing_response(self, proxy):
|
|
contents = [
|
|
{"role": "user", "parts": [{"functionResponse": {"name": "noop"}}]},
|
|
{"role": "user", "parts": [{"functionResponse": {"name": "none", "response": None}}]},
|
|
]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
|
|
assert messages == []
|
|
|
|
def test_flag_emits_text_before_tool_within_entry(self, proxy):
|
|
contents = [
|
|
{
|
|
"role": "user",
|
|
"parts": [
|
|
{"text": "tool said:"},
|
|
{"functionResponse": {"name": "f", "response": "output"}},
|
|
],
|
|
}
|
|
]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
|
|
assert [m["role"] for m in messages] == ["user", "tool"]
|
|
assert messages[0]["content"] == "tool said:"
|
|
assert messages[1]["content"] == "output"
|
|
|
|
def test_unserializable_response_falls_back_to_str(self, proxy):
|
|
circular: dict = {"name": "loop"}
|
|
circular["self"] = circular
|
|
text = proxy._function_response_text({"response": circular})
|
|
assert "loop" in text
|
|
|
|
|
|
class TestMalformedPartsToleration:
|
|
"""A request-controlled `parts` that is null or carries non-dict elements
|
|
must not crash the compression-path conversion helpers."""
|
|
|
|
def test_null_parts_does_not_crash(self, proxy):
|
|
contents = [
|
|
{"role": "user", "parts": None},
|
|
{"role": "user", "parts": [{"text": "real"}]},
|
|
]
|
|
messages, preserved = proxy._gemini_contents_to_messages(contents)
|
|
assert messages == [{"role": "user", "content": "real"}]
|
|
assert preserved == set()
|
|
|
|
def test_string_part_elements_do_not_crash(self, proxy):
|
|
# A client that treats `parts` as a string array sends bare strings;
|
|
# they carry no `text` key, so they contribute nothing but must not
|
|
# crash `.get`.
|
|
contents = [{"role": "user", "parts": ["bare string", {"text": "kept"}]}]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents)
|
|
assert messages == [{"role": "user", "content": "kept"}]
|
|
|
|
def test_null_part_element_is_skipped(self, proxy):
|
|
contents = [{"role": "user", "parts": [None, {"text": "kept"}]}]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents)
|
|
assert messages == [{"role": "user", "content": "kept"}]
|
|
|
|
def test_has_non_text_parts_tolerates_null_parts(self, proxy):
|
|
assert proxy._has_non_text_parts({"role": "user", "parts": None}) is False
|
|
assert proxy._has_non_text_parts({"role": "user", "parts": ["str"]}) is False
|
|
assert proxy._has_non_text_parts({"parts": [{"inlineData": {"data": "x"}}]}) is True
|
|
|
|
def test_non_dict_content_entry_is_tolerated(self, proxy):
|
|
# A non-dict entry in contents[] is treated as an empty user turn rather
|
|
# than crashing content.get / the parts iteration.
|
|
contents = ["not a dict", {"role": "user", "parts": [{"text": "kept"}]}]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents)
|
|
assert messages == [{"role": "user", "content": "kept"}]
|
|
|
|
|
|
class TestFunctionResponseWasteParsing:
|
|
def test_function_response_payload_reaches_waste_signals(self, proxy, tokenizer):
|
|
contents = [
|
|
{"role": "user", "parts": [{"text": "fetch the data"}]},
|
|
_function_response_content(_big_payload()),
|
|
]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
|
|
blocks, _, waste = parse_messages(messages, tokenizer)
|
|
assert any(b.kind == "tool_result" for b in blocks)
|
|
assert waste.json_bloat_tokens > 0
|
|
|
|
def test_repeated_function_response_counts_as_reread(self, proxy, tokenizer):
|
|
payload = _big_payload()
|
|
filler = [{"role": "user", "parts": [{"text": f"working on step {i}"}]} for i in range(5)]
|
|
contents = [
|
|
_function_response_content(payload),
|
|
*filler,
|
|
_function_response_content(payload),
|
|
]
|
|
messages, _ = proxy._gemini_contents_to_messages(contents, include_function_responses=True)
|
|
_, _, waste = parse_messages(messages, tokenizer)
|
|
assert waste.reread_tokens > 0
|
|
|
|
|
|
class TestPipelineWasteMessages:
|
|
@staticmethod
|
|
def _base_messages() -> list[dict]:
|
|
# Compressible enough that the pipeline clears the >100 saved-token
|
|
# gate that guards waste-signal detection.
|
|
return [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Inspect the data set."},
|
|
{"role": "tool", "content": json.dumps(_big_payload(400)["result"])},
|
|
]
|
|
|
|
def test_waste_messages_override_waste_source(self, tokenizer):
|
|
messages = self._base_messages()
|
|
extra_tool = {"role": "tool", "content": json.dumps(_big_payload(300))}
|
|
|
|
baseline = TransformPipeline(HeadroomConfig()).apply(
|
|
[dict(m) for m in messages], model="gpt-4o", model_limit=128000
|
|
)
|
|
enriched = TransformPipeline(HeadroomConfig()).apply(
|
|
[dict(m) for m in messages],
|
|
model="gpt-4o",
|
|
model_limit=128000,
|
|
waste_messages=[*messages, extra_tool],
|
|
)
|
|
|
|
assert baseline.waste_signals is not None
|
|
assert enriched.waste_signals is not None
|
|
assert enriched.waste_signals.json_bloat_tokens > baseline.waste_signals.json_bloat_tokens
|
|
|
|
def test_waste_messages_do_not_affect_transform_output(self, tokenizer):
|
|
messages = self._base_messages()
|
|
extra_tool = {"role": "tool", "content": json.dumps(_big_payload(300))}
|
|
|
|
baseline = TransformPipeline(HeadroomConfig()).apply(
|
|
[dict(m) for m in messages], model="gpt-4o", model_limit=128000
|
|
)
|
|
enriched = TransformPipeline(HeadroomConfig()).apply(
|
|
[dict(m) for m in messages],
|
|
model="gpt-4o",
|
|
model_limit=128000,
|
|
waste_messages=[*messages, extra_tool],
|
|
)
|
|
|
|
assert enriched.messages == baseline.messages
|
|
assert enriched.tokens_before == baseline.tokens_before
|
|
assert enriched.tokens_after == baseline.tokens_after
|
|
|
|
def test_no_waste_messages_falls_back_to_transform_input(self, tokenizer):
|
|
result = TransformPipeline(HeadroomConfig()).apply(
|
|
[dict(m) for m in self._base_messages()], model="gpt-4o", model_limit=128000
|
|
)
|
|
assert result.waste_signals is not None
|
|
assert result.waste_signals.json_bloat_tokens > 0
|