fix(proxy): keep Gemini memory context on entries with non-text parts (#4506)

## Description

Memory context on the Gemini path is meant to go to the live-zone tail,
the latest user turn. When that turn's `contents[]` entry carries a
non-text part next to its text (a `functionResponse`, an `inlineData`
image, ...), the memory block never reaches the upstream request,
although the proxy logs it as injected. This is the Gemini half of
#4419.

The handler converts `contents[]` to OpenAI-shaped messages, appends the
memory block to the latest user message (`handlers/gemini.py`,
memory-injection block), logs `injected_live_zone_tail_gemini`, and then
rebuilds `contents[]`. `_gemini_contents_to_messages` marks every entry
with a non-text part as preserved, and `_rebuild_gemini_contents`
forwards preserved entries verbatim and discards their converted
message. That is right for compression, which must not rewrite those
entries, but it also discards the memory text that was appended to that
message.

This PR carries the appended suffix onto a copy of the preserved entry's
last text part (option 3 in the issue). The forwarded bytes match what a
text-only entry gets.

The OpenAI chat half of #4419 (memory landing on an older user turn when
the transcript ends in `assistant` tool calls and `tool` results) is a
choice between the issue's options 1 and 2, which the reporter left to
maintainers. It is not changed here.

Refs #4419

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/gemini.py`:
- New `_carry_tail_append_into_preserved` static method. It finds the
one message the tail append changed and maps it to its `contents[]`
index using the alignment `_rebuild_gemini_contents` already relies on:
the k-th non-system message pairs with the k-th entry that has a text
part. If that entry is preserved, the method stores a copy of it with
the appended suffix added to its last text part. It returns without
changes when the target entry is not preserved, when more or fewer than
one message changed, or when the new content is not the old content plus
a suffix.
- `handle_gemini_generate_content` calls it right after
`_append_to_latest_user_tail` reports `bytes_appended > 0`, before the
existing log and rebuild. If the append targets a preserved entry that
is older than the latest user entry (the latest one has no text part),
the helper returns False and the handler skips the injection: writing
memory into that older turn would change an already-cached prefix, and
logging it would report bytes that never reach the wire.
- Every part of the preserved entry is kept as-is, including non-dict
parts. The parsed request entry is copied, not mutated in place.
- `tests/test_gemini_memory_tail_preserved.py` (new): drives
`handle_gemini_generate_content` with the real
`MemoryHandler._append_to_latest_user_tail` and captures the body sent
upstream:
- latest user entry = `functionResponse` + text: memory is on the text
part, the `functionResponse` part and older turns are unchanged, and the
request entry is not mutated;
- latest user entry = text + `inlineData` + text: memory is after the
last text part;
  - text-only latest entry (control): unchanged behavior;
  - the helper does nothing when the target entry is not preserved;
- latest user entry has no text and the append targets an older
preserved entry: nothing is rewritten and no injection is logged;
- each early return of the helper (message count changed, no single
changed message, non-string content, content that is not an append, no
matching text entry) leaves the preserved entry untouched.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): 2 errors, the same 2 as on
`main` (`headroom/memory/adapters/embedders.py:389`,
`headroom/integrations/langchain/chat_model.py:380`), none in
`gemini.py`.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests against main's gemini.py
$ pytest tests/test_gemini_memory_tail_preserved.py -q
4 failed, 1 passed        # the text-only control passes on main

# this branch
$ pytest tests/test_gemini_memory_tail_preserved.py -q
10 passed

$ pytest tests/test_gemini_memory_tail_preserved.py tests/test_google_multimodal.py tests/test_gemini*.py \
    tests/test_memory_decision.py tests/test_memory_invariants.py tests/test_memory_injection_logging.py \
    tests/test_proxy_handlers_batch.py tests/test_memory_tool_mode.py tests/test_memory_tool_session_sticky.py \
    tests/test_proxy_byte_faithful_forwarding.py tests/test_tokenizer_count_offload.py -q
287 passed                # plus test_tokenizer_count_offload.py, which passes

$ ruff check . && ruff format --check .
All checks passed!
1835 files already formatted

# full suite (Linux, Python 3.12, --timeout=120)
$ pytest tests -q --timeout=120
8 failed, 17098 passed, 599 skipped
```

None of the 8 failures comes from this change.
`test_mcp_reconcile::test_ordinary_install_does_not_adopt_serena`,
`test_wrap_sighup_proxy_leak::test_sighup_on_launch_tool_releases_the_proxy`
and both `test_cc_switch_reconciler` tests are environment failures that
also occur on `main` here. `test_offline_egress_chokepoint` and
`test_stateless_no_disk` pass when run on their own on this branch.
`test_direct_chat_ccr_resolution` is flaky here: a different
parametrized case fails on each run, on `main` too.
`test_tokenizer_count_offload` failed only because I edited `gemini.py`
while the suite was running (`inspect.getsource` read the new file with
the old line numbers); it passes on a fresh run.

## Real Behavior Proof

- Environment: Linux x86_64 (Amazon Linux 2023), Python 3.12, this
branch on top of `main` at `17d38e56`. The real proxy app is started
with `run_server(ProxyConfig(memory_enabled=True,
memory_backend="local", memory_storage_mode="global",
memory_db_path=..., gemini_api_url="http://127.0.0.1:18931"))`, the same
entry point `headroom proxy` calls (the CLI wrapper itself needs the
`mcp` package, which this environment does not have). Memory uses the
real local backend with the cached
`sentence-transformers/all-MiniLM-L6-v2` embedder. The upstream at
`:18931` is a loopback HTTP server that records each body it receives
and returns a Gemini-shaped 200. Nothing in the proxy is stubbed.
- Exact command / steps: seed one memory ("The user wants weather
temperatures reported in Celsius.") for user `alice` in the proxy's
memory DB through `MemoryHandler`, start the proxy, then `curl -X POST
http://127.0.0.1:18930/v1beta/models/gemini-2.5-flash:generateContent -H
'x-headroom-user-id: alice'` with a body whose last entry is
`{"role":"user","parts":[{"functionResponse":{"name":"get_weather",...}},{"text":"Report
the weather temperature for Paris in one line."}]}`. Run once with
`main`'s `gemini.py` and once with this branch's, and compare the last
entry the upstream received.
- Observed result: on `main` the proxy logs
`decision=injected_live_zone_tail_gemini bytes_injected=785`, but the
upstream receives the user text with no memory block. On this branch the
same log line appears and the upstream receives the text followed by the
memory block, with the `functionResponse` part unchanged.

```text
# main (17d38e56)
INFO - Memory: Injected 785 chars into latest user message tail for user alice (gemini)
INFO - event=memory_injection ... decision=injected_live_zone_tail_gemini bytes_injected=785
upstream received contents[-1]:
{"role":"user","parts":[{"functionResponse":{"name":"get_weather","response":{"temp_c":21,"sky":"sunny"}}},{"text":"Report the weather temperature for Paris in one line."}]}

# this branch
INFO - Memory: Injected 785 chars into latest user message tail for user alice (gemini)
INFO - event=memory_injection ... decision=injected_live_zone_tail_gemini bytes_injected=785
upstream received contents[-1]:
{"role":"user","parts":[{"functionResponse":{"name":"get_weather","response":{"temp_c":21,"sky":"sunny"}}},{"text":"Report the weather temperature for Paris in one line.\n\n## Relevant Memories (scope: global)\n\n... 1. [6463749b-...] The user wants weather temperatures reported in Celsius. ..."}]}
```

- Not tested: a real Gemini or Vertex upstream (the loopback server only
records the body); `streamGenerateContent` beyond the shared code path
(it delegates to `handle_gemini_generate_content`); the Qdrant/Neo4j
memory backend; Windows and macOS. A request whose every entry has a
non-text part still skips compression and memory entirely through the
existing early exit, as on `main`.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, only for Gemini requests with
memory injection enabled whose target user entry has a non-text part:
the memory block is now forwarded on that entry's last text part instead
of being dropped. Text-only entries, compression, CCR and other
providers are unchanged.
- Kill switch / disable path: `HEADROOM_MEMORY_INJECTION_MODE=disabled`
or `tool` turns off tail injection, as today.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- No documentation change: the handler comment already says memory goes
to the latest user message tail; this makes the Gemini path do that.
- The OpenAI chat half of #4419 is untouched (see Description), so the
issue should stay open for it.

---
🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an
open-source contribution agent.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Gaurav Dubey
2026-11-04 22:55:29 -06:00
committed by GitHub
co-authored by JerrettDavis
parent dd361bb7a3
commit 35a2b763f0
2 changed files with 350 additions and 0 deletions
+81
View File
@@ -225,6 +225,76 @@ class GeminiHandlerMixin:
# else: dropped by compression — omit
return result
@staticmethod
def _carry_tail_append_into_preserved(
original_contents: list[dict],
preserved_indices: set[int],
preserved_contents: dict[int, dict],
before_messages: list[dict],
after_messages: list[dict],
) -> bool:
"""Carry a memory tail append onto a preserved contents[] entry.
``_rebuild_gemini_contents`` forwards preserved (non-text) entries
verbatim and discards their converted message, so text appended to
that message never reaches the upstream body (#4419). Find the one
message the tail append changed, map it to its contents[] index the
same way the rebuild does (the k-th non-system message pairs with the
k-th entry that has a text part), and, when that entry is preserved
and is the latest user entry, append the same suffix to a copy of its
last text part.
Returns False when the append would be dropped: the target is a
preserved entry older than the latest user entry. Writing memory into
that older turn would change an already-cached prefix, so the caller
skips the injection instead of logging one that never reaches the wire.
"""
if len(before_messages) != len(after_messages):
return True
changed = [i for i, (b, a) in enumerate(zip(before_messages, after_messages)) if b is not a]
if len(changed) != 1:
return True
msg_idx = changed[0]
before = before_messages[msg_idx].get("content")
after = after_messages[msg_idx].get("content")
if not (isinstance(before, str) and isinstance(after, str)):
return True
if not after.startswith(before) or len(after) == len(before):
return True
suffix = after[len(before) :]
ordinal = sum(1 for m in before_messages[:msg_idx] if m.get("role") != "system")
text_entry_indices = [
idx
for idx, content in enumerate(original_contents)
if any(isinstance(p.get("text"), str) for p in GeminiHandlerMixin._dict_parts(content))
]
if ordinal >= len(text_entry_indices):
return True
content_idx = text_entry_indices[ordinal]
if content_idx not in preserved_indices:
return True
latest_user_idx = max(
(
idx
for idx, content in enumerate(original_contents)
if (content.get("role", "user") if isinstance(content, dict) else "user") == "user"
),
default=None,
)
if content_idx != latest_user_idx:
return False
entry = preserved_contents[content_idx]
# Copy the raw list so every part (including non-dict ones) is kept as-is.
parts = list(entry["parts"])
last_text = max(
i for i, p in enumerate(parts) if isinstance(p, dict) and isinstance(p.get("text"), str)
)
parts[last_text] = {**parts[last_text], "text": parts[last_text]["text"] + suffix}
preserved_contents[content_idx] = {**entry, "parts": parts}
return True
def _gemini_contents_to_messages(
self,
contents: list[dict],
@@ -778,6 +848,17 @@ class GeminiHandlerMixin:
provider="openai",
)
)
# Entries with non-text parts are restored verbatim by
# _rebuild_gemini_contents; carry the append onto them so
# it is not dropped after being logged, or skip it.
if bytes_appended > 0 and not self._carry_tail_append_into_preserved(
contents,
preserved_indices,
preserved_contents,
optimized_messages,
new_messages,
):
bytes_appended = 0
if bytes_appended > 0:
optimized_messages = new_messages
logger.info(
+269
View File
@@ -0,0 +1,269 @@
"""Gemini memory context must reach the wire on entries with non-text parts (#4419).
The Gemini handler converts ``contents[]`` to OpenAI-shaped messages, appends
the memory block to the latest user message, then rebuilds ``contents[]``.
Entries carrying a non-text part (``functionResponse``, ``inlineData``, ...)
are restored verbatim by ``_rebuild_gemini_contents`` so compression never
rewrites them. When the latest user entry was one of those, the rebuild also
discarded the memory text, while ``injected_live_zone_tail_gemini`` had already
been logged. The memory block must instead land on the preserved entry's text.
"""
from __future__ import annotations
import copy
import json
from types import SimpleNamespace
from typing import Any
import pytest
from headroom.proxy.handlers.gemini import GeminiHandlerMixin
from headroom.proxy.memory_handler import MemoryHandler
MEMORY_BLOCK = "<memory>user prefers metric units</memory>"
class _FakeRequest:
def __init__(self) -> None:
self.headers: dict[str, str] = {"x-headroom-user-id": "alice"}
self.query_params: dict[str, str] = {}
self.url = SimpleNamespace(path="/v1beta/models/gemini-pro:generateContent", query="")
self.scope: dict = {"type": "http", "method": "POST"}
self.client = SimpleNamespace(host="127.0.0.1", port=50000)
class _UpstreamResponse:
status_code = 200
content = json.dumps(
{
"candidates": [{"content": {"parts": [{"text": "ok"}]}}],
"usageMetadata": {"promptTokenCount": 10, "candidatesTokenCount": 1},
}
).encode()
headers = {"content-type": "application/json"}
def json(self) -> object:
return json.loads(self.content)
class _MemoryHandler:
"""Memory handler double: real tail-append, fixed search result."""
def __init__(self) -> None:
self.config = SimpleNamespace(inject_context=True, project_root_override="")
async def search_and_format_context(self, *args: Any, **kwargs: Any) -> str:
return MEMORY_BLOCK
_append_to_latest_user_tail = staticmethod(MemoryHandler._append_to_latest_user_tail)
class _NoCcr:
config = SimpleNamespace(enabled=False)
def has_ccr_tool_calls(self, resp_json: Any, provider: str) -> bool:
return False
class _Handler(GeminiHandlerMixin):
GEMINI_API_URL = "https://gemini.example"
def __init__(self) -> None:
from headroom.cache.prefix_tracker import SessionTrackerStore
self.memory_handler = _MemoryHandler()
self.rate_limiter = None
self.usage_reporter = None
self.session_tracker_store = SessionTrackerStore()
self.config = SimpleNamespace(
optimize=False,
anthropic_pre_upstream_memory_context_timeout_seconds=1.0,
ccr_inject_tool=False,
)
self.metrics = SimpleNamespace()
self.ccr_response_handler = _NoCcr()
self.outcomes: list = []
self.forwarded: list[dict] = []
async def _next_request_id(self) -> str:
return "req-mem-1"
async def _retry_request(self, method, url, headers, body): # noqa: ANN001, ANN201
self.forwarded.append(copy.deepcopy(body))
return _UpstreamResponse()
async def _record_request_outcome(self, outcome) -> None: # noqa: ANN001
self.outcomes.append(outcome)
async def _count_tokens_offloaded(self, model, messages): # noqa: ANN001, ANN201
return SimpleNamespace(), 10
async def _send(monkeypatch: pytest.MonkeyPatch, body: dict) -> dict:
async def payload(request): # noqa: ANN001, ANN202
return body
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
monkeypatch.setenv("HEADROOM_MEMORY_INJECTION_MODE", "live_zone_tail")
handler = _Handler()
response = await handler.handle_gemini_generate_content(_FakeRequest(), "gemini-pro")
assert response.status_code == 200
(forwarded,) = handler.forwarded
return forwarded
def _texts(entry: dict) -> list[str]:
return [p["text"] for p in entry["parts"] if isinstance(p.get("text"), str)]
@pytest.mark.asyncio
async def test_memory_reaches_latest_user_entry_with_function_response(
monkeypatch: pytest.MonkeyPatch,
) -> None:
function_response = {"functionResponse": {"name": "get_weather", "response": {"temp_c": 21}}}
body = {
"contents": [
{"role": "user", "parts": [{"text": "What is the weather?"}]},
{"role": "model", "parts": [{"functionCall": {"name": "get_weather", "args": {}}}]},
{"role": "user", "parts": [function_response, {"text": "Answer in one line."}]},
]
}
original = copy.deepcopy(body)
latest_entry = body["contents"][-1]
forwarded = await _send(monkeypatch, body)
latest = forwarded["contents"][-1]
# The non-text part is still forwarded untouched...
assert latest["parts"][0] == function_response
# ...and the memory block is on that entry's text, not dropped.
assert _texts(latest) == ["Answer in one line.\n\n" + MEMORY_BLOCK]
# Older turns are not touched.
assert forwarded["contents"][:2] == original["contents"][:2]
# The parsed request entry is copied, not mutated in place.
assert latest_entry == original["contents"][-1]
@pytest.mark.asyncio
async def test_memory_reaches_latest_user_entry_with_inline_image(
monkeypatch: pytest.MonkeyPatch,
) -> None:
image = {"inlineData": {"mimeType": "image/png", "data": "iVBORw0KGgo="}}
body = {
"contents": [
{"role": "user", "parts": [{"text": "Hi"}]},
{"role": "model", "parts": [{"text": "Hello"}]},
{"role": "user", "parts": [{"text": "Describe this"}, image, {"text": "briefly"}]},
]
}
forwarded = await _send(monkeypatch, body)
entry = forwarded["contents"][-1]
assert entry["parts"][1] == image
# The block is appended after the last text part, matching the bytes the
# text-only path produces for the joined message text.
assert _texts(entry) == ["Describe this", "briefly\n\n" + MEMORY_BLOCK]
@pytest.mark.asyncio
async def test_text_only_latest_entry_is_unchanged(monkeypatch: pytest.MonkeyPatch) -> None:
body = {
"contents": [
{"role": "user", "parts": [{"text": "hello"}]},
{"role": "model", "parts": [{"text": "hi"}]},
{"role": "user", "parts": [{"text": "what did I say?"}]},
]
}
forwarded = await _send(monkeypatch, body)
assert forwarded["contents"][-1] == {
"role": "user",
"parts": [{"text": "what did I say?\n\n" + MEMORY_BLOCK}],
}
assert forwarded["contents"][0] == {"role": "user", "parts": [{"text": "hello"}]}
@pytest.mark.asyncio
async def test_older_preserved_entry_is_not_rewritten_or_logged(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The latest user entry has no text, so the tail append targets an older
user entry. That entry is preserved and already in the provider's prompt
cache: skip the injection instead of rewriting it or logging a drop."""
logged: list[dict] = []
monkeypatch.setattr(
"headroom.proxy.helpers.log_memory_injection", lambda **kw: logged.append(kw)
)
image = {"inlineData": {"mimeType": "image/png", "data": "iVBORw0KGgo="}}
body = {
"contents": [
{"role": "user", "parts": [{"text": "Hi"}]},
{"role": "model", "parts": [{"text": "Hello"}]},
{"role": "user", "parts": [image, {"text": "What city is this?"}]},
{"role": "model", "parts": [{"functionCall": {"name": "lookup", "args": {}}}]},
{"role": "user", "parts": [{"functionResponse": {"name": "lookup", "response": {}}}]},
]
}
original = copy.deepcopy(body)
forwarded = await _send(monkeypatch, body)
assert forwarded["contents"] == original["contents"]
assert logged == []
def test_carry_skips_when_target_entry_is_not_preserved() -> None:
contents = [{"role": "user", "parts": [{"text": "a"}]}]
preserved: dict[int, dict] = {}
before = [{"role": "user", "content": "a"}]
after = [{"role": "user", "content": "a\n\nmem"}]
assert GeminiHandlerMixin._carry_tail_append_into_preserved(
contents, set(), preserved, before, after
)
assert preserved == {}
_IMAGE_ENTRY = {
"role": "user",
"parts": [{"inlineData": {"mimeType": "image/png", "data": "AA=="}}, {"text": "a"}],
}
_BEFORE = [{"role": "user", "content": "a"}]
_SYSTEM = {"role": "system", "content": "s"}
@pytest.mark.parametrize(
("before", "after"),
[
pytest.param(_BEFORE, [*_BEFORE, {"role": "user", "content": "b"}], id="length-changed"),
pytest.param(_BEFORE, list(_BEFORE), id="nothing-changed"),
pytest.param(
_BEFORE,
[{"role": "user", "content": [{"type": "text", "text": "a\n\nmem"}]}],
id="non-string-content",
),
pytest.param(_BEFORE, [{"role": "user", "content": "rewritten"}], id="not-an-append"),
pytest.param(
[_SYSTEM, {"role": "user", "content": "x"}],
[_SYSTEM, {"role": "user", "content": "x\n\nmem"}],
id="no-matching-entry",
),
],
)
def test_carry_leaves_preserved_entry_alone_for_unrecognized_shapes(
before: list[dict], after: list[dict]
) -> None:
"""Anything other than one string message gaining a suffix is not a tail
append this helper understands: keep today's behavior untouched."""
# "no-matching-entry" maps to a second text entry that does not exist.
contents = [_IMAGE_ENTRY] if len(before) == 1 else [{"role": "user", "parts": []}]
preserved = {0: _IMAGE_ENTRY}
assert GeminiHandlerMixin._carry_tail_append_into_preserved(
contents, {0}, preserved, before, after
)
assert preserved == {0: _IMAGE_ENTRY}