fix(proxy): freeze and replay the forwarded prefix on Gemini paths (#3394) (#3865)

## Description

The Gemini handler never participated in the prefix-freeze machinery: no
`PrefixCacheTracker` was wired, and `frozen_message_count` never reached
any of the three `apply()` sites. Gemini clients that resend the
original (uncompressed) prefix on every turn therefore bust the
provider's context cache every turn. Freezing alone would not fix it:
the router's freeze branch returns the supplied message unchanged, so
resent originals would still go out raw. This change freezes the
compressed prefix **and replays the exact bytes Gemini received last
turn** over the resent originals, byte-identically, before the body goes
out.

Closes #3394

Design credit to the issue discussion: chopratejas's correction comment
(the freeze floor belongs in gemini.py) and UniversePeak's earlier offer
to implement.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/proxy/handlers/gemini.py`: `_gemini_resolve_prefix_tracker`
(session id + resolve_tracker + freeze floor, mirroring the openai
handler) and `_gemini_replay_forwarded_prefix` (finalize_turn overlay
wrapper); wired at all three apply() sites (generateContent, Cloud Code
stream, countTokens). Tracker fed from the final `usageMetadata` only on
clean 200 + parsed body; streaming paths pass the tracker and original
messages into `_stream_response`.
- `headroom/proxy/handlers/streaming.py`: provider-scoped branches in
the shared finalizer read Gemini `contents[]` (or Cloud Code's nested
`request`) into the forwarded-message list the tracker walks;
tracker-local cache-write inference for Gemini.
- `tests/test_gemini_prefix_freeze.py` (new, 3 tests): buffered 3-turn
byte-identity, streaming tracker-advance + replay, countTokens wiring.
- `tests/test_gemini_ccr_continuation_usage.py`,
`tests/test_gemini_nonjson_status.py`: hand-rolled stubs given a real
`SessionTrackerStore` (the mixin uses it now, like its sibling
handlers). No production fallback added.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) - not run
- [x] New tests added for new functionality
- [ ] Manual testing performed - proxy-level tests use
httpx.MockTransport against a mock Gemini upstream

### Test Output

```text
$ pytest tests/test_gemini_prefix_freeze.py -q
3 passed
frozen_message_count seen by pipeline across 3 turns: [0, 2, 4]

$ pytest (gemini + proxy + cache adjacent sweep)
274 passed, 10 skipped (pre-existing skips requiring live API keys)

$ ruff check . && ruff format --check .
All checks passed!
```

## Real Behavior Proof

- Environment: Linux container, Python 3 venv, repo at main 8ff46dc9,
pytest with httpx.MockTransport; the mock Gemini upstream captures the
real forwarded request bodies.
- Exact command / steps: 1) Fails-before: ran the new 3-turn test
against the unpatched handler - the pipeline spy saw
`frozen_message_count: [None, None, None]` and the tracker never
advanced. 2) After the fix: `pytest tests/test_gemini_prefix_freeze.py
-q`, then the adjacent sweep above.
- Observed result: 3 passed. With the client resending the original
uncompressed prefix on every turn, the mock upstream received turn 1's
compressed prefix **byte-for-byte identical** on turns 2 and 3 (asserted
at the Gemini request-body level); frozen counts advanced 0 -> 2 -> 4;
the streaming path advances the tracker from SSE `usageMetadata`
(`cachedContentTokenCount`).
- Not tested: live Gemini API (no API key in the test environment);
mypy.

## Runtime Rollout Safety

- Rollout-managed feature(s): none new - uses the existing
prefix-freeze/tracker machinery (same as #3380 on the openai path)
- Minimum rollout channel: default
- Stable/default behavior changed: yes, on Gemini paths only - the
compressed prefix is now frozen and replayed (that is the fix); when the
tracker cannot resolve a session, both helpers bail to pass-through and
behavior is unchanged
- Kill switch / disable path: existing prefix-cache configuration; the
overlay declines and forwards the pipeline's own output whenever it
cannot prove append-only shape
- Unsafe override required: none
- Qualification impact: none - the Gemini handler previously had no
tracker; adjacent sweep (274 passed) covers the shared finalizer
- Rollback path: revert this branch's commits; freeze floor and replay
are additive and fail-open

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation - N/A:
internal proxy mechanics, no doc surface changes
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Two caveats, both by design and asserted in the tests:

1. **Cache-write inference**: Gemini's `usageMetadata` has no
cache-write field, so writes are inferred as uncached input tokens (same
approach as `_infer_openai_cache_write_tokens`). The inference is
tracker-local and kept off the metric funnel, matching the handler's
existing "Gemini has no cache-write concept" choice.
2. **Replay guard**: the replay only fires when the turn stays
positionally aligned with the client's messages (same message count). A
future tail transform that changed the count would make the overlay
decline for that turn - safe degradation to the pipeline's own output,
never stale content.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
davidscottpope-gif
2026-10-02 17:24:55 -07:00
committed by GitHub
co-authored by JerrettDavis
parent 540f4da199
commit dd84321daa
6 changed files with 442 additions and 1 deletions
+126
View File
@@ -61,6 +61,50 @@ class GeminiHandlerMixin:
return await count_texts_offloaded(self, model, texts)
def _gemini_replay_forwarded_prefix(
self,
optimized_messages: list[dict],
messages: list[dict],
prefix_tracker, # noqa: ANN001 - PrefixCacheTracker (avoid runtime import)
frozen_count: int,
tokenizer, # noqa: ANN001
):
"""Replay the previously forwarded (compressed) prefix byte-identical (#3394).
``frozen_message_count`` only stops the pipeline from RECOMPRESSING the
prefix; the freeze branch then forwards the client's ORIGINAL bytes -
but Gemini's implicit cache holds the COMPRESSED form forwarded last
turn, so raw originals would bust it. ``finalize_turn`` overlays the
tracker's last-forwarded (compressed) bytes over the pipeline output.
Self-guarded: no-op unless this turn provably extends the previous one
(positional, append-only, non-inflating); a decline forwards the
pipeline's own output.
"""
from headroom.proxy.session_engine import finalize_turn
return finalize_turn(
optimized_messages,
messages,
prefix_tracker.get_last_original_messages(),
prefix_tracker.get_last_forwarded_messages(),
count_tokens=tokenizer.count_messages,
confirmed_frozen_count=frozen_count,
)
def _gemini_resolve_prefix_tracker(self, request, model, messages): # noqa: ANN001, ANN202
"""Session tracker + freeze floor for this Gemini conversation (#3394).
Mirrors the Anthropic/OpenAI handlers: the tracker is fed from the
response's cache usage (buffered: ``update_from_response`` after the
upstream reply; streaming: the ``prefix_tracker`` param of
``_stream_response``) so the next turn freezes the forwarded prefix.
"""
session_id = self.session_tracker_store.compute_session_id(request, model, messages)
tracker = self.session_tracker_store.resolve_tracker(
session_id, "gemini", messages=messages
)
return tracker, tracker.get_frozen_message_count()
def _is_cloudcode_antigravity_request(
self, body: dict[str, Any], headers: dict[str, str]
) -> bool:
@@ -529,6 +573,11 @@ class GeminiHandlerMixin:
headers=response_headers,
)
# Prefix freeze floor + compressed-prefix replay (#3394).
gemini_prefix_tracker, gemini_frozen_count = self._gemini_resolve_prefix_tracker(
request, model, messages
)
# Token counting (offloaded off the event loop — GH #1701)
tokenizer, original_tokens = await self._count_tokens_offloaded(model, messages)
@@ -574,6 +623,7 @@ class GeminiHandlerMixin:
model=model,
model_limit=context_limit,
context=extract_user_query(messages),
frozen_message_count=gemini_frozen_count,
waste_messages=waste_messages,
**proxy_pipeline_kwargs(self.config),
),
@@ -601,6 +651,21 @@ class GeminiHandlerMixin:
optimized_tokens = original_tokens
transforms_applied = []
if _decision.should_compress:
# Replay last turn's compressed prefix over this turn's output so
# the frozen region goes out byte-identical (#3394). Gated on the
# compression decision: a bypassed request's bytes stay untouched.
_final = self._gemini_replay_forwarded_prefix(
optimized_messages,
messages,
gemini_prefix_tracker,
gemini_frozen_count,
tokenizer,
)
optimized_messages = _final.messages
if _final.tokens is not None:
optimized_tokens = _final.tokens
tokens_saved = original_tokens - optimized_tokens
optimization_latency = (time.time() - start_time) * 1000
@@ -786,6 +851,8 @@ class GeminiHandlerMixin:
tags,
optimization_latency,
outcome_provider=provider_name,
prefix_tracker=gemini_prefix_tracker,
original_messages=messages,
)
else:
response = await self._retry_request("POST", url, headers, body)
@@ -915,6 +982,20 @@ class GeminiHandlerMixin:
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens)
# Feed the prefix tracker (#3394). Gemini reports cache reads
# only (cachedContentTokenCount); implicit caching exposes no
# write counter, so the uncached input portion is the write
# proxy - the same inference the OpenAI path makes. Only a
# clean 200 with a parsed body feeds the tracker: nothing was
# cached for an error or an unparseable response.
if response.status_code == 200 and isinstance(resp_json, dict):
gemini_prefix_tracker.update_from_response(
cache_read_tokens=cache_read_tokens,
cache_write_tokens=max(total_input_tokens - cache_read_tokens, 0),
messages=optimized_messages,
original_messages=messages,
)
# optimized_tokens carries Gemini's own promptTokenCount, which is
# on the provider's tokenizer scale (it feeds billing/dashboard),
# while original_tokens is a LOCAL estimator count. When Gemini
@@ -1107,6 +1188,11 @@ class GeminiHandlerMixin:
if isinstance(contents, list) and idx < len(contents)
}
# Prefix freeze floor + compressed-prefix replay (#3394).
gemini_prefix_tracker, gemini_frozen_count = self._gemini_resolve_prefix_tracker(
request, model, messages
)
# Token counting (offloaded off the event loop — GH #1701)
tokenizer, original_tokens = await self._count_tokens_offloaded(model, messages)
if not messages:
@@ -1140,6 +1226,7 @@ class GeminiHandlerMixin:
model=model,
model_limit=context_limit,
context=extract_user_query(messages),
frozen_message_count=gemini_frozen_count,
waste_messages=waste_messages,
**proxy_pipeline_kwargs(self.config),
),
@@ -1162,6 +1249,24 @@ class GeminiHandlerMixin:
optimized_tokens = original_tokens
transforms_applied = []
# Replay BEFORE the contents/systemInstruction conversion below: the
# payload is rebuilt from the replayed messages, so the frozen prefix
# goes upstream byte-identical (#3394 review).
if _decision.should_compress:
# Replay last turn's compressed prefix over this turn's output so
# the frozen region goes out byte-identical (#3394). Gated on the
# compression decision: a bypassed request's bytes stay untouched.
_final = self._gemini_replay_forwarded_prefix(
optimized_messages,
messages,
gemini_prefix_tracker,
gemini_frozen_count,
tokenizer,
)
optimized_messages = _final.messages
if _final.tokens is not None:
optimized_tokens = _final.tokens
if optimized_messages != messages:
optimized_contents, optimized_system = self._messages_to_gemini_contents(
optimized_messages
@@ -1199,6 +1304,8 @@ class GeminiHandlerMixin:
transforms_applied,
tags,
optimization_latency,
prefix_tracker=gemini_prefix_tracker,
original_messages=messages,
)
async def handle_gemini_stream_generate_content(
@@ -1384,6 +1491,13 @@ class GeminiHandlerMixin:
headers=response_headers,
)
# Prefix freeze floor + compressed-prefix replay (#3394). Read-only:
# countTokens feeds the shared session's freeze floor into the pipeline
# but has no usage to feed back into the tracker.
gemini_prefix_tracker, gemini_frozen_count = self._gemini_resolve_prefix_tracker(
request, model, messages
)
# Token counting (original, offloaded off the event loop — GH #1701)
tokenizer, original_tokens = await self._count_tokens_offloaded(model, messages)
@@ -1419,6 +1533,7 @@ class GeminiHandlerMixin:
model=model,
model_limit=context_limit,
context=extract_user_query(messages),
frozen_message_count=gemini_frozen_count,
**proxy_pipeline_kwargs(self.config),
),
timeout=COMPRESSION_TIMEOUT_SECONDS,
@@ -1429,6 +1544,17 @@ class GeminiHandlerMixin:
except Exception as e:
logger.warning(f"[{request_id}] Gemini countTokens optimization failed: {e}")
if _decision.should_compress:
# Same compressed-prefix replay as generateContent (#3394), so the
# count previews what the real call would forward.
optimized_messages = self._gemini_replay_forwarded_prefix(
optimized_messages,
messages,
gemini_prefix_tracker,
gemini_frozen_count,
tokenizer,
).messages
# Convert back to Gemini format for the API call
if optimized_messages != messages:
optimized_contents, optimized_system = self._messages_to_gemini_contents(
+21 -1
View File
@@ -659,6 +659,16 @@ class StreamingMixin:
import copy as _copy
forwarded_messages = body.get("messages", [])
if not forwarded_messages and provider == "gemini":
# Gemini bodies carry contents[] rather than messages[] (Cloud
# Code Assist nests the payload under body["request"]).
# Convert with the same helper the Gemini handler used so the
# tracker walks the same message shape (#3394).
_gemini_payload = body["request"] if isinstance(body.get("request"), dict) else body
forwarded_messages, _ = self._gemini_contents_to_messages(
_gemini_payload.get("contents", []),
_gemini_payload.get("systemInstruction"),
)
next_forwarded = _copy.deepcopy(forwarded_messages)
next_original = _copy.deepcopy(original_messages or forwarded_messages)
@@ -695,9 +705,19 @@ class StreamingMixin:
)
await self.metrics.record_cache_miss_attribution(provider, miss.reason)
tracker_cache_write = cache_write_tokens
if provider == "gemini" and tracker_cache_write == 0:
# Gemini's stream usage reports cache reads only
# (cachedContentTokenCount); implicit caching has no write
# counter, so the uncached input portion is the write proxy
# (same inference as the OpenAI buffered path). Kept as a
# tracker-local value: Gemini outcomes intentionally report no
# cache-write concept (#3394).
tracker_cache_write = max(effective_optimized_tokens - cache_read_tokens, 0)
prefix_tracker.update_from_response(
cache_read_tokens=cache_read_tokens,
cache_write_tokens=cache_write_tokens,
cache_write_tokens=tracker_cache_write,
messages=next_forwarded,
original_messages=next_original,
)
@@ -91,6 +91,10 @@ class _Handler(GeminiHandlerMixin):
self.memory_handler = None
self.rate_limiter = None
self.usage_reporter = None
# The mixin resolves a prefix tracker for the freeze floor (#3394).
from headroom.cache.prefix_tracker import SessionTrackerStore
self.session_tracker_store = SessionTrackerStore()
self.config = SimpleNamespace(
optimize=False,
anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
+4
View File
@@ -42,6 +42,10 @@ class _Handler(GeminiHandlerMixin):
self.memory_handler = None
self.rate_limiter = None
self.usage_reporter = None
# The mixin resolves a prefix tracker for the freeze floor (#3394).
from headroom.cache.prefix_tracker import SessionTrackerStore
self.session_tracker_store = SessionTrackerStore()
self.config = SimpleNamespace(
optimize=False,
anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
+283
View File
@@ -0,0 +1,283 @@
"""Regression coverage for #3394: Gemini prefix freeze + compressed-prefix replay.
The Anthropic and OpenAI handlers keep a per-session PrefixCacheTracker and use
it two ways on every turn:
* a freeze floor (``frozen_message_count``) so the pipeline does not recompress
already-forwarded history, and
* a ``finalize_turn`` replay so the previously forwarded COMPRESSED prefix goes
out byte-identical - the provider cached the compressed bytes, so forwarding
the raw originals for a frozen message would bust the prefix cache from the
first changed byte.
The Gemini handler had neither: ``gemini.py`` resolved no tracker, passed no
``frozen_message_count`` on any of its three ``openai_pipeline.apply()`` call
sites, and never replayed the forwarded prefix - so every Gemini turn
recompressed the whole history and any compressor drift busted Gemini's
implicit prefix cache (whose hits the handler already reads back as
``cachedContentTokenCount``).
These tests drive multi-turn conversations through the real proxy with a
mocked Gemini upstream (MockTransport) and assert on the actual forwarded
bodies: a fix that only adds the kwarg (forwarding raw originals for the
frozen prefix) fails the byte-identity assertions.
"""
from __future__ import annotations
import json
from types import SimpleNamespace
from typing import Any
import pytest
fastapi = pytest.importorskip("fastapi")
httpx = pytest.importorskip("httpx")
from fastapi.testclient import TestClient # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
SESSION_ID = "gemini-prefix-freeze-test"
MODEL = "gemini-2.5-flash"
# Well above the default min_cached_tokens=1024 and above the local token
# estimate of the small fixture messages, so the freeze walk covers every
# forwarded message each turn.
PROVIDER_PROMPT_TOKENS = 10_000
def _config() -> ProxyConfig:
return ProxyConfig(
optimize=True,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
def _text(tag: str) -> str:
return f"{tag} " + "lorem ipsum dolor sit amet " * 40
def _turn_body(n_turns: int) -> dict[str, Any]:
"""systemInstruction + n user/model exchanges + one trailing user turn."""
contents: list[dict[str, Any]] = []
for i in range(n_turns):
contents.append({"role": "user", "parts": [{"text": _text(f"user-{i}")}]})
contents.append({"role": "model", "parts": [{"text": _text(f"model-{i}")}]})
contents.append({"role": "user", "parts": [{"text": _text(f"user-{n_turns}")}]})
return {
"systemInstruction": {"parts": [{"text": _text("system")}]},
"contents": contents,
}
def _fake_apply_factory(frozen_seen: list[int | None]):
"""Fake pipeline: marks only UNfrozen messages, like the real router.
Records the raw ``frozen_message_count`` kwarg (None = never passed).
"""
def _fake_apply(**kwargs: Any) -> SimpleNamespace:
frozen_seen.append(kwargs.get("frozen_message_count"))
frozen = kwargs.get("frozen_message_count") or 0
out = [
({**m, "content": f"[z] {m['content']}"} if i >= frozen else m)
for i, m in enumerate(kwargs["messages"])
]
return SimpleNamespace(
messages=out,
transforms_applied=["fake:mark-unfrozen"],
timing={},
tokens_before=1000,
tokens_after=900,
waste_signals=None,
)
return _fake_apply
def _gemini_provider(sent: list[dict[str, Any]]):
def _provider(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content)
sent.append(payload)
if ":countTokens" in str(request.url):
return httpx.Response(200, json={"totalTokens": 123})
usage = {
"promptTokenCount": PROVIDER_PROMPT_TOKENS,
"candidatesTokenCount": 5,
"totalTokenCount": PROVIDER_PROMPT_TOKENS + 5,
"cachedContentTokenCount": 0,
}
if ":streamGenerateContent" in str(request.url):
chunks = (
'data: {"candidates": [{"content": {"role": "model", '
'"parts": [{"text": "ack"}]}}]}\n\n'
+ f"data: {json.dumps({'usageMetadata': usage})}\n\n"
)
return httpx.Response(
200,
content=chunks.encode(),
headers={"content-type": "text/event-stream"},
)
return httpx.Response(
200,
json={
"candidates": [
{
"content": {"role": "model", "parts": [{"text": "ack"}]},
"finishReason": "STOP",
}
],
"usageMetadata": usage,
},
)
return _provider
def _install_fakes(proxy, sent: list[dict[str, Any]], frozen_seen: list[int | None]) -> None:
proxy.openai_pipeline.apply = _fake_apply_factory(frozen_seen)
proxy.http_client = httpx.AsyncClient(transport=httpx.MockTransport(_gemini_provider(sent)))
def _post(client: TestClient, body: dict[str, Any], *, stream: bool = False) -> None:
url = f"/v1beta/models/{MODEL}:generateContent?key=test-key"
if stream:
url += "&alt=sse"
resp = client.post(url, json=body, headers={"x-headroom-session-id": SESSION_ID})
assert resp.status_code == 200, resp.text[:300]
resp.read()
def test_gemini_buffered_replays_compressed_prefix_byte_identical() -> None:
"""Three buffered turns: turn N forwards turn N-1's prefix byte-identical.
Fails before the fix on two independent grounds: the pipeline never sees
``frozen_message_count`` (None at every call) and the forwarded prefix
bytes are the freshly re-marked forms rather than turn 1's forwarded ones.
"""
sent: list[dict[str, Any]] = []
frozen_seen: list[int | None] = []
app = create_app(_config())
with TestClient(app) as client:
_install_fakes(client.app.state.proxy, sent, frozen_seen)
for n in (0, 1, 2):
_post(client, _turn_body(n))
# Freeze floor reaches the pipeline: cold on turn 1, then the whole
# forwarded history of the previous turn.
assert frozen_seen == [0, 2, 4]
# The compressed prefix is replayed byte-identical - not re-emitted raw.
assert "[z]" in sent[0]["systemInstruction"]["parts"][0]["text"]
assert sent[1]["systemInstruction"] == sent[0]["systemInstruction"]
assert sent[2]["systemInstruction"] == sent[0]["systemInstruction"]
assert sent[1]["contents"][:1] == sent[0]["contents"]
assert sent[2]["contents"][:3] == sent[1]["contents"]
# The new tail still gets compressed (only the frozen prefix is replayed).
for payload in sent[1:]:
assert "[z]" in payload["contents"][-1]["parts"][0]["text"]
def test_gemini_streaming_advances_tracker_and_replays_prefix() -> None:
"""A streaming turn must feed the tracker; the next turn freezes + replays.
Before the fix the streaming branch never passed a prefix_tracker to
``_stream_response`` (and the finalizer could not read Gemini ``contents``
anyway), so the tracker never advanced and no freeze ever engaged.
"""
sent: list[dict[str, Any]] = []
frozen_seen: list[int | None] = []
app = create_app(_config())
with TestClient(app) as client:
proxy = client.app.state.proxy
_install_fakes(proxy, sent, frozen_seen)
_post(client, _turn_body(0), stream=True)
tracker = proxy.session_tracker_store.peek(SESSION_ID)
assert tracker is not None, "streaming turn never resolved a prefix tracker"
assert tracker._turn_number == 1, (
"streaming turn never called update_from_response - tracker stuck at turn 0"
)
assert tracker.get_frozen_message_count() == 2
assert tracker.get_last_forwarded_messages(), (
"tracker recorded no forwarded messages from the streaming turn"
)
_post(client, _turn_body(1), stream=True)
assert frozen_seen == [0, 2]
assert sent[1]["systemInstruction"] == sent[0]["systemInstruction"]
assert sent[1]["contents"][:1] == sent[0]["contents"]
def test_gemini_count_tokens_passes_frozen_count() -> None:
"""countTokens compresses with the same freeze floor as generateContent."""
sent: list[dict[str, Any]] = []
frozen_seen: list[int | None] = []
app = create_app(_config())
with TestClient(app) as client:
_install_fakes(client.app.state.proxy, sent, frozen_seen)
_post(client, _turn_body(0))
frozen_seen.clear()
resp = client.post(
f"/v1beta/models/{MODEL}:countTokens?key=test-key",
json=_turn_body(1),
headers={"x-headroom-session-id": SESSION_ID},
)
assert resp.status_code == 200, resp.text[:300]
assert frozen_seen == [2]
def _post_cloudcode(client: TestClient, body: dict[str, Any]) -> None:
resp = client.post(
"/v1internal:streamGenerateContent",
json={"model": MODEL, "request": body},
headers={"x-headroom-session-id": SESSION_ID},
)
assert resp.status_code == 200, resp.text[:300]
resp.read()
def test_cloudcode_streaming_advances_tracker_and_replays_prefix() -> None:
"""Cloud Code Assist (v1internal): same freeze + replay as the native path.
Regression for the requested-changes review on #3394: the replay ran only
after ``request_payload`` was already built (so the forwarded body never
contained the replayed prefix), and ``_stream_response`` never received
the tracker. Fails before this fix: the tracker stays at turn 0, the
freeze floor never reaches the pipeline ([0, 0]), and turn 2's nested
payload prefix is not turn 1's forwarded bytes.
"""
sent: list[dict[str, Any]] = []
frozen_seen: list[int | None] = []
app = create_app(_config())
with TestClient(app) as client:
proxy = client.app.state.proxy
_install_fakes(proxy, sent, frozen_seen)
_post_cloudcode(client, _turn_body(0))
tracker = proxy.session_tracker_store.peek(SESSION_ID)
assert tracker is not None, "cloudcode stream never resolved a prefix tracker"
assert tracker._turn_number == 1, (
"cloudcode stream never called update_from_response - tracker stuck at turn 0"
)
assert tracker.get_frozen_message_count() == 2
assert tracker.get_last_forwarded_messages(), (
"tracker recorded no forwarded messages from the cloudcode stream"
)
_post_cloudcode(client, _turn_body(1))
# Freeze floor reaches the pipeline on the cloudcode route too.
assert frozen_seen == [0, 2]
# The nested request payload carries the replayed prefix byte-identical.
req0, req1 = sent[0]["request"], sent[1]["request"]
assert "[z]" in req0["systemInstruction"]["parts"][0]["text"]
assert req1["systemInstruction"] == req0["systemInstruction"]
assert req1["contents"][:1] == req0["contents"]
+4
View File
@@ -180,6 +180,10 @@ class NativeGeminiHandler(DummyBatchHandler):
self.memory_handler = None
self.rate_limiter = None
self.usage_reporter = None
# The mixin resolves a prefix tracker for the freeze floor (#3394).
from headroom.cache.prefix_tracker import SessionTrackerStore
self.session_tracker_store = SessionTrackerStore()
self.responses = iter(responses)
self.sent_bodies: list[dict] = []
from headroom.ccr.response_handler import CCRResponseHandler