mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
The Gemini handler never participated in the prefix-freeze machinery: no
`PrefixCacheTracker` was wired, and `frozen_message_count` never reached
any of the three `apply()` sites. Gemini clients that resend the
original (uncompressed) prefix on every turn therefore bust the
provider's context cache every turn. Freezing alone would not fix it:
the router's freeze branch returns the supplied message unchanged, so
resent originals would still go out raw. This change freezes the
compressed prefix **and replays the exact bytes Gemini received last
turn** over the resent originals, byte-identically, before the body goes
out.
Closes #3394
Design credit to the issue discussion: chopratejas's correction comment
(the freeze floor belongs in gemini.py) and UniversePeak's earlier offer
to implement.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: `_gemini_resolve_prefix_tracker`
(session id + resolve_tracker + freeze floor, mirroring the openai
handler) and `_gemini_replay_forwarded_prefix` (finalize_turn overlay
wrapper); wired at all three apply() sites (generateContent, Cloud Code
stream, countTokens). Tracker fed from the final `usageMetadata` only on
clean 200 + parsed body; streaming paths pass the tracker and original
messages into `_stream_response`.
- `headroom/proxy/handlers/streaming.py`: provider-scoped branches in
the shared finalizer read Gemini `contents[]` (or Cloud Code's nested
`request`) into the forwarded-message list the tracker walks;
tracker-local cache-write inference for Gemini.
- `tests/test_gemini_prefix_freeze.py` (new, 3 tests): buffered 3-turn
byte-identity, streaming tracker-advance + replay, countTokens wiring.
- `tests/test_gemini_ccr_continuation_usage.py`,
`tests/test_gemini_nonjson_status.py`: hand-rolled stubs given a real
`SessionTrackerStore` (the mixin uses it now, like its sibling
handlers). No production fallback added.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) - not run
- [x] New tests added for new functionality
- [ ] Manual testing performed - proxy-level tests use
httpx.MockTransport against a mock Gemini upstream
### Test Output
```text
$ pytest tests/test_gemini_prefix_freeze.py -q
3 passed
frozen_message_count seen by pipeline across 3 turns: [0, 2, 4]
$ pytest (gemini + proxy + cache adjacent sweep)
274 passed, 10 skipped (pre-existing skips requiring live API keys)
$ ruff check . && ruff format --check .
All checks passed!
```
## Real Behavior Proof
- Environment: Linux container, Python 3 venv, repo at main 8ff46dc9,
pytest with httpx.MockTransport; the mock Gemini upstream captures the
real forwarded request bodies.
- Exact command / steps: 1) Fails-before: ran the new 3-turn test
against the unpatched handler - the pipeline spy saw
`frozen_message_count: [None, None, None]` and the tracker never
advanced. 2) After the fix: `pytest tests/test_gemini_prefix_freeze.py
-q`, then the adjacent sweep above.
- Observed result: 3 passed. With the client resending the original
uncompressed prefix on every turn, the mock upstream received turn 1's
compressed prefix **byte-for-byte identical** on turns 2 and 3 (asserted
at the Gemini request-body level); frozen counts advanced 0 -> 2 -> 4;
the streaming path advances the tracker from SSE `usageMetadata`
(`cachedContentTokenCount`).
- Not tested: live Gemini API (no API key in the test environment);
mypy.
## Runtime Rollout Safety
- Rollout-managed feature(s): none new - uses the existing
prefix-freeze/tracker machinery (same as #3380 on the openai path)
- Minimum rollout channel: default
- Stable/default behavior changed: yes, on Gemini paths only - the
compressed prefix is now frozen and replayed (that is the fix); when the
tracker cannot resolve a session, both helpers bail to pass-through and
behavior is unchanged
- Kill switch / disable path: existing prefix-cache configuration; the
overlay declines and forwards the pipeline's own output whenever it
cannot prove append-only shape
- Unsafe override required: none
- Qualification impact: none - the Gemini handler previously had no
tracker; adjacent sweep (274 passed) covers the shared finalizer
- Rollback path: revert this branch's commits; freeze floor and replay
are additive and fail-open
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation - N/A:
internal proxy mechanics, no doc surface changes
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Two caveats, both by design and asserted in the tests:
1. **Cache-write inference**: Gemini's `usageMetadata` has no
cache-write field, so writes are inferred as uncached input tokens (same
approach as `_infer_openai_cache_write_tokens`). The inference is
tracker-local and kept off the metric funnel, matching the handler's
existing "Gemini has no cache-write concept" choice.
2. **Replay guard**: the replay only fires when the turn stays
positionally aligned with the client's messages (same message count). A
future tail transform that changed the count would make the overlay
decline for that turn - safe degradation to the pipeline's own output,
never stale content.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
540f4da199
commit
dd84321daa
@@ -61,6 +61,50 @@ class GeminiHandlerMixin:
|
||||
|
||||
return await count_texts_offloaded(self, model, texts)
|
||||
|
||||
def _gemini_replay_forwarded_prefix(
|
||||
self,
|
||||
optimized_messages: list[dict],
|
||||
messages: list[dict],
|
||||
prefix_tracker, # noqa: ANN001 - PrefixCacheTracker (avoid runtime import)
|
||||
frozen_count: int,
|
||||
tokenizer, # noqa: ANN001
|
||||
):
|
||||
"""Replay the previously forwarded (compressed) prefix byte-identical (#3394).
|
||||
|
||||
``frozen_message_count`` only stops the pipeline from RECOMPRESSING the
|
||||
prefix; the freeze branch then forwards the client's ORIGINAL bytes -
|
||||
but Gemini's implicit cache holds the COMPRESSED form forwarded last
|
||||
turn, so raw originals would bust it. ``finalize_turn`` overlays the
|
||||
tracker's last-forwarded (compressed) bytes over the pipeline output.
|
||||
Self-guarded: no-op unless this turn provably extends the previous one
|
||||
(positional, append-only, non-inflating); a decline forwards the
|
||||
pipeline's own output.
|
||||
"""
|
||||
from headroom.proxy.session_engine import finalize_turn
|
||||
|
||||
return finalize_turn(
|
||||
optimized_messages,
|
||||
messages,
|
||||
prefix_tracker.get_last_original_messages(),
|
||||
prefix_tracker.get_last_forwarded_messages(),
|
||||
count_tokens=tokenizer.count_messages,
|
||||
confirmed_frozen_count=frozen_count,
|
||||
)
|
||||
|
||||
def _gemini_resolve_prefix_tracker(self, request, model, messages): # noqa: ANN001, ANN202
|
||||
"""Session tracker + freeze floor for this Gemini conversation (#3394).
|
||||
|
||||
Mirrors the Anthropic/OpenAI handlers: the tracker is fed from the
|
||||
response's cache usage (buffered: ``update_from_response`` after the
|
||||
upstream reply; streaming: the ``prefix_tracker`` param of
|
||||
``_stream_response``) so the next turn freezes the forwarded prefix.
|
||||
"""
|
||||
session_id = self.session_tracker_store.compute_session_id(request, model, messages)
|
||||
tracker = self.session_tracker_store.resolve_tracker(
|
||||
session_id, "gemini", messages=messages
|
||||
)
|
||||
return tracker, tracker.get_frozen_message_count()
|
||||
|
||||
def _is_cloudcode_antigravity_request(
|
||||
self, body: dict[str, Any], headers: dict[str, str]
|
||||
) -> bool:
|
||||
@@ -529,6 +573,11 @@ class GeminiHandlerMixin:
|
||||
headers=response_headers,
|
||||
)
|
||||
|
||||
# Prefix freeze floor + compressed-prefix replay (#3394).
|
||||
gemini_prefix_tracker, gemini_frozen_count = self._gemini_resolve_prefix_tracker(
|
||||
request, model, messages
|
||||
)
|
||||
|
||||
# Token counting (offloaded off the event loop — GH #1701)
|
||||
tokenizer, original_tokens = await self._count_tokens_offloaded(model, messages)
|
||||
|
||||
@@ -574,6 +623,7 @@ class GeminiHandlerMixin:
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(messages),
|
||||
frozen_message_count=gemini_frozen_count,
|
||||
waste_messages=waste_messages,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
@@ -601,6 +651,21 @@ class GeminiHandlerMixin:
|
||||
optimized_tokens = original_tokens
|
||||
transforms_applied = []
|
||||
|
||||
if _decision.should_compress:
|
||||
# Replay last turn's compressed prefix over this turn's output so
|
||||
# the frozen region goes out byte-identical (#3394). Gated on the
|
||||
# compression decision: a bypassed request's bytes stay untouched.
|
||||
_final = self._gemini_replay_forwarded_prefix(
|
||||
optimized_messages,
|
||||
messages,
|
||||
gemini_prefix_tracker,
|
||||
gemini_frozen_count,
|
||||
tokenizer,
|
||||
)
|
||||
optimized_messages = _final.messages
|
||||
if _final.tokens is not None:
|
||||
optimized_tokens = _final.tokens
|
||||
|
||||
tokens_saved = original_tokens - optimized_tokens
|
||||
optimization_latency = (time.time() - start_time) * 1000
|
||||
|
||||
@@ -786,6 +851,8 @@ class GeminiHandlerMixin:
|
||||
tags,
|
||||
optimization_latency,
|
||||
outcome_provider=provider_name,
|
||||
prefix_tracker=gemini_prefix_tracker,
|
||||
original_messages=messages,
|
||||
)
|
||||
else:
|
||||
response = await self._retry_request("POST", url, headers, body)
|
||||
@@ -915,6 +982,20 @@ class GeminiHandlerMixin:
|
||||
|
||||
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens)
|
||||
|
||||
# Feed the prefix tracker (#3394). Gemini reports cache reads
|
||||
# only (cachedContentTokenCount); implicit caching exposes no
|
||||
# write counter, so the uncached input portion is the write
|
||||
# proxy - the same inference the OpenAI path makes. Only a
|
||||
# clean 200 with a parsed body feeds the tracker: nothing was
|
||||
# cached for an error or an unparseable response.
|
||||
if response.status_code == 200 and isinstance(resp_json, dict):
|
||||
gemini_prefix_tracker.update_from_response(
|
||||
cache_read_tokens=cache_read_tokens,
|
||||
cache_write_tokens=max(total_input_tokens - cache_read_tokens, 0),
|
||||
messages=optimized_messages,
|
||||
original_messages=messages,
|
||||
)
|
||||
|
||||
# optimized_tokens carries Gemini's own promptTokenCount, which is
|
||||
# on the provider's tokenizer scale (it feeds billing/dashboard),
|
||||
# while original_tokens is a LOCAL estimator count. When Gemini
|
||||
@@ -1107,6 +1188,11 @@ class GeminiHandlerMixin:
|
||||
if isinstance(contents, list) and idx < len(contents)
|
||||
}
|
||||
|
||||
# Prefix freeze floor + compressed-prefix replay (#3394).
|
||||
gemini_prefix_tracker, gemini_frozen_count = self._gemini_resolve_prefix_tracker(
|
||||
request, model, messages
|
||||
)
|
||||
|
||||
# Token counting (offloaded off the event loop — GH #1701)
|
||||
tokenizer, original_tokens = await self._count_tokens_offloaded(model, messages)
|
||||
if not messages:
|
||||
@@ -1140,6 +1226,7 @@ class GeminiHandlerMixin:
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(messages),
|
||||
frozen_message_count=gemini_frozen_count,
|
||||
waste_messages=waste_messages,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
@@ -1162,6 +1249,24 @@ class GeminiHandlerMixin:
|
||||
optimized_tokens = original_tokens
|
||||
transforms_applied = []
|
||||
|
||||
# Replay BEFORE the contents/systemInstruction conversion below: the
|
||||
# payload is rebuilt from the replayed messages, so the frozen prefix
|
||||
# goes upstream byte-identical (#3394 review).
|
||||
if _decision.should_compress:
|
||||
# Replay last turn's compressed prefix over this turn's output so
|
||||
# the frozen region goes out byte-identical (#3394). Gated on the
|
||||
# compression decision: a bypassed request's bytes stay untouched.
|
||||
_final = self._gemini_replay_forwarded_prefix(
|
||||
optimized_messages,
|
||||
messages,
|
||||
gemini_prefix_tracker,
|
||||
gemini_frozen_count,
|
||||
tokenizer,
|
||||
)
|
||||
optimized_messages = _final.messages
|
||||
if _final.tokens is not None:
|
||||
optimized_tokens = _final.tokens
|
||||
|
||||
if optimized_messages != messages:
|
||||
optimized_contents, optimized_system = self._messages_to_gemini_contents(
|
||||
optimized_messages
|
||||
@@ -1199,6 +1304,8 @@ class GeminiHandlerMixin:
|
||||
transforms_applied,
|
||||
tags,
|
||||
optimization_latency,
|
||||
prefix_tracker=gemini_prefix_tracker,
|
||||
original_messages=messages,
|
||||
)
|
||||
|
||||
async def handle_gemini_stream_generate_content(
|
||||
@@ -1384,6 +1491,13 @@ class GeminiHandlerMixin:
|
||||
headers=response_headers,
|
||||
)
|
||||
|
||||
# Prefix freeze floor + compressed-prefix replay (#3394). Read-only:
|
||||
# countTokens feeds the shared session's freeze floor into the pipeline
|
||||
# but has no usage to feed back into the tracker.
|
||||
gemini_prefix_tracker, gemini_frozen_count = self._gemini_resolve_prefix_tracker(
|
||||
request, model, messages
|
||||
)
|
||||
|
||||
# Token counting (original, offloaded off the event loop — GH #1701)
|
||||
tokenizer, original_tokens = await self._count_tokens_offloaded(model, messages)
|
||||
|
||||
@@ -1419,6 +1533,7 @@ class GeminiHandlerMixin:
|
||||
model=model,
|
||||
model_limit=context_limit,
|
||||
context=extract_user_query(messages),
|
||||
frozen_message_count=gemini_frozen_count,
|
||||
**proxy_pipeline_kwargs(self.config),
|
||||
),
|
||||
timeout=COMPRESSION_TIMEOUT_SECONDS,
|
||||
@@ -1429,6 +1544,17 @@ class GeminiHandlerMixin:
|
||||
except Exception as e:
|
||||
logger.warning(f"[{request_id}] Gemini countTokens optimization failed: {e}")
|
||||
|
||||
if _decision.should_compress:
|
||||
# Same compressed-prefix replay as generateContent (#3394), so the
|
||||
# count previews what the real call would forward.
|
||||
optimized_messages = self._gemini_replay_forwarded_prefix(
|
||||
optimized_messages,
|
||||
messages,
|
||||
gemini_prefix_tracker,
|
||||
gemini_frozen_count,
|
||||
tokenizer,
|
||||
).messages
|
||||
|
||||
# Convert back to Gemini format for the API call
|
||||
if optimized_messages != messages:
|
||||
optimized_contents, optimized_system = self._messages_to_gemini_contents(
|
||||
|
||||
@@ -659,6 +659,16 @@ class StreamingMixin:
|
||||
import copy as _copy
|
||||
|
||||
forwarded_messages = body.get("messages", [])
|
||||
if not forwarded_messages and provider == "gemini":
|
||||
# Gemini bodies carry contents[] rather than messages[] (Cloud
|
||||
# Code Assist nests the payload under body["request"]).
|
||||
# Convert with the same helper the Gemini handler used so the
|
||||
# tracker walks the same message shape (#3394).
|
||||
_gemini_payload = body["request"] if isinstance(body.get("request"), dict) else body
|
||||
forwarded_messages, _ = self._gemini_contents_to_messages(
|
||||
_gemini_payload.get("contents", []),
|
||||
_gemini_payload.get("systemInstruction"),
|
||||
)
|
||||
next_forwarded = _copy.deepcopy(forwarded_messages)
|
||||
next_original = _copy.deepcopy(original_messages or forwarded_messages)
|
||||
|
||||
@@ -695,9 +705,19 @@ class StreamingMixin:
|
||||
)
|
||||
await self.metrics.record_cache_miss_attribution(provider, miss.reason)
|
||||
|
||||
tracker_cache_write = cache_write_tokens
|
||||
if provider == "gemini" and tracker_cache_write == 0:
|
||||
# Gemini's stream usage reports cache reads only
|
||||
# (cachedContentTokenCount); implicit caching has no write
|
||||
# counter, so the uncached input portion is the write proxy
|
||||
# (same inference as the OpenAI buffered path). Kept as a
|
||||
# tracker-local value: Gemini outcomes intentionally report no
|
||||
# cache-write concept (#3394).
|
||||
tracker_cache_write = max(effective_optimized_tokens - cache_read_tokens, 0)
|
||||
|
||||
prefix_tracker.update_from_response(
|
||||
cache_read_tokens=cache_read_tokens,
|
||||
cache_write_tokens=cache_write_tokens,
|
||||
cache_write_tokens=tracker_cache_write,
|
||||
messages=next_forwarded,
|
||||
original_messages=next_original,
|
||||
)
|
||||
|
||||
@@ -91,6 +91,10 @@ class _Handler(GeminiHandlerMixin):
|
||||
self.memory_handler = None
|
||||
self.rate_limiter = None
|
||||
self.usage_reporter = None
|
||||
# The mixin resolves a prefix tracker for the freeze floor (#3394).
|
||||
from headroom.cache.prefix_tracker import SessionTrackerStore
|
||||
|
||||
self.session_tracker_store = SessionTrackerStore()
|
||||
self.config = SimpleNamespace(
|
||||
optimize=False,
|
||||
anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
|
||||
|
||||
@@ -42,6 +42,10 @@ class _Handler(GeminiHandlerMixin):
|
||||
self.memory_handler = None
|
||||
self.rate_limiter = None
|
||||
self.usage_reporter = None
|
||||
# The mixin resolves a prefix tracker for the freeze floor (#3394).
|
||||
from headroom.cache.prefix_tracker import SessionTrackerStore
|
||||
|
||||
self.session_tracker_store = SessionTrackerStore()
|
||||
self.config = SimpleNamespace(
|
||||
optimize=False,
|
||||
anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
"""Regression coverage for #3394: Gemini prefix freeze + compressed-prefix replay.
|
||||
|
||||
The Anthropic and OpenAI handlers keep a per-session PrefixCacheTracker and use
|
||||
it two ways on every turn:
|
||||
|
||||
* a freeze floor (``frozen_message_count``) so the pipeline does not recompress
|
||||
already-forwarded history, and
|
||||
* a ``finalize_turn`` replay so the previously forwarded COMPRESSED prefix goes
|
||||
out byte-identical - the provider cached the compressed bytes, so forwarding
|
||||
the raw originals for a frozen message would bust the prefix cache from the
|
||||
first changed byte.
|
||||
|
||||
The Gemini handler had neither: ``gemini.py`` resolved no tracker, passed no
|
||||
``frozen_message_count`` on any of its three ``openai_pipeline.apply()`` call
|
||||
sites, and never replayed the forwarded prefix - so every Gemini turn
|
||||
recompressed the whole history and any compressor drift busted Gemini's
|
||||
implicit prefix cache (whose hits the handler already reads back as
|
||||
``cachedContentTokenCount``).
|
||||
|
||||
These tests drive multi-turn conversations through the real proxy with a
|
||||
mocked Gemini upstream (MockTransport) and assert on the actual forwarded
|
||||
bodies: a fix that only adds the kwarg (forwarding raw originals for the
|
||||
frozen prefix) fails the byte-identity assertions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
fastapi = pytest.importorskip("fastapi")
|
||||
httpx = pytest.importorskip("httpx")
|
||||
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
|
||||
|
||||
SESSION_ID = "gemini-prefix-freeze-test"
|
||||
MODEL = "gemini-2.5-flash"
|
||||
# Well above the default min_cached_tokens=1024 and above the local token
|
||||
# estimate of the small fixture messages, so the freeze walk covers every
|
||||
# forwarded message each turn.
|
||||
PROVIDER_PROMPT_TOKENS = 10_000
|
||||
|
||||
|
||||
def _config() -> ProxyConfig:
|
||||
return ProxyConfig(
|
||||
optimize=True,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
)
|
||||
|
||||
|
||||
def _text(tag: str) -> str:
|
||||
return f"{tag} " + "lorem ipsum dolor sit amet " * 40
|
||||
|
||||
|
||||
def _turn_body(n_turns: int) -> dict[str, Any]:
|
||||
"""systemInstruction + n user/model exchanges + one trailing user turn."""
|
||||
contents: list[dict[str, Any]] = []
|
||||
for i in range(n_turns):
|
||||
contents.append({"role": "user", "parts": [{"text": _text(f"user-{i}")}]})
|
||||
contents.append({"role": "model", "parts": [{"text": _text(f"model-{i}")}]})
|
||||
contents.append({"role": "user", "parts": [{"text": _text(f"user-{n_turns}")}]})
|
||||
return {
|
||||
"systemInstruction": {"parts": [{"text": _text("system")}]},
|
||||
"contents": contents,
|
||||
}
|
||||
|
||||
|
||||
def _fake_apply_factory(frozen_seen: list[int | None]):
|
||||
"""Fake pipeline: marks only UNfrozen messages, like the real router.
|
||||
|
||||
Records the raw ``frozen_message_count`` kwarg (None = never passed).
|
||||
"""
|
||||
|
||||
def _fake_apply(**kwargs: Any) -> SimpleNamespace:
|
||||
frozen_seen.append(kwargs.get("frozen_message_count"))
|
||||
frozen = kwargs.get("frozen_message_count") or 0
|
||||
out = [
|
||||
({**m, "content": f"[z] {m['content']}"} if i >= frozen else m)
|
||||
for i, m in enumerate(kwargs["messages"])
|
||||
]
|
||||
return SimpleNamespace(
|
||||
messages=out,
|
||||
transforms_applied=["fake:mark-unfrozen"],
|
||||
timing={},
|
||||
tokens_before=1000,
|
||||
tokens_after=900,
|
||||
waste_signals=None,
|
||||
)
|
||||
|
||||
return _fake_apply
|
||||
|
||||
|
||||
def _gemini_provider(sent: list[dict[str, Any]]):
|
||||
def _provider(request: httpx.Request) -> httpx.Response:
|
||||
payload = json.loads(request.content)
|
||||
sent.append(payload)
|
||||
if ":countTokens" in str(request.url):
|
||||
return httpx.Response(200, json={"totalTokens": 123})
|
||||
usage = {
|
||||
"promptTokenCount": PROVIDER_PROMPT_TOKENS,
|
||||
"candidatesTokenCount": 5,
|
||||
"totalTokenCount": PROVIDER_PROMPT_TOKENS + 5,
|
||||
"cachedContentTokenCount": 0,
|
||||
}
|
||||
if ":streamGenerateContent" in str(request.url):
|
||||
chunks = (
|
||||
'data: {"candidates": [{"content": {"role": "model", '
|
||||
'"parts": [{"text": "ack"}]}}]}\n\n'
|
||||
+ f"data: {json.dumps({'usageMetadata': usage})}\n\n"
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
content=chunks.encode(),
|
||||
headers={"content-type": "text/event-stream"},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"candidates": [
|
||||
{
|
||||
"content": {"role": "model", "parts": [{"text": "ack"}]},
|
||||
"finishReason": "STOP",
|
||||
}
|
||||
],
|
||||
"usageMetadata": usage,
|
||||
},
|
||||
)
|
||||
|
||||
return _provider
|
||||
|
||||
|
||||
def _install_fakes(proxy, sent: list[dict[str, Any]], frozen_seen: list[int | None]) -> None:
|
||||
proxy.openai_pipeline.apply = _fake_apply_factory(frozen_seen)
|
||||
proxy.http_client = httpx.AsyncClient(transport=httpx.MockTransport(_gemini_provider(sent)))
|
||||
|
||||
|
||||
def _post(client: TestClient, body: dict[str, Any], *, stream: bool = False) -> None:
|
||||
url = f"/v1beta/models/{MODEL}:generateContent?key=test-key"
|
||||
if stream:
|
||||
url += "&alt=sse"
|
||||
resp = client.post(url, json=body, headers={"x-headroom-session-id": SESSION_ID})
|
||||
assert resp.status_code == 200, resp.text[:300]
|
||||
resp.read()
|
||||
|
||||
|
||||
def test_gemini_buffered_replays_compressed_prefix_byte_identical() -> None:
|
||||
"""Three buffered turns: turn N forwards turn N-1's prefix byte-identical.
|
||||
|
||||
Fails before the fix on two independent grounds: the pipeline never sees
|
||||
``frozen_message_count`` (None at every call) and the forwarded prefix
|
||||
bytes are the freshly re-marked forms rather than turn 1's forwarded ones.
|
||||
"""
|
||||
sent: list[dict[str, Any]] = []
|
||||
frozen_seen: list[int | None] = []
|
||||
app = create_app(_config())
|
||||
with TestClient(app) as client:
|
||||
_install_fakes(client.app.state.proxy, sent, frozen_seen)
|
||||
for n in (0, 1, 2):
|
||||
_post(client, _turn_body(n))
|
||||
|
||||
# Freeze floor reaches the pipeline: cold on turn 1, then the whole
|
||||
# forwarded history of the previous turn.
|
||||
assert frozen_seen == [0, 2, 4]
|
||||
|
||||
# The compressed prefix is replayed byte-identical - not re-emitted raw.
|
||||
assert "[z]" in sent[0]["systemInstruction"]["parts"][0]["text"]
|
||||
assert sent[1]["systemInstruction"] == sent[0]["systemInstruction"]
|
||||
assert sent[2]["systemInstruction"] == sent[0]["systemInstruction"]
|
||||
assert sent[1]["contents"][:1] == sent[0]["contents"]
|
||||
assert sent[2]["contents"][:3] == sent[1]["contents"]
|
||||
|
||||
# The new tail still gets compressed (only the frozen prefix is replayed).
|
||||
for payload in sent[1:]:
|
||||
assert "[z]" in payload["contents"][-1]["parts"][0]["text"]
|
||||
|
||||
|
||||
def test_gemini_streaming_advances_tracker_and_replays_prefix() -> None:
|
||||
"""A streaming turn must feed the tracker; the next turn freezes + replays.
|
||||
|
||||
Before the fix the streaming branch never passed a prefix_tracker to
|
||||
``_stream_response`` (and the finalizer could not read Gemini ``contents``
|
||||
anyway), so the tracker never advanced and no freeze ever engaged.
|
||||
"""
|
||||
sent: list[dict[str, Any]] = []
|
||||
frozen_seen: list[int | None] = []
|
||||
app = create_app(_config())
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
_install_fakes(proxy, sent, frozen_seen)
|
||||
_post(client, _turn_body(0), stream=True)
|
||||
|
||||
tracker = proxy.session_tracker_store.peek(SESSION_ID)
|
||||
assert tracker is not None, "streaming turn never resolved a prefix tracker"
|
||||
assert tracker._turn_number == 1, (
|
||||
"streaming turn never called update_from_response - tracker stuck at turn 0"
|
||||
)
|
||||
assert tracker.get_frozen_message_count() == 2
|
||||
assert tracker.get_last_forwarded_messages(), (
|
||||
"tracker recorded no forwarded messages from the streaming turn"
|
||||
)
|
||||
|
||||
_post(client, _turn_body(1), stream=True)
|
||||
|
||||
assert frozen_seen == [0, 2]
|
||||
assert sent[1]["systemInstruction"] == sent[0]["systemInstruction"]
|
||||
assert sent[1]["contents"][:1] == sent[0]["contents"]
|
||||
|
||||
|
||||
def test_gemini_count_tokens_passes_frozen_count() -> None:
|
||||
"""countTokens compresses with the same freeze floor as generateContent."""
|
||||
sent: list[dict[str, Any]] = []
|
||||
frozen_seen: list[int | None] = []
|
||||
app = create_app(_config())
|
||||
with TestClient(app) as client:
|
||||
_install_fakes(client.app.state.proxy, sent, frozen_seen)
|
||||
_post(client, _turn_body(0))
|
||||
frozen_seen.clear()
|
||||
|
||||
resp = client.post(
|
||||
f"/v1beta/models/{MODEL}:countTokens?key=test-key",
|
||||
json=_turn_body(1),
|
||||
headers={"x-headroom-session-id": SESSION_ID},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text[:300]
|
||||
|
||||
assert frozen_seen == [2]
|
||||
|
||||
|
||||
def _post_cloudcode(client: TestClient, body: dict[str, Any]) -> None:
|
||||
resp = client.post(
|
||||
"/v1internal:streamGenerateContent",
|
||||
json={"model": MODEL, "request": body},
|
||||
headers={"x-headroom-session-id": SESSION_ID},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text[:300]
|
||||
resp.read()
|
||||
|
||||
|
||||
def test_cloudcode_streaming_advances_tracker_and_replays_prefix() -> None:
|
||||
"""Cloud Code Assist (v1internal): same freeze + replay as the native path.
|
||||
|
||||
Regression for the requested-changes review on #3394: the replay ran only
|
||||
after ``request_payload`` was already built (so the forwarded body never
|
||||
contained the replayed prefix), and ``_stream_response`` never received
|
||||
the tracker. Fails before this fix: the tracker stays at turn 0, the
|
||||
freeze floor never reaches the pipeline ([0, 0]), and turn 2's nested
|
||||
payload prefix is not turn 1's forwarded bytes.
|
||||
"""
|
||||
sent: list[dict[str, Any]] = []
|
||||
frozen_seen: list[int | None] = []
|
||||
app = create_app(_config())
|
||||
with TestClient(app) as client:
|
||||
proxy = client.app.state.proxy
|
||||
_install_fakes(proxy, sent, frozen_seen)
|
||||
_post_cloudcode(client, _turn_body(0))
|
||||
|
||||
tracker = proxy.session_tracker_store.peek(SESSION_ID)
|
||||
assert tracker is not None, "cloudcode stream never resolved a prefix tracker"
|
||||
assert tracker._turn_number == 1, (
|
||||
"cloudcode stream never called update_from_response - tracker stuck at turn 0"
|
||||
)
|
||||
assert tracker.get_frozen_message_count() == 2
|
||||
assert tracker.get_last_forwarded_messages(), (
|
||||
"tracker recorded no forwarded messages from the cloudcode stream"
|
||||
)
|
||||
|
||||
_post_cloudcode(client, _turn_body(1))
|
||||
|
||||
# Freeze floor reaches the pipeline on the cloudcode route too.
|
||||
assert frozen_seen == [0, 2]
|
||||
|
||||
# The nested request payload carries the replayed prefix byte-identical.
|
||||
req0, req1 = sent[0]["request"], sent[1]["request"]
|
||||
assert "[z]" in req0["systemInstruction"]["parts"][0]["text"]
|
||||
assert req1["systemInstruction"] == req0["systemInstruction"]
|
||||
assert req1["contents"][:1] == req0["contents"]
|
||||
@@ -180,6 +180,10 @@ class NativeGeminiHandler(DummyBatchHandler):
|
||||
self.memory_handler = None
|
||||
self.rate_limiter = None
|
||||
self.usage_reporter = None
|
||||
# The mixin resolves a prefix tracker for the freeze floor (#3394).
|
||||
from headroom.cache.prefix_tracker import SessionTrackerStore
|
||||
|
||||
self.session_tracker_store = SessionTrackerStore()
|
||||
self.responses = iter(responses)
|
||||
self.sent_bodies: list[dict] = []
|
||||
from headroom.ccr.response_handler import CCRResponseHandler
|
||||
|
||||
Reference in New Issue
Block a user