mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): do not book Gemini countTokens as a request outcome (#4412)
## Description `handle_gemini_count_tokens` recorded every `:countTokens` call through `_record_request_outcome` with status 200, `optimized_tokens=totalTokens` and a computed `tokens_saved`. The provider generates and bills nothing for `:countTokens`, so the call inflated the request count and saved-token stats and fed the cost tracker. A successful `:countTokens` is no longer recorded; an upstream rejection (429, 5xx) is still passed to the outcome funnel, which counts it as an upstream rate limit or a failure and stops before any savings or cost accounting, as before. The upstream response is returned unchanged and the count is still logged. Fixes #4408 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/gemini.py`: in `handle_gemini_count_tokens`, call `_record_request_outcome(RequestOutcome(...))` only when `response.status_code >= 400`. A successful sizing call books nothing. - `tests/test_gemini_counttokens_booking.py`: a 200 `:countTokens` returns the upstream count and records no outcome; a 429, 500 and 529 still record an outcome with that status. - `tests/test_proxy_count_tokens_integration.py`: the live-API stats tests (skipped without `GEMINI_API_KEY`) now expect `:countTokens` not to move the request counters. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text == new test file, with the second commit's source change reverted (first commit only) FAILED tests/test_gemini_counttokens_booking.py::test_count_tokens_upstream_error_is_still_recorded[429] FAILED tests/test_gemini_counttokens_booking.py::test_count_tokens_upstream_error_is_still_recorded[500] FAILED tests/test_gemini_counttokens_booking.py::test_count_tokens_upstream_error_is_still_recorded[529] ========================= 3 failed, 1 passed in 1.25s ========================== == tests/test_5xx_accounting_all_providers.py, first commit only FAILED tests/test_5xx_accounting_all_providers.py::test_gemini_count_tokens_handler_threads_real_529_onto_outcome ========================= 1 failed, 8 passed in 4.95s ========================== == with the fix: booking test + 5xx accounting tests ============================== 13 passed in 5.96s ============================== == with the fix: plus the neighbouring Gemini / provider-route / count-tokens tests ================== 66 passed, 37 skipped, 1 warning in 30.78s ================== $ ruff check (3 changed files) All checks passed! $ ruff format --check (3 changed files) 3 files already formatted ``` Neighbouring files run: `test_5xx_accounting_all_providers.py`, `test_proxy_count_tokens_integration.py`, `test_gemini_nonjson_status.py`, `test_proxy_gemini_native_integration.py`, `test_provider_proxy_routes.py`, `test_gemini_compression_offload.py`, `test_proxy/test_gemini_plugin_routed_streaming.py`. The skips are the live-API tests that need a Gemini key; I edited the stats expectations in `test_proxy_count_tokens_integration.py` but could not run them. ## Real Behavior Proof - Environment: Ubuntu on WSL2 (Linux 6.18 x86_64), Python 3.12.15, branch `fix-gemini-counttokens-booking` (head `b8a9bfd2`) on base e995eb9f; `headroom/_core.abi3.so` copied (untracked, not committed) from the official headroom-ai 0.41.0 wheel. Local stub Gemini upstream on 127.0.0.1:18851; real proxy server on 127.0.0.1:18850. The launcher prints the stub URL as the proxy's Gemini upstream, and the run aborts if it does not. - Exact command / steps: start the stub, start the proxy with `--no-cache --no-rate-limit`, POST to `/v1beta/models/gemini-2.0-flash:countTokens` four times (two normal bodies the stub answers 200 `{"totalTokens": 40}`, one the stub answers 429, one it answers 529), then `GET /stats`. Three runs on the same script: (A) upstream/main source, (B) the first commit only, (C) this head. - Observed result: the client gets 200, 200, 429, 529 in all three runs, and the stub saw the same 8 upstream POSTs each time (retries included). `/stats`: (A) `requests.total 2`, `tokens.input 80`, `tokens.saved 936`, `rate_limited.upstream 1`, `failed 1`; (B) all zero, so the 429 and 529 vanish from the error counters; (C) `requests.total 0`, `tokens.input 0`, `tokens.saved 0`, `rate_limited.upstream 1`, `failed 1` (gemini). - Not tested: a real Gemini API (stub only); a configured budget tripping (the stub model has no price entry); the Vertex / cloudcode / Copilot-routed `:countTokens` variants (same handler, only the default upstream URL path was run); `mypy`; the full test suite. Details: ```text ===A upstream/main source STATS requests.total: 2 by_provider: {'gemini': 2} STATS requests.rate_limited: 1 by_source: {'headroom': 0, 'upstream': 1} STATS requests.failed: 1 failed_by_provider: {'unknown': 0, 'gemini': 1} STATS tokens.input: 80 tokens.saved: 936 ===B first commit only STATS requests.total: 0 by_provider: {} STATS requests.rate_limited: 0 by_source: {'headroom': 0, 'upstream': 0} STATS requests.failed: 0 failed_by_provider: {'unknown': 0} STATS tokens.input: 0 tokens.saved: 0 ===C this head STATS requests.total: 0 by_provider: {} STATS requests.rate_limited: 1 by_source: {'headroom': 0, 'upstream': 1} STATS requests.failed: 1 failed_by_provider: {'unknown': 0, 'gemini': 1} STATS tokens.input: 0 tokens.saved: 0 ``` The server's own CLI parser has no Gemini upstream option and `headroom proxy` needs the `mcp` extra, which my venv lacks, so a 10-line launcher runs `python -m headroom.proxy.server` unchanged and only sets the default `ProxyConfig.gemini_api_url` to the stub. Scripts and raw output: `/var/tmp/cb-pub/proof/fix-gemini-counttokens-booking/` (WSL; `run15.sh`, `A_upstream.*`, `B_first.*`, `C_head.*`). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, in stats only. Successful `:countTokens` calls no longer appear in request, savings or cost figures; upstream 429/5xx are still counted as before. The response to the client is unchanged. - Kill switch / disable path: none; revert the commit - Unsafe override required: no - Qualification impact: none expected - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (n/a, no documented behavior covers this) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (the ones listed above, not the full suite) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) n/a ## Additional Notes - Not run: `mypy headroom`, the full test suite. - Outside this change: the handler still has no rate-limiter check; I did not touch that. Produced with AI assistance (Claude Code) operated by me (HetCreep); the reproduction was run on my machine on a clone of the repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
d9958bfdff
commit
4d2dd62b7c
@@ -1551,28 +1551,29 @@ class GeminiHandlerMixin:
|
||||
max(0, original_tokens - compressed_tokens) if compressed_tokens > 0 else 0
|
||||
)
|
||||
|
||||
# Fallback denominator (see comment on the main gemini
|
||||
# record_request site) — pre-comp request size.
|
||||
# countTokens is a sizing helper; it never generates output
|
||||
# tokens and never touches cache. The funnel handles the
|
||||
# "nothing to report" shape with all-zero cache defaults.
|
||||
await self._record_request_outcome(
|
||||
RequestOutcome(
|
||||
request_id=request_id,
|
||||
provider=provider_name,
|
||||
model=model,
|
||||
status_code=response.status_code,
|
||||
original_tokens=original_tokens,
|
||||
optimized_tokens=compressed_tokens,
|
||||
output_tokens=0,
|
||||
tokens_saved=tokens_saved,
|
||||
attempted_input_tokens=compressed_tokens + tokens_saved,
|
||||
total_latency_ms=total_latency,
|
||||
transforms_applied=tuple(transforms_applied),
|
||||
tags=tags,
|
||||
client=client,
|
||||
# countTokens is a sizing helper: the provider generates and bills
|
||||
# nothing for it, so a successful call is logged below but never
|
||||
# booked as a request or as savings. An upstream rejection (429,
|
||||
# 5xx) still goes through the funnel, which records it under the
|
||||
# rate-limit / failure counters and stops before any success stats.
|
||||
if response.status_code >= 400:
|
||||
await self._record_request_outcome(
|
||||
RequestOutcome(
|
||||
request_id=request_id,
|
||||
provider=provider_name,
|
||||
model=model,
|
||||
status_code=response.status_code,
|
||||
original_tokens=original_tokens,
|
||||
optimized_tokens=compressed_tokens,
|
||||
output_tokens=0,
|
||||
tokens_saved=tokens_saved,
|
||||
attempted_input_tokens=compressed_tokens + tokens_saved,
|
||||
total_latency_ms=total_latency,
|
||||
transforms_applied=tuple(transforms_applied),
|
||||
tags=tags,
|
||||
client=client,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
if tokens_saved > 0:
|
||||
logger.info(
|
||||
|
||||
Executable
+111
@@ -0,0 +1,111 @@
|
||||
"""Gemini :countTokens is a sizing call: not booked as a billed request, errors still recorded."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.handlers.gemini import GeminiHandlerMixin
|
||||
|
||||
|
||||
class _FakeRequest:
|
||||
def __init__(self) -> None:
|
||||
self.headers: dict[str, str] = {}
|
||||
self.query_params: dict[str, str] = {}
|
||||
self.url = SimpleNamespace(path="/v1beta/models/gemini-pro:countTokens", query="")
|
||||
self.scope: dict = {"type": "http", "method": "POST"}
|
||||
|
||||
|
||||
class _CountResponse:
|
||||
content = b'{"totalTokens": 40}'
|
||||
headers = {"content-type": "application/json", "content-length": str(len(content))}
|
||||
|
||||
def __init__(self, status_code: int = 200) -> None:
|
||||
self.status_code = status_code
|
||||
|
||||
def json(self) -> object:
|
||||
return json.loads(self.content)
|
||||
|
||||
|
||||
class _FakeMetrics:
|
||||
async def record_failed(self, *, provider: str, model: str = "") -> None:
|
||||
pass
|
||||
|
||||
|
||||
class _Tokenizer:
|
||||
def count_messages(self, messages): # noqa: ANN001, ANN201
|
||||
return 100
|
||||
|
||||
|
||||
class _Handler(GeminiHandlerMixin):
|
||||
GEMINI_API_URL = "https://gemini.example"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.memory_handler = None
|
||||
self.rate_limiter = None
|
||||
self.usage_reporter = None
|
||||
from headroom.cache.prefix_tracker import SessionTrackerStore
|
||||
|
||||
self.session_tracker_store = SessionTrackerStore()
|
||||
self.config = SimpleNamespace(
|
||||
optimize=False,
|
||||
anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
|
||||
)
|
||||
self.metrics = _FakeMetrics()
|
||||
self.outcomes: list = []
|
||||
self.status = 200
|
||||
|
||||
async def _next_request_id(self) -> str:
|
||||
return "req-1"
|
||||
|
||||
async def _retry_request(self, method, url, headers, body): # noqa: ANN001, ANN201
|
||||
return _CountResponse(self.status)
|
||||
|
||||
async def _record_request_outcome(self, outcome) -> None: # noqa: ANN001
|
||||
self.outcomes.append(outcome)
|
||||
|
||||
async def _count_tokens_offloaded(self, model, messages): # noqa: ANN001, ANN201
|
||||
return _Tokenizer(), _Tokenizer().count_messages(messages)
|
||||
|
||||
|
||||
async def _call(handler: _Handler, monkeypatch: pytest.MonkeyPatch): # noqa: ANN202
|
||||
async def payload(request): # noqa: ANN001, ANN201
|
||||
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
|
||||
|
||||
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
|
||||
return await handler.handle_gemini_count_tokens(_FakeRequest(), "gemini-pro")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_count_tokens_is_not_recorded_as_a_request_outcome(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
async def payload(request): # noqa: ANN001, ANN201
|
||||
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
|
||||
|
||||
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
|
||||
|
||||
handler = _Handler()
|
||||
response = await handler.handle_gemini_count_tokens(_FakeRequest(), "gemini-pro")
|
||||
|
||||
# The upstream count is still returned to the client unchanged.
|
||||
assert response.status_code == 200
|
||||
assert json.loads(response.body) == {"totalTokens": 40}
|
||||
# Nothing was generated or billed, so no outcome (spend / savings) is booked.
|
||||
assert handler.outcomes == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("status", [429, 500, 529])
|
||||
async def test_count_tokens_upstream_error_is_still_recorded(
|
||||
monkeypatch: pytest.MonkeyPatch, status: int
|
||||
) -> None:
|
||||
handler = _Handler()
|
||||
handler.status = status
|
||||
response = await _call(handler, monkeypatch)
|
||||
|
||||
assert response.status_code == status
|
||||
# The outcome funnel turns this into a rate-limit / failure count.
|
||||
assert [o.status_code for o in handler.outcomes] == [status]
|
||||
@@ -234,9 +234,9 @@ class TestGeminiCountTokensCompression:
|
||||
assert stats_response.status_code == 200
|
||||
stats = stats_response.json()
|
||||
|
||||
# Verify Gemini requests are tracked
|
||||
assert stats["requests"]["total"] >= 1
|
||||
assert "gemini" in stats["requests"]["by_provider"]
|
||||
# countTokens is a sizing call, not a billed request: it is not booked
|
||||
assert stats["requests"]["total"] == 0
|
||||
assert "gemini" not in stats["requests"]["by_provider"]
|
||||
|
||||
|
||||
class TestGeminiCountTokensLargeContent:
|
||||
@@ -413,28 +413,16 @@ class TestGeminiCountTokensErrors:
|
||||
class TestGeminiCountTokensStats:
|
||||
"""Test proxy stats tracking for countTokens requests."""
|
||||
|
||||
def test_stats_track_gemini_provider(self, gemini_client_optimized, api_key):
|
||||
"""Stats correctly track Gemini provider."""
|
||||
# Clear stats by getting a fresh client
|
||||
def test_stats_do_not_book_count_tokens(self, gemini_client_optimized, api_key):
|
||||
"""countTokens is a sizing call and is not booked as a request."""
|
||||
gemini_client_optimized.post(
|
||||
f"/v1beta/models/gemini-2.0-flash:countTokens?key={api_key}",
|
||||
json={"contents": create_simple_content()},
|
||||
)
|
||||
|
||||
stats = gemini_client_optimized.get("/stats").json()
|
||||
assert "gemini" in stats["requests"]["by_provider"]
|
||||
assert stats["requests"]["by_provider"]["gemini"] >= 1
|
||||
|
||||
def test_stats_track_model(self, gemini_client_optimized, api_key):
|
||||
"""Stats correctly track model used."""
|
||||
gemini_client_optimized.post(
|
||||
f"/v1beta/models/gemini-2.0-flash:countTokens?key={api_key}",
|
||||
json={"contents": create_simple_content()},
|
||||
)
|
||||
|
||||
stats = gemini_client_optimized.get("/stats").json()
|
||||
# Model should be tracked in by_model
|
||||
assert len(stats["requests"]["by_model"]) >= 1
|
||||
assert "gemini" not in stats["requests"]["by_provider"]
|
||||
assert stats["requests"]["total"] == 0
|
||||
|
||||
def test_stats_track_tokens_saved(self, gemini_client_optimized, api_key):
|
||||
"""Stats track tokens saved from compression."""
|
||||
@@ -476,16 +464,16 @@ class TestGeminiCountTokensIntegration:
|
||||
token_count = response.json()["totalTokens"]
|
||||
assert token_count > 0
|
||||
|
||||
# Step 3: Verify stats updated
|
||||
# Step 3: countTokens is not booked as a request
|
||||
updated_stats = gemini_client_optimized.get("/stats").json()
|
||||
assert updated_stats["requests"]["total"] > initial_stats["requests"]["total"]
|
||||
assert updated_stats["requests"]["total"] == initial_stats["requests"]["total"]
|
||||
|
||||
# Step 4: Verify tokens saved is tracked (may be negative for small overhead)
|
||||
# Allow for some compression overhead
|
||||
assert updated_stats["tokens"]["saved"] >= initial_tokens_saved - 100
|
||||
|
||||
def test_multiple_requests_accumulate_stats(self, gemini_client_optimized, api_key):
|
||||
"""Multiple requests correctly accumulate stats."""
|
||||
def test_multiple_requests_are_not_booked(self, gemini_client_optimized, api_key):
|
||||
"""Repeated countTokens calls leave the request counter unchanged."""
|
||||
initial_stats = gemini_client_optimized.get("/stats").json()
|
||||
initial_total = initial_stats["requests"]["total"]
|
||||
|
||||
@@ -497,4 +485,4 @@ class TestGeminiCountTokensIntegration:
|
||||
)
|
||||
|
||||
updated_stats = gemini_client_optimized.get("/stats").json()
|
||||
assert updated_stats["requests"]["total"] >= initial_total + 3
|
||||
assert updated_stats["requests"]["total"] == initial_total
|
||||
|
||||
Reference in New Issue
Block a user