fix(proxy): do not book Gemini countTokens as a request outcome (#4412)

## Description

`handle_gemini_count_tokens` recorded every `:countTokens` call through
`_record_request_outcome` with status 200,
`optimized_tokens=totalTokens` and a computed `tokens_saved`. The
provider generates and bills nothing for `:countTokens`, so the call
inflated the request count and saved-token stats and fed the cost
tracker. A successful `:countTokens` is no longer recorded; an upstream
rejection (429, 5xx) is still passed to the outcome funnel, which counts
it as an upstream rate limit or a failure and stops before any savings
or cost accounting, as before. The upstream response is returned
unchanged and the count is still logged.

Fixes #4408

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/gemini.py`: in `handle_gemini_count_tokens`,
call `_record_request_outcome(RequestOutcome(...))` only when
`response.status_code >= 400`. A successful sizing call books nothing.
- `tests/test_gemini_counttokens_booking.py`: a 200 `:countTokens`
returns the upstream count and records no outcome; a 429, 500 and 529
still record an outcome with that status.
- `tests/test_proxy_count_tokens_integration.py`: the live-API stats
tests (skipped without `GEMINI_API_KEY`) now expect `:countTokens` not
to move the request counters.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
== new test file, with the second commit's source change reverted (first commit only)
FAILED tests/test_gemini_counttokens_booking.py::test_count_tokens_upstream_error_is_still_recorded[429]
FAILED tests/test_gemini_counttokens_booking.py::test_count_tokens_upstream_error_is_still_recorded[500]
FAILED tests/test_gemini_counttokens_booking.py::test_count_tokens_upstream_error_is_still_recorded[529]
========================= 3 failed, 1 passed in 1.25s ==========================
== tests/test_5xx_accounting_all_providers.py, first commit only
FAILED tests/test_5xx_accounting_all_providers.py::test_gemini_count_tokens_handler_threads_real_529_onto_outcome
========================= 1 failed, 8 passed in 4.95s ==========================
== with the fix: booking test + 5xx accounting tests
============================== 13 passed in 5.96s ==============================
== with the fix: plus the neighbouring Gemini / provider-route / count-tokens tests
================== 66 passed, 37 skipped, 1 warning in 30.78s ==================
$ ruff check (3 changed files)
All checks passed!
$ ruff format --check (3 changed files)
3 files already formatted
```

Neighbouring files run: `test_5xx_accounting_all_providers.py`,
`test_proxy_count_tokens_integration.py`,
`test_gemini_nonjson_status.py`,
`test_proxy_gemini_native_integration.py`,
`test_provider_proxy_routes.py`, `test_gemini_compression_offload.py`,
`test_proxy/test_gemini_plugin_routed_streaming.py`. The skips are the
live-API tests that need a Gemini key; I edited the stats expectations
in `test_proxy_count_tokens_integration.py` but could not run them.

## Real Behavior Proof

- Environment: Ubuntu on WSL2 (Linux 6.18 x86_64), Python 3.12.15,
branch `fix-gemini-counttokens-booking` (head `b8a9bfd2`) on base
e995eb9f; `headroom/_core.abi3.so` copied (untracked, not committed)
from the official headroom-ai 0.41.0 wheel. Local stub Gemini upstream
on 127.0.0.1:18851; real proxy server on 127.0.0.1:18850. The launcher
prints the stub URL as the proxy's Gemini upstream, and the run aborts
if it does not.
- Exact command / steps: start the stub, start the proxy with
`--no-cache --no-rate-limit`, POST to
`/v1beta/models/gemini-2.0-flash:countTokens` four times (two normal
bodies the stub answers 200 `{"totalTokens": 40}`, one the stub answers
429, one it answers 529), then `GET /stats`. Three runs on the same
script: (A) upstream/main source, (B) the first commit only, (C) this
head.
- Observed result: the client gets 200, 200, 429, 529 in all three runs,
and the stub saw the same 8 upstream POSTs each time (retries included).
`/stats`: (A) `requests.total 2`, `tokens.input 80`, `tokens.saved 936`,
`rate_limited.upstream 1`, `failed 1`; (B) all zero, so the 429 and 529
vanish from the error counters; (C) `requests.total 0`, `tokens.input
0`, `tokens.saved 0`, `rate_limited.upstream 1`, `failed 1` (gemini).
- Not tested: a real Gemini API (stub only); a configured budget
tripping (the stub model has no price entry); the Vertex / cloudcode /
Copilot-routed `:countTokens` variants (same handler, only the default
upstream URL path was run); `mypy`; the full test suite.

Details:

```text
===A upstream/main source
STATS requests.total: 2 by_provider: {'gemini': 2}
STATS requests.rate_limited: 1 by_source: {'headroom': 0, 'upstream': 1}
STATS requests.failed: 1 failed_by_provider: {'unknown': 0, 'gemini': 1}
STATS tokens.input: 80 tokens.saved: 936
===B first commit only
STATS requests.total: 0 by_provider: {}
STATS requests.rate_limited: 0 by_source: {'headroom': 0, 'upstream': 0}
STATS requests.failed: 0 failed_by_provider: {'unknown': 0}
STATS tokens.input: 0 tokens.saved: 0
===C this head
STATS requests.total: 0 by_provider: {}
STATS requests.rate_limited: 1 by_source: {'headroom': 0, 'upstream': 1}
STATS requests.failed: 1 failed_by_provider: {'unknown': 0, 'gemini': 1}
STATS tokens.input: 0 tokens.saved: 0
```

The server's own CLI parser has no Gemini upstream option and `headroom
proxy` needs the `mcp` extra, which my venv lacks, so a 10-line launcher
runs `python -m headroom.proxy.server` unchanged and only sets the
default `ProxyConfig.gemini_api_url` to the stub. Scripts and raw
output: `/var/tmp/cb-pub/proof/fix-gemini-counttokens-booking/` (WSL;
`run15.sh`, `A_upstream.*`, `B_first.*`, `C_head.*`).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, in stats only. Successful
`:countTokens` calls no longer appear in request, savings or cost
figures; upstream 429/5xx are still counted as before. The response to
the client is unchanged.
- Kill switch / disable path: none; revert the commit
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (n/a, no
documented behavior covers this)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (the ones
listed above, not the full suite)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

n/a

## Additional Notes

- Not run: `mypy headroom`, the full test suite.
- Outside this change: the handler still has no rate-limiter check; I
did not touch that.

Produced with AI assistance (Claude Code) operated by me (HetCreep); the
reproduction was run on my machine on a clone of the repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
HetCreep
2026-10-28 23:29:56 -05:00
committed by GitHub
co-authored by Claude Opus 5.5 JD Davis
parent d9958bfdff
commit 4d2dd62b7c
3 changed files with 145 additions and 45 deletions
+22 -21
View File
@@ -1551,28 +1551,29 @@ class GeminiHandlerMixin:
max(0, original_tokens - compressed_tokens) if compressed_tokens > 0 else 0
)
# Fallback denominator (see comment on the main gemini
# record_request site) — pre-comp request size.
# countTokens is a sizing helper; it never generates output
# tokens and never touches cache. The funnel handles the
# "nothing to report" shape with all-zero cache defaults.
await self._record_request_outcome(
RequestOutcome(
request_id=request_id,
provider=provider_name,
model=model,
status_code=response.status_code,
original_tokens=original_tokens,
optimized_tokens=compressed_tokens,
output_tokens=0,
tokens_saved=tokens_saved,
attempted_input_tokens=compressed_tokens + tokens_saved,
total_latency_ms=total_latency,
transforms_applied=tuple(transforms_applied),
tags=tags,
client=client,
# countTokens is a sizing helper: the provider generates and bills
# nothing for it, so a successful call is logged below but never
# booked as a request or as savings. An upstream rejection (429,
# 5xx) still goes through the funnel, which records it under the
# rate-limit / failure counters and stops before any success stats.
if response.status_code >= 400:
await self._record_request_outcome(
RequestOutcome(
request_id=request_id,
provider=provider_name,
model=model,
status_code=response.status_code,
original_tokens=original_tokens,
optimized_tokens=compressed_tokens,
output_tokens=0,
tokens_saved=tokens_saved,
attempted_input_tokens=compressed_tokens + tokens_saved,
total_latency_ms=total_latency,
transforms_applied=tuple(transforms_applied),
tags=tags,
client=client,
)
)
)
if tokens_saved > 0:
logger.info(
+111
View File
@@ -0,0 +1,111 @@
"""Gemini :countTokens is a sizing call: not booked as a billed request, errors still recorded."""
from __future__ import annotations
import json
from types import SimpleNamespace
import pytest
from headroom.proxy.handlers.gemini import GeminiHandlerMixin
class _FakeRequest:
def __init__(self) -> None:
self.headers: dict[str, str] = {}
self.query_params: dict[str, str] = {}
self.url = SimpleNamespace(path="/v1beta/models/gemini-pro:countTokens", query="")
self.scope: dict = {"type": "http", "method": "POST"}
class _CountResponse:
content = b'{"totalTokens": 40}'
headers = {"content-type": "application/json", "content-length": str(len(content))}
def __init__(self, status_code: int = 200) -> None:
self.status_code = status_code
def json(self) -> object:
return json.loads(self.content)
class _FakeMetrics:
async def record_failed(self, *, provider: str, model: str = "") -> None:
pass
class _Tokenizer:
def count_messages(self, messages): # noqa: ANN001, ANN201
return 100
class _Handler(GeminiHandlerMixin):
GEMINI_API_URL = "https://gemini.example"
def __init__(self) -> None:
self.memory_handler = None
self.rate_limiter = None
self.usage_reporter = None
from headroom.cache.prefix_tracker import SessionTrackerStore
self.session_tracker_store = SessionTrackerStore()
self.config = SimpleNamespace(
optimize=False,
anthropic_pre_upstream_memory_context_timeout_seconds=0.1,
)
self.metrics = _FakeMetrics()
self.outcomes: list = []
self.status = 200
async def _next_request_id(self) -> str:
return "req-1"
async def _retry_request(self, method, url, headers, body): # noqa: ANN001, ANN201
return _CountResponse(self.status)
async def _record_request_outcome(self, outcome) -> None: # noqa: ANN001
self.outcomes.append(outcome)
async def _count_tokens_offloaded(self, model, messages): # noqa: ANN001, ANN201
return _Tokenizer(), _Tokenizer().count_messages(messages)
async def _call(handler: _Handler, monkeypatch: pytest.MonkeyPatch): # noqa: ANN202
async def payload(request): # noqa: ANN001, ANN201
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
return await handler.handle_gemini_count_tokens(_FakeRequest(), "gemini-pro")
@pytest.mark.asyncio
async def test_count_tokens_is_not_recorded_as_a_request_outcome(
monkeypatch: pytest.MonkeyPatch,
) -> None:
async def payload(request): # noqa: ANN001, ANN201
return {"contents": [{"role": "user", "parts": [{"text": "hello"}]}]}
monkeypatch.setattr("headroom.proxy.helpers._read_request_json", payload)
handler = _Handler()
response = await handler.handle_gemini_count_tokens(_FakeRequest(), "gemini-pro")
# The upstream count is still returned to the client unchanged.
assert response.status_code == 200
assert json.loads(response.body) == {"totalTokens": 40}
# Nothing was generated or billed, so no outcome (spend / savings) is booked.
assert handler.outcomes == []
@pytest.mark.asyncio
@pytest.mark.parametrize("status", [429, 500, 529])
async def test_count_tokens_upstream_error_is_still_recorded(
monkeypatch: pytest.MonkeyPatch, status: int
) -> None:
handler = _Handler()
handler.status = status
response = await _call(handler, monkeypatch)
assert response.status_code == status
# The outcome funnel turns this into a rate-limit / failure count.
assert [o.status_code for o in handler.outcomes] == [status]
+12 -24
View File
@@ -234,9 +234,9 @@ class TestGeminiCountTokensCompression:
assert stats_response.status_code == 200
stats = stats_response.json()
# Verify Gemini requests are tracked
assert stats["requests"]["total"] >= 1
assert "gemini" in stats["requests"]["by_provider"]
# countTokens is a sizing call, not a billed request: it is not booked
assert stats["requests"]["total"] == 0
assert "gemini" not in stats["requests"]["by_provider"]
class TestGeminiCountTokensLargeContent:
@@ -413,28 +413,16 @@ class TestGeminiCountTokensErrors:
class TestGeminiCountTokensStats:
"""Test proxy stats tracking for countTokens requests."""
def test_stats_track_gemini_provider(self, gemini_client_optimized, api_key):
"""Stats correctly track Gemini provider."""
# Clear stats by getting a fresh client
def test_stats_do_not_book_count_tokens(self, gemini_client_optimized, api_key):
"""countTokens is a sizing call and is not booked as a request."""
gemini_client_optimized.post(
f"/v1beta/models/gemini-2.0-flash:countTokens?key={api_key}",
json={"contents": create_simple_content()},
)
stats = gemini_client_optimized.get("/stats").json()
assert "gemini" in stats["requests"]["by_provider"]
assert stats["requests"]["by_provider"]["gemini"] >= 1
def test_stats_track_model(self, gemini_client_optimized, api_key):
"""Stats correctly track model used."""
gemini_client_optimized.post(
f"/v1beta/models/gemini-2.0-flash:countTokens?key={api_key}",
json={"contents": create_simple_content()},
)
stats = gemini_client_optimized.get("/stats").json()
# Model should be tracked in by_model
assert len(stats["requests"]["by_model"]) >= 1
assert "gemini" not in stats["requests"]["by_provider"]
assert stats["requests"]["total"] == 0
def test_stats_track_tokens_saved(self, gemini_client_optimized, api_key):
"""Stats track tokens saved from compression."""
@@ -476,16 +464,16 @@ class TestGeminiCountTokensIntegration:
token_count = response.json()["totalTokens"]
assert token_count > 0
# Step 3: Verify stats updated
# Step 3: countTokens is not booked as a request
updated_stats = gemini_client_optimized.get("/stats").json()
assert updated_stats["requests"]["total"] > initial_stats["requests"]["total"]
assert updated_stats["requests"]["total"] == initial_stats["requests"]["total"]
# Step 4: Verify tokens saved is tracked (may be negative for small overhead)
# Allow for some compression overhead
assert updated_stats["tokens"]["saved"] >= initial_tokens_saved - 100
def test_multiple_requests_accumulate_stats(self, gemini_client_optimized, api_key):
"""Multiple requests correctly accumulate stats."""
def test_multiple_requests_are_not_booked(self, gemini_client_optimized, api_key):
"""Repeated countTokens calls leave the request counter unchanged."""
initial_stats = gemini_client_optimized.get("/stats").json()
initial_total = initial_stats["requests"]["total"]
@@ -497,4 +485,4 @@ class TestGeminiCountTokensIntegration:
)
updated_stats = gemini_client_optimized.get("/stats").json()
assert updated_stats["requests"]["total"] >= initial_total + 3
assert updated_stats["requests"]["total"] == initial_total