perf(providers): cache Google/Cohere token counters across requests (#3273)

## Description

`GoogleProvider.get_token_counter` and
`CohereProvider.get_token_counter` built a **fresh** counter on every
call:

```python
def get_token_counter(self, model: str) -> TokenCounter:
    ...
    return GeminiTokenCounter(model, client=self._client)   # new object every call
```

The Anthropic and OpenAI providers do **not** do this — they memoize
counters in a per-model dict (`self._token_counters`) and reuse them
across requests. That matters because each counter owns an
`EstimatingTokenCounter` carrying a `TokenCountCache` (the same
memoization added for the OpenAI counter in #3168). Providers are
long-lived singletons on the proxy hot path
(`ProxyProviderRuntime.pipeline_providers` is built once), so rebuilding
the counter every request threw that cache away every request — and a
stable prefix (system prompt + tools), which is identical turn to turn,
was re-tokenized from scratch on every request for Gemini and Cohere
deployments.

This adds the same `self._token_counters` memoization to both providers,
so the token-count cache (and any lazily-built genai model on the Gemini
counter) persists across requests, exactly like the Anthropic/OpenAI
providers.

Benchmark (`GoogleProvider`, ~11 KB stable prefix, mean of 2000
`get_token_counter(...).count_text(prefix)` calls):

```
fresh counter each call (old):    42.5 us/call
cached counter          (new):     0.4 us/call
speedup: 107x
```

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/google.py`: `GoogleProvider.__init__` now
initializes `self._token_counters: dict[str, TokenCounter] = {}`;
`get_token_counter` caches the counter per model instead of constructing
a new `GeminiTokenCounter` each call.
- `headroom/providers/cohere.py`: same change for `CohereProvider` /
`CohereTokenCounter`.
- `tests/test_providers/test_google.py` (new): asserts the counter is
memoized per model (same instance returned; distinct models get distinct
counters) and that an unknown model still raises.
- `tests/test_providers/test_cohere.py`: added the same per-model
caching assertion.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_providers/test_google.py tests/test_providers/test_cohere.py  ->  20 passed
uvx ruff@0.16.2 check headroom/providers/{google,cohere}.py tests/test_providers/test_{google,cohere}.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/providers/google.py headroom/providers/cohere.py  ->  Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: stashed the two source edits and ran the two
new caching tests -> both FAILED (a second `get_token_counter` returned
a different instance); restored the edits -> 20 passed for the two
files. Benchmarked `GoogleProvider` on an ~11 KB stable prefix: fresh
counter per call 42.5us vs cached counter 0.4us (107x), same count.
- Observed result: `get_token_counter(model)` returns the same counter
instance across calls, so its `TokenCountCache` survives across requests
and the stable prefix is tokenized once instead of every turn. Token
counts are unchanged (same counter class, same estimation).
- Not tested: live Gemini/Cohere API `countTokens`/`tokenize` calls (no
API client configured in this environment); the estimation path and
counter identity are exercised.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — no feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same counter class and same token
counts; only the counter's lifetime changes (reused instead of rebuilt).
- Kill switch / disable path: N/A (no config surface added).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; counters go back to being rebuilt
per call.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

This brings the Google and Cohere providers in line with the Anthropic
and OpenAI providers, which already cache their counters in
`self._token_counters` for exactly this reason. No behavior change
beyond counter lifetime; the counters and the shared cache class are
otherwise untouched.
This commit is contained in:
Abhay Singh
2026-09-25 10:25:53 -05:00
committed by GitHub
parent b9e8462a3a
commit cc6a07cd65
4 changed files with 65 additions and 2 deletions
+8 -1
View File
@@ -230,6 +230,11 @@ class CohereProvider(Provider):
If provided, uses tokenize API for accurate counts.
"""
self._client = client
# Cache counters per model so their internal TokenCountCache persists
# across requests, like the Anthropic and OpenAI providers. A fresh
# counter per call threw the cache away every request, re-tokenizing a
# stable prefix (system prompt + tools) on every turn.
self._token_counters: dict[str, TokenCounter] = {}
@property
def name(self) -> str:
@@ -256,7 +261,9 @@ class CohereProvider(Provider):
f"Model '{model}' is not recognized as a Cohere model. "
f"Supported models: {list(_CONTEXT_LIMITS.keys())}"
)
return CohereTokenCounter(model, client=self._client)
if model not in self._token_counters:
self._token_counters[model] = CohereTokenCounter(model, client=self._client)
return self._token_counters[model]
def get_context_limit(self, model: str) -> int:
"""Get context limit for a Cohere model.
+9 -1
View File
@@ -265,6 +265,12 @@ class GoogleProvider(Provider):
If provided, uses countTokens API for accurate counts.
"""
self._client = client
# Cache counters per model so their internal TokenCountCache (and any
# lazily-built genai model) persist across requests, like the Anthropic
# and OpenAI providers. Rebuilding a fresh counter every call threw the
# cache away every request, so a stable prefix (system prompt + tools)
# was re-tokenized on every turn.
self._token_counters: dict[str, TokenCounter] = {}
@property
def name(self) -> str:
@@ -291,7 +297,9 @@ class GoogleProvider(Provider):
f"Model '{model}' is not recognized as a Google model. "
f"Supported models: {list(_CONTEXT_LIMITS.keys())}"
)
return GeminiTokenCounter(model, client=self._client)
if model not in self._token_counters:
self._token_counters[model] = GeminiTokenCounter(model, client=self._client)
return self._token_counters[model]
def get_context_limit(self, model: str) -> int:
"""Get context limit for a Gemini model.
+10
View File
@@ -39,6 +39,16 @@ class TestCohereProvider:
count = counter.count_text("Hello, world!")
assert count > 0
def test_get_token_counter_is_cached_per_model(self, provider):
"""Counters are memoized per model so their internal token-count cache
survives across requests, matching the Anthropic/OpenAI providers."""
first = provider.get_token_counter("command-r-plus")
assert provider.get_token_counter("command-r-plus") is first
# A different model gets its own counter.
other = provider.get_token_counter("command-r")
assert other is not first
assert provider.get_token_counter("command-r") is other
def test_get_context_limit_command_a(self, provider):
"""Test context limit for Command A (256K)."""
limit = provider.get_context_limit("command-a")
+38
View File
@@ -0,0 +1,38 @@
"""Tests for Google (Gemini) provider."""
from __future__ import annotations
import pytest
from headroom.providers import GoogleProvider
class TestGoogleProvider:
"""Tests for GoogleProvider."""
@pytest.fixture
def provider(self):
"""Create Google provider without client (estimation mode)."""
return GoogleProvider()
def test_name(self, provider):
assert provider.name == "google"
def test_get_token_counter(self, provider):
counter = provider.get_token_counter("gemini-2.0-flash")
assert counter is not None
assert counter.count_text("Hello, world!") > 0
def test_get_token_counter_is_cached_per_model(self, provider):
"""Counters are memoized per model so their internal token-count cache
survives across requests, matching the Anthropic/OpenAI providers."""
first = provider.get_token_counter("gemini-2.0-flash")
assert provider.get_token_counter("gemini-2.0-flash") is first
# A different model gets its own counter.
other = provider.get_token_counter("gemini-1.5-pro")
assert other is not first
assert provider.get_token_counter("gemini-1.5-pro") is other
def test_get_token_counter_rejects_unknown_model(self, provider):
with pytest.raises(ValueError):
provider.get_token_counter("gpt-4o")