mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
`GoogleProvider.get_token_counter` and
`CohereProvider.get_token_counter` built a **fresh** counter on every
call:
```python
def get_token_counter(self, model: str) -> TokenCounter:
...
return GeminiTokenCounter(model, client=self._client) # new object every call
```
The Anthropic and OpenAI providers do **not** do this — they memoize
counters in a per-model dict (`self._token_counters`) and reuse them
across requests. That matters because each counter owns an
`EstimatingTokenCounter` carrying a `TokenCountCache` (the same
memoization added for the OpenAI counter in #3168). Providers are
long-lived singletons on the proxy hot path
(`ProxyProviderRuntime.pipeline_providers` is built once), so rebuilding
the counter every request threw that cache away every request — and a
stable prefix (system prompt + tools), which is identical turn to turn,
was re-tokenized from scratch on every request for Gemini and Cohere
deployments.
This adds the same `self._token_counters` memoization to both providers,
so the token-count cache (and any lazily-built genai model on the Gemini
counter) persists across requests, exactly like the Anthropic/OpenAI
providers.
Benchmark (`GoogleProvider`, ~11 KB stable prefix, mean of 2000
`get_token_counter(...).count_text(prefix)` calls):
```
fresh counter each call (old): 42.5 us/call
cached counter (new): 0.4 us/call
speedup: 107x
```
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/google.py`: `GoogleProvider.__init__` now
initializes `self._token_counters: dict[str, TokenCounter] = {}`;
`get_token_counter` caches the counter per model instead of constructing
a new `GeminiTokenCounter` each call.
- `headroom/providers/cohere.py`: same change for `CohereProvider` /
`CohereTokenCounter`.
- `tests/test_providers/test_google.py` (new): asserts the counter is
memoized per model (same instance returned; distinct models get distinct
counters) and that an unknown model still raises.
- `tests/test_providers/test_cohere.py`: added the same per-model
caching assertion.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
tests/test_providers/test_google.py tests/test_providers/test_cohere.py -> 20 passed
uvx ruff@0.16.2 check headroom/providers/{google,cohere}.py tests/test_providers/test_{google,cohere}.py -> All checks passed!
uvx mypy@1.20.2 headroom/providers/google.py headroom/providers/cohere.py -> Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: stashed the two source edits and ran the two
new caching tests -> both FAILED (a second `get_token_counter` returned
a different instance); restored the edits -> 20 passed for the two
files. Benchmarked `GoogleProvider` on an ~11 KB stable prefix: fresh
counter per call 42.5us vs cached counter 0.4us (107x), same count.
- Observed result: `get_token_counter(model)` returns the same counter
instance across calls, so its `TokenCountCache` survives across requests
and the stable prefix is tokenized once instead of every turn. Token
counts are unchanged (same counter class, same estimation).
- Not tested: live Gemini/Cohere API `countTokens`/`tokenize` calls (no
API client configured in this environment); the estimation path and
counter identity are exercised.
## Runtime Rollout Safety
- Rollout-managed feature(s): none — no feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. Same counter class and same token
counts; only the counter's lifetime changes (reused instead of rebuilt).
- Kill switch / disable path: N/A (no config surface added).
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit; counters go back to being rebuilt
per call.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal behavior, counts unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
This brings the Google and Cohere providers in line with the Anthropic
and OpenAI providers, which already cache their counters in
`self._token_counters` for exactly this reason. No behavior change
beyond counter lifetime; the counters and the shared cache class are
otherwise untouched.
39 lines
1.3 KiB
Python
39 lines
1.3 KiB
Python
"""Tests for Google (Gemini) provider."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.providers import GoogleProvider
|
|
|
|
|
|
class TestGoogleProvider:
|
|
"""Tests for GoogleProvider."""
|
|
|
|
@pytest.fixture
|
|
def provider(self):
|
|
"""Create Google provider without client (estimation mode)."""
|
|
return GoogleProvider()
|
|
|
|
def test_name(self, provider):
|
|
assert provider.name == "google"
|
|
|
|
def test_get_token_counter(self, provider):
|
|
counter = provider.get_token_counter("gemini-2.0-flash")
|
|
assert counter is not None
|
|
assert counter.count_text("Hello, world!") > 0
|
|
|
|
def test_get_token_counter_is_cached_per_model(self, provider):
|
|
"""Counters are memoized per model so their internal token-count cache
|
|
survives across requests, matching the Anthropic/OpenAI providers."""
|
|
first = provider.get_token_counter("gemini-2.0-flash")
|
|
assert provider.get_token_counter("gemini-2.0-flash") is first
|
|
# A different model gets its own counter.
|
|
other = provider.get_token_counter("gemini-1.5-pro")
|
|
assert other is not first
|
|
assert provider.get_token_counter("gemini-1.5-pro") is other
|
|
|
|
def test_get_token_counter_rejects_unknown_model(self, provider):
|
|
with pytest.raises(ValueError):
|
|
provider.get_token_counter("gpt-4o")
|