Files
headroom/tests/test_tokenizer_count_offload.py
Tejas Chopra 31d378b0e5 refactor(proxy): remove test-only and dead helpers from the handlers (#4117)
## Description

This removes proxy handler helpers that production code never calls, and
one test that always skipped. Behaviour does not change.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/openai.py`: removed
`_compact_openai_tool_schema_value`. It had no callers anywhere.
- `headroom/proxy/handlers/streaming.py`:
- Removed `StreamingMixin._parse_sse_usage`, the 92-line single-chunk
SSE usage parser. Production only calls `_parse_sse_usage_from_buffer`.
- Removed `_sse_contains_error_event`, a 5-line wrapper over
`_sse_event_outcome` that only `tests/test_route_advice.py` called.
- `headroom/proxy/handlers/gemini.py`: removed
`GeminiHandlerMixin._count_texts_offloaded`, which only tests called.
`tests/test_tokenizer_count_offload.py:77` already asserts that no
handler uses it.
- Test changes:
- `tests/test_proxy_cache_ttl_metrics.py`: the TTL-bucket test now
drives `_parse_sse_usage_from_buffer`.
- `tests/test_streaming_sse_malformed_events.py`: dropped the
chunk-parser variants. The buffered-parser variants of the same cases
stay.
- `tests/test_route_advice.py`: carries the error-event helper locally,
so its `_sse_event_outcome` coverage is unchanged.
- `tests/test_tokenizer_count_offload.py`: calls
`token_counting.count_texts_offloaded(proxy, ...)` directly, still on a
real proxy and its worker thread.
- `tests/test_codex_ws_compression_scheduler.py`: removed
`test_codex_ws_emits_perf_log_with_cache_keys`, which called
`pytest.skip` unconditionally ("Pending: full WS lifecycle harness"),
along with the fixtures only it used.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.16.8 check headroom/proxy/handlers tests/test_codex_ws_compression_scheduler.py tests/test_tokenizer_count_offload.py tests/test_route_advice.py tests/test_streaming_sse_malformed_events.py tests/test_proxy_cache_ttl_metrics.py
All checks passed!
$ pytest -q tests/test_streaming_sse_malformed_events.py tests/test_proxy_cache_ttl_metrics.py tests/test_route_advice.py tests/test_tokenizer_count_offload.py tests/test_codex_ws_compression_scheduler.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_additional_tools.py tests/test_gemini_compression_offload.py tests/test_proxy_gemini_integration.py tests/test_proxy_gemini_native_integration.py
470 passed, 27 skipped in 24.13s
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo venv, branch
`cleanup/proxy-handlers-dead-helpers` @ c15e679e, temp
`HOME`/`HEADROOM_WORKSPACE_DIR`/`CODEX_HOME`, `HEADROOM_OFFLINE=1`
- Exact command / steps: (1) a `grep -rnE
'_compact_openai_tool_schema_value|_parse_sse_usage\b|_sse_contains_error_event|_count_texts_offloaded'`
over headroom, tests, crates, plugins, sdk, examples, benchmarks,
scripts, e2e, docs, wiki, .github, pyproject.toml and every consumer
repo clone; (2) the shared smoke harness: a real `headroom proxy` from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI turn; (3) a direct check that the removed names are gone from the
assembled `HeadroomProxy` and that the live SSE usage parser still works
- Observed result: (1) before the change, only the definitions and tests
matched; after it, nothing in `headroom/` references the removed names.
(2) and (3) are shown below: both turns return 200 with savings, the
names are absent, and the live parser returns the usage.
- Not tested: a real hosted upstream

```text
$ bash smoke.sh <worktree>
branch: cleanup/proxy-handlers-dead-helpers @ c15e679e
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines:
   (only the pre-existing offline-mode notice and Kompress offline-prefetch warnings)

$ HOME=$T/h HEADROOM_WORKSPACE_DIR=$T/ws CODEX_HOME=$T/c python -c "<check HeadroomProxy attrs; run _parse_sse_usage_from_buffer>"
_parse_sse_usage on HeadroomProxy: False
_sse_contains_error_event on HeadroomProxy: False
_count_texts_offloaded on HeadroomProxy: False
_compact_openai_tool_schema_value in openai: False
live SSE usage parser: {'input_tokens': 9, 'output_tokens': 2, 'cache_read_input_tokens': 0}
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: no
- Kill switch / disable path: not applicable (dead code removal)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

- `headroom.proxy.token_counting.count_texts_offloaded` is now called
only by tests. It is outside this cleanup slice, so it is left in place
and flagged for the token-counting owner.
- No `headroom.proxy.handlers` symbol appears in the consumer import
index.
- mypy and comments: N/A, since this PR only deletes code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-10 09:00:49 -07:00

282 lines
12 KiB
Python

"""Token counting must run off the event loop (GH #1701): the Anthropic messages
handler resolved the tokenizer and counted the conversation inline in the async
handler. For HF-backed models (e.g. deepseek-*) first use triggers an unbounded
network download, freezing the whole server (610s request, then /livez, /readyz
and /health hang until kill). The fix routes resolution + counting through
HeadroomProxy._count_tokens_offloaded (compression executor, bounded by
COMPRESSION_TIMEOUT_SECONDS, fail-open to estimation) — shared by every provider
handler (Anthropic, OpenAI, Gemini), since the OpenAI passthrough endpoints
receive the same HF-backed models — and offloads the inline batch
pipeline.apply() calls the same way.
"""
from __future__ import annotations
import asyncio
import inspect
import threading
import time
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
from headroom.proxy.handlers.batch import BatchHandlerMixin
from headroom.proxy.handlers.gemini import GeminiHandlerMixin
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
from headroom.proxy.server import (
CompressionQuarantinedError,
ProxyConfig,
create_app,
)
from headroom.proxy.token_counting import (
_count_offloaded,
count_texts_offloaded,
count_tokens_offloaded,
)
from headroom.tokenizers import EstimatingTokenCounter
def _make_proxy(): # noqa: ANN202 — returns the internal HeadroomProxy
app = create_app(
ProxyConfig(
optimize=True,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
)
)
return app.state.proxy
def test_handlers_offload_token_counting_and_batch_apply() -> None:
"""Wiring guard: the request paths must use the offloaded helpers, not inline
get_tokenizer/count_messages or pipeline.apply on the event loop."""
# Every provider handler that counts the original conversation must route
# resolution + counting through the shared fail-open helper, never inline on
# the loop. OpenAI /chat + /responses are multi-provider passthroughs, so an
# HF-routed model (qwen, deepseek, llama, ...) can reach them and cold-load.
for mixin, method in (
(AnthropicHandlerMixin, "handle_anthropic_messages"),
(OpenAIHandlerMixin, "handle_openai_chat"),
(OpenAIHandlerMixin, "handle_openai_responses"),
(GeminiHandlerMixin, "handle_gemini_generate_content"),
(GeminiHandlerMixin, "handle_google_cloudcode_stream"),
(GeminiHandlerMixin, "handle_gemini_count_tokens"),
):
fn = getattr(mixin, method)
assert inspect.iscoroutinefunction(fn), f"{method} must be async"
src = inspect.getsource(fn)
assert "_count_tokens_offloaded(" in src, f"{method}: token counting not offloaded"
assert "tokenizer = get_tokenizer(" not in src, (
f"{method}: tokenizer resolved inline on the loop"
)
fn = GeminiHandlerMixin.handle_gemini_stream_generate_content
assert inspect.iscoroutinefunction(fn)
src = inspect.getsource(fn)
assert "return await self.handle_gemini_generate_content(" in src
assert "upstream_base_url=upstream_base_url" in src
assert "_count_texts_offloaded(" not in src
assert "_dict_parts(" not in src
for mixin, method in (
(AnthropicHandlerMixin, "handle_anthropic_batch_create"),
(BatchHandlerMixin, "handle_google_batch_create"),
(BatchHandlerMixin, "_compress_batch_jsonl"),
):
fn = getattr(mixin, method)
assert inspect.iscoroutinefunction(fn), f"{method} must be async"
src = inspect.getsource(fn)
if "pipeline.apply(" in src:
assert "_run_compression_in_executor(" in src, f"{method}: apply() not offloaded"
assert "COMPRESSION_TIMEOUT_SECONDS" in src, f"{method}: offload missing timeout"
helper_src = inspect.getsource(_count_offloaded)
assert "COMPRESSION_TIMEOUT_SECONDS" in helper_src
assert "EstimatingTokenCounter" in helper_src, "helper must fail open to estimation"
async def test_count_tokens_offloaded_runs_on_worker_thread(monkeypatch) -> None: # noqa: ANN001
proxy = _make_proxy()
loop_thread = threading.current_thread().name
seen: dict[str, str] = {}
class _SpyTokenizer(EstimatingTokenCounter):
def count_messages(self, messages): # noqa: ANN001, ANN201
seen["thread"] = threading.current_thread().name
return super().count_messages(messages)
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda *a, **k: _SpyTokenizer())
_, tokens = await proxy._count_tokens_offloaded("gpt-4", [{"role": "user", "content": "hi"}])
assert tokens > 0
assert seen["thread"].startswith("headroom-compress")
assert seen["thread"] != loop_thread
async def test_count_tokens_offloaded_keeps_loop_responsive(monkeypatch) -> None: # noqa: ANN001
"""A slow tokenizer (stand-in for an HF network load) must not starve the loop —
the pre-fix inline call yielded ~0 ticks here."""
proxy = _make_proxy()
ticks = 0
async def _ticker() -> None:
nonlocal ticks
while True:
await asyncio.sleep(0.01)
ticks += 1
class _SlowTokenizer(EstimatingTokenCounter):
def count_messages(self, messages): # noqa: ANN001, ANN201
time.sleep(0.3)
return super().count_messages(messages)
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda *a, **k: _SlowTokenizer())
tick_task = asyncio.create_task(_ticker())
try:
_, tokens = await proxy._count_tokens_offloaded("m", [{"role": "user", "content": "hi"}])
finally:
tick_task.cancel()
assert tokens > 0
assert ticks >= 5
async def test_count_tokens_offloaded_fails_open(monkeypatch) -> None: # noqa: ANN001
"""Resolution errors and timeouts downgrade to estimation instead of raising."""
proxy = _make_proxy()
def _boom(*a, **k): # noqa: ANN002, ANN003, ANN202
raise RuntimeError("tokenizer backend exploded")
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", _boom)
tokenizer, tokens = await proxy._count_tokens_offloaded(
"deepseek-chat", [{"role": "user", "content": "hello world"}]
)
assert isinstance(tokenizer, EstimatingTokenCounter)
assert tokens > 0
# Logged-once bookkeeping records the downgraded model.
assert "deepseek-chat" in proxy._token_count_fallback_models
async def test_count_tokens_offloaded_fails_open_on_executor_quarantine() -> None:
"""Now that OpenAI/Gemini counting shares the compression executor, an
unrelated request's compression timeout can quarantine it — the next
``_run_compression_in_executor`` call raises ``CompressionQuarantinedError``
immediately (process-wide state). A request that is only counting tokens
must not 500 on that; it fails open to estimation like any other error."""
# The executor's ``except Exception`` fail-open only catches the quarantine
# error because it subclasses Exception — pin that contract.
assert issubclass(CompressionQuarantinedError, Exception)
proxy = _make_proxy()
# Record a concurrent compression as timed out so the real executor guard
# quarantines the next call — no mock of the helper itself. Since the
# quarantine became time-capped (#2412), standing debt alone no longer
# quarantines: the deadline armed by the fresh timeout must still be in
# the future, so arm it the way a real timeout would. The debt must reach
# the quarantine threshold (half the auto-sized pool).
proxy._compression_timed_out_in_flight = proxy._compression_quarantine_threshold
proxy._compression_quarantine_deadline = time.monotonic() + 60.0
tokenizer, tokens = await proxy._count_tokens_offloaded(
"qwen2.5-coder", [{"role": "user", "content": "hello world"}]
)
assert isinstance(tokenizer, EstimatingTokenCounter)
assert tokens > 0
assert "qwen2.5-coder" in proxy._token_count_fallback_models
async def test_count_tokens_offloaded_returns_count_text_capable_tokenizer() -> None:
"""The fail-open tokenizer should still support text counting for callers
that need per-fragment accounting."""
proxy = _make_proxy()
# Quarantine forces the fail-open branch (an EstimatingTokenCounter).
# Post-#2412 the quarantine is time-capped, so the deadline must be armed
# alongside the standing debt, which must reach the quarantine threshold.
proxy._compression_timed_out_in_flight = proxy._compression_quarantine_threshold
proxy._compression_quarantine_deadline = time.monotonic() + 60.0
# The empty-messages count is intentionally discarded by that handler
# (it sums text parts itself), so only the tokenizer matters here.
tokenizer, _ = await proxy._count_tokens_offloaded("qwen2.5-coder", [])
assert isinstance(tokenizer, EstimatingTokenCounter)
# The streaming handler's per-part loop must not raise on the fallback.
assert tokenizer.count_text("hello world") > 0
async def test_count_texts_offloaded_runs_on_worker_thread(monkeypatch) -> None: # noqa: ANN001
proxy = _make_proxy()
loop_thread = threading.current_thread().name
seen: dict[str, str] = {}
class _SpyTokenizer(EstimatingTokenCounter):
def count_text(self, text): # noqa: ANN001, ANN201
seen["thread"] = threading.current_thread().name
return super().count_text(text)
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda *a, **k: _SpyTokenizer())
_, tokens = await count_texts_offloaded(proxy, "gemini-pro", ["hello", "world"])
assert tokens > 0
assert seen["thread"].startswith("headroom-compress")
assert seen["thread"] != loop_thread
async def test_count_texts_offloaded_fails_open(monkeypatch) -> None: # noqa: ANN001
"""The texts variant downgrades to estimation on a resolution error, the same
as the messages variant (its fail-open branch was previously uncovered)."""
proxy = _make_proxy()
def _boom(*a, **k): # noqa: ANN002, ANN003, ANN202
raise RuntimeError("tokenizer backend exploded")
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", _boom)
tokenizer, tokens = await count_texts_offloaded(proxy, "deepseek-chat", ["hello", "world"])
assert isinstance(tokenizer, EstimatingTokenCounter)
assert tokens > 0
assert "deepseek-chat" in proxy._token_count_fallback_models
async def test_count_offloaded_without_executor_estimates() -> None:
"""An owner with no compression executor (a lightweight caller or test double)
fails open to estimation inline instead of crashing on the missing runner."""
class _NoExecutorOwner:
pass
owner = _NoExecutorOwner()
tok, n_msg = await count_tokens_offloaded(
owner, "gpt-4", [{"role": "user", "content": "hello world"}]
)
assert isinstance(tok, EstimatingTokenCounter)
assert n_msg > 0
tok2, n_txt = await count_texts_offloaded(owner, "gemini-pro", ["hello", "world"])
assert isinstance(tok2, EstimatingTokenCounter)
assert n_txt > 0
async def test_count_texts_offloaded_sums_fragments(monkeypatch) -> None: # noqa: ANN001
"""The streaming rewrite sums per-fragment counts, matching the old per-part
count_text loop it replaced."""
proxy = _make_proxy()
monkeypatch.setattr(
"headroom.tokenizers.get_tokenizer", lambda *a, **k: EstimatingTokenCounter()
)
fragments = ["hello", "world", "foo"]
_, total = await count_texts_offloaded(proxy, "gemini-pro", fragments)
est = EstimatingTokenCounter()
assert total == sum(est.count_text(f) for f in fragments)
assert total > 0