mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description This removes proxy handler helpers that production code never calls, and one test that always skipped. Behaviour does not change. Refs #4136 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: removed `_compact_openai_tool_schema_value`. It had no callers anywhere. - `headroom/proxy/handlers/streaming.py`: - Removed `StreamingMixin._parse_sse_usage`, the 92-line single-chunk SSE usage parser. Production only calls `_parse_sse_usage_from_buffer`. - Removed `_sse_contains_error_event`, a 5-line wrapper over `_sse_event_outcome` that only `tests/test_route_advice.py` called. - `headroom/proxy/handlers/gemini.py`: removed `GeminiHandlerMixin._count_texts_offloaded`, which only tests called. `tests/test_tokenizer_count_offload.py:77` already asserts that no handler uses it. - Test changes: - `tests/test_proxy_cache_ttl_metrics.py`: the TTL-bucket test now drives `_parse_sse_usage_from_buffer`. - `tests/test_streaming_sse_malformed_events.py`: dropped the chunk-parser variants. The buffered-parser variants of the same cases stay. - `tests/test_route_advice.py`: carries the error-event helper locally, so its `_sse_event_outcome` coverage is unchanged. - `tests/test_tokenizer_count_offload.py`: calls `token_counting.count_texts_offloaded(proxy, ...)` directly, still on a real proxy and its worker thread. - `tests/test_codex_ws_compression_scheduler.py`: removed `test_codex_ws_emits_perf_log_with_cache_keys`, which called `pytest.skip` unconditionally ("Pending: full WS lifecycle harness"), along with the fixtures only it used. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.16.8 check headroom/proxy/handlers tests/test_codex_ws_compression_scheduler.py tests/test_tokenizer_count_offload.py tests/test_route_advice.py tests/test_streaming_sse_malformed_events.py tests/test_proxy_cache_ttl_metrics.py All checks passed! $ pytest -q tests/test_streaming_sse_malformed_events.py tests/test_proxy_cache_ttl_metrics.py tests/test_route_advice.py tests/test_tokenizer_count_offload.py tests/test_codex_ws_compression_scheduler.py tests/test_openai_responses_context_compaction.py tests/test_openai_responses_additional_tools.py tests/test_gemini_compression_offload.py tests/test_proxy_gemini_integration.py tests/test_proxy_gemini_native_integration.py 470 passed, 27 skipped in 24.13s ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, repo venv, branch `cleanup/proxy-handlers-dead-helpers` @c15e679e, temp `HOME`/`HEADROOM_WORKSPACE_DIR`/`CODEX_HOME`, `HEADROOM_OFFLINE=1` - Exact command / steps: (1) a `grep -rnE '_compact_openai_tool_schema_value|_parse_sse_usage\b|_sse_contains_error_event|_count_texts_offloaded'` over headroom, tests, crates, plugins, sdk, examples, benchmarks, scripts, e2e, docs, wiki, .github, pyproject.toml and every consumer repo clone; (2) the shared smoke harness: a real `headroom proxy` from this checkout against a local fake upstream, with one Anthropic and one OpenAI turn; (3) a direct check that the removed names are gone from the assembled `HeadroomProxy` and that the live SSE usage parser still works - Observed result: (1) before the change, only the definitions and tests matched; after it, nothing in `headroom/` references the removed names. (2) and (3) are shown below: both turns return 200 with savings, the names are absent, and the live parser returns the usage. - Not tested: a real hosted upstream ```text $ bash smoke.sh <worktree> branch: cleanup/proxy-handlers-dead-helpers @c15e679ehealth: 200 POST /v1/messages -> HTTP200 smoke-ok POST /v1/chat/completions -> HTTP200 smoke-ok stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82 proxy.log WARNING/ERROR lines: (only the pre-existing offline-mode notice and Kompress offline-prefetch warnings) $ HOME=$T/h HEADROOM_WORKSPACE_DIR=$T/ws CODEX_HOME=$T/c python -c "<check HeadroomProxy attrs; run _parse_sse_usage_from_buffer>" _parse_sse_usage on HeadroomProxy: False _sse_contains_error_event on HeadroomProxy: False _count_texts_offloaded on HeadroomProxy: False _compact_openai_tool_schema_value in openai: False live SSE usage parser: {'input_tokens': 9, 'output_tokens': 2, 'cache_read_input_tokens': 0} ``` ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: stable - Stable/default behavior changed: no - Kill switch / disable path: not applicable (dead code removal) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Part of maintainer-requested cleanup #4136 (dead code, duplication, debuggability), split into small PRs for review. - `headroom.proxy.token_counting.count_texts_offloaded` is now called only by tests. It is outside this cleanup slice, so it is left in place and flagged for the token-counting owner. - No `headroom.proxy.handlers` symbol appears in the consumer import index. - mypy and comments: N/A, since this PR only deletes code. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
282 lines
12 KiB
Python
282 lines
12 KiB
Python
"""Token counting must run off the event loop (GH #1701): the Anthropic messages
|
|
handler resolved the tokenizer and counted the conversation inline in the async
|
|
handler. For HF-backed models (e.g. deepseek-*) first use triggers an unbounded
|
|
network download, freezing the whole server (610s request, then /livez, /readyz
|
|
and /health hang until kill). The fix routes resolution + counting through
|
|
HeadroomProxy._count_tokens_offloaded (compression executor, bounded by
|
|
COMPRESSION_TIMEOUT_SECONDS, fail-open to estimation) — shared by every provider
|
|
handler (Anthropic, OpenAI, Gemini), since the OpenAI passthrough endpoints
|
|
receive the same HF-backed models — and offloads the inline batch
|
|
pipeline.apply() calls the same way.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import inspect
|
|
import threading
|
|
import time
|
|
|
|
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
|
|
from headroom.proxy.handlers.batch import BatchHandlerMixin
|
|
from headroom.proxy.handlers.gemini import GeminiHandlerMixin
|
|
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
|
|
from headroom.proxy.server import (
|
|
CompressionQuarantinedError,
|
|
ProxyConfig,
|
|
create_app,
|
|
)
|
|
from headroom.proxy.token_counting import (
|
|
_count_offloaded,
|
|
count_texts_offloaded,
|
|
count_tokens_offloaded,
|
|
)
|
|
from headroom.tokenizers import EstimatingTokenCounter
|
|
|
|
|
|
def _make_proxy(): # noqa: ANN202 — returns the internal HeadroomProxy
|
|
app = create_app(
|
|
ProxyConfig(
|
|
optimize=True,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
)
|
|
)
|
|
return app.state.proxy
|
|
|
|
|
|
def test_handlers_offload_token_counting_and_batch_apply() -> None:
|
|
"""Wiring guard: the request paths must use the offloaded helpers, not inline
|
|
get_tokenizer/count_messages or pipeline.apply on the event loop."""
|
|
# Every provider handler that counts the original conversation must route
|
|
# resolution + counting through the shared fail-open helper, never inline on
|
|
# the loop. OpenAI /chat + /responses are multi-provider passthroughs, so an
|
|
# HF-routed model (qwen, deepseek, llama, ...) can reach them and cold-load.
|
|
for mixin, method in (
|
|
(AnthropicHandlerMixin, "handle_anthropic_messages"),
|
|
(OpenAIHandlerMixin, "handle_openai_chat"),
|
|
(OpenAIHandlerMixin, "handle_openai_responses"),
|
|
(GeminiHandlerMixin, "handle_gemini_generate_content"),
|
|
(GeminiHandlerMixin, "handle_google_cloudcode_stream"),
|
|
(GeminiHandlerMixin, "handle_gemini_count_tokens"),
|
|
):
|
|
fn = getattr(mixin, method)
|
|
assert inspect.iscoroutinefunction(fn), f"{method} must be async"
|
|
src = inspect.getsource(fn)
|
|
assert "_count_tokens_offloaded(" in src, f"{method}: token counting not offloaded"
|
|
assert "tokenizer = get_tokenizer(" not in src, (
|
|
f"{method}: tokenizer resolved inline on the loop"
|
|
)
|
|
|
|
fn = GeminiHandlerMixin.handle_gemini_stream_generate_content
|
|
assert inspect.iscoroutinefunction(fn)
|
|
src = inspect.getsource(fn)
|
|
assert "return await self.handle_gemini_generate_content(" in src
|
|
assert "upstream_base_url=upstream_base_url" in src
|
|
assert "_count_texts_offloaded(" not in src
|
|
assert "_dict_parts(" not in src
|
|
|
|
for mixin, method in (
|
|
(AnthropicHandlerMixin, "handle_anthropic_batch_create"),
|
|
(BatchHandlerMixin, "handle_google_batch_create"),
|
|
(BatchHandlerMixin, "_compress_batch_jsonl"),
|
|
):
|
|
fn = getattr(mixin, method)
|
|
assert inspect.iscoroutinefunction(fn), f"{method} must be async"
|
|
src = inspect.getsource(fn)
|
|
if "pipeline.apply(" in src:
|
|
assert "_run_compression_in_executor(" in src, f"{method}: apply() not offloaded"
|
|
assert "COMPRESSION_TIMEOUT_SECONDS" in src, f"{method}: offload missing timeout"
|
|
|
|
helper_src = inspect.getsource(_count_offloaded)
|
|
assert "COMPRESSION_TIMEOUT_SECONDS" in helper_src
|
|
assert "EstimatingTokenCounter" in helper_src, "helper must fail open to estimation"
|
|
|
|
|
|
async def test_count_tokens_offloaded_runs_on_worker_thread(monkeypatch) -> None: # noqa: ANN001
|
|
proxy = _make_proxy()
|
|
loop_thread = threading.current_thread().name
|
|
seen: dict[str, str] = {}
|
|
|
|
class _SpyTokenizer(EstimatingTokenCounter):
|
|
def count_messages(self, messages): # noqa: ANN001, ANN201
|
|
seen["thread"] = threading.current_thread().name
|
|
return super().count_messages(messages)
|
|
|
|
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda *a, **k: _SpyTokenizer())
|
|
|
|
_, tokens = await proxy._count_tokens_offloaded("gpt-4", [{"role": "user", "content": "hi"}])
|
|
|
|
assert tokens > 0
|
|
assert seen["thread"].startswith("headroom-compress")
|
|
assert seen["thread"] != loop_thread
|
|
|
|
|
|
async def test_count_tokens_offloaded_keeps_loop_responsive(monkeypatch) -> None: # noqa: ANN001
|
|
"""A slow tokenizer (stand-in for an HF network load) must not starve the loop —
|
|
the pre-fix inline call yielded ~0 ticks here."""
|
|
proxy = _make_proxy()
|
|
ticks = 0
|
|
|
|
async def _ticker() -> None:
|
|
nonlocal ticks
|
|
while True:
|
|
await asyncio.sleep(0.01)
|
|
ticks += 1
|
|
|
|
class _SlowTokenizer(EstimatingTokenCounter):
|
|
def count_messages(self, messages): # noqa: ANN001, ANN201
|
|
time.sleep(0.3)
|
|
return super().count_messages(messages)
|
|
|
|
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda *a, **k: _SlowTokenizer())
|
|
|
|
tick_task = asyncio.create_task(_ticker())
|
|
try:
|
|
_, tokens = await proxy._count_tokens_offloaded("m", [{"role": "user", "content": "hi"}])
|
|
finally:
|
|
tick_task.cancel()
|
|
|
|
assert tokens > 0
|
|
assert ticks >= 5
|
|
|
|
|
|
async def test_count_tokens_offloaded_fails_open(monkeypatch) -> None: # noqa: ANN001
|
|
"""Resolution errors and timeouts downgrade to estimation instead of raising."""
|
|
proxy = _make_proxy()
|
|
|
|
def _boom(*a, **k): # noqa: ANN002, ANN003, ANN202
|
|
raise RuntimeError("tokenizer backend exploded")
|
|
|
|
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", _boom)
|
|
|
|
tokenizer, tokens = await proxy._count_tokens_offloaded(
|
|
"deepseek-chat", [{"role": "user", "content": "hello world"}]
|
|
)
|
|
|
|
assert isinstance(tokenizer, EstimatingTokenCounter)
|
|
assert tokens > 0
|
|
# Logged-once bookkeeping records the downgraded model.
|
|
assert "deepseek-chat" in proxy._token_count_fallback_models
|
|
|
|
|
|
async def test_count_tokens_offloaded_fails_open_on_executor_quarantine() -> None:
|
|
"""Now that OpenAI/Gemini counting shares the compression executor, an
|
|
unrelated request's compression timeout can quarantine it — the next
|
|
``_run_compression_in_executor`` call raises ``CompressionQuarantinedError``
|
|
immediately (process-wide state). A request that is only counting tokens
|
|
must not 500 on that; it fails open to estimation like any other error."""
|
|
# The executor's ``except Exception`` fail-open only catches the quarantine
|
|
# error because it subclasses Exception — pin that contract.
|
|
assert issubclass(CompressionQuarantinedError, Exception)
|
|
|
|
proxy = _make_proxy()
|
|
# Record a concurrent compression as timed out so the real executor guard
|
|
# quarantines the next call — no mock of the helper itself. Since the
|
|
# quarantine became time-capped (#2412), standing debt alone no longer
|
|
# quarantines: the deadline armed by the fresh timeout must still be in
|
|
# the future, so arm it the way a real timeout would. The debt must reach
|
|
# the quarantine threshold (half the auto-sized pool).
|
|
proxy._compression_timed_out_in_flight = proxy._compression_quarantine_threshold
|
|
proxy._compression_quarantine_deadline = time.monotonic() + 60.0
|
|
|
|
tokenizer, tokens = await proxy._count_tokens_offloaded(
|
|
"qwen2.5-coder", [{"role": "user", "content": "hello world"}]
|
|
)
|
|
|
|
assert isinstance(tokenizer, EstimatingTokenCounter)
|
|
assert tokens > 0
|
|
assert "qwen2.5-coder" in proxy._token_count_fallback_models
|
|
|
|
|
|
async def test_count_tokens_offloaded_returns_count_text_capable_tokenizer() -> None:
|
|
"""The fail-open tokenizer should still support text counting for callers
|
|
that need per-fragment accounting."""
|
|
proxy = _make_proxy()
|
|
# Quarantine forces the fail-open branch (an EstimatingTokenCounter).
|
|
# Post-#2412 the quarantine is time-capped, so the deadline must be armed
|
|
# alongside the standing debt, which must reach the quarantine threshold.
|
|
proxy._compression_timed_out_in_flight = proxy._compression_quarantine_threshold
|
|
proxy._compression_quarantine_deadline = time.monotonic() + 60.0
|
|
|
|
# The empty-messages count is intentionally discarded by that handler
|
|
# (it sums text parts itself), so only the tokenizer matters here.
|
|
tokenizer, _ = await proxy._count_tokens_offloaded("qwen2.5-coder", [])
|
|
|
|
assert isinstance(tokenizer, EstimatingTokenCounter)
|
|
# The streaming handler's per-part loop must not raise on the fallback.
|
|
assert tokenizer.count_text("hello world") > 0
|
|
|
|
|
|
async def test_count_texts_offloaded_runs_on_worker_thread(monkeypatch) -> None: # noqa: ANN001
|
|
proxy = _make_proxy()
|
|
loop_thread = threading.current_thread().name
|
|
seen: dict[str, str] = {}
|
|
|
|
class _SpyTokenizer(EstimatingTokenCounter):
|
|
def count_text(self, text): # noqa: ANN001, ANN201
|
|
seen["thread"] = threading.current_thread().name
|
|
return super().count_text(text)
|
|
|
|
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", lambda *a, **k: _SpyTokenizer())
|
|
|
|
_, tokens = await count_texts_offloaded(proxy, "gemini-pro", ["hello", "world"])
|
|
|
|
assert tokens > 0
|
|
assert seen["thread"].startswith("headroom-compress")
|
|
assert seen["thread"] != loop_thread
|
|
|
|
|
|
async def test_count_texts_offloaded_fails_open(monkeypatch) -> None: # noqa: ANN001
|
|
"""The texts variant downgrades to estimation on a resolution error, the same
|
|
as the messages variant (its fail-open branch was previously uncovered)."""
|
|
proxy = _make_proxy()
|
|
|
|
def _boom(*a, **k): # noqa: ANN002, ANN003, ANN202
|
|
raise RuntimeError("tokenizer backend exploded")
|
|
|
|
monkeypatch.setattr("headroom.tokenizers.get_tokenizer", _boom)
|
|
|
|
tokenizer, tokens = await count_texts_offloaded(proxy, "deepseek-chat", ["hello", "world"])
|
|
|
|
assert isinstance(tokenizer, EstimatingTokenCounter)
|
|
assert tokens > 0
|
|
assert "deepseek-chat" in proxy._token_count_fallback_models
|
|
|
|
|
|
async def test_count_offloaded_without_executor_estimates() -> None:
|
|
"""An owner with no compression executor (a lightweight caller or test double)
|
|
fails open to estimation inline instead of crashing on the missing runner."""
|
|
|
|
class _NoExecutorOwner:
|
|
pass
|
|
|
|
owner = _NoExecutorOwner()
|
|
|
|
tok, n_msg = await count_tokens_offloaded(
|
|
owner, "gpt-4", [{"role": "user", "content": "hello world"}]
|
|
)
|
|
assert isinstance(tok, EstimatingTokenCounter)
|
|
assert n_msg > 0
|
|
|
|
tok2, n_txt = await count_texts_offloaded(owner, "gemini-pro", ["hello", "world"])
|
|
assert isinstance(tok2, EstimatingTokenCounter)
|
|
assert n_txt > 0
|
|
|
|
|
|
async def test_count_texts_offloaded_sums_fragments(monkeypatch) -> None: # noqa: ANN001
|
|
"""The streaming rewrite sums per-fragment counts, matching the old per-part
|
|
count_text loop it replaced."""
|
|
proxy = _make_proxy()
|
|
monkeypatch.setattr(
|
|
"headroom.tokenizers.get_tokenizer", lambda *a, **k: EstimatingTokenCounter()
|
|
)
|
|
fragments = ["hello", "world", "foo"]
|
|
|
|
_, total = await count_texts_offloaded(proxy, "gemini-pro", fragments)
|
|
|
|
est = EstimatingTokenCounter()
|
|
assert total == sum(est.count_text(f) for f in fragments)
|
|
assert total > 0
|