mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description The counter called leaked_threads_total is cumulative. It counts workers that finished after their deadline and never goes down, so it reads like a growing leak even when no thread is stuck. This renames it to timed_out_workers_total. The old key stays in /health with the same value so existing dashboards keep working. Closes #3653 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Renamed the counter to timed_out_workers_total and kept leaked_threads_total as an alias with the same value - Fixed comments that called it a gauge - Rebased onto current main, keeping the upstream half-pool quarantine threshold logic ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_proxy_compression_executor.py .................... tests/test_anthropic_pre_upstream_backpressure.py ...................... ................ 58 passed in 24.52s ``` Ruff check on the three changed files: all checks passed. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, repo venv - Exact command / steps: ran the two executor and backpressure suites above - Observed result: 58 passed, /health keeps both keys - Not tested: mypy ### Maintainer executor runtime qualification (2026-10-08) - Environment: actual Windows 11, Python 3.12.13, uvicorn/httpx over loopback TCP, the real bounded compression executor with two workers. This head contains current `main`; only the synchronization was added to the contributor's metric rename. - Exact steps: start the proxy, read `/health`, submit a real executor worker blocked on a threading Event with a 30ms timeout, read `/health` while it remains in flight, release it, and read `/health` until the worker exits. No executor or HTTP route was mocked. - Observed: cumulative/in-flight counts were **(0,0) before**, **(0,1) while timed out but still running**, and **(1,0) after exit**. A later read remained (1,0), and `leaked_threads_total` matched `timed_out_workers_total` throughout. This confirms the new field is cumulative, while `timed_out_workers` measures live timeout debt. **58 executor/backpressure tests passed** on the synchronized tree. - Scope: the probe drives an actual worker through the proxy's executor entrypoint and observes its published runtime metric over HTTP; no provider inference is needed for a metric rename. Hosted checks at this new head are under maintainer qualification; PR approval remains pending. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: no, old key is kept - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Maintainer edits are enabled on my fork. Please let me know if you prefer a different name for the metric. Thanks. ### Maintainer current-main requalification Exact head `c825041c0fe5065451b01d7c6d618bb4f0a228b5`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head758f75453and current discussion, including author verification of branch history and the compatibility alias. Clean integration with upstream/main2d3ee51846. The cumulative late-finishing-worker counter remains distinct from the live timeout-debt gauge, and leaked_threads_total still aliases timed_out_workers_total. No remaining PR-specific source blocker found in scope. All 136 executor/backpressure/release/version/MCP tests pass using the existing native extension for Python startup; Ruff check/format and whitespace pass. This is not a fresh native build. Earlier dependency audit findings were against dependencies inherited from main, so they are not contributor-code blockers. Maintainers handle fresh exact-head qualification and separate inherited failures from PR regressions. No contributor metric behavior rewritten; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
622 lines
23 KiB
Python
622 lines
23 KiB
Python
"""Audit follow-up C3: bounded compression executor + cancel-aware metrics.
|
|
|
|
Replaces ``asyncio.to_thread`` for ``pipeline.apply()`` calls with a dedicated
|
|
``ThreadPoolExecutor`` that's bounded by ``ProxyConfig.compression_max_workers``.
|
|
|
|
Locks the following invariants:
|
|
|
|
1. The pool exists and respects ``compression_max_workers`` (auto and explicit).
|
|
2. ``compression_in_flight`` increments while a compression is running and
|
|
decrements after it completes — under load, the high-water mark moves up
|
|
as expected.
|
|
3. When a compression call exceeds its timeout, the awaiter unblocks with
|
|
``TimeoutError`` — but the worker thread keeps running (Python cannot
|
|
preempt running CPython bytecode or in-flight Rust calls), and when the
|
|
work eventually completes, ``_compression_timed_out_workers_total``
|
|
increments.
|
|
4. Jobs that time out while still queued do not leak the running gauge.
|
|
5. ``/stats runtime.compression_executor`` surfaces the gauges + counters so
|
|
operators can see leaked-thread rate and queue pressure.
|
|
6. Once a timed-out worker is known to still be running, new compression work
|
|
raises an asyncio timeout immediately until that worker exits instead of
|
|
multiplying the timeout debt across the executor.
|
|
|
|
These tests also serve as documentation: anyone reading them sees that
|
|
"timeout fired" does not mean "compression was cancelled" — it means "we
|
|
stopped waiting; the worker is still going". A bounded pool plus the
|
|
timed-out-worker counter is how we make that visible.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import threading
|
|
import time
|
|
|
|
import pytest
|
|
|
|
pytest.importorskip("fastapi")
|
|
|
|
from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS # noqa: F401
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
from headroom.proxy.tenant_key import get_current_tenant_key, set_request_tenant_key
|
|
|
|
|
|
def _make_proxy(compression_max_workers: int | None = None):
|
|
"""Construct a HeadroomProxy with a no-op pipeline. Returns the proxy."""
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
compression_max_workers=compression_max_workers,
|
|
)
|
|
app = create_app(config)
|
|
return app.state.proxy
|
|
|
|
|
|
def test_compression_executor_default_size_matches_cpu_count() -> None:
|
|
"""When ``compression_max_workers`` is None, the resolved size should
|
|
match the host CPU count.
|
|
"""
|
|
import os
|
|
|
|
proxy = _make_proxy(compression_max_workers=None)
|
|
expected = max(1, os.cpu_count() or 1)
|
|
assert proxy.compression_max_workers == expected
|
|
assert proxy._compression_executor._max_workers == expected
|
|
|
|
|
|
def test_compression_executor_explicit_override() -> None:
|
|
"""``ProxyConfig.compression_max_workers=N`` is honored verbatim."""
|
|
proxy = _make_proxy(compression_max_workers=3)
|
|
assert proxy.compression_max_workers == 3
|
|
assert proxy._compression_executor._max_workers == 3
|
|
|
|
|
|
def test_compression_executor_minimum_one_worker() -> None:
|
|
"""A non-positive override clamps to 1 (zero workers would deadlock)."""
|
|
proxy = _make_proxy(compression_max_workers=0)
|
|
assert proxy.compression_max_workers == 1
|
|
|
|
|
|
def test_compression_executor_preserves_request_contextvars() -> None:
|
|
"""Tenant context survives the thread-pool hop used by compression."""
|
|
proxy = _make_proxy(compression_max_workers=1)
|
|
|
|
async def _drive() -> str:
|
|
set_request_tenant_key("tenant_executor")
|
|
try:
|
|
return await proxy._run_compression_in_executor(
|
|
get_current_tenant_key,
|
|
timeout=10.0,
|
|
)
|
|
finally:
|
|
set_request_tenant_key(None)
|
|
|
|
assert asyncio.run(_drive()) == "tenant_executor"
|
|
|
|
|
|
def test_in_flight_gauge_tracks_running_compressions() -> None:
|
|
"""While a compression is running, ``_compression_in_flight`` reads ≥ 1.
|
|
After it completes, it returns to 0. The high-water mark records the
|
|
peak observed.
|
|
"""
|
|
proxy = _make_proxy(compression_max_workers=4)
|
|
|
|
enter_event = threading.Event()
|
|
release_event = threading.Event()
|
|
observed: dict[str, int] = {}
|
|
|
|
def _slow_compression():
|
|
enter_event.set()
|
|
# Block until the test thread reads in_flight from the gauge.
|
|
release_event.wait(timeout=5.0)
|
|
return "done"
|
|
|
|
async def _drive():
|
|
task = asyncio.create_task(
|
|
proxy._run_compression_in_executor(_slow_compression, timeout=10.0)
|
|
)
|
|
# Wait for the worker to actually start.
|
|
for _ in range(50):
|
|
if enter_event.is_set():
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
with proxy._compression_metrics_lock:
|
|
observed["mid_flight"] = proxy._compression_in_flight
|
|
observed["mid_flight_max"] = proxy._compression_in_flight_max
|
|
release_event.set()
|
|
result = await task
|
|
return result
|
|
|
|
result = asyncio.run(_drive())
|
|
assert result == "done"
|
|
assert observed["mid_flight"] == 1, (
|
|
f"in_flight should be 1 mid-call, got {observed['mid_flight']}"
|
|
)
|
|
assert observed["mid_flight_max"] >= 1
|
|
# Decremented after task completes.
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_in_flight == 0
|
|
|
|
|
|
def test_contextvars_set_before_submission_are_visible_inside_the_worker() -> None:
|
|
"""`run_in_executor` doesn't copy contextvars into the worker thread by
|
|
default (unlike `asyncio.to_thread`) -- without the fix, code reading
|
|
`get_registered_cwd()` there would silently see the default, not what
|
|
request middleware bound. Locks the fix with a generic ContextVar and
|
|
the real one astgrep.py's disk-verify depends on."""
|
|
import contextvars
|
|
|
|
from headroom.proxy.project_context import get_registered_cwd, set_registered_cwd
|
|
|
|
proxy = _make_proxy(compression_max_workers=2)
|
|
probe: contextvars.ContextVar[str | None] = contextvars.ContextVar("test_probe", default=None)
|
|
observed: dict[str, object] = {}
|
|
|
|
def _read_contextvars_on_worker_thread():
|
|
observed["probe"] = probe.get()
|
|
observed["registered_cwd"] = get_registered_cwd()
|
|
return "done"
|
|
|
|
async def _drive():
|
|
probe.set("set-on-calling-coroutine")
|
|
set_registered_cwd("/some/registered/project")
|
|
return await proxy._run_compression_in_executor(
|
|
_read_contextvars_on_worker_thread, timeout=5.0
|
|
)
|
|
|
|
result = asyncio.run(_drive())
|
|
assert result == "done"
|
|
assert observed["probe"] == "set-on-calling-coroutine"
|
|
assert observed["registered_cwd"] == "/some/registered/project"
|
|
|
|
|
|
def test_contextvars_propagate_through_background_executor_too() -> None:
|
|
"""Same fix, same bug class, but the *other* submission point --
|
|
`_run_compression_background` (no timeout, no leaked-thread accounting)
|
|
-- has its own `contextvars.copy_context()` call, not shared code with
|
|
`_run_compression_in_executor`. Covered separately since a fix at one
|
|
call site doesn't imply the other was fixed too."""
|
|
from headroom.proxy.project_context import get_registered_cwd, set_registered_cwd
|
|
|
|
proxy = _make_proxy(compression_max_workers=1)
|
|
observed: dict[str, object] = {}
|
|
|
|
def _read_registered_cwd_on_worker_thread():
|
|
observed["registered_cwd"] = get_registered_cwd()
|
|
return "done"
|
|
|
|
async def _drive():
|
|
set_registered_cwd("/some/other/registered/project")
|
|
return await proxy._run_compression_background(_read_registered_cwd_on_worker_thread)
|
|
|
|
result = asyncio.run(_drive())
|
|
assert result == "done"
|
|
assert observed["registered_cwd"] == "/some/other/registered/project"
|
|
|
|
|
|
def test_high_water_mark_persists_after_completion() -> None:
|
|
"""``_compression_in_flight_max`` is monotonic — never decreases."""
|
|
proxy = _make_proxy(compression_max_workers=8)
|
|
|
|
enter_events = [threading.Event() for _ in range(3)]
|
|
release_events = [threading.Event() for _ in range(3)]
|
|
|
|
def _make_slow(idx: int):
|
|
def _slow():
|
|
enter_events[idx].set()
|
|
release_events[idx].wait(timeout=5.0)
|
|
return idx
|
|
|
|
return _slow
|
|
|
|
async def _drive():
|
|
tasks = [
|
|
asyncio.create_task(proxy._run_compression_in_executor(_make_slow(i), timeout=10.0))
|
|
for i in range(3)
|
|
]
|
|
# Wait for all 3 to enter.
|
|
for ev in enter_events:
|
|
for _ in range(50):
|
|
if ev.is_set():
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
peak = proxy._compression_in_flight
|
|
for ev in release_events:
|
|
ev.set()
|
|
for t in tasks:
|
|
await t
|
|
return peak
|
|
|
|
peak = asyncio.run(_drive())
|
|
assert peak == 3, f"Should have observed 3 concurrent compressions, got {peak}"
|
|
# After all complete, in_flight is back to 0 but max remains 3.
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_in_flight == 0
|
|
assert proxy._compression_in_flight_max >= 3
|
|
|
|
|
|
def test_timeout_fires_and_leaked_thread_is_counted() -> None:
|
|
"""When the compression exceeds ``timeout``, the awaiter sees
|
|
``TimeoutError`` immediately. The worker keeps running; when it finishes,
|
|
``_compression_timed_out_workers_total`` increments by 1.
|
|
"""
|
|
proxy = _make_proxy(compression_max_workers=2)
|
|
finished_event = threading.Event()
|
|
timeout_seconds = 0.10
|
|
|
|
def _slow_compression():
|
|
# Sleep well past the timeout so the asyncio side cancels first.
|
|
time.sleep(timeout_seconds * 5)
|
|
finished_event.set()
|
|
return "completed-after-deadline"
|
|
|
|
async def _drive():
|
|
with pytest.raises(asyncio.TimeoutError):
|
|
await proxy._run_compression_in_executor(_slow_compression, timeout=timeout_seconds)
|
|
|
|
asyncio.run(_drive())
|
|
|
|
# Wait for the worker to actually finish (it ran past the deadline).
|
|
finished_event.wait(timeout=2.0)
|
|
# Give the worker thread a moment to update the counter under the lock.
|
|
deadline = time.monotonic() + 1.0
|
|
while time.monotonic() < deadline:
|
|
with proxy._compression_metrics_lock:
|
|
if proxy._compression_timed_out_workers_total >= 1:
|
|
break
|
|
time.sleep(0.01)
|
|
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_timed_out_workers_total >= 1, (
|
|
f"timed-out-worker count should be ≥ 1; got {proxy._compression_timed_out_workers_total}. "
|
|
f"The worker either didn't finish past the deadline, or the wrapper "
|
|
f"didn't increment the counter."
|
|
)
|
|
# In-flight gauge restored.
|
|
assert proxy._compression_in_flight == 0
|
|
|
|
|
|
def test_timeout_quarantines_new_work_until_timed_out_worker_finishes() -> None:
|
|
"""One post-timeout worker must not admit more compression work.
|
|
|
|
This is the production failure mode behind the executor cascade: the
|
|
asyncio waiter times out, but its thread continues running. Without a
|
|
quarantine, every subsequent request can occupy another worker and repeat
|
|
the same timeout until the pool is exhausted.
|
|
"""
|
|
proxy = _make_proxy(compression_max_workers=2)
|
|
first_started = threading.Event()
|
|
release_first = threading.Event()
|
|
second_started = threading.Event()
|
|
|
|
def _timed_out_compression():
|
|
first_started.set()
|
|
release_first.wait(timeout=5.0)
|
|
return "late"
|
|
|
|
def _second_compression():
|
|
second_started.set()
|
|
return "second"
|
|
|
|
async def _drive():
|
|
with pytest.raises(asyncio.TimeoutError):
|
|
await proxy._run_compression_in_executor(_timed_out_compression, timeout=0.05)
|
|
assert first_started.is_set()
|
|
|
|
bypass_started = time.monotonic()
|
|
try:
|
|
# asyncio.TimeoutError is distinct from builtin TimeoutError on
|
|
# Python 3.10. The quarantine signal must follow the former so the
|
|
# existing handler failure policy classifies it as a timeout.
|
|
with pytest.raises(asyncio.TimeoutError, match="quarantin"):
|
|
await proxy._run_compression_in_executor(_second_compression, timeout=1.0)
|
|
bypass_elapsed = time.monotonic() - bypass_started
|
|
assert bypass_elapsed < 0.2
|
|
assert not second_started.is_set()
|
|
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_timed_out_in_flight == 1
|
|
assert proxy._compression_quarantine_skips == 1
|
|
assert proxy._compression_quarantine_activations == 1
|
|
finally:
|
|
release_first.set()
|
|
|
|
for _ in range(100):
|
|
with proxy._compression_metrics_lock:
|
|
if proxy._compression_timed_out_in_flight == 0:
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_timed_out_in_flight == 0
|
|
assert proxy._compression_timed_out_workers_total == 1
|
|
|
|
# Quarantine is self-clearing: normal compression resumes after the
|
|
# timed-out worker has genuinely left the executor.
|
|
assert (
|
|
await proxy._run_compression_in_executor(_second_compression, timeout=1.0) == "second"
|
|
)
|
|
return await proxy.metrics.export()
|
|
|
|
prometheus_text = asyncio.run(_drive())
|
|
assert 'headroom_compression_quarantine_total{event="activated"} 1' in prometheus_text
|
|
assert 'headroom_compression_quarantine_total{event="skipped"} 1' in prometheus_text
|
|
|
|
|
|
def test_quarantine_waits_for_half_the_pool_to_be_stuck() -> None:
|
|
"""One straggler must not refuse compression while most workers are idle.
|
|
|
|
The quarantine exists so timed-out work cannot saturate the executor. On
|
|
an 8-worker pool, one or three stuck workers leave the pool mostly idle,
|
|
so compression keeps running; at four (half the pool) the quarantine
|
|
engages, is counted once, and ``/health`` reports it. A timeout after the
|
|
time cap released it counts as a fresh activation.
|
|
"""
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import CompressionQuarantinedError
|
|
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
compression_max_workers=8,
|
|
)
|
|
app = create_app(config)
|
|
proxy = app.state.proxy
|
|
release = threading.Event()
|
|
|
|
async def _strand(count: int) -> None:
|
|
for _ in range(count):
|
|
with pytest.raises(asyncio.TimeoutError):
|
|
await proxy._run_compression_in_executor(release.wait, timeout=0.05)
|
|
|
|
def _executor_health() -> dict:
|
|
with TestClient(app) as client:
|
|
return client.get("/health").json()["runtime"]["compression_executor"]
|
|
|
|
try:
|
|
asyncio.run(_strand(1))
|
|
assert asyncio.run(proxy._run_compression_in_executor(lambda: "ran", timeout=1.0)) == "ran"
|
|
health = _executor_health()
|
|
assert health["timed_out_workers"] == 1
|
|
assert health["quarantine_active"] is False
|
|
|
|
asyncio.run(_strand(2))
|
|
assert asyncio.run(proxy._run_compression_in_executor(lambda: "ran", timeout=1.0)) == "ran"
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_quarantine_activations == 0
|
|
assert proxy._compression_quarantine_skips == 0
|
|
assert proxy._compression_quarantine_releases == 0
|
|
|
|
asyncio.run(_strand(1))
|
|
with pytest.raises(CompressionQuarantinedError):
|
|
asyncio.run(proxy._run_compression_in_executor(lambda: "unused", timeout=1.0))
|
|
health = _executor_health()
|
|
assert health["timed_out_workers"] == 4
|
|
assert health["quarantine_active"] is True
|
|
assert health["quarantine_activations_total"] == 1
|
|
assert health["quarantine_skips_total"] == 1
|
|
|
|
proxy._compression_quarantine_deadline = time.monotonic() - 1.0
|
|
assert asyncio.run(proxy._run_compression_in_executor(lambda: "ran", timeout=1.0)) == "ran"
|
|
assert proxy._compression_quarantine_releases == 1
|
|
asyncio.run(_strand(1))
|
|
with pytest.raises(CompressionQuarantinedError):
|
|
asyncio.run(proxy._run_compression_in_executor(lambda: "unused", timeout=1.0))
|
|
assert proxy._compression_quarantine_activations == 2
|
|
finally:
|
|
release.set()
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("workers", "env", "expected"),
|
|
[(8, None, 4), (3, None, 1), (1, None, 1), (8, "1", 1), (8, "0", 4)],
|
|
)
|
|
def test_quarantine_threshold_is_half_the_pool_unless_overridden(
|
|
monkeypatch, workers: int, env: str | None, expected: int
|
|
) -> None:
|
|
if env is None:
|
|
monkeypatch.delenv("HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD", raising=False)
|
|
else:
|
|
monkeypatch.setenv("HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD", env)
|
|
proxy = _make_proxy(compression_max_workers=workers)
|
|
assert proxy._compression_quarantine_threshold == expected
|
|
|
|
|
|
def test_timeout_before_worker_start_does_not_leak_in_flight() -> None:
|
|
"""If a queued job times out before a worker starts, queued accounting
|
|
is cleaned up without touching the running gauge.
|
|
"""
|
|
proxy = _make_proxy(compression_max_workers=1)
|
|
first_started = threading.Event()
|
|
release_first = threading.Event()
|
|
second_started = threading.Event()
|
|
|
|
def _blocking_compression():
|
|
first_started.set()
|
|
release_first.wait(timeout=5.0)
|
|
return "first"
|
|
|
|
def _queued_compression():
|
|
second_started.set()
|
|
return "second"
|
|
|
|
async def _drive():
|
|
first_task = asyncio.create_task(
|
|
proxy._run_compression_in_executor(_blocking_compression, timeout=10.0)
|
|
)
|
|
for _ in range(50):
|
|
if first_started.is_set():
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
assert first_started.is_set()
|
|
|
|
with pytest.raises(asyncio.TimeoutError):
|
|
await proxy._run_compression_in_executor(_queued_compression, timeout=0.05)
|
|
|
|
with proxy._compression_metrics_lock:
|
|
mid_queued = proxy._compression_queued
|
|
mid_in_flight = proxy._compression_in_flight
|
|
queue_timeouts = proxy._compression_queue_timeouts
|
|
|
|
release_first.set()
|
|
assert await first_task == "first"
|
|
return mid_queued, mid_in_flight, queue_timeouts
|
|
|
|
mid_queued, mid_in_flight, queue_timeouts = asyncio.run(_drive())
|
|
|
|
assert not second_started.is_set()
|
|
assert mid_queued == 0
|
|
assert mid_in_flight == 1
|
|
assert queue_timeouts == 1
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_queued == 0
|
|
assert proxy._compression_in_flight == 0
|
|
assert proxy._compression_timed_out_workers_total == 0
|
|
assert proxy._compression_timed_out_in_flight == 0
|
|
assert proxy._compression_quarantine_activations == 0
|
|
assert proxy._compression_quarantine_skips == 0
|
|
|
|
|
|
def test_compression_executor_skip_signal_remains_visible() -> None:
|
|
"""A compression executor queue timeout increments visible runtime counters."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
compression_max_workers=1,
|
|
)
|
|
app = create_app(config)
|
|
proxy = app.state.proxy
|
|
|
|
with TestClient(app) as client:
|
|
baseline = client.get("/health").json()["runtime"]["compression_executor"][
|
|
"queue_timeouts_total"
|
|
]
|
|
|
|
first_started = threading.Event()
|
|
release_first = threading.Event()
|
|
|
|
def _blocking_compression():
|
|
first_started.set()
|
|
release_first.wait(timeout=5.0)
|
|
return "first"
|
|
|
|
def _queued_compression():
|
|
return "second"
|
|
|
|
async def _drive():
|
|
first_task = asyncio.create_task(
|
|
proxy._run_compression_in_executor(_blocking_compression, timeout=10.0)
|
|
)
|
|
for _ in range(50):
|
|
if first_started.is_set():
|
|
break
|
|
await asyncio.sleep(0.01)
|
|
assert first_started.is_set()
|
|
|
|
with pytest.raises(asyncio.TimeoutError):
|
|
await proxy._run_compression_in_executor(_queued_compression, timeout=0.05)
|
|
|
|
with proxy._compression_metrics_lock:
|
|
assert proxy._compression_queued == 0
|
|
|
|
release_first.set()
|
|
return await first_task
|
|
|
|
asyncio.run(_drive())
|
|
|
|
with TestClient(app) as client:
|
|
after = client.get("/health").json()["runtime"]["compression_executor"]
|
|
assert after["queue_timeouts_total"] == baseline + 1
|
|
|
|
|
|
def test_compression_executor_metrics_appear_in_runtime_payload() -> None:
|
|
"""``/stats runtime.compression_executor`` surfaces the new gauges."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
compression_max_workers=5,
|
|
)
|
|
app = create_app(config)
|
|
|
|
with TestClient(app) as client:
|
|
# The compression_executor metrics are published from the runtime
|
|
# payload (also surfaced in /health). Hit /health and look there.
|
|
r = client.get("/health")
|
|
assert r.status_code == 200
|
|
runtime = r.json()["runtime"]
|
|
assert "compression_executor" in runtime
|
|
ce = runtime["compression_executor"]
|
|
assert ce["max_workers"] == 5
|
|
assert ce["queued"] == 0
|
|
assert ce["running"] == 0
|
|
assert ce["in_flight"] == 0
|
|
assert ce["queue_timeouts_total"] == 0
|
|
assert ce["queue_wait_seconds_total"] == 0.0
|
|
assert ce["run_seconds_total"] == 0.0
|
|
assert ce["leaked_threads_total"] == 0
|
|
# Deprecated alias must track the canonical counter (#3653).
|
|
assert ce["timed_out_workers_total"] == 0
|
|
assert ce["timed_out_workers_total"] == ce["leaked_threads_total"]
|
|
assert ce["quarantine_active"] is False
|
|
assert ce["timed_out_workers"] == 0
|
|
assert ce["timed_out_workers_max"] == 0
|
|
assert ce["quarantine_activations_total"] == 0
|
|
assert ce["quarantine_skips_total"] == 0
|
|
assert ce["source"] == "explicit"
|
|
|
|
|
|
def test_explicit_None_resolves_to_auto_source() -> None:
|
|
"""When max_workers is None (default), the runtime payload reports
|
|
``source: auto``."""
|
|
from fastapi.testclient import TestClient
|
|
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
ccr_inject_tool=False,
|
|
ccr_handle_responses=False,
|
|
ccr_context_tracking=False,
|
|
image_optimize=False,
|
|
)
|
|
app = create_app(config)
|
|
with TestClient(app) as client:
|
|
r = client.get("/health")
|
|
assert r.json()["runtime"]["compression_executor"]["source"] == "auto"
|