mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): rename leaked_threads metric to timed_out_workers_total (#3908)
## Description The counter called leaked_threads_total is cumulative. It counts workers that finished after their deadline and never goes down, so it reads like a growing leak even when no thread is stuck. This renames it to timed_out_workers_total. The old key stays in /health with the same value so existing dashboards keep working. Closes #3653 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Renamed the counter to timed_out_workers_total and kept leaked_threads_total as an alias with the same value - Fixed comments that called it a gauge - Rebased onto current main, keeping the upstream half-pool quarantine threshold logic ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_proxy_compression_executor.py .................... tests/test_anthropic_pre_upstream_backpressure.py ...................... ................ 58 passed in 24.52s ``` Ruff check on the three changed files: all checks passed. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, repo venv - Exact command / steps: ran the two executor and backpressure suites above - Observed result: 58 passed, /health keeps both keys - Not tested: mypy ### Maintainer executor runtime qualification (2026-10-08) - Environment: actual Windows 11, Python 3.12.13, uvicorn/httpx over loopback TCP, the real bounded compression executor with two workers. This head contains current `main`; only the synchronization was added to the contributor's metric rename. - Exact steps: start the proxy, read `/health`, submit a real executor worker blocked on a threading Event with a 30ms timeout, read `/health` while it remains in flight, release it, and read `/health` until the worker exits. No executor or HTTP route was mocked. - Observed: cumulative/in-flight counts were **(0,0) before**, **(0,1) while timed out but still running**, and **(1,0) after exit**. A later read remained (1,0), and `leaked_threads_total` matched `timed_out_workers_total` throughout. This confirms the new field is cumulative, while `timed_out_workers` measures live timeout debt. **58 executor/backpressure tests passed** on the synchronized tree. - Scope: the probe drives an actual worker through the proxy's executor entrypoint and observes its published runtime metric over HTTP; no provider inference is needed for a metric rename. Hosted checks at this new head are under maintainer qualification; PR approval remains pending. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: N/A - Stable/default behavior changed: no, old key is kept - Kill switch / disable path: N/A - Unsafe override required: no - Qualification impact: none - Rollback path: revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Maintainer edits are enabled on my fork. Please let me know if you prefer a different name for the metric. Thanks. ### Maintainer current-main requalification Exact head `c825041c0fe5065451b01d7c6d618bb4f0a228b5`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head758f75453and current discussion, including author verification of branch history and the compatibility alias. Clean integration with upstream/main2d3ee51846. The cumulative late-finishing-worker counter remains distinct from the live timeout-debt gauge, and leaked_threads_total still aliases timed_out_workers_total. No remaining PR-specific source blocker found in scope. All 136 executor/backpressure/release/version/MCP tests pass using the existing native extension for Python startup; Ruff check/format and whitespace pass. This is not a fresh native build. Earlier dependency audit findings were against dependencies inherited from main, so they are not contributor-code blockers. Maintainers handle fresh exact-head qualification and separate inherited failures from PR regressions. No contributor metric behavior rewritten; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
+21
-12
@@ -1314,9 +1314,10 @@ class HeadroomProxy(
|
||||
# sharing the loop's default executor (file IO, etc.).
|
||||
# 2. Tasks that exceed ``COMPRESSION_TIMEOUT_SECONDS`` and complete
|
||||
# *after* the asyncio future was cancelled are counted in the
|
||||
# ``compression_leaked_threads`` gauge — Python cannot preempt
|
||||
# the worker, so this is the only signal that some pool slots
|
||||
# are sitting on stuck work.
|
||||
# ``compression_timed_out_workers_total`` counter — Python cannot
|
||||
# preempt the worker, so this is the only signal that some pool
|
||||
# slots sat on work past its deadline. Cumulative: it counts
|
||||
# workers that finished late, not threads still stuck (#3653).
|
||||
_compression_max_cfg = config.compression_max_workers
|
||||
if _compression_max_cfg is None:
|
||||
_compression_max = max(1, os.cpu_count() or 1)
|
||||
@@ -1362,9 +1363,12 @@ class HeadroomProxy(
|
||||
self._compression_in_flight_max: int = 0
|
||||
self._compression_run_seconds_total: float = 0.0
|
||||
self._compression_run_seconds_max: float = 0.0
|
||||
# Counter: threads that finished AFTER their asyncio future hit the
|
||||
# timeout. Stuck-thread leak indicator.
|
||||
self._compression_leaked_threads: int = 0
|
||||
# Counter: workers that finished AFTER their asyncio future hit the
|
||||
# timeout. Cumulative total, never decremented: a late-finishing
|
||||
# worker is counted once when it exits. It does NOT report live
|
||||
# stuck threads — a steady value means past timeouts, not a
|
||||
# growing leak (#3653).
|
||||
self._compression_timed_out_workers_total: int = 0
|
||||
# Timeout-debt quarantine. Python cannot preempt a worker after its
|
||||
# asyncio waiter times out, so accepting more compression while those
|
||||
# workers are still running can multiply slow calls into a saturated
|
||||
@@ -1619,7 +1623,7 @@ class HeadroomProxy(
|
||||
preempt running CPython bytecode or in-flight Rust calls. The
|
||||
worker keeps running to completion, ignored. We detect this by
|
||||
marking the call timed out on the asyncio side and incrementing
|
||||
``_compression_leaked_threads`` from the worker's ``finally``
|
||||
``_compression_timed_out_workers_total`` from the worker's ``finally``
|
||||
block after it eventually finishes. While such workers hold half the
|
||||
pool (``_compression_quarantine_threshold``), new calls raise
|
||||
:class:`CompressionQuarantinedError` immediately so callers apply the
|
||||
@@ -1633,11 +1637,11 @@ class HeadroomProxy(
|
||||
Args:
|
||||
fn: A no-arg sync callable that runs the compression. Must not
|
||||
raise asyncio Cancellation; if it does, the wrapper still
|
||||
decrements the in-flight gauge but the leaked-thread
|
||||
decrements the in-flight gauge but the timed-out-worker
|
||||
counter may double-count.
|
||||
timeout: Wall-clock timeout for the asyncio side. The
|
||||
executor worker keeps running past this (Python limitation
|
||||
— see above), but at least the awaiter unblocks.
|
||||
executor worker keeps running past this (Python limitation
|
||||
— see above), but at least the awaiter unblocks.
|
||||
|
||||
Returns:
|
||||
Whatever ``fn()`` returns.
|
||||
@@ -1768,7 +1772,7 @@ class HeadroomProxy(
|
||||
if elapsed > self._compression_run_seconds_max:
|
||||
self._compression_run_seconds_max = elapsed
|
||||
if state["timed_out"]:
|
||||
self._compression_leaked_threads += 1
|
||||
self._compression_timed_out_workers_total += 1
|
||||
if state["timeout_debt_recorded"]:
|
||||
self._compression_timed_out_in_flight -= 1
|
||||
state["timeout_debt_recorded"] = False
|
||||
@@ -3858,7 +3862,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
_comp_in_flight_max = proxy._compression_in_flight_max
|
||||
_comp_run_total = proxy._compression_run_seconds_total
|
||||
_comp_run_max = proxy._compression_run_seconds_max
|
||||
_comp_leaked = proxy._compression_leaked_threads
|
||||
_comp_leaked = proxy._compression_timed_out_workers_total
|
||||
_comp_timed_out_in_flight = proxy._compression_timed_out_in_flight
|
||||
_comp_quarantine_active = (
|
||||
_comp_timed_out_in_flight >= proxy._compression_quarantine_threshold
|
||||
@@ -3894,6 +3898,11 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
"run_seconds_total": _comp_run_total,
|
||||
"run_seconds_max": _comp_run_max,
|
||||
"leaked_threads_total": _comp_leaked,
|
||||
# Canonical name: this is a cumulative count of workers that
|
||||
# finished after their deadline, not live stuck threads.
|
||||
# `leaked_threads_total` is kept as a deprecated alias so
|
||||
# existing dashboards and watchdogs keep working (#3653).
|
||||
"timed_out_workers_total": _comp_leaked,
|
||||
"quarantine_active": _comp_quarantine_active,
|
||||
"timed_out_workers": _comp_timed_out_in_flight,
|
||||
"timed_out_workers_max": _comp_timed_out_in_flight_max,
|
||||
|
||||
@@ -201,7 +201,7 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
self.compression_max_workers = 4
|
||||
self._compression_in_flight = 0
|
||||
self._compression_in_flight_max = 0
|
||||
self._compression_leaked_threads = 0
|
||||
self._compression_timed_out_workers_total = 0
|
||||
self._compression_metrics_lock = _threading.Lock()
|
||||
self._upstream_delay_s = upstream_delay_s
|
||||
self._raise_during_critical = raise_during_critical
|
||||
@@ -234,7 +234,7 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
|
||||
with self._compression_metrics_lock:
|
||||
self._compression_in_flight -= 1
|
||||
if elapsed > timeout:
|
||||
self._compression_leaked_threads += 1
|
||||
self._compression_timed_out_workers_total += 1
|
||||
|
||||
future = loop.run_in_executor(self._compression_executor, _wrapped)
|
||||
return await asyncio.wait_for(future, timeout=timeout)
|
||||
|
||||
@@ -12,7 +12,8 @@ Locks the following invariants:
|
||||
3. When a compression call exceeds its timeout, the awaiter unblocks with
|
||||
``TimeoutError`` — but the worker thread keeps running (Python cannot
|
||||
preempt running CPython bytecode or in-flight Rust calls), and when the
|
||||
work eventually completes, ``compression_leaked_threads`` increments.
|
||||
work eventually completes, ``_compression_timed_out_workers_total``
|
||||
increments.
|
||||
4. Jobs that time out while still queued do not leak the running gauge.
|
||||
5. ``/stats runtime.compression_executor`` surfaces the gauges + counters so
|
||||
operators can see leaked-thread rate and queue pressure.
|
||||
@@ -23,7 +24,7 @@ Locks the following invariants:
|
||||
These tests also serve as documentation: anyone reading them sees that
|
||||
"timeout fired" does not mean "compression was cancelled" — it means "we
|
||||
stopped waiting; the worker is still going". A bounded pool plus the
|
||||
leaked-thread counter is how we make that visible.
|
||||
timed-out-worker counter is how we make that visible.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -245,7 +246,7 @@ def test_high_water_mark_persists_after_completion() -> None:
|
||||
def test_timeout_fires_and_leaked_thread_is_counted() -> None:
|
||||
"""When the compression exceeds ``timeout``, the awaiter sees
|
||||
``TimeoutError`` immediately. The worker keeps running; when it finishes,
|
||||
``_compression_leaked_threads`` increments by 1.
|
||||
``_compression_timed_out_workers_total`` increments by 1.
|
||||
"""
|
||||
proxy = _make_proxy(compression_max_workers=2)
|
||||
finished_event = threading.Event()
|
||||
@@ -269,13 +270,13 @@ def test_timeout_fires_and_leaked_thread_is_counted() -> None:
|
||||
deadline = time.monotonic() + 1.0
|
||||
while time.monotonic() < deadline:
|
||||
with proxy._compression_metrics_lock:
|
||||
if proxy._compression_leaked_threads >= 1:
|
||||
if proxy._compression_timed_out_workers_total >= 1:
|
||||
break
|
||||
time.sleep(0.01)
|
||||
|
||||
with proxy._compression_metrics_lock:
|
||||
assert proxy._compression_leaked_threads >= 1, (
|
||||
f"leaked_threads should be ≥ 1; got {proxy._compression_leaked_threads}. "
|
||||
assert proxy._compression_timed_out_workers_total >= 1, (
|
||||
f"timed-out-worker count should be ≥ 1; got {proxy._compression_timed_out_workers_total}. "
|
||||
f"The worker either didn't finish past the deadline, or the wrapper "
|
||||
f"didn't increment the counter."
|
||||
)
|
||||
@@ -336,7 +337,7 @@ def test_timeout_quarantines_new_work_until_timed_out_worker_finishes() -> None:
|
||||
|
||||
with proxy._compression_metrics_lock:
|
||||
assert proxy._compression_timed_out_in_flight == 0
|
||||
assert proxy._compression_leaked_threads == 1
|
||||
assert proxy._compression_timed_out_workers_total == 1
|
||||
|
||||
# Quarantine is self-clearing: normal compression resumes after the
|
||||
# timed-out worker has genuinely left the executor.
|
||||
@@ -486,7 +487,7 @@ def test_timeout_before_worker_start_does_not_leak_in_flight() -> None:
|
||||
with proxy._compression_metrics_lock:
|
||||
assert proxy._compression_queued == 0
|
||||
assert proxy._compression_in_flight == 0
|
||||
assert proxy._compression_leaked_threads == 0
|
||||
assert proxy._compression_timed_out_workers_total == 0
|
||||
assert proxy._compression_timed_out_in_flight == 0
|
||||
assert proxy._compression_quarantine_activations == 0
|
||||
assert proxy._compression_quarantine_skips == 0
|
||||
@@ -587,6 +588,9 @@ def test_compression_executor_metrics_appear_in_runtime_payload() -> None:
|
||||
assert ce["queue_wait_seconds_total"] == 0.0
|
||||
assert ce["run_seconds_total"] == 0.0
|
||||
assert ce["leaked_threads_total"] == 0
|
||||
# Deprecated alias must track the canonical counter (#3653).
|
||||
assert ce["timed_out_workers_total"] == 0
|
||||
assert ce["timed_out_workers_total"] == ce["leaked_threads_total"]
|
||||
assert ce["quarantine_active"] is False
|
||||
assert ce["timed_out_workers"] == 0
|
||||
assert ce["timed_out_workers_max"] == 0
|
||||
|
||||
Reference in New Issue
Block a user