fix(proxy): rename leaked_threads metric to timed_out_workers_total (#3908)

## Description

The counter called leaked_threads_total is cumulative. It counts workers
that finished after their deadline and never goes down, so it reads like
a growing leak even when no thread is stuck. This renames it to
timed_out_workers_total. The old key stays in /health with the same
value so existing dashboards keep working.

Closes #3653

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Renamed the counter to timed_out_workers_total and kept
leaked_threads_total as an alias with the same value
- Fixed comments that called it a gauge
- Rebased onto current main, keeping the upstream half-pool quarantine
threshold logic

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
tests/test_proxy_compression_executor.py ....................
tests/test_anthropic_pre_upstream_backpressure.py ......................
................
58 passed in 24.52s
```

Ruff check on the three changed files: all checks passed.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, repo venv
- Exact command / steps: ran the two executor and backpressure suites
above
- Observed result: 58 passed, /health keeps both keys
- Not tested: mypy

### Maintainer executor runtime qualification (2026-10-08)

- Environment: actual Windows 11, Python 3.12.13, uvicorn/httpx over
loopback TCP, the real bounded compression executor with two workers.
This head contains current `main`; only the synchronization was added to
the contributor's metric rename.
- Exact steps: start the proxy, read `/health`, submit a real executor
worker blocked on a threading Event with a 30ms timeout, read `/health`
while it remains in flight, release it, and read `/health` until the
worker exits. No executor or HTTP route was mocked.
- Observed: cumulative/in-flight counts were **(0,0) before**, **(0,1)
while timed out but still running**, and **(1,0) after exit**. A later
read remained (1,0), and `leaked_threads_total` matched
`timed_out_workers_total` throughout. This confirms the new field is
cumulative, while `timed_out_workers` measures live timeout debt. **58
executor/backpressure tests passed** on the synchronized tree.
- Scope: the probe drives an actual worker through the proxy's executor
entrypoint and observes its published runtime metric over HTTP; no
provider inference is needed for a metric rename. Hosted checks at this
new head are under maintainer qualification; PR approval remains
pending.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: no, old key is kept
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Maintainer edits are enabled on my fork. Please let me know if you
prefer a different name for the metric. Thanks.

### Maintainer current-main requalification

Exact head `c825041c0fe5065451b01d7c6d618bb4f0a228b5`, upstream main
`2d3ee51846435ef5b320eea4806af14e602b0d8c`.

Re-reviewed latest integration head 758f75453 and current discussion,
including author verification of branch history and the compatibility
alias. Clean integration with upstream/main
2d3ee51846. The cumulative
late-finishing-worker counter remains distinct from the live
timeout-debt gauge, and leaked_threads_total still aliases
timed_out_workers_total. No remaining PR-specific source blocker found
in scope.

All 136 executor/backpressure/release/version/MCP tests pass using the
existing native extension for Python startup; Ruff check/format and
whitespace pass. This is not a fresh native build. Earlier dependency
audit findings were against dependencies inherited from main, so they
are not contributor-code blockers. Maintainers handle fresh exact-head
qualification and separate inherited failures from PR regressions. No
contributor metric behavior rewritten; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Omar
2026-11-05 20:17:17 -06:00
committed by GitHub
co-authored by JerrettDavis
parent c05466a469
commit bd164a2c3b
3 changed files with 35 additions and 22 deletions
+21 -12
View File
@@ -1314,9 +1314,10 @@ class HeadroomProxy(
# sharing the loop's default executor (file IO, etc.).
# 2. Tasks that exceed ``COMPRESSION_TIMEOUT_SECONDS`` and complete
# *after* the asyncio future was cancelled are counted in the
# ``compression_leaked_threads`` gauge — Python cannot preempt
# the worker, so this is the only signal that some pool slots
# are sitting on stuck work.
# ``compression_timed_out_workers_total`` counter — Python cannot
# preempt the worker, so this is the only signal that some pool
# slots sat on work past its deadline. Cumulative: it counts
# workers that finished late, not threads still stuck (#3653).
_compression_max_cfg = config.compression_max_workers
if _compression_max_cfg is None:
_compression_max = max(1, os.cpu_count() or 1)
@@ -1362,9 +1363,12 @@ class HeadroomProxy(
self._compression_in_flight_max: int = 0
self._compression_run_seconds_total: float = 0.0
self._compression_run_seconds_max: float = 0.0
# Counter: threads that finished AFTER their asyncio future hit the
# timeout. Stuck-thread leak indicator.
self._compression_leaked_threads: int = 0
# Counter: workers that finished AFTER their asyncio future hit the
# timeout. Cumulative total, never decremented: a late-finishing
# worker is counted once when it exits. It does NOT report live
# stuck threads — a steady value means past timeouts, not a
# growing leak (#3653).
self._compression_timed_out_workers_total: int = 0
# Timeout-debt quarantine. Python cannot preempt a worker after its
# asyncio waiter times out, so accepting more compression while those
# workers are still running can multiply slow calls into a saturated
@@ -1619,7 +1623,7 @@ class HeadroomProxy(
preempt running CPython bytecode or in-flight Rust calls. The
worker keeps running to completion, ignored. We detect this by
marking the call timed out on the asyncio side and incrementing
``_compression_leaked_threads`` from the worker's ``finally``
``_compression_timed_out_workers_total`` from the worker's ``finally``
block after it eventually finishes. While such workers hold half the
pool (``_compression_quarantine_threshold``), new calls raise
:class:`CompressionQuarantinedError` immediately so callers apply the
@@ -1633,11 +1637,11 @@ class HeadroomProxy(
Args:
fn: A no-arg sync callable that runs the compression. Must not
raise asyncio Cancellation; if it does, the wrapper still
decrements the in-flight gauge but the leaked-thread
decrements the in-flight gauge but the timed-out-worker
counter may double-count.
timeout: Wall-clock timeout for the asyncio side. The
executor worker keeps running past this (Python limitation
— see above), but at least the awaiter unblocks.
executor worker keeps running past this (Python limitation
— see above), but at least the awaiter unblocks.
Returns:
Whatever ``fn()`` returns.
@@ -1768,7 +1772,7 @@ class HeadroomProxy(
if elapsed > self._compression_run_seconds_max:
self._compression_run_seconds_max = elapsed
if state["timed_out"]:
self._compression_leaked_threads += 1
self._compression_timed_out_workers_total += 1
if state["timeout_debt_recorded"]:
self._compression_timed_out_in_flight -= 1
state["timeout_debt_recorded"] = False
@@ -3858,7 +3862,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
_comp_in_flight_max = proxy._compression_in_flight_max
_comp_run_total = proxy._compression_run_seconds_total
_comp_run_max = proxy._compression_run_seconds_max
_comp_leaked = proxy._compression_leaked_threads
_comp_leaked = proxy._compression_timed_out_workers_total
_comp_timed_out_in_flight = proxy._compression_timed_out_in_flight
_comp_quarantine_active = (
_comp_timed_out_in_flight >= proxy._compression_quarantine_threshold
@@ -3894,6 +3898,11 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"run_seconds_total": _comp_run_total,
"run_seconds_max": _comp_run_max,
"leaked_threads_total": _comp_leaked,
# Canonical name: this is a cumulative count of workers that
# finished after their deadline, not live stuck threads.
# `leaked_threads_total` is kept as a deprecated alias so
# existing dashboards and watchdogs keep working (#3653).
"timed_out_workers_total": _comp_leaked,
"quarantine_active": _comp_quarantine_active,
"timed_out_workers": _comp_timed_out_in_flight,
"timed_out_workers_max": _comp_timed_out_in_flight_max,
@@ -201,7 +201,7 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
self.compression_max_workers = 4
self._compression_in_flight = 0
self._compression_in_flight_max = 0
self._compression_leaked_threads = 0
self._compression_timed_out_workers_total = 0
self._compression_metrics_lock = _threading.Lock()
self._upstream_delay_s = upstream_delay_s
self._raise_during_critical = raise_during_critical
@@ -234,7 +234,7 @@ class _DummyAnthropicHandler(AnthropicHandlerMixin):
with self._compression_metrics_lock:
self._compression_in_flight -= 1
if elapsed > timeout:
self._compression_leaked_threads += 1
self._compression_timed_out_workers_total += 1
future = loop.run_in_executor(self._compression_executor, _wrapped)
return await asyncio.wait_for(future, timeout=timeout)
+12 -8
View File
@@ -12,7 +12,8 @@ Locks the following invariants:
3. When a compression call exceeds its timeout, the awaiter unblocks with
``TimeoutError`` — but the worker thread keeps running (Python cannot
preempt running CPython bytecode or in-flight Rust calls), and when the
work eventually completes, ``compression_leaked_threads`` increments.
work eventually completes, ``_compression_timed_out_workers_total``
increments.
4. Jobs that time out while still queued do not leak the running gauge.
5. ``/stats runtime.compression_executor`` surfaces the gauges + counters so
operators can see leaked-thread rate and queue pressure.
@@ -23,7 +24,7 @@ Locks the following invariants:
These tests also serve as documentation: anyone reading them sees that
"timeout fired" does not mean "compression was cancelled" — it means "we
stopped waiting; the worker is still going". A bounded pool plus the
leaked-thread counter is how we make that visible.
timed-out-worker counter is how we make that visible.
"""
from __future__ import annotations
@@ -245,7 +246,7 @@ def test_high_water_mark_persists_after_completion() -> None:
def test_timeout_fires_and_leaked_thread_is_counted() -> None:
"""When the compression exceeds ``timeout``, the awaiter sees
``TimeoutError`` immediately. The worker keeps running; when it finishes,
``_compression_leaked_threads`` increments by 1.
``_compression_timed_out_workers_total`` increments by 1.
"""
proxy = _make_proxy(compression_max_workers=2)
finished_event = threading.Event()
@@ -269,13 +270,13 @@ def test_timeout_fires_and_leaked_thread_is_counted() -> None:
deadline = time.monotonic() + 1.0
while time.monotonic() < deadline:
with proxy._compression_metrics_lock:
if proxy._compression_leaked_threads >= 1:
if proxy._compression_timed_out_workers_total >= 1:
break
time.sleep(0.01)
with proxy._compression_metrics_lock:
assert proxy._compression_leaked_threads >= 1, (
f"leaked_threads should be ≥ 1; got {proxy._compression_leaked_threads}. "
assert proxy._compression_timed_out_workers_total >= 1, (
f"timed-out-worker count should be ≥ 1; got {proxy._compression_timed_out_workers_total}. "
f"The worker either didn't finish past the deadline, or the wrapper "
f"didn't increment the counter."
)
@@ -336,7 +337,7 @@ def test_timeout_quarantines_new_work_until_timed_out_worker_finishes() -> None:
with proxy._compression_metrics_lock:
assert proxy._compression_timed_out_in_flight == 0
assert proxy._compression_leaked_threads == 1
assert proxy._compression_timed_out_workers_total == 1
# Quarantine is self-clearing: normal compression resumes after the
# timed-out worker has genuinely left the executor.
@@ -486,7 +487,7 @@ def test_timeout_before_worker_start_does_not_leak_in_flight() -> None:
with proxy._compression_metrics_lock:
assert proxy._compression_queued == 0
assert proxy._compression_in_flight == 0
assert proxy._compression_leaked_threads == 0
assert proxy._compression_timed_out_workers_total == 0
assert proxy._compression_timed_out_in_flight == 0
assert proxy._compression_quarantine_activations == 0
assert proxy._compression_quarantine_skips == 0
@@ -587,6 +588,9 @@ def test_compression_executor_metrics_appear_in_runtime_payload() -> None:
assert ce["queue_wait_seconds_total"] == 0.0
assert ce["run_seconds_total"] == 0.0
assert ce["leaked_threads_total"] == 0
# Deprecated alias must track the canonical counter (#3653).
assert ce["timed_out_workers_total"] == 0
assert ce["timed_out_workers_total"] == ce["leaked_threads_total"]
assert ce["quarantine_active"] is False
assert ce["timed_out_workers"] == 0
assert ce["timed_out_workers_max"] == 0