mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): quarantine compression only once half the pool is stuck (#3932)
## Description The timeout-debt quarantine in `HeadroomProxy._run_compression_in_executor` refused every new compression call while even one timed-out worker was still running. On an auto-sized pool (one worker per CPU, 8 on a typical laptop) a single straggler forwarded every other request uncompressed for up to the 60s cap while 7 of 8 workers sat idle. A common trigger is the cold-start fast pass missing its deliberately short 10s fail-open budget. The quarantine exists so timed-out work cannot saturate the executor (#2292, #2360). That risk starts near saturation, not at the first straggler, so the check now engages only once timed-out workers hold half the pool (`max(1, max_workers // 2)`). Pools of 1-3 workers keep today's behaviour. Closes #3931 Related, not closed by this PR: #3711 (what makes one Kompress stage overrun its budget; this PR narrows how far one overrun spreads, not the overrun itself) and #3653 (thread/fd reclamation for stuck workers; see Additional Notes for why this change does not add to it). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: new `_compression_quarantine_threshold`, defaulting to `max(1, compression_max_workers // 2)` and overridable with `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD` (read with the existing `_get_env_int`, `min_value=1`; `=1` restores the old any-straggler behaviour). - The quarantine check and its 60s time-cap release both key on `timed_out_in_flight >= threshold` instead of `> 0`. The cap is still re-armed on every fresh timeout, as before. - `quarantine_activations_total` (and the `activated` Prometheus event and warning) now fire on the transition into the quarantined state, i.e. when debt reaches the threshold, or when a fresh timeout re-arms it after the time cap released it. Previously a re-arm after a release while debt still stood was not counted. - "Quarantine cleared" is logged when debt drops back below the threshold, with the number of timed-out workers still running. - `/health` `runtime.compression_executor.quarantine_active` now reports the condition the check enforces (debt at or above the threshold and the cap not lapsed). It used to report `timed_out_workers > 0`, which was already wrong after a time-cap release. - Tests: `test_quarantine_waits_for_half_the_pool_to_be_stuck` (8-worker pool: 1 and 3 stuck still compress, 4 stuck refuses, `/health` matches, re-arm after release counts as a second activation) and `test_quarantine_threshold_is_half_the_pool_unless_overridden` (default sizing and the env override). Two tests in `tests/test_tokenizer_count_offload.py` that forced a quarantine with a debt of 1 on an auto-sized pool now set the debt to the threshold. ## Testing - [x] Unit tests pass (`pytest`) - targeted files only, listed below; the full suite was not run - [x] Linting passes (`ruff check .`) - on the changed files, plus `ruff format --check` - [x] Type checking passes (`mypy headroom`) - `mypy headroom/proxy/server.py` - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Before the fix (new tests against unmodified `server.py`; the two tokenizer tests fail only because they read the new attribute): ```text $ python -m pytest tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py -q -p no:cacheprovider FAILED tests/test_proxy_compression_executor.py::test_quarantine_waits_for_half_the_pool_to_be_stuck FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-None-4] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[3-None-1] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[1-None-1] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-1-1] FAILED tests/test_proxy_compression_executor.py::test_quarantine_threshold_is_half_the_pool_unless_overridden[8-0-4] FAILED tests/test_tokenizer_count_offload.py::test_count_tokens_offloaded_fails_open_on_executor_quarantine FAILED tests/test_tokenizer_count_offload.py::test_count_tokens_offloaded_returns_count_text_capable_tokenizer =================== 8 failed, 20 passed, 1 warning in 20.73s =================== ``` The behavioural failure on main is: ```text tests/test_proxy_compression_executor.py:336: in test_quarantine_waits_for_half_the_pool_to_be_stuck E headroom.proxy.server.CompressionQuarantinedError: compression quarantined: 1 timed-out worker(s) still running ``` After the fix: ```text $ python -m pytest tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py tests/test_platform_stabilization_functional.py tests/test_prometheus_obs_counters.py -q -p no:cacheprovider ======================== 51 passed, 1 warning in 28.02s ======================== $ ruff check headroom/proxy/server.py tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py All checks passed! $ ruff format --check headroom/proxy/server.py tests/test_proxy_compression_executor.py tests/test_tokenizer_count_offload.py 3 files already formatted $ mypy headroom/proxy/server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.11, upstream main `d5318ac2` (before) and the same commit plus this branch (after). Proxy booted through `headroom.proxy.server.run_server(ProxyConfig(port=..., compression_max_workers=8, disable_kompress=True, cache_enabled=False, rate_limit_enabled=False, log_requests=False))`, which is what `headroom proxy` calls after argument parsing (the local venv lacked the `mcp`/`magika`/`onnxruntime` extras that the CLI's dependency preflight requires). Env: `HEADROOM_SKIP_UPSTREAM_CHECK=1 HEADROOM_COMPRESSION_TIMEOUT_SECONDS=3 HEADROOM_TELEMETRY=off`, a fresh `HEADROOM_WORKSPACE_DIR` per run. - Exact command / steps: over HTTP, one warm-up `/v1/compress` (200-row JSON `tool_result`, 10,623 tokens), one oversized `/v1/compress` (40,000 rows, about 22s of work against the 3s budget, so its worker keeps running after the request fails open), then three small `/v1/compress` calls while that worker is still running, reading `/health` `runtime.compression_executor` in between. A second run fires four oversized calls concurrently to strand 4 of 8 workers, sends a small call, waits until one stuck worker exits, and sends another. - Observed result: on main one straggler refuses every following request; on this branch the other 7 workers keep compressing, and half the pool stuck still quarantines and clears again (output below). Before (main), one straggler refuses everything: ```text health after big timed out: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 0} small request 1: 0.02s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout small request 2: 0.01s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout small request 3: 0.01s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout health after small requests: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 3} ``` After, the same straggler leaves the other 7 workers serving: ```text health after big timed out: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': False, 'quarantine_activations_total': 0, 'quarantine_skips_total': 0} small request 1: 0.12s tokens_before=10623 tokens_saved=4411 skipped=False reason=None small request 2: 0.12s tokens_before=10623 tokens_saved=4411 skipped=False reason=None small request 3: 0.18s tokens_before=10623 tokens_saved=4411 skipped=False reason=None health after small requests: {'max_workers': 8, 'running': 1, 'timed_out_workers': 1, 'quarantine_active': False, 'quarantine_activations_total': 0, 'quarantine_skips_total': 0} ``` After, half the pool stuck still quarantines, and it clears when debt drops below half: ```text health after 4 concurrent big requests timed out: {'max_workers': 8, 'running': 4, 'timed_out_workers': 4, 'quarantine_active': True, 'quarantine_activations_total': 1, 'quarantine_skips_total': 0} small request while 4 of 8 stuck: 0.10s tokens_before=0 tokens_saved=0 skipped=True reason=compression_timeout health once a stuck worker exited: {'max_workers': 8, 'running': 3, 'timed_out_workers': 3, 'quarantine_active': False, 'quarantine_activations_total': 1, 'quarantine_skips_total': 1} small request with 3 stuck: 0.19s tokens_before=10623 tokens_saved=4411 skipped=False reason=None ``` ```text WARNING - Compression worker exceeded its request deadline and is still running; new compression is quarantined until fewer than 4 timed-out workers remain INFO - Compression quarantine cleared; 3 timed-out worker(s) still running ``` - Not tested: the `headroom proxy` CLI entry point itself (booted via `run_server`, see Environment); the `/v1/messages`, `/v1/responses` and Codex WS paths end to end (they share the same `_run_compression_in_executor`, but I only drove `/v1/compress`); Windows and Linux (the field evidence covers them, my repro does not); the 60s time-cap release over HTTP (covered by `test_compression_quarantine_releases_after_time_cap` and the new re-arm assertion, not by a live run); a long soak sampling OS thread and fd counts. ## Runtime Rollout Safety - Rollout-managed feature(s): none. The quarantine is not in `rollout.py` FEATURES. - Minimum rollout channel: n/a (applies on every channel). - Stable/default behavior changed: yes. On pools of 4 or more workers, compression keeps running while fewer than half the workers are held by timed-out jobs; previously the first timed-out job refused all compression for up to 60s. Pools of 1-3 workers are unchanged. - Kill switch / disable path: `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1` restores the previous behaviour on any pool size. - Unsafe override required: no. - Qualification impact: more requests get compressed while a straggler runs, so `tokens_saved` rises and the `skipped` quarantine counter falls on affected installs; no change to what compression does to a request it runs on, and no change to provider cache behaviour beyond the requests that are now compressed instead of forwarded raw. - Rollback path: set `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1`, or revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Pool capacity and #3653. The compression executor is a `ThreadPoolExecutor` with a fixed `max_workers`; it never creates threads beyond that cap, and a stuck worker keeps the thread it already holds. Admitting work while a straggler runs only puts that work on the pool's idle threads, which finish and release their resources normally, so this change cannot raise the process's thread count above `max_workers` or add a leaked thread/fd per quarantine event. What it does change is how many timed-out jobs can stand before new work is refused: up to `threshold - 1` (3 on an 8-worker pool) instead of 0. That bound is no looser than today's in practice: the old rule never prevented requests that were already admitted from timing out together (it only refuses calls that arrive after a timeout is recorded), and the 60s cap already lets work back in while debt stands. At `threshold` stuck workers the quarantine engages exactly as before, so at least half the pool stays free for work that is not stuck. Field evidence that motivated this (Headroom Desktop fleet canary, 2026-09-21 to 2026-09-23): 9 hosts on macOS, Windows and Linux had between 2.1% and 38.9% of requests refused compression by the quarantine. On one machine's proxy logs, 211 of 262 refusals happened with ONE stuck worker and none with more than 3 of 8 stuck; on 0.37.0 that machine forwarded 42 of 180 requests (23%) uncompressed in a day while `/health` showed `max_workers` 8, `queued_max` 1 and `queue_wait_seconds_max` 0.23, i.e. an idle pool. Headroom Desktop has shipped an equivalent downstream patch (standing the quarantine down below half the pool) since 2026-09-23 (desktop 0.9.22-rc.1); this PR implements it in the check itself so it can be dropped there. Documentation: `HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD` follows the existing undocumented `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`, so no docs page was updated; happy to add both to the proxy env reference if wanted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3932" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
1132a54b21
commit
be2b205c51
+47
-26
@@ -1276,11 +1276,19 @@ class HeadroomProxy(
|
||||
# timeout. Stuck-thread leak indicator.
|
||||
self._compression_leaked_threads: int = 0
|
||||
# Timeout-debt quarantine. Python cannot preempt a worker after its
|
||||
# asyncio waiter times out, so accepting more compression while that
|
||||
# worker is still running can multiply one slow call into a saturated
|
||||
# executor. New work raises immediately until every known post-timeout
|
||||
# worker has genuinely exited, then follows the caller's existing
|
||||
# compression-failure policy.
|
||||
# asyncio waiter times out, so accepting more compression while those
|
||||
# workers are still running can multiply slow calls into a saturated
|
||||
# executor. Once half the pool is held by post-timeout workers, new work
|
||||
# raises immediately until the debt drops below that again, and callers
|
||||
# follow their existing compression-failure policy. One straggler on a
|
||||
# mostly idle pool does not saturate it, so it does not refuse work;
|
||||
# pools of 1-3 workers still quarantine on the first timeout.
|
||||
# HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD=1 restores that everywhere.
|
||||
self._compression_quarantine_threshold: int = _get_env_int(
|
||||
"HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD",
|
||||
max(1, _compression_max // 2),
|
||||
min_value=1,
|
||||
)
|
||||
self._compression_timed_out_in_flight: int = 0
|
||||
self._compression_timed_out_in_flight_max: int = 0
|
||||
self._compression_quarantine_activations: int = 0
|
||||
@@ -1531,10 +1539,11 @@ class HeadroomProxy(
|
||||
worker keeps running to completion, ignored. We detect this by
|
||||
marking the call timed out on the asyncio side and incrementing
|
||||
``_compression_leaked_threads`` from the worker's ``finally``
|
||||
block after it eventually finishes. While any such worker remains,
|
||||
new calls raise :class:`CompressionQuarantinedError` immediately so
|
||||
callers apply the existing compression-failure policy instead of
|
||||
filling the rest of the pool with the same timeout debt. Jobs that are
|
||||
block after it eventually finishes. While such workers hold half the
|
||||
pool (``_compression_quarantine_threshold``), new calls raise
|
||||
:class:`CompressionQuarantinedError` immediately so callers apply the
|
||||
existing compression-failure policy instead of filling the rest of the
|
||||
pool with the same timeout debt. Jobs that are
|
||||
successfully cancelled before a worker starts are removed from the
|
||||
queued gauge and do not activate the quarantine. If cancellation races
|
||||
with worker startup, the now-running job is tracked as timeout debt and
|
||||
@@ -1556,20 +1565,19 @@ class HeadroomProxy(
|
||||
``asyncio.TimeoutError`` if the callable doesn't return within
|
||||
``timeout``. Any exception raised by ``fn`` propagates
|
||||
unchanged. :class:`CompressionQuarantinedError` (an
|
||||
``asyncio.TimeoutError`` subclass) if a prior timed-out worker is
|
||||
still running.
|
||||
``asyncio.TimeoutError`` subclass) if prior timed-out workers still
|
||||
hold half the pool.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
with self._compression_metrics_lock:
|
||||
timed_out_in_flight = self._compression_timed_out_in_flight
|
||||
quarantined = timed_out_in_flight > 0 and now < self._compression_quarantine_deadline
|
||||
saturating = timed_out_in_flight >= self._compression_quarantine_threshold
|
||||
quarantined = saturating and now < self._compression_quarantine_deadline
|
||||
# Debt outlived the cap: presume the worker leaked/hung and stop
|
||||
# blocking on it. Count the release once per lapse (while debt stands
|
||||
# and the deadline has passed) so operators can see it happened.
|
||||
released = (
|
||||
timed_out_in_flight > 0
|
||||
and not quarantined
|
||||
and self._compression_quarantine_deadline > 0.0
|
||||
saturating and not quarantined and self._compression_quarantine_deadline > 0.0
|
||||
)
|
||||
if quarantined:
|
||||
self._compression_quarantine_skips += 1
|
||||
@@ -1613,7 +1621,7 @@ class HeadroomProxy(
|
||||
"""Record a still-running post-timeout worker; lock must be held.
|
||||
|
||||
Returns ``True`` only when this worker transitions the executor
|
||||
from clear to quarantined.
|
||||
from clear (or released by the time cap) to quarantined.
|
||||
"""
|
||||
if (
|
||||
not state["timed_out"]
|
||||
@@ -1622,7 +1630,12 @@ class HeadroomProxy(
|
||||
or state["timeout_debt_recorded"]
|
||||
):
|
||||
return False
|
||||
was_clear = self._compression_timed_out_in_flight == 0
|
||||
now = time.monotonic()
|
||||
threshold = self._compression_quarantine_threshold
|
||||
was_quarantined = (
|
||||
self._compression_timed_out_in_flight >= threshold
|
||||
and now < self._compression_quarantine_deadline
|
||||
)
|
||||
self._compression_timed_out_in_flight += 1
|
||||
self._compression_timed_out_in_flight_max = max(
|
||||
self._compression_timed_out_in_flight_max,
|
||||
@@ -1631,19 +1644,19 @@ class HeadroomProxy(
|
||||
# (Re)arm the quarantine time cap on every fresh timeout, so ongoing
|
||||
# slowness keeps quarantining while a single leaked worker cannot
|
||||
# hold it past the cap (#2360).
|
||||
self._compression_quarantine_deadline = (
|
||||
time.monotonic() + self._compression_quarantine_max_seconds
|
||||
)
|
||||
self._compression_quarantine_deadline = now + self._compression_quarantine_max_seconds
|
||||
state["timeout_debt_recorded"] = True
|
||||
if was_clear:
|
||||
activated = not was_quarantined and self._compression_timed_out_in_flight >= threshold
|
||||
if activated:
|
||||
self._compression_quarantine_activations += 1
|
||||
return was_clear
|
||||
return activated
|
||||
|
||||
def _announce_quarantine() -> None:
|
||||
self.metrics.record_compression_quarantine("activated")
|
||||
logger.warning(
|
||||
"Compression worker exceeded its request deadline and is still running; "
|
||||
"new compression is quarantined until timed-out workers exit"
|
||||
"new compression is quarantined until fewer than %d timed-out workers remain",
|
||||
self._compression_quarantine_threshold,
|
||||
)
|
||||
|
||||
def _wrapped(): # noqa: ANN202
|
||||
@@ -1678,11 +1691,15 @@ class HeadroomProxy(
|
||||
if state["timeout_debt_recorded"]:
|
||||
self._compression_timed_out_in_flight -= 1
|
||||
state["timeout_debt_recorded"] = False
|
||||
quarantine_cleared = self._compression_timed_out_in_flight == 0
|
||||
remaining = self._compression_timed_out_in_flight
|
||||
quarantine_cleared = remaining == self._compression_quarantine_threshold - 1
|
||||
else:
|
||||
quarantine_cleared = False
|
||||
if quarantine_cleared:
|
||||
logger.info("Compression quarantine cleared after all timed-out workers exited")
|
||||
logger.info(
|
||||
"Compression quarantine cleared; %d timed-out worker(s) still running",
|
||||
remaining,
|
||||
)
|
||||
|
||||
future = loop.run_in_executor(self._compression_executor, _wrapped)
|
||||
try:
|
||||
@@ -3542,6 +3559,10 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
_comp_run_max = proxy._compression_run_seconds_max
|
||||
_comp_leaked = proxy._compression_leaked_threads
|
||||
_comp_timed_out_in_flight = proxy._compression_timed_out_in_flight
|
||||
_comp_quarantine_active = (
|
||||
_comp_timed_out_in_flight >= proxy._compression_quarantine_threshold
|
||||
and time.monotonic() < proxy._compression_quarantine_deadline
|
||||
)
|
||||
_comp_timed_out_in_flight_max = proxy._compression_timed_out_in_flight_max
|
||||
_comp_quarantine_activations = proxy._compression_quarantine_activations
|
||||
_comp_quarantine_skips = proxy._compression_quarantine_skips
|
||||
@@ -3572,7 +3593,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
||||
"run_seconds_total": _comp_run_total,
|
||||
"run_seconds_max": _comp_run_max,
|
||||
"leaked_threads_total": _comp_leaked,
|
||||
"quarantine_active": _comp_timed_out_in_flight > 0,
|
||||
"quarantine_active": _comp_quarantine_active,
|
||||
"timed_out_workers": _comp_timed_out_in_flight,
|
||||
"timed_out_workers_max": _comp_timed_out_in_flight_max,
|
||||
"quarantine_activations_total": _comp_quarantine_activations,
|
||||
|
||||
@@ -294,6 +294,93 @@ def test_timeout_quarantines_new_work_until_timed_out_worker_finishes() -> None:
|
||||
assert 'headroom_compression_quarantine_total{event="skipped"} 1' in prometheus_text
|
||||
|
||||
|
||||
def test_quarantine_waits_for_half_the_pool_to_be_stuck() -> None:
|
||||
"""One straggler must not refuse compression while most workers are idle.
|
||||
|
||||
The quarantine exists so timed-out work cannot saturate the executor. On
|
||||
an 8-worker pool, one or three stuck workers leave the pool mostly idle,
|
||||
so compression keeps running; at four (half the pool) the quarantine
|
||||
engages, is counted once, and ``/health`` reports it. A timeout after the
|
||||
time cap released it counts as a fresh activation.
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from headroom.proxy.server import CompressionQuarantinedError
|
||||
|
||||
config = ProxyConfig(
|
||||
optimize=False,
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
cost_tracking_enabled=False,
|
||||
log_requests=False,
|
||||
ccr_inject_tool=False,
|
||||
ccr_handle_responses=False,
|
||||
ccr_context_tracking=False,
|
||||
image_optimize=False,
|
||||
compression_max_workers=8,
|
||||
)
|
||||
app = create_app(config)
|
||||
proxy = app.state.proxy
|
||||
release = threading.Event()
|
||||
|
||||
async def _strand(count: int) -> None:
|
||||
for _ in range(count):
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await proxy._run_compression_in_executor(release.wait, timeout=0.05)
|
||||
|
||||
def _executor_health() -> dict:
|
||||
with TestClient(app) as client:
|
||||
return client.get("/health").json()["runtime"]["compression_executor"]
|
||||
|
||||
try:
|
||||
asyncio.run(_strand(1))
|
||||
assert asyncio.run(proxy._run_compression_in_executor(lambda: "ran", timeout=1.0)) == "ran"
|
||||
health = _executor_health()
|
||||
assert health["timed_out_workers"] == 1
|
||||
assert health["quarantine_active"] is False
|
||||
|
||||
asyncio.run(_strand(2))
|
||||
assert asyncio.run(proxy._run_compression_in_executor(lambda: "ran", timeout=1.0)) == "ran"
|
||||
with proxy._compression_metrics_lock:
|
||||
assert proxy._compression_quarantine_activations == 0
|
||||
assert proxy._compression_quarantine_skips == 0
|
||||
assert proxy._compression_quarantine_releases == 0
|
||||
|
||||
asyncio.run(_strand(1))
|
||||
with pytest.raises(CompressionQuarantinedError):
|
||||
asyncio.run(proxy._run_compression_in_executor(lambda: "unused", timeout=1.0))
|
||||
health = _executor_health()
|
||||
assert health["timed_out_workers"] == 4
|
||||
assert health["quarantine_active"] is True
|
||||
assert health["quarantine_activations_total"] == 1
|
||||
assert health["quarantine_skips_total"] == 1
|
||||
|
||||
proxy._compression_quarantine_deadline = time.monotonic() - 1.0
|
||||
assert asyncio.run(proxy._run_compression_in_executor(lambda: "ran", timeout=1.0)) == "ran"
|
||||
assert proxy._compression_quarantine_releases == 1
|
||||
asyncio.run(_strand(1))
|
||||
with pytest.raises(CompressionQuarantinedError):
|
||||
asyncio.run(proxy._run_compression_in_executor(lambda: "unused", timeout=1.0))
|
||||
assert proxy._compression_quarantine_activations == 2
|
||||
finally:
|
||||
release.set()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("workers", "env", "expected"),
|
||||
[(8, None, 4), (3, None, 1), (1, None, 1), (8, "1", 1), (8, "0", 4)],
|
||||
)
|
||||
def test_quarantine_threshold_is_half_the_pool_unless_overridden(
|
||||
monkeypatch, workers: int, env: str | None, expected: int
|
||||
) -> None:
|
||||
if env is None:
|
||||
monkeypatch.delenv("HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD", raising=False)
|
||||
else:
|
||||
monkeypatch.setenv("HEADROOM_COMPRESSION_QUARANTINE_THRESHOLD", env)
|
||||
proxy = _make_proxy(compression_max_workers=workers)
|
||||
assert proxy._compression_quarantine_threshold == expected
|
||||
|
||||
|
||||
def test_timeout_before_worker_start_does_not_leak_in_flight() -> None:
|
||||
"""If a queued job times out before a worker starts, queued accounting
|
||||
is cleaned up without touching the running gauge.
|
||||
|
||||
@@ -179,8 +179,9 @@ async def test_count_tokens_offloaded_fails_open_on_executor_quarantine() -> Non
|
||||
# quarantines the next call — no mock of the helper itself. Since the
|
||||
# quarantine became time-capped (#2412), standing debt alone no longer
|
||||
# quarantines: the deadline armed by the fresh timeout must still be in
|
||||
# the future, so arm it the way a real timeout would.
|
||||
proxy._compression_timed_out_in_flight = 1
|
||||
# the future, so arm it the way a real timeout would. The debt must reach
|
||||
# the quarantine threshold (half the auto-sized pool).
|
||||
proxy._compression_timed_out_in_flight = proxy._compression_quarantine_threshold
|
||||
proxy._compression_quarantine_deadline = time.monotonic() + 60.0
|
||||
|
||||
tokenizer, tokens = await proxy._count_tokens_offloaded(
|
||||
@@ -198,8 +199,8 @@ async def test_count_tokens_offloaded_returns_count_text_capable_tokenizer() ->
|
||||
proxy = _make_proxy()
|
||||
# Quarantine forces the fail-open branch (an EstimatingTokenCounter).
|
||||
# Post-#2412 the quarantine is time-capped, so the deadline must be armed
|
||||
# alongside the standing debt.
|
||||
proxy._compression_timed_out_in_flight = 1
|
||||
# alongside the standing debt, which must reach the quarantine threshold.
|
||||
proxy._compression_timed_out_in_flight = proxy._compression_quarantine_threshold
|
||||
proxy._compression_quarantine_deadline = time.monotonic() + 60.0
|
||||
|
||||
# The empty-messages count is intentionally discarded by that handler
|
||||
|
||||
Reference in New Issue
Block a user