mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(proxy): kill the image worker a timed-out call abandons (#3940)
## Description When an image compression call times out, crashes or raises, `run_image_compression_isolated` forwards the original payload and calls `_reset_image_pool()`, which drops the pool with `shutdown(wait=False, cancel_futures=True)`. That never stops a worker that is still running the abandoned call. The worker runs the image to the end and its result is discarded, while the next image request starts a fresh worker beside it that cold-loads the ONNX models (technique router, SigLIP, RapidOCR) again. On a loaded machine this feeds itself: model loads make the next call slower, so it times out too and leaves another worker behind. On one macOS proxy I watched four such workers at about one core each, with 6 to 8 worker cold starts per 10 minutes and the machine at load average 140. That proxy logged 32 image worker timeouts that day. `_reset_image_pool` now kills the dropped pool's worker. It takes the pool the failed call ran on, and only unpublishes it if it is still the current one. A call that fails after another request has already rebuilt the pool leaves the rebuilt pool and its worker alone. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/image_isolation.py`: `_reset_image_pool(pool=None)` reads the pool's worker processes before `shutdown()` (which sets `_processes` to `None`) and kills them after it. It clears `_IMAGE_POOL` only when that is still the pool being reset. `run_image_compression_isolated` passes the pool it submitted to. Adds a `_hang_worker` test hook next to the existing `_sleep_worker` / `_abort_worker`. - `tests/test_image_compression_isolation.py`: `test_worker_timeout_kills_the_abandoned_worker` warms the pool, times out a hanging call and checks the original messages come back and the worker is dead. `test_late_failure_leaves_a_rebuilt_pool_alone` resets a stale pool after a rebuild and checks the rebuilt pool is still published. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Before the fix (upstream/maind5318ac2), new tests only: ```text FAILED tests/test_image_compression_isolation.py::test_worker_timeout_kills_the_abandoned_worker FAILED tests/test_image_compression_isolation.py::test_late_failure_leaves_a_rebuilt_pool_alone ========================= 2 failed, 5 passed in 7.79s ========================== ``` With the fix: ```text $ uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py tests/test_image_compressor_singleton_reuse.py ============================== 14 passed in 3.71s ============================== $ uv run ruff check headroom/proxy/image_isolation.py tests/test_image_compression_isolation.py All checks passed! $ uv run mypy headroom/proxy/image_isolation.py Success: no issues found in 1 source file ``` Every test file that touches the image stack or the proxy's image path, with the `image` extra: ```text $ uv run --extra dev --extra image pytest tests/test_image_*.py tests/test_onnx_router_threads.py tests/test_builtin_compressor_adapters.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handler_helpers.py 226 passed, 2 skipped, 1 warning in 44.97s ``` The two skips are the tree-sitter and trafilatura adapters, which aren't installed here. The full suite did not finish locally within 25 minutes, so CI covers the rest. ## Real Behavior Proof - Environment: macOS 27 (M2, 8 cores), Python 3.11.13, `uv run --extra dev --extra image`, upstream/maind5318ac2vs this branch. - Exact command / steps: a script calls the real `run_image_compression_isolated` with the real `_compress_messages_worker` on a 1 MB PNG screenshot, three times, each with `timeout=0.5`. That is shorter than the worker's cold model load, so every call times out, as it does on an overloaded machine. Two seconds later the script counts `multiprocessing.active_children()`. ```python for _ in range(3): out, res = await m.run_image_compression_isolated(msgs, "anthropic", timeout=0.5) assert out is msgs and res is None time.sleep(2) print(len(multiprocessing.active_children())) ``` - Observed result: upstream leaves one live worker behind per timeout; with the fix none survive, and every call still returns the original messages (output below). ```text upstream: {"timeouts": 3, "workers_still_alive": 3} branch: {"timeouts": 3, "workers_still_alive": 0} ``` Upstream leaves one live worker per timeout, each still loading models for a call whose result is gone. With the fix every timed-out call's worker is gone, and all three calls still return the original messages. - Live: the same change, vendored into a desktop build on headroom-ai 0.39.0 behind real Claude Code traffic, logged 5 image worker timeouts between 16:34 and 16:41, and no check found more than one worker alive. - Not tested: Linux and Windows. `Process.kill()` is SIGKILL on POSIX and `TerminateProcess` on Windows, and the existing `_abort_worker` test already covers a pool whose worker died. The `BrokenProcessPool` and generic-exception branches go through the same reset and were not exercised separately for real. The race covered by `test_late_failure_leaves_a_rebuilt_pool_alone` was tested at unit level only. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a, not rollout-managed. - Stable/default behavior changed: only after an image compression call has already failed (timeout, worker crash, exception). The request still forwards its original image payload as before. The abandoned worker is now killed instead of left running. Successful calls are untouched. A request queued behind the killed worker on the same pool now gets `BrokenProcessPool` and forwards its original payload right away, instead of waiting out its own timeout behind the abandoned call. Live, that shows up as one `worker crashed` warning right after a `worker timed out`. - Kill switch / disable path: none. This is the failure path only. - Unsafe override required: no. - Qualification impact: none on compression or cache. Less CPU after image timeouts. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
2c4b20fcba
commit
793bb85659
@@ -84,6 +84,14 @@ def _sleep_worker(
|
||||
return messages, None
|
||||
|
||||
|
||||
def _hang_worker(
|
||||
messages: list[dict[str, Any]],
|
||||
provider: str,
|
||||
) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
|
||||
time.sleep(60)
|
||||
return messages, None
|
||||
|
||||
|
||||
def _abort_worker(
|
||||
messages: list[dict[str, Any]],
|
||||
provider: str,
|
||||
@@ -108,13 +116,28 @@ def _image_pool() -> ProcessPoolExecutor:
|
||||
return _IMAGE_POOL
|
||||
|
||||
|
||||
def _reset_image_pool() -> None:
|
||||
def _reset_image_pool(pool: ProcessPoolExecutor | None = None) -> None:
|
||||
"""Drop ``pool`` (default: the current one) and kill its worker.
|
||||
|
||||
``shutdown(wait=False)`` alone never stops a worker that is still running
|
||||
an abandoned call: after a timeout it ran the image to the end, its result
|
||||
discarded, while the next request cold-loaded the models in a fresh worker
|
||||
beside it. Only the current pool is unpublished, so a call that fails
|
||||
after another request already rebuilt the pool leaves the rebuilt one alone.
|
||||
"""
|
||||
global _IMAGE_POOL
|
||||
with _IMAGE_POOL_LOCK:
|
||||
pool = _IMAGE_POOL
|
||||
_IMAGE_POOL = None
|
||||
if pool is not None:
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
if pool is None:
|
||||
pool = _IMAGE_POOL
|
||||
if _IMAGE_POOL is pool:
|
||||
_IMAGE_POOL = None
|
||||
if pool is None:
|
||||
return
|
||||
# shutdown() sets _processes to None, so take the workers first.
|
||||
workers = list((pool._processes or {}).values())
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
for worker in workers:
|
||||
worker.kill()
|
||||
|
||||
|
||||
async def run_image_compression_isolated(
|
||||
@@ -124,21 +147,23 @@ async def run_image_compression_isolated(
|
||||
timeout: float,
|
||||
) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
|
||||
loop = asyncio.get_running_loop()
|
||||
pool: ProcessPoolExecutor | None = None
|
||||
try:
|
||||
future = loop.run_in_executor(_image_pool(), _IMAGE_WORKER, messages, provider)
|
||||
pool = _image_pool()
|
||||
future = loop.run_in_executor(pool, _IMAGE_WORKER, messages, provider)
|
||||
return await asyncio.wait_for(future, timeout=timeout)
|
||||
except BrokenProcessPool:
|
||||
logger.warning("Image compression worker crashed; forwarding original image payload")
|
||||
_reset_image_pool()
|
||||
_reset_image_pool(pool)
|
||||
return messages, None
|
||||
except TimeoutError:
|
||||
logger.warning("Image compression worker timed out; forwarding original image payload")
|
||||
_reset_image_pool()
|
||||
_reset_image_pool(pool)
|
||||
return messages, None
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Image compression worker failed (%s); forwarding original image payload",
|
||||
type(exc).__name__,
|
||||
)
|
||||
_reset_image_pool()
|
||||
_reset_image_pool(pool)
|
||||
return messages, None
|
||||
|
||||
@@ -85,6 +85,39 @@ async def test_worker_timeout_fails_open(monkeypatch) -> None:
|
||||
assert recovered_result is not None
|
||||
|
||||
|
||||
async def test_worker_timeout_kills_the_abandoned_worker(monkeypatch) -> None:
|
||||
# shutdown(wait=False) alone left the worker running the abandoned call to
|
||||
# the end while the next request cold-started a fresh worker beside it.
|
||||
messages = _messages()
|
||||
monkeypatch.setattr(image_isolation, "_IMAGE_WORKER", image_isolation._success_worker)
|
||||
await image_isolation.run_image_compression_isolated(messages, provider="openai", timeout=30.0)
|
||||
(worker,) = image_isolation._IMAGE_POOL._processes.values()
|
||||
|
||||
monkeypatch.setattr(image_isolation, "_IMAGE_WORKER", image_isolation._hang_worker)
|
||||
returned, result = await image_isolation.run_image_compression_isolated(
|
||||
messages,
|
||||
provider="openai",
|
||||
timeout=0.5,
|
||||
)
|
||||
|
||||
assert returned is messages
|
||||
assert result is None
|
||||
worker.join(timeout=5.0)
|
||||
assert not worker.is_alive()
|
||||
|
||||
|
||||
def test_late_failure_leaves_a_rebuilt_pool_alone() -> None:
|
||||
# A call that fails after another request already replaced its pool must
|
||||
# not drop (and kill the worker of) the replacement.
|
||||
old = image_isolation._image_pool()
|
||||
image_isolation._reset_image_pool(old)
|
||||
rebuilt = image_isolation._image_pool()
|
||||
|
||||
image_isolation._reset_image_pool(old)
|
||||
|
||||
assert image_isolation._IMAGE_POOL is rebuilt
|
||||
|
||||
|
||||
async def test_worker_success_returns_compressed(monkeypatch) -> None:
|
||||
messages = _messages()
|
||||
monkeypatch.setattr(image_isolation, "_IMAGE_WORKER", image_isolation._success_worker)
|
||||
|
||||
Reference in New Issue
Block a user