fix(proxy): kill the image worker a timed-out call abandons (#3940)

## Description

When an image compression call times out, crashes or raises,
`run_image_compression_isolated` forwards the original payload and calls
`_reset_image_pool()`, which drops the pool with `shutdown(wait=False,
cancel_futures=True)`. That never stops a worker that is still running
the abandoned call. The worker runs the image to the end and its result
is discarded, while the next image request starts a fresh worker beside
it that cold-loads the ONNX models (technique router, SigLIP, RapidOCR)
again.

On a loaded machine this feeds itself: model loads make the next call
slower, so it times out too and leaves another worker behind. On one
macOS proxy I watched four such workers at about one core each, with 6
to 8 worker cold starts per 10 minutes and the machine at load average
140. That proxy logged 32 image worker timeouts that day.

`_reset_image_pool` now kills the dropped pool's worker. It takes the
pool the failed call ran on, and only unpublishes it if it is still the
current one. A call that fails after another request has already rebuilt
the pool leaves the rebuilt pool and its worker alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/image_isolation.py`: `_reset_image_pool(pool=None)`
reads the pool's worker processes before `shutdown()` (which sets
`_processes` to `None`) and kills them after it. It clears `_IMAGE_POOL`
only when that is still the pool being reset.
`run_image_compression_isolated` passes the pool it submitted to. Adds a
`_hang_worker` test hook next to the existing `_sleep_worker` /
`_abort_worker`.
- `tests/test_image_compression_isolation.py`:
`test_worker_timeout_kills_the_abandoned_worker` warms the pool, times
out a hanging call and checks the original messages come back and the
worker is dead. `test_late_failure_leaves_a_rebuilt_pool_alone` resets a
stale pool after a rebuild and checks the rebuilt pool is still
published.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Before the fix (upstream/main d5318ac2), new tests only:
```text
FAILED tests/test_image_compression_isolation.py::test_worker_timeout_kills_the_abandoned_worker
FAILED tests/test_image_compression_isolation.py::test_late_failure_leaves_a_rebuilt_pool_alone
========================= 2 failed, 5 passed in 7.79s ==========================
```

With the fix:
```text
$ uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py tests/test_image_compressor_singleton_reuse.py
============================== 14 passed in 3.71s ==============================
$ uv run ruff check headroom/proxy/image_isolation.py tests/test_image_compression_isolation.py
All checks passed!
$ uv run mypy headroom/proxy/image_isolation.py
Success: no issues found in 1 source file
```

Every test file that touches the image stack or the proxy's image path,
with the `image` extra:
```text
$ uv run --extra dev --extra image pytest tests/test_image_*.py tests/test_onnx_router_threads.py tests/test_builtin_compressor_adapters.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handler_helpers.py
226 passed, 2 skipped, 1 warning in 44.97s
```
The two skips are the tree-sitter and trafilatura adapters, which aren't
installed here. The full suite did not finish locally within 25 minutes,
so CI covers the rest.

## Real Behavior Proof

- Environment: macOS 27 (M2, 8 cores), Python 3.11.13, `uv run --extra
dev --extra image`, upstream/main d5318ac2 vs this branch.
- Exact command / steps: a script calls the real
`run_image_compression_isolated` with the real
`_compress_messages_worker` on a 1 MB PNG screenshot, three times, each
with `timeout=0.5`. That is shorter than the worker's cold model load,
so every call times out, as it does on an overloaded machine. Two
seconds later the script counts `multiprocessing.active_children()`.
  ```python
  for _ in range(3):
out, res = await m.run_image_compression_isolated(msgs, "anthropic",
timeout=0.5)
      assert out is msgs and res is None
  time.sleep(2)
  print(len(multiprocessing.active_children()))
  ```
- Observed result: upstream leaves one live worker behind per timeout;
with the fix none survive, and every call still returns the original
messages (output below).
  ```text
  upstream: {"timeouts": 3, "workers_still_alive": 3}
  branch:   {"timeouts": 3, "workers_still_alive": 0}
  ```
Upstream leaves one live worker per timeout, each still loading models
for a call whose result is gone. With the fix every timed-out call's
worker is gone, and all three calls still return the original messages.
- Live: the same change, vendored into a desktop build on headroom-ai
0.39.0 behind real Claude Code traffic, logged 5 image worker timeouts
between 16:34 and 16:41, and no check found more than one worker alive.
- Not tested: Linux and Windows. `Process.kill()` is SIGKILL on POSIX
and `TerminateProcess` on Windows, and the existing `_abort_worker` test
already covers a pool whose worker died. The `BrokenProcessPool` and
generic-exception branches go through the same reset and were not
exercised separately for real. The race covered by
`test_late_failure_leaves_a_rebuilt_pool_alone` was tested at unit level
only.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, not rollout-managed.
- Stable/default behavior changed: only after an image compression call
has already failed (timeout, worker crash, exception). The request still
forwards its original image payload as before. The abandoned worker is
now killed instead of left running. Successful calls are untouched. A
request queued behind the killed worker on the same pool now gets
`BrokenProcessPool` and forwards its original payload right away,
instead of waiting out its own timeout behind the abandoned call. Live,
that shows up as one `worker crashed` warning right after a `worker
timed out`.
- Kill switch / disable path: none. This is the failure path only.
- Unsafe override required: no.
- Qualification impact: none on compression or cache. Less CPU after
image timeouts.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
gglucass
2026-10-02 18:58:09 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 2c4b20fcba
commit 793bb85659
2 changed files with 67 additions and 9 deletions
+34 -9
View File
@@ -84,6 +84,14 @@ def _sleep_worker(
return messages, None
def _hang_worker(
messages: list[dict[str, Any]],
provider: str,
) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
time.sleep(60)
return messages, None
def _abort_worker(
messages: list[dict[str, Any]],
provider: str,
@@ -108,13 +116,28 @@ def _image_pool() -> ProcessPoolExecutor:
return _IMAGE_POOL
def _reset_image_pool() -> None:
def _reset_image_pool(pool: ProcessPoolExecutor | None = None) -> None:
"""Drop ``pool`` (default: the current one) and kill its worker.
``shutdown(wait=False)`` alone never stops a worker that is still running
an abandoned call: after a timeout it ran the image to the end, its result
discarded, while the next request cold-loaded the models in a fresh worker
beside it. Only the current pool is unpublished, so a call that fails
after another request already rebuilt the pool leaves the rebuilt one alone.
"""
global _IMAGE_POOL
with _IMAGE_POOL_LOCK:
pool = _IMAGE_POOL
_IMAGE_POOL = None
if pool is not None:
pool.shutdown(wait=False, cancel_futures=True)
if pool is None:
pool = _IMAGE_POOL
if _IMAGE_POOL is pool:
_IMAGE_POOL = None
if pool is None:
return
# shutdown() sets _processes to None, so take the workers first.
workers = list((pool._processes or {}).values())
pool.shutdown(wait=False, cancel_futures=True)
for worker in workers:
worker.kill()
async def run_image_compression_isolated(
@@ -124,21 +147,23 @@ async def run_image_compression_isolated(
timeout: float,
) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
loop = asyncio.get_running_loop()
pool: ProcessPoolExecutor | None = None
try:
future = loop.run_in_executor(_image_pool(), _IMAGE_WORKER, messages, provider)
pool = _image_pool()
future = loop.run_in_executor(pool, _IMAGE_WORKER, messages, provider)
return await asyncio.wait_for(future, timeout=timeout)
except BrokenProcessPool:
logger.warning("Image compression worker crashed; forwarding original image payload")
_reset_image_pool()
_reset_image_pool(pool)
return messages, None
except TimeoutError:
logger.warning("Image compression worker timed out; forwarding original image payload")
_reset_image_pool()
_reset_image_pool(pool)
return messages, None
except Exception as exc:
logger.warning(
"Image compression worker failed (%s); forwarding original image payload",
type(exc).__name__,
)
_reset_image_pool()
_reset_image_pool(pool)
return messages, None
+33
View File
@@ -85,6 +85,39 @@ async def test_worker_timeout_fails_open(monkeypatch) -> None:
assert recovered_result is not None
async def test_worker_timeout_kills_the_abandoned_worker(monkeypatch) -> None:
# shutdown(wait=False) alone left the worker running the abandoned call to
# the end while the next request cold-started a fresh worker beside it.
messages = _messages()
monkeypatch.setattr(image_isolation, "_IMAGE_WORKER", image_isolation._success_worker)
await image_isolation.run_image_compression_isolated(messages, provider="openai", timeout=30.0)
(worker,) = image_isolation._IMAGE_POOL._processes.values()
monkeypatch.setattr(image_isolation, "_IMAGE_WORKER", image_isolation._hang_worker)
returned, result = await image_isolation.run_image_compression_isolated(
messages,
provider="openai",
timeout=0.5,
)
assert returned is messages
assert result is None
worker.join(timeout=5.0)
assert not worker.is_alive()
def test_late_failure_leaves_a_rebuilt_pool_alone() -> None:
# A call that fails after another request already replaced its pool must
# not drop (and kill the worker of) the replacement.
old = image_isolation._image_pool()
image_isolation._reset_image_pool(old)
rebuilt = image_isolation._image_pool()
image_isolation._reset_image_pool(old)
assert image_isolation._IMAGE_POOL is rebuilt
async def test_worker_success_returns_compressed(monkeypatch) -> None:
messages = _messages()
monkeypatch.setattr(image_isolation, "_IMAGE_WORKER", image_isolation._success_worker)