mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description
Fixes #4418. A timeout in one image-compression request must not tear
down the shared worker pool while another request still uses it.
Abandoned queued work must also be cancelled so it does not execute
after its request has failed open.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Pair pool acquisition and inflight accounting under the pool lock;
avoid destructive reset while another call uses that pool.
- Distinguish pool-job cancellation from caller cancellation through a
private waiter, including Python 3.10 compatibility.
- Best-effort cancel the underlying executor future when its caller
times out or is cancelled.
- Add spawned-process regressions for pending-job abandonment, peer
isolation, and concurrent event-loop threads.
- Maintainer synced current main without conflicts; the three
contributor files are unchanged by that sync.
## Testing
- [x] Unit tests pass (`pytest`): focused files below
- [x] Linting passes: Ruff on the three changed files
- [ ] Type checking passes (`mypy headroom`): not run in this maintainer
pass
- [x] New tests added for new functionality
### Test Output
```text
python -m pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q
Before main sync: 17 passed in 16.93s
After main sync: 17 passed in 13.97s
ruff check <three changed files>: All checks passed!
git diff --check: passed
```
## Real Behavior Proof
- Environment: maintainer validation on Windows, Python 3.12, existing
local native extension; real spawn ProcessPoolExecutor workers with
synthetic image-worker payloads.
- Exact command / steps: run the two focused test files above. The
pending-job regressions occupy the worker with a peer, enqueue fillers
and a target, then time out or cancel the target before releasing the
peer.
- Observed result: abandoned target future is cancelled and its
execution marker is never created; the unrelated peer and fillers
complete. Concurrent requests and event-loop threads retain their
compression. All 17 tests pass before and after the main sync.
- Not tested: live image-model/ONNX inference, provider traffic, Python
3.10 runtime, full repository test suite, or production load. Hosted
qualification is pending.
## Runtime Rollout Safety
- Rollout-managed feature(s): no new feature or rollout gate;
image-worker lifecycle correction.
- Minimum rollout channel: existing image-compression configuration.
- Stable/default behavior changed: a failed call no longer destroys a
pool still used by peers; abandoned pending work is cancelled
best-effort.
- Kill switch / disable path: existing image-compression configuration;
no new switch.
- Unsafe override required: no.
- Qualification impact: focused process-lifecycle regressions added;
hosted CI remains required.
- Rollback path: revert the implementation and regression commits.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
These readiness statements describe the maintainer review of the current
combined tree, not an assertion of contributor human review or completed
hosted CI. Final approval remains pending hosted qualification.
## Additional Notes
Original contribution attribution: OpenCode using Autoloops wrote the
AI-assisted patch. The original automated harness reported 11 passing
tests and one failure with the patch removed, based on
8d5dde7765. Subsequent maintainer reviews
requested cancellation and isolation corrections; the current results
above supersede that earlier test count. No contributor logic was
modified during the maintainer main sync. Nothing has been merged.
---------
Co-authored-by: nandanadileep <116407139+nandanadileep@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Nandana Dileep <nandanadileep@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
69 lines
2.3 KiB
Python
69 lines
2.3 KiB
Python
"""Image compression must stay off the event loop and behind the isolation runner.
|
|
|
|
The handlers no longer call `compressor.compress(...)` on the thread pool. They
|
|
delegate to `run_image_compression_isolated(...)`, which moves the native image
|
|
stack into a spawned subprocess so OpenCV crashes fail open without taking down
|
|
the proxy.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import inspect
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy import image_isolation
|
|
from headroom.proxy.handlers.anthropic import AnthropicHandlerMixin
|
|
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_image_pool() -> None:
|
|
image_isolation._reset_image_pool()
|
|
image_isolation._IMAGE_POOL_INFLIGHT = 0
|
|
try:
|
|
yield
|
|
finally:
|
|
image_isolation._reset_image_pool()
|
|
image_isolation._IMAGE_POOL_INFLIGHT = 0
|
|
image_isolation._IMAGE_WORKER = image_isolation._compress_messages_worker
|
|
|
|
|
|
def test_image_blocks_use_isolation_runner_and_fail_open() -> None:
|
|
for mixin, method in (
|
|
(AnthropicHandlerMixin, "handle_anthropic_messages"),
|
|
(OpenAIHandlerMixin, "handle_openai_chat"),
|
|
):
|
|
fn = getattr(mixin, method)
|
|
assert inspect.iscoroutinefunction(fn), f"{method} must be async to await the isolation"
|
|
src = inspect.getsource(fn)
|
|
assert "run_image_compression_isolated(" in src, f"{method}: isolation runner missing"
|
|
assert "COMPRESSION_TIMEOUT_SECONDS" in src, f"{method}: isolation missing a timeout"
|
|
assert "Image compression failed" in src, f"{method}: image compress not fail-open"
|
|
|
|
|
|
async def test_image_isolation_keeps_event_loop_responsive() -> None:
|
|
ticks = 0
|
|
image_isolation._IMAGE_WORKER = image_isolation._sleep_worker
|
|
|
|
async def _ticker() -> None:
|
|
nonlocal ticks
|
|
while True:
|
|
await asyncio.sleep(0.01)
|
|
ticks += 1
|
|
|
|
tick_task = asyncio.create_task(_ticker())
|
|
try:
|
|
returned, result = await image_isolation.run_image_compression_isolated(
|
|
[{"role": "user", "content": "image payload"}],
|
|
provider="openai",
|
|
timeout=1.0,
|
|
)
|
|
finally:
|
|
tick_task.cancel()
|
|
|
|
assert returned == [{"role": "user", "content": "image payload"}]
|
|
assert result is None
|
|
assert ticks >= 5
|