Commit Graph
3 Commits
Author SHA1 Message Date
e9bcfe73fa fix: [BUG] A timeout in one image compression call resets the shared worker pool and affects concurrent requests (#4528)
## Description

Fixes #4418. A timeout in one image-compression request must not tear
down the shared worker pool while another request still uses it.
Abandoned queued work must also be cancelled so it does not execute
after its request has failed open.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Pair pool acquisition and inflight accounting under the pool lock;
avoid destructive reset while another call uses that pool.
- Distinguish pool-job cancellation from caller cancellation through a
private waiter, including Python 3.10 compatibility.
- Best-effort cancel the underlying executor future when its caller
times out or is cancelled.
- Add spawned-process regressions for pending-job abandonment, peer
isolation, and concurrent event-loop threads.
- Maintainer synced current main without conflicts; the three
contributor files are unchanged by that sync.

## Testing

- [x] Unit tests pass (`pytest`): focused files below
- [x] Linting passes: Ruff on the three changed files
- [ ] Type checking passes (`mypy headroom`): not run in this maintainer
pass
- [x] New tests added for new functionality

### Test Output

```text
python -m pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q
Before main sync: 17 passed in 16.93s
After main sync: 17 passed in 13.97s
ruff check <three changed files>: All checks passed!
git diff --check: passed
```

## Real Behavior Proof

- Environment: maintainer validation on Windows, Python 3.12, existing
local native extension; real spawn ProcessPoolExecutor workers with
synthetic image-worker payloads.
- Exact command / steps: run the two focused test files above. The
pending-job regressions occupy the worker with a peer, enqueue fillers
and a target, then time out or cancel the target before releasing the
peer.
- Observed result: abandoned target future is cancelled and its
execution marker is never created; the unrelated peer and fillers
complete. Concurrent requests and event-loop threads retain their
compression. All 17 tests pass before and after the main sync.
- Not tested: live image-model/ONNX inference, provider traffic, Python
3.10 runtime, full repository test suite, or production load. Hosted
qualification is pending.

## Runtime Rollout Safety

- Rollout-managed feature(s): no new feature or rollout gate;
image-worker lifecycle correction.
- Minimum rollout channel: existing image-compression configuration.
- Stable/default behavior changed: a failed call no longer destroys a
pool still used by peers; abandoned pending work is cancelled
best-effort.
- Kill switch / disable path: existing image-compression configuration;
no new switch.
- Unsafe override required: no.
- Qualification impact: focused process-lifecycle regressions added;
hosted CI remains required.
- Rollback path: revert the implementation and regression commits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

These readiness statements describe the maintainer review of the current
combined tree, not an assertion of contributor human review or completed
hosted CI. Final approval remains pending hosted qualification.

## Additional Notes

Original contribution attribution: OpenCode using Autoloops wrote the
AI-assisted patch. The original automated harness reported 11 passing
tests and one failure with the patch removed, based on
8d5dde7765. Subsequent maintainer reviews
requested cancellation and isolation corrections; the current results
above supersede that earlier test count. No contributor logic was
modified during the maintainer main sync. Nothing has been merged.

---------

Co-authored-by: nandanadileep <116407139+nandanadileep@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Nandana Dileep <nandanadileep@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-03 23:19:54 -06:00
gglucassandClaude Opus 5.5 793bb85659 fix(proxy): kill the image worker a timed-out call abandons (#3940)
## Description

When an image compression call times out, crashes or raises,
`run_image_compression_isolated` forwards the original payload and calls
`_reset_image_pool()`, which drops the pool with `shutdown(wait=False,
cancel_futures=True)`. That never stops a worker that is still running
the abandoned call. The worker runs the image to the end and its result
is discarded, while the next image request starts a fresh worker beside
it that cold-loads the ONNX models (technique router, SigLIP, RapidOCR)
again.

On a loaded machine this feeds itself: model loads make the next call
slower, so it times out too and leaves another worker behind. On one
macOS proxy I watched four such workers at about one core each, with 6
to 8 worker cold starts per 10 minutes and the machine at load average
140. That proxy logged 32 image worker timeouts that day.

`_reset_image_pool` now kills the dropped pool's worker. It takes the
pool the failed call ran on, and only unpublishes it if it is still the
current one. A call that fails after another request has already rebuilt
the pool leaves the rebuilt pool and its worker alone.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/image_isolation.py`: `_reset_image_pool(pool=None)`
reads the pool's worker processes before `shutdown()` (which sets
`_processes` to `None`) and kills them after it. It clears `_IMAGE_POOL`
only when that is still the pool being reset.
`run_image_compression_isolated` passes the pool it submitted to. Adds a
`_hang_worker` test hook next to the existing `_sleep_worker` /
`_abort_worker`.
- `tests/test_image_compression_isolation.py`:
`test_worker_timeout_kills_the_abandoned_worker` warms the pool, times
out a hanging call and checks the original messages come back and the
worker is dead. `test_late_failure_leaves_a_rebuilt_pool_alone` resets a
stale pool after a rebuild and checks the rebuilt pool is still
published.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Before the fix (upstream/main d5318ac2), new tests only:
```text
FAILED tests/test_image_compression_isolation.py::test_worker_timeout_kills_the_abandoned_worker
FAILED tests/test_image_compression_isolation.py::test_late_failure_leaves_a_rebuilt_pool_alone
========================= 2 failed, 5 passed in 7.79s ==========================
```

With the fix:
```text
$ uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py tests/test_image_compressor_singleton_reuse.py
============================== 14 passed in 3.71s ==============================
$ uv run ruff check headroom/proxy/image_isolation.py tests/test_image_compression_isolation.py
All checks passed!
$ uv run mypy headroom/proxy/image_isolation.py
Success: no issues found in 1 source file
```

Every test file that touches the image stack or the proxy's image path,
with the `image` extra:
```text
$ uv run --extra dev --extra image pytest tests/test_image_*.py tests/test_onnx_router_threads.py tests/test_builtin_compressor_adapters.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handler_helpers.py
226 passed, 2 skipped, 1 warning in 44.97s
```
The two skips are the tree-sitter and trafilatura adapters, which aren't
installed here. The full suite did not finish locally within 25 minutes,
so CI covers the rest.

## Real Behavior Proof

- Environment: macOS 27 (M2, 8 cores), Python 3.11.13, `uv run --extra
dev --extra image`, upstream/main d5318ac2 vs this branch.
- Exact command / steps: a script calls the real
`run_image_compression_isolated` with the real
`_compress_messages_worker` on a 1 MB PNG screenshot, three times, each
with `timeout=0.5`. That is shorter than the worker's cold model load,
so every call times out, as it does on an overloaded machine. Two
seconds later the script counts `multiprocessing.active_children()`.
  ```python
  for _ in range(3):
out, res = await m.run_image_compression_isolated(msgs, "anthropic",
timeout=0.5)
      assert out is msgs and res is None
  time.sleep(2)
  print(len(multiprocessing.active_children()))
  ```
- Observed result: upstream leaves one live worker behind per timeout;
with the fix none survive, and every call still returns the original
messages (output below).
  ```text
  upstream: {"timeouts": 3, "workers_still_alive": 3}
  branch:   {"timeouts": 3, "workers_still_alive": 0}
  ```
Upstream leaves one live worker per timeout, each still loading models
for a call whose result is gone. With the fix every timed-out call's
worker is gone, and all three calls still return the original messages.
- Live: the same change, vendored into a desktop build on headroom-ai
0.39.0 behind real Claude Code traffic, logged 5 image worker timeouts
between 16:34 and 16:41, and no check found more than one worker alive.
- Not tested: Linux and Windows. `Process.kill()` is SIGKILL on POSIX
and `TerminateProcess` on Windows, and the existing `_abort_worker` test
already covers a pool whose worker died. The `BrokenProcessPool` and
generic-exception branches go through the same reset and were not
exercised separately for real. The race covered by
`test_late_failure_leaves_a_rebuilt_pool_alone` was tested at unit level
only.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, not rollout-managed.
- Stable/default behavior changed: only after an image compression call
has already failed (timeout, worker crash, exception). The request still
forwards its original image payload as before. The abandoned worker is
now killed instead of left running. Successful calls are untouched. A
request queued behind the killed worker on the same pool now gets
`BrokenProcessPool` and forwards its original payload right away,
instead of waiting out its own timeout behind the abandoned call. Live,
that shows up as one `worker crashed` warning right after a `worker
timed out`.
- Kill switch / disable path: none. This is the failure path only.
- Unsafe override required: no.
- Qualification impact: none on compression or cache. Less CPU after
image timeouts.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` - it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 18:58:09 -07:00
Rod BoevandJerrettDavis 09e72125b4 fix(proxy): isolate image compression in a subprocess so a native SIGSEGV can't take down the proxy (#2107) (#2162)
## Description

On Apple Silicon (arm64) macOS the proxy hard-crashes with SIGSEGV the
moment it compresses an image, and because the proxy is the single API
endpoint for every routed client
(`ANTHROPIC_BASE_URL=http://127.0.0.1:8787`), one image request takes
down every agent on the machine at once — they then fail with
`ConnectionRefused` and retry into a closed port until the proxy is
manually restarted.

The faulting stack is inside OpenCV's KleidiCV ARM NEON resize
(`kleidicv::neon::kleidicv_resize_generic_stripe_u8`), reached from the
SigLIP ONNX image encoder during `ImageCompressor.compress()`. The proxy
runs that call on a `ThreadPoolExecutor`
(`headroom/proxy/server.py:946`), and both handler call sites
(`headroom/proxy/handlers/anthropic.py:1148-1172`,
`headroom/proxy/handlers/openai.py:2274-2296`) wrap it in `try/except
Exception` intending to fail open. That guard cannot help: a native
SIGSEGV is not a Python exception, and a segfault on any worker thread
aborts the whole interpreter. Thread isolation is not crash isolation.

The defect Headroom owns is that an optional, best-effort, native-heavy
transform runs in-process with no crash boundary, so any native fault in
it is fatal to the proxy and to every unrelated client it fronts. This
change gives image compression a real crash boundary by executing it in
a spawned subprocess, so a native crash degrades to "image forwarded
uncompressed" instead of killing the proxy.

Closes #2107.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `headroom/proxy/image_isolation.py` with
`run_image_compression_isolated(messages, provider, *, timeout) ->
tuple[list[dict], dict | None]`, which runs an
`ImageCompressor.compress` worker inside a lazily-created module-level
`ProcessPoolExecutor(max_workers=1)` on a **spawn** multiprocessing
context (ONNX sessions are not fork-safe) and carries the compression
result (technique, `savings_percent`, token counts) back across the
process boundary. The native OpenCV/KleidiCV work now runs in the child
address space.
- Made the runner fail open for **any** child outcome:
`BrokenProcessPool` (the class raised when the child is killed by a
signal, i.e. SIGSEGV/SIGABRT), `TimeoutError`, or any other `Exception`
all return `(messages, None)` — the original `messages` unchanged, no
telemetry — and reset the pool so the next request re-spawns a fresh
child.
- Routed the two request-path image-compression sites
(`handlers/anthropic.py`, `handlers/openai.py`) through the runner,
keeping the existing `ImageCompressionDecision` gate and the
`image_compression` mutation tag, and emitting the savings `INFO` log
line from the runner's returned result on the success path.
- Scoped strictly to crash containment: `config.image_optimize` default
is unchanged, no dependency pins, no new env switches.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_image_compression_isolation.py
tests/test_image_compression_offload.py`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_image_compression_isolation.py tests/test_image_compression_offload.py -q
.......                                                                   [100%]
7 passed in 1.73s
```

The reproduction test (`test_worker_sigsegv_fails_open_parent_survives`)
spawns a real subprocess whose worker dies by signal (`os.abort()`),
then asserts `run_image_compression_isolated` returns the original
messages and that the parent test process is still alive and continues
past the call.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uv run`; no live provider.
- Exact command / steps: `uv run pytest
tests/test_image_compression_isolation.py
tests/test_image_compression_offload.py -q`, which drives the runner
against a real spawned subprocess that is killed by signal, one that
raises, one that times out, and one that returns normally, and also
locks the handler wiring to `run_image_compression_isolated(...)`.
- Observed result: on a signal-killed child the runner returns the
original message list and the parent survives; on a raising or
timing-out child it fails open the same way; on a normal child the
compressed messages are returned. The handler source-level regression
keeps the savings log and mutation-tag path wired through the new
isolation helper. Before the change, the handlers called
`compressor.compress(...)` through `_run_compression_in_executor(...)`,
so a native crash on that worker thread would abort the interpreter.
- Not tested: live arm64 macOS run against a real `opencv-python` 5.x
KleidiCV fault.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Non-goals, kept out deliberately to keep this slice shippable now:
pinning `opencv-python<5` (a transitive-dependency change that only
masks this one fault and does not contain the next native crash), a
`HEADROOM_IMAGE_OPTIMIZE=0` env off-switch (distinct config surface;
`config.image_optimize=False` already disables the feature), the
per-request ONNX model reload, and the negative-savings (`preserve`
logged as `-100%`) reporting bug. The last two are independent defects
noted in the same report and are better fixed on their own.
- `mypy` left unchecked: not part of the focused validation for this
change.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-14 11:52:20 -04:00