fix(onnx): stop magika's ORT intra-op pool from spinning idle cores (#4189)

## Description

The Rust Magika detector
(`crates/headroom-core/src/transforms/magika_detector.rs`) built its
ONNX session with `Session::new()`. ONNX Runtime then sized the intra-op
pool to the core count, and those threads spin-wait between inferences.
The proxy calls this detector for every content-routing decision through
`headroom._core.detect_content_type`, so under steady light load the
pool burns roughly 2 cores. #2540 fixed this for the Python sessions by
setting `session.intra_op.allow_spinning=0`, but the `magika` 1.1.0
`Builder` doesn't expose session config entries, so that key can't be
set here.

The fix builds the session with
`Session::builder().with_intra_threads(1)`. With one intra-op thread,
ONNX Runtime doesn't create a pool at all, so nothing is left to spin.
The inter-op pool only exists in parallel execution mode, which isn't
used here, so it needs no change. I confirmed that adding
`with_inter_threads(1)` changes nothing.

Closes #4187

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `magika_detector.rs`: replace `Session::new()` with
`Session::builder().with_intra_threads(1).build()`, with a one-line
comment explaining why.
- `tests/magika_idle_threads.rs`: new regression test. It counts OS
threads (`/proc/self/task` on Linux, `ps -M` on macOS) before and after
the first `magika_detect` call and asserts that at most 2 new threads
remain. Those 2 are ORT's always-on telemetry worker and, on macOS, a
libdispatch worker; both are blocked rather than spinning. The test is
in its own file so it runs in its own process, because the session is a
process-global singleton. It skips when the ONNX Runtime library isn't
available, matching the existing magika tests.

## Testing

- [x] Rust: `cargo fmt --all -- --check`, `cargo clippy --workspace --
-D warnings`, `cargo test --workspace` (with `ORT_DYLIB_PATH` set so the
magika tests actually run)
- [ ] Unit tests pass (`pytest`): only the pre-push `ci-precheck-python`
subset was run (177 passed, 4 skipped); the full suite was not
- [x] Linting passes (`ruff check .`): no Python files changed;
pre-commit hooks passed
- [ ] Type checking passes (`mypy headroom`): N/A, no Python changes
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

New test: fails on `main`, passes with the fix (macOS arm64, 16 cores,
onnxruntime 1.29 dylib):

```text
# main
magika session left 9 extra OS thread(s) running (before=2, after=11)
test result: FAILED. 0 passed; 1 failed
# this branch
test result: ok. 1 passed; 0 failed
```

The same test on Linux (Docker `rust:1.95-bookworm`, aarch64, 8 CPUs,
pip onnxruntime 1.31.0):

```text
# main
magika session left 8 extra OS thread(s) running (before=2, after=10)
test result: FAILED. 0 passed; 1 failed
# this branch
test result: ok. 1 passed; 0 failed
```

Full Rust gate on this branch (`make ci-precheck` via the pre-push
hook):

```text
cargo test --workspace: 68 suites, 1658 passed, 0 failed, 3 ignored
ci-precheck-python: 177 passed, 4 skipped
✅ ci-precheck PASSED — safe to push.
```

## Real Behavior Proof

**1. Linux: CPU under light load (reproduces the issue's ~180%)**

- Environment: Docker `rust:1.95-bookworm` on Apple Silicon (Linux 6.10
aarch64, 8 CPUs), `pip install onnxruntime` (1.31.0), `ORT_DYLIB_PATH`
pointed at the pip library.
- Exact command / steps: a release-build probe binary calls
`headroom_core::transforms::magika_detector::magika_detect` on a ~5.6 KB
Python snippet, first 200 times back-to-back (latency), then once every
50 ms for 10 s ("trickle", like a proxy under light load). Process CPU
was read from `/proc/<pid>/stat` (utime+stime) over a 9 s window during
the trickle, and threads from `/proc/<pid>/task`. Built from `main` and
from this branch in the same container.
- Observed result: on `main` the probe ran at ~191% of one core during
the trickle; on this branch ~24%. Threads 9 → 2, idle CPU ~0% in both,
back-to-back latency 2.4 → 8.4 ms. Full table:

| | `main` | this branch |
|---|---|---|
| OS threads after init | 9 | 2 |
| CPU during 50 ms trickle | **~191% of one core** | **~24% of one
core** |
| CPU over 10 s fully idle | ~0% | ~0% |
| Average detection latency (back-to-back) | 2.4 ms | 8.4 ms |

**2. macOS: the real proxy (`headroom proxy`) on the `/v1/compress`
path**

- Environment: macOS 26 arm64 (16 cores), Python 3.12 `.venv` from `uv
sync --extra dev --extra proxy`, onnxruntime 1.26.0. Extension built
from `main`, then from this branch, with `uv pip install -e .`.
- Exact command / steps: `headroom proxy --port 18787 --code-aware` with
a throwaway `HOME`. A client sent one `/v1/compress` request every 200
ms, each with a unique ~6 KB code tool result from a non-excluded tool.
The router log confirmed `cache_miss` / `compressed=1`, so the native
detector ran. During the load, `sample <pid> 10 1` was taken and
top-of-stack samples were summed per library: `libonnxruntime.*.dylib`
(Rust `ort`) vs `onnxruntime_pybind11_state.so` (Python ORT, i.e.
Kompress).
- Observed result: Rust-ORT worker-pool threads 7 → 1, CPU inside the
Rust ORT dylib 0.11 → 0.00 cores, `/v1/compress` p50 latency unchanged
(840 vs 846 ms). Full table:

| | `main` | this branch |
|---|---|---|
| Rust-ORT worker-pool threads (stacks in `ThreadPoolTempl::WorkerLoop`)
| 7 | 1 (the calling thread) |
| Proxy OS threads after warm-up | 44 | 37 |
| CPU inside the Rust ORT dylib during load | 0.11 cores, mostly
`SpinPause` / `WorkerLoop` | 0.00 cores |
| `/v1/compress` p50 latency | 840 ms | 846 ms (noise; dominated by
Kompress) |

On macOS the spin is smaller than on Linux, but the profile shows the
same pool spinning in `SpinPause`, and with the fix it's gone.

- Not tested: x86_64 Linux, Windows, the full proxy on Linux, very high
request rates, very low-core hosts, and `--no-magika`. Details:
- x86_64 Linux, and Windows (Windows defaults to the pure-Python
detector, so it's unaffected unless `HEADROOM_DETECT_BACKEND=rust` is
set).
- The full proxy on Linux; the Linux numbers come from the detector
probe, not the proxy.
  - Very high request rates where per-call latency could matter.
- Very low-core hosts (1–2 cores): the regression test allows 2 idle
background threads, so a 1–2 worker pool can fit inside that allowance
there. CI's 4-vCPU runner is above it.
- The optional `--no-magika` flag suggested in the issue, which is left
out of scope.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A
- Stable/default behavior changed: the Rust Magika session now uses one
intra-op thread. Detection results are unchanged; single-call latency
rises (about 2.4 → 8.4 ms on Linux aarch64, about 1.3 → 1.9 ms on macOS)
in exchange for not spinning idle cores.
- Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` already
bypasses the native detector.
- Unsafe override required: no.
- Qualification impact: none expected; well within the <50 ms transform
budget.
- Rollback path: revert this commit.

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, which is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Trade-off: one intra-op thread makes each Magika call slower (+~6 ms
worst case measured), but stops ~2 cores of spin under light load. A
fuller fix would set `allow_spinning=0` while keeping a pool, which
needs the `magika` crate to expose session config entries; that would be
an upstream change and a dependency bump.
- `--no-magika` from the issue isn't included; it's a feature and should
be raised separately per CONTRIBUTING.
- This change was developed with GitHub Copilot CLI assistance; all
measurements above were run locally by me.

---------

Co-authored-by: Romin Parekh <rominparekh@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Romin Parekh
2026-10-13 16:23:42 -07:00
committed by GitHub
co-authored by Romin Parekh JerrettDavis
parent 1cf4d053d2
commit 2016dc6d60
2 changed files with 62 additions and 1 deletions
@@ -382,7 +382,11 @@ fn session() -> &'static Mutex<Result<Session, String>> {
let spawned = std::thread::Builder::new()
.name("magika-init".into())
.spawn(move || {
let _ = tx.send(Session::new().map_err(|e| e.to_string()));
// One intra-op thread means ORT builds no pool: pool threads
// spin-wait between inferences, burning ~2 cores under light
// load (#4187), and magika's builder has no allow_spinning knob.
let built = Session::builder().with_intra_threads(1).build();
let _ = tx.send(built.map_err(|e| e.to_string()));
});
if let Err(e) = spawned {
tracing::warn!("magika init thread spawn failed: {e}");
@@ -0,0 +1,57 @@
//! Magika must not leave an ONNX Runtime intra-op thread pool behind (#4187).
//!
//! ORT's pool threads spin-wait between inferences, so a pool sized to the
//! core count pegs idle CPUs in a long-lived proxy. The magika session is a
//! process-global singleton and the assertion counts OS threads, so this file
//! holds exactly ONE `#[test]` fn and must run in its own process.
#![cfg(all(feature = "ml", any(target_os = "linux", target_os = "macos")))]
use std::time::Duration;
use headroom_core::transforms::magika_detector::{magika_detect, MagikaDetectorError};
#[cfg(target_os = "linux")]
fn os_thread_count() -> usize {
std::fs::read_dir("/proc/self/task")
.expect("read /proc/self/task")
.count()
}
#[cfg(target_os = "macos")]
fn os_thread_count() -> usize {
let out = std::process::Command::new("ps")
.args(["-M", "-p", &std::process::id().to_string()])
.output()
.expect("run ps -M");
// One header line, then one line per thread.
String::from_utf8_lossy(&out.stdout).lines().count() - 1
}
#[test]
fn magika_session_does_not_spawn_intra_op_pool() {
let before = os_thread_count();
match magika_detect("fn main() { println!(\"hi\"); }") {
Ok(_) => {}
// With an explicit ORT_DYLIB_PATH (as in CI) the runtime is present,
// so an init failure is a real session-build regression, not a skip.
Err(MagikaDetectorError::Init(reason))
if std::env::var("ORT_DYLIB_PATH").map_or(true, |v| v.trim().is_empty()) =>
{
eprintln!("skipping: magika runtime unavailable on this host: {reason}");
return;
}
Err(e) => panic!("magika detection failed: {e}"),
}
// Let the short-lived `magika-init` thread finish exiting.
std::thread::sleep(Duration::from_millis(200));
let after = os_thread_count();
// ORT and macOS may leave up to 2 idle background threads (ORT telemetry,
// libdispatch); neither spins. A pool adds its workers on top (7 on a
// 16-core Mac), but on very small hosts a 1-2 worker pool can hide inside
// this allowance, so a pass there is not proof of no spinning.
let extra = after.saturating_sub(before);
assert!(
extra <= 2,
"magika session left {extra} extra OS thread(s) running (before={before}, after={after})"
);
}