mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(onnx): stop magika's ORT intra-op pool from spinning idle cores (#4189)
## Description The Rust Magika detector (`crates/headroom-core/src/transforms/magika_detector.rs`) built its ONNX session with `Session::new()`. ONNX Runtime then sized the intra-op pool to the core count, and those threads spin-wait between inferences. The proxy calls this detector for every content-routing decision through `headroom._core.detect_content_type`, so under steady light load the pool burns roughly 2 cores. #2540 fixed this for the Python sessions by setting `session.intra_op.allow_spinning=0`, but the `magika` 1.1.0 `Builder` doesn't expose session config entries, so that key can't be set here. The fix builds the session with `Session::builder().with_intra_threads(1)`. With one intra-op thread, ONNX Runtime doesn't create a pool at all, so nothing is left to spin. The inter-op pool only exists in parallel execution mode, which isn't used here, so it needs no change. I confirmed that adding `with_inter_threads(1)` changes nothing. Closes #4187 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `magika_detector.rs`: replace `Session::new()` with `Session::builder().with_intra_threads(1).build()`, with a one-line comment explaining why. - `tests/magika_idle_threads.rs`: new regression test. It counts OS threads (`/proc/self/task` on Linux, `ps -M` on macOS) before and after the first `magika_detect` call and asserts that at most 2 new threads remain. Those 2 are ORT's always-on telemetry worker and, on macOS, a libdispatch worker; both are blocked rather than spinning. The test is in its own file so it runs in its own process, because the session is a process-global singleton. It skips when the ONNX Runtime library isn't available, matching the existing magika tests. ## Testing - [x] Rust: `cargo fmt --all -- --check`, `cargo clippy --workspace -- -D warnings`, `cargo test --workspace` (with `ORT_DYLIB_PATH` set so the magika tests actually run) - [ ] Unit tests pass (`pytest`): only the pre-push `ci-precheck-python` subset was run (177 passed, 4 skipped); the full suite was not - [x] Linting passes (`ruff check .`): no Python files changed; pre-commit hooks passed - [ ] Type checking passes (`mypy headroom`): N/A, no Python changes - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output New test: fails on `main`, passes with the fix (macOS arm64, 16 cores, onnxruntime 1.29 dylib): ```text # main magika session left 9 extra OS thread(s) running (before=2, after=11) test result: FAILED. 0 passed; 1 failed # this branch test result: ok. 1 passed; 0 failed ``` The same test on Linux (Docker `rust:1.95-bookworm`, aarch64, 8 CPUs, pip onnxruntime 1.31.0): ```text # main magika session left 8 extra OS thread(s) running (before=2, after=10) test result: FAILED. 0 passed; 1 failed # this branch test result: ok. 1 passed; 0 failed ``` Full Rust gate on this branch (`make ci-precheck` via the pre-push hook): ```text cargo test --workspace: 68 suites, 1658 passed, 0 failed, 3 ignored ci-precheck-python: 177 passed, 4 skipped ✅ ci-precheck PASSED — safe to push. ``` ## Real Behavior Proof **1. Linux: CPU under light load (reproduces the issue's ~180%)** - Environment: Docker `rust:1.95-bookworm` on Apple Silicon (Linux 6.10 aarch64, 8 CPUs), `pip install onnxruntime` (1.31.0), `ORT_DYLIB_PATH` pointed at the pip library. - Exact command / steps: a release-build probe binary calls `headroom_core::transforms::magika_detector::magika_detect` on a ~5.6 KB Python snippet, first 200 times back-to-back (latency), then once every 50 ms for 10 s ("trickle", like a proxy under light load). Process CPU was read from `/proc/<pid>/stat` (utime+stime) over a 9 s window during the trickle, and threads from `/proc/<pid>/task`. Built from `main` and from this branch in the same container. - Observed result: on `main` the probe ran at ~191% of one core during the trickle; on this branch ~24%. Threads 9 → 2, idle CPU ~0% in both, back-to-back latency 2.4 → 8.4 ms. Full table: | | `main` | this branch | |---|---|---| | OS threads after init | 9 | 2 | | CPU during 50 ms trickle | **~191% of one core** | **~24% of one core** | | CPU over 10 s fully idle | ~0% | ~0% | | Average detection latency (back-to-back) | 2.4 ms | 8.4 ms | **2. macOS: the real proxy (`headroom proxy`) on the `/v1/compress` path** - Environment: macOS 26 arm64 (16 cores), Python 3.12 `.venv` from `uv sync --extra dev --extra proxy`, onnxruntime 1.26.0. Extension built from `main`, then from this branch, with `uv pip install -e .`. - Exact command / steps: `headroom proxy --port 18787 --code-aware` with a throwaway `HOME`. A client sent one `/v1/compress` request every 200 ms, each with a unique ~6 KB code tool result from a non-excluded tool. The router log confirmed `cache_miss` / `compressed=1`, so the native detector ran. During the load, `sample <pid> 10 1` was taken and top-of-stack samples were summed per library: `libonnxruntime.*.dylib` (Rust `ort`) vs `onnxruntime_pybind11_state.so` (Python ORT, i.e. Kompress). - Observed result: Rust-ORT worker-pool threads 7 → 1, CPU inside the Rust ORT dylib 0.11 → 0.00 cores, `/v1/compress` p50 latency unchanged (840 vs 846 ms). Full table: | | `main` | this branch | |---|---|---| | Rust-ORT worker-pool threads (stacks in `ThreadPoolTempl::WorkerLoop`) | 7 | 1 (the calling thread) | | Proxy OS threads after warm-up | 44 | 37 | | CPU inside the Rust ORT dylib during load | 0.11 cores, mostly `SpinPause` / `WorkerLoop` | 0.00 cores | | `/v1/compress` p50 latency | 840 ms | 846 ms (noise; dominated by Kompress) | On macOS the spin is smaller than on Linux, but the profile shows the same pool spinning in `SpinPause`, and with the fix it's gone. - Not tested: x86_64 Linux, Windows, the full proxy on Linux, very high request rates, very low-core hosts, and `--no-magika`. Details: - x86_64 Linux, and Windows (Windows defaults to the pure-Python detector, so it's unaffected unless `HEADROOM_DETECT_BACKEND=rust` is set). - The full proxy on Linux; the Linux numbers come from the detector probe, not the proxy. - Very high request rates where per-call latency could matter. - Very low-core hosts (1–2 cores): the regression test allows 2 idle background threads, so a 1–2 worker pool can fit inside that allowance there. CI's 4-vCPU runner is above it. - The optional `--no-magika` flag suggested in the issue, which is left out of scope. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A - Stable/default behavior changed: the Rust Magika session now uses one intra-op thread. Detection results are unchanged; single-call latency rises (about 2.4 → 8.4 ms on Linux aarch64, about 1.3 → 1.9 ms on macOS) in exchange for not spinning idle cores. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` already bypasses the native detector. - Unsafe override required: no. - Qualification impact: none expected; well within the <50 ms transform budget. - Rollback path: revert this commit. ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (none needed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, which is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Trade-off: one intra-op thread makes each Magika call slower (+~6 ms worst case measured), but stops ~2 cores of spin under light load. A fuller fix would set `allow_spinning=0` while keeping a pool, which needs the `magika` crate to expose session config entries; that would be an upstream change and a dependency bump. - `--no-magika` from the issue isn't included; it's a feature and should be raised separately per CONTRIBUTING. - This change was developed with GitHub Copilot CLI assistance; all measurements above were run locally by me. --------- Co-authored-by: Romin Parekh <rominparekh@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Romin Parekh
JerrettDavis
parent
1cf4d053d2
commit
2016dc6d60
@@ -382,7 +382,11 @@ fn session() -> &'static Mutex<Result<Session, String>> {
|
||||
let spawned = std::thread::Builder::new()
|
||||
.name("magika-init".into())
|
||||
.spawn(move || {
|
||||
let _ = tx.send(Session::new().map_err(|e| e.to_string()));
|
||||
// One intra-op thread means ORT builds no pool: pool threads
|
||||
// spin-wait between inferences, burning ~2 cores under light
|
||||
// load (#4187), and magika's builder has no allow_spinning knob.
|
||||
let built = Session::builder().with_intra_threads(1).build();
|
||||
let _ = tx.send(built.map_err(|e| e.to_string()));
|
||||
});
|
||||
if let Err(e) = spawned {
|
||||
tracing::warn!("magika init thread spawn failed: {e}");
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
//! Magika must not leave an ONNX Runtime intra-op thread pool behind (#4187).
|
||||
//!
|
||||
//! ORT's pool threads spin-wait between inferences, so a pool sized to the
|
||||
//! core count pegs idle CPUs in a long-lived proxy. The magika session is a
|
||||
//! process-global singleton and the assertion counts OS threads, so this file
|
||||
//! holds exactly ONE `#[test]` fn and must run in its own process.
|
||||
#![cfg(all(feature = "ml", any(target_os = "linux", target_os = "macos")))]
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use headroom_core::transforms::magika_detector::{magika_detect, MagikaDetectorError};
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
fn os_thread_count() -> usize {
|
||||
std::fs::read_dir("/proc/self/task")
|
||||
.expect("read /proc/self/task")
|
||||
.count()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn os_thread_count() -> usize {
|
||||
let out = std::process::Command::new("ps")
|
||||
.args(["-M", "-p", &std::process::id().to_string()])
|
||||
.output()
|
||||
.expect("run ps -M");
|
||||
// One header line, then one line per thread.
|
||||
String::from_utf8_lossy(&out.stdout).lines().count() - 1
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn magika_session_does_not_spawn_intra_op_pool() {
|
||||
let before = os_thread_count();
|
||||
match magika_detect("fn main() { println!(\"hi\"); }") {
|
||||
Ok(_) => {}
|
||||
// With an explicit ORT_DYLIB_PATH (as in CI) the runtime is present,
|
||||
// so an init failure is a real session-build regression, not a skip.
|
||||
Err(MagikaDetectorError::Init(reason))
|
||||
if std::env::var("ORT_DYLIB_PATH").map_or(true, |v| v.trim().is_empty()) =>
|
||||
{
|
||||
eprintln!("skipping: magika runtime unavailable on this host: {reason}");
|
||||
return;
|
||||
}
|
||||
Err(e) => panic!("magika detection failed: {e}"),
|
||||
}
|
||||
// Let the short-lived `magika-init` thread finish exiting.
|
||||
std::thread::sleep(Duration::from_millis(200));
|
||||
let after = os_thread_count();
|
||||
// ORT and macOS may leave up to 2 idle background threads (ORT telemetry,
|
||||
// libdispatch); neither spins. A pool adds its workers on top (7 on a
|
||||
// 16-core Mac), but on very small hosts a 1-2 worker pool can hide inside
|
||||
// this allowance, so a pass there is not proof of no spinning.
|
||||
let extra = after.saturating_sub(before);
|
||||
assert!(
|
||||
extra <= 2,
|
||||
"magika session left {extra} extra OS thread(s) running (before={before}, after={after})"
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user