22 KiB
Shared Transcription Engine Service (hex engine)
Status: Measured spike on engine-service; plan ready.
Executive Verdict
- Does this work? Yes. Once
LocalApi::servepolls its listener withpoll(2)instead of sleeping 10 ms between accepts, loopback HTTP (127.0.0.1+ Bearer token) adds 0.12 ms on/health, 0.28 ms per 100 ms/streamsaudio push, 0.39 ms median SSE partial latency, and 1.4–3.3 ms on whole-WAV/transcriptions. Streaming/streams/:id/finishcompletes in 44.0 ms over HTTP vs 53.0 ms in-processLiveStream::finish_with. - Is there any problem with it being a background service? One real OS trap, with a verified fix: macOS clamps default background daemons (
TASK_BACKGROUND_APPLICATION/taskpolicy -b) to E-cores even when the inference thread setsQOS_CLASS_USER_INTERACTIVE, slowing quiet transcription 1.7–2.1× (46.5 ms → 76.7–105.5 ms) and contended transcription up to 15.7× (106 ms → 1,852 ms). Running the service with interactive process policy (ProcessType = Interactiveinlaunchdor inherited from a GUI parent +NSProcessInfo.beginActivity(.userInitiatedAllowingIdleSystemSleep)) keeps Metal + CPU decoder speed identical to in-process (47.4 ms short, 121.6 ms long). - Is that dangerous? No. The engine owns no microphone, no Accessibility/Input Monitoring taps, no Screen Recording, no clipboard, and no network sockets outside
127.0.0.1. It triggers zero TCC prompts, stores no audio or history on disk, and exposes only authenticated loopback inference over caller-pushed PCM/WAV bytes.
Sharing the Engine and the Models: the Guide
There are two ways for Hex, Seer, Rune Desktop and OpenCode Desktop to share speech recognition. Either each app links the engine as a library, or one engine process serves them all.
library in every app one engine process
├── 4 copies of the scheduler ├── 1 scheduler: dictation first, always
├── ~150 MiB Metal + weights per app ├── ~250 MiB once (model file pages are shared either way)
├── apps fight over the GPU, no order ├── one queue across apps
├── 4 builds to update for a fix ├── 1 binary to update
└── each app can download models └── 1 writer for the model folder
The library route can't keep dictation first, because priority only exists
inside one process. One engine process is the right shape. It's how macOS
runs its own speech service and how Ollama serves local models. Each app links
only a thin client (@kitlangton/hex, Seer's hex-client).
Who owns what
- Hex.app ships the engine. The
hex enginebinary exists once, insideHex.app, signed and updated with it. Other apps don't bundle a copy: they find Hex by its bundle idanomaly.hex, and without it they say "Install Hex for transcription" (Seer already falls back like this). Bundling copies in every app means four signatures, version handoffs and newest-wins rules; revisit only if a Seer-only install becomes a real need. - The engine owns the models. One folder,
~/Library/Application Support/hex/models/, checksum-pinned. Only the engine downloads, verifies, loads or deletes models. Apps never open model files; they ask (GET /models,POST /models/:id/prepare) and send audio. - Each app owns its microphone, permissions and UI. The engine records nothing, so it never asks for a permission.
Which model runs: the engine keeps one model loaded, the one the user
picked in Hex. A request that names no model uses it. A request that names
another model waits until the engine is idle, loads it beside the current one,
then switches (the ModelSwap that Linux and Windows already use). Dictation
always uses the user's pick.
Making it interactive is two small, fixed steps, not an ongoing risk.
macOS throttles a process only when it's marked as background. The engine
clears that mark when it starts (setpriority(PRIO_DARWIN_PROCESS, 0, 0))
and holds a "user-initiated" activity so App Nap leaves it alone. Under
launchd the same thing is one plist key, ProcessType = Interactive. A test
asserts the engine isn't background, and /health reports it, so a regression
shows up immediately.
Starting it: the first app that needs the engine launches
Hex.app/Contents/MacOS/hex engine. The engine takes engine.lock, so a
second launch exits, then writes local-api.json. Every app reads that file
to find the port and token. Hex.app starts it at launch, so it's normally
already running. There's no login item and no "Background item added" notice.
Measured Spike (Mac Studio M5 Max, 36 GiB, parakeet-unified-en-0.6b-Q8_0.gguf)
Fixtures: term-hex-3.wav (2.64 s, 84.5 KB) and 5 concatenated vocabulary clips (13.08 s, 418.6 KB).
| Metric | Measured | Notes |
|---|---|---|
| Process memory before model load | 6.2 MiB phys_footprint (16.0 MiB RSS) |
LocalApi listening, WarmTranscriber empty |
| Process memory with Parakeet 0.6B loaded & warm | 249.3–252.7 MiB phys_footprint (199.3–203.0 MiB RSS) |
697 MiB of the 731 MB GGUF is clean read-only mmap wrapped via MTLBuffer (bytesNoCopy, 0 KiB dirty); dirty footprint is 117.1 MiB Metal driver/pipelines + 31.4 MiB fused BN weights + 50 MiB malloc cache |
In-process model unload (*model = WarmTranscriber::default()) |
0.84 ms; drops to 213.3–222.4 MiB phys_footprint (153.7–167.3 MiB RSS) |
Unmaps the 697 MiB GGUF + frees 31 MiB BN buffers; ggml-metal's process-global g_device (117 MiB) stays resident until process exit |
| Cold load (first run of a new build, cold Metal shader cache) | 3,665–4,483 ms (3,322–3,453 ms Metal init + 76–501 ms prewarm + 267–528 ms stream probe) |
MTLCompilerService compiles ggml-metal kernels once into the Darwin user shader cache |
| Cold process start (warm OS Metal shader cache) | 386.5–585.0 ms (178 ms Metal load + 89 ms offline prewarm + 317 ms stream probe) |
Fresh child hex engine from fork/exec to HTTP ready |
| Warm in-process reload (after idle unload) | 301–308 ms total (19–25 ms load_from + 33–39 ms offline prewarm + 247 ms stream probe) |
52–64 ms ready for offline inference; ggml-metal pipelines stay compiled in g_device |
GET /health over loopback HTTP |
122.3 µs median (145.0 µs p95) | Was 12,522.5 µs (12.5 ms) before replacing thread::sleep(10ms) with libc::poll in LocalApi::serve |
POST /transcriptions — 2.64 s WAV |
46.46 ms in-proc vs 47.83 ms HTTP (+1.37 ms) | Includes TCP connect, Bearer auth, WAV parse, JSON reply |
POST /transcriptions — 13.08 s WAV |
106.08 ms in-proc vs 109.42 ms HTTP (+3.34 ms) | Includes 418.6 KB loopback upload + WAV decode |
/streams — 13.08 s paced (100 ms / 6.4 KB chunks) |
Create 0.31 ms; audio push 282.9 µs med (357.8 µs p95); SSE partial 387.5 µs med (541.4 µs p95); /finish 44.02 ms med (vs 53.01 ms in-proc LiveStream) |
Pause-cut windows commit during speech so /finish only decodes the tail |
QoS: Interactive vs taskpolicy -b (quiet) |
Short: 46.5 ms vs 76.7–105.5 ms (1.7–2.2× slower); Long: 106.1 ms vs 200.6–204.9 ms (1.9× slower) | TASK_BACKGROUND_APPLICATION clamps threads to E-cores even when pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE) succeeds |
QoS: Interactive vs taskpolicy -b (12-thread CPU load) |
Long: 111.6–126.1 ms med (133.1 ms max) vs 272.7–808.5 ms med (1,851.9 ms max, up to 15.7× slower) | Proves background task policy is unacceptable for interactive dictation |
SMAppService.agent (ProcessType = Interactive, Aqua) |
Short: 47.41 ms med (47.84 ms p95); Long: 121.58 ms med (PID 72058, status = Enabled) |
Matches foreground in-process speed within 1 ms; unregistered cleanly (status = NotRegistered) |
1. Service Lifecycle on macOS
| Option | Pros | Cons |
|---|---|---|
SMAppService.agent(plistName:) in Hex.app/Contents/Library/LaunchAgents/ |
launchd supervises crashes (KeepAlive.SuccessfulExit = false) and applies ProcessType = Interactive + LimitLoadToSessionType = Aqua. |
Verified via sfltool dumpbtm: macOS posts a "Background Items Added" notification (Disposition: [enabled, allowed, notified]) and adds a separate toggle in System Settings that breaks transcription if turned off (RequiresApproval). Also requires Hex.app to be installed, failing in Seer-only / Rune-only installs. |
SMAppService.mainApp Login Item |
Already used by Hex.app for Launch at Login. |
Starts the full Hex.app menu-bar app, not a headless engine for Seer-only users. |
On-demand user-session daemon (hex engine) coordinated by engine.lock + local-api.json (Recommended) |
Zero BTM "Background Items Added" notifications; works identically whether the user has Hex.app, only Seer, only OpenCode Desktop, or all four; inherits foreground interactive task policy when spawned by a GUI app. |
Client SDK/host helper respawns the daemon on connection failure instead of launchd KeepAlive. |
Recommendation: Use an on-demand detached user daemon (hex engine) coordinated through ~/Library/Application Support/hex/engine.lock and local-api.json, while Hex.app keeps its existing SMAppService.mainApp Login Item.
client.connectOrStartEngine()
├── read ~/Library/Application Support/hex/local-api.json -> GET /health (150 ms)
│ └── 200 OK & compatible apiVersion -> return endpoint
└── missing, stale, or refused
├── spawn `<bundled-or-installed-hex> engine` detached (setsid, stdio -> /dev/null)
├── engine acquires flock(engine.lock, LOCK_EX | LOCK_NB) (losers exit 0)
├── engine binds 127.0.0.1:0, writes 0600 local-api.json atomically, serves
└── client polls local-api.json until healthy (budget: 2 s)
- Seer-only / non-Hex installs: for now the engine ships only in
Hex.app(see the guide above); other apps launch it from there and fall back to "Install Hex" without it. If a Seer-only install becomes a real need, the same lock and discovery let an app bundle its ownhex engine: whichever app starts first becomes the single engine for the user session, andHex.appshares it when it launches later. RENAME_SWAPself-updates: On macOS,renamex_np(RENAME_SWAP)swaps the directory entry ofHex.appwhile the running engine's open executable inode andmmaped GGUF file descriptor remain valid.local-api.jsoncarriesengineVersion(crate version + git SHA); when an updatedHex.applaunches and sees an olderengineVersionin/health, it postsPOST /control {"command":"shutdown-when-idle"}(which drains any in-flight/streamsor/transcriptionsjob and exits 0) and spawns its newer bundledhex engine.
2. Transport and Authentication
Recommendation: Keep authenticated loopback HTTP (127.0.0.1:<ephemeral-port>) with 0600 local-api.json discovery and random 256-bit Bearer token, and ship the 12-line libc::poll accept fix in src/local_api.rs (plus HTTP/1.1 keep-alive for /streams/:id/audio connection reuse).
- With
libc::pollonlistener.as_raw_fd(), loopback HTTP takes 0.12 ms per control call, 0.28 ms per 100 ms audio chunk, and 0.39 ms for SSE partials—over 40× below human perceptual thresholds (~15 ms). - A Unix domain socket saves ~40 µs per call while breaking standard
fetch/EventSourcein TypeScript clients (@kitlangton/hex, Seer, OpenCode Desktop).
3. Process Priority, QoS, and Metal Access
Recommendation: Enforce interactive task + thread QoS and an App Nap activity assertion inside hex engine:
- Task policy: Never run
hex engineunderTASK_BACKGROUND_APPLICATION. At startup,hex enginecallssetpriority(PRIO_DARWIN_PROCESS, 0, 0)to clearPRIO_DARWIN_BGif inherited, and holds anNSProcessInfo.processInfo.beginActivity(options: [.userInitiatedAllowingIdleSystemSleep], reason: "Hex speech transcription engine")token for the life of the process. This prevents App Nap timer/CPU throttling while still allowing the Mac to sleep when idle. - Thread QoS: The
TranscriptionEngineworker thread callsprioritize_inference_thread()(pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0)), keeping the single-threaded Parakeet CPU token decoder on P-cores. - Metal in user session: Because
hex engineis spawned by a GUI client inside the user'sAqualogin session (or underLimitLoadToSessionType = Aqua),MTLCreateSystemDefaultDevice()has full hardware GPU access. During active inference (which already yields to dictation), meeting/background chunks continue to lower thread QoS toQOS_CLASS_UTILITYand restoreQOS_CLASS_USER_INTERACTIVEimmediately afterward.
4. Memory and Model Residency
Recommendation: Keep the engine process resident (6.2 MiB base; ~222 MiB after first Metal init), unload WarmTranscriber after 10 minutes of idle (*model = WarmTranscriber::default()), and pre-warm on dictation key-down (POST /warm).
- Why: Parakeet 0.6B Q8_0 (
731 MBGGUF) uses only 249–253 MiB dirtyphys_footprintwhile loaded becausetranscribe.cppwraps 697 MiB of cleanmmapfile pages directly in anMTLBuffer(bytesNoCopy). DroppingWarmTranscriberunmaps the 697 MiB file and releasesggml-metal's 3-minuteMTLResidencySet+ 31 MiB of fused BatchNorm buffers in 0.84 ms while keepingggml-metal's compiled pipeline cache warm in-process. - Warm reload latency: Reloading
Parakeetin-process takes 19–25 ms (load_from) + 33–39 ms (offline prewarm) = 52–64 ms (or 301 ms with the 8 s streaming probe), vs 387–585 ms to cold-start a new process. - Zero-latency dictation after idle: When the user presses the dictation shortcut in
Hex.app,Hex.appimmediately firesPOST /warm(or opens its speculativePOST /streamssession) at key-down (t = 0 ms). BecauseHex.app's intentional-hold threshold is 300 ms plus speech duration and the 200 ms post-release tail, an unloaded model reloads in 52–301 ms before the user finishes their first word.
5. Crash, Hang, and Restart Behavior
Recommendation: Stateless fail-fast recovery with retained-audio fallback in Hex.app.
- In-flight
/streamsand/transcriptions: Ifhex enginecrashes or hangs past the client's deadline, the socket closes (EOF/ECONNRESET). External SDK clients (Seer,OpenCode) receive a typedEngineDisconnectederror on that stream, callconnectOrStartEngine(), and reconnect within ~390–585 ms on the next utterance. Hex.appdictation resilience:Hex.appretains the authoritative 16 kHz PCM capture buffer in its own memory (dictation_audio) until paste completes. If the speculative/streamssession tohex enginedrops mid-capture or times out at/finish,Hex.apprespawnshex engineand submits the retained whole clip viaPOST /transcriptions(the exact same speculative-stream-to-whole-clip fallbackTranscriptionEngine::finish_withperforms today). No dictated speech is ever lost to an engine crash.- Crash loop throttle: Clients track engine spawn timestamps in-process and cap restarts at 3 attempts per 30 seconds (with
100 ms → 1 s → 5 sbackoff) before surfacingEngineUnavailable.
6. Priority and Preemption Across Processes (Dictation First)
Recommendation: Tag requests with a priority tier (dictation vs interactiveStream vs background) and keep the existing dictation_busy atomic preemption gate inside hex engine.
POST /streams { priority: "dictation" | "stream" }
├── priority: "dictation" (Hex.app hotkey / Voice Action)
│ ├── increments dictation_busy in engine -> immediately closes any active YieldingStream (Seer/OpenCode)
│ ├── claims exclusive LiveStream on TranscriptionEngine
│ └── any competing /streams push buffers in its 30 s backlog and catches up via offline window after dictation finishes
└── POST /transcriptions { priority: "dictation" | "normal" | "background" }
└── engine queue orders: dictation > stream window > normal batch > background meeting chunk
- IPC latency budget for
Hex.appdictation:- Shortcut press →
POST /streams(priority: "dictation"): 0.31 ms (runs during the 300 ms hold threshold; 0 ms added to post-release latency). - During speech →
POST /streams/:id/audioevery 100 ms: 0.28 ms per 6.4 KB chunk off the audio thread (0 ms added to post-release latency). - Shortcut release (+200 ms tail) →
POST /streams/:id/finishwith final tail samples: 44.0 ms total (of which < 0.5 ms is loopback HTTP framing/serialization).
- Shortcut press →
7. API Versioning and Client Skew
Recommendation: Single additive API_VERSION = "2" contract with capability discovery in GET /health and GET /capabilities.
local-api.jsonandGET /healthreportapiVersion: "2",engineVersion: "2.3.1+sha",pid, andcapabilities: ["transcriptions", "streams", "warm", "priority-dictation"].- Additive evolution: New fields in JSON requests/replies default cleanly (
#[serde(default)]). Incompatible breaking changes incrementapiVersion. - Version replacement rule: When
Hex.appor a newer SDK client starts and sees a running engine whoseengineVersionis older than its own bundled engine binary, it requests a graceful handoff (POST /control {"command":"shutdown-when-idle"}) so the newest installed engine binary always serves the session without breaking older v2 clients.
8. Security and Privacy ("Is It Dangerous?")
Recommendation: Keep the engine privilege-free, loopback-only, and storage-free.
- Zero TCC surface:
hex enginenever links or opens CoreAudio input,CGEventTap,AXUIElement,ScreenCaptureKit, orNSPasteboard. It cannot record the user or read keys/screens; macOS never prompts for permissions on its behalf. - Loopback +
0600token auth: Binds strictly to127.0.0.1:0(never0.0.0.0or::).local-api.jsonis created withmode(0o600)inside0700~/Library/Application Support/hex/. Every route requiresAuthorization: Bearer <token>(checked before reading bodies), and each/streams/:idsession requires its per-streamX-Hex-Stream-Token. - Hard resource bounds (already enforced in
local_api.rs/streams.rs): 4 HTTP workers, connection queue 16, 16 KiB header cap, 2 s header deadline, 64 MiB WAV upload cap, 1 concurrent stream upload, 256 KiB stream audio chunk cap, 30 s stream backlog cap, and 15 s stream lease timeout. - No retained user data:
hex enginewrites no transcripts, nohistory.json, and no WAV files;Hex.appcontinues to ownhistory.jsonand meeting artifacts in its own process.
9. How Hex.app Talks to It & Incremental PR Plan
Hex.app keeps all macOS-specific desktop responsibilities in-process: dictation_audio (CoreAudio mic + pre-roll), suppression (event taps + shortcut state machine), dictation_indicator (Metal HUD), audio/recording_environment (mute/media pause/sleep), dictation_processor (corrections, OpenCode rewrite, transformations), paste, history, recognition (Moonshine voice commands), meeting_capture, and the Rune settings window. Only TranscriptionEngine (Parakeet/Cohere/Whisper GGUF + Apple Speech) moves behind an EngineClient seam that speaks to hex engine over local-api.json:
- PR 1 —
LocalApi::servepoll(2)accept + HTTP keep-alive: Land the 12-linelibc::pollfix insrc/local_api.rs(dropping loopback latency from 12.5 ms to 0.12 ms) andPOST /warm. - PR 2 — Standalone
hex enginecommand & lock/discovery lifecycle: Promotehex service/hex engineto start and shareTranscriptionEnginewith interactive QoS, App Nap assertion, 10-minute idle model unload, andshutdown-when-idle. - PR 3 — Priority dictation stream & meeting-chunk routes: Add
priority: "dictation"preemption on/streamsand background priority on/transcriptionsso external clients yield toHex.apphotkey dictation just as in-processYieldingStreamdoes today. - PR 4 —
Hex.appas an engine client +@kitlangton/hexauto-start: SwitchHex.app's dictation and meeting workers toconnectOrStartEngine()(retaining the PCM clip inHex.appfor crash-safe whole-clip retry) and update@kitlangton/hexto auto-spawnhex enginewhenlocal-api.jsonis absent.
10. Linux and Windows Counterparts
- Linux: The existing
hex.serviceper-usersystemdunit (Type=notify,PartOf=graphical-session.target,Restart=on-failure) already runs a single long-lived process per user and data root (docs/plans/single-process.md). SharingTranscriptionEngineinLocalApion Linux exposes the exact samelocal-api.json+/streams+/transcriptionscontract on127.0.0.1, with non-systemd desktops using the sameengine.lock+ detachedhex enginespawn. - Windows:
WindowsDesktopHost(or a headlesshex.exe enginespawned withCREATE_NO_WINDOW | DETACHED_PROCESSand coordinated by a named mutex +%APPDATA%\hex\local-api.jsonwith owner-only DACL) serves the same loopback HTTP API, usingSetProcessInformation(..., ProcessPowerThrottling)to disable EcoQoS throttling so Vulkan/CPU inference stays at full speed.
What the Spike Did Not Prove
- Multi-hour macOS memory pressure / sleep-wake cycles: We measured explicit in-process unload (
0.84 ms, 52–301 ms reload) andtaskpolicy -bvsInteractive, but did not run a 24-hour soak across macOS display sleep/hibernation (darkwake). - Simultaneous 4-app contention trace: We verified
/streams+/transcriptionsover loopback HTTP and 12-thread CPU contention, but thepriority: "dictation"preemption route across two separate OS processes (Hex.apppreempting an activeSeer/streamssession) is planned for PR 3 (todayYieldingStreampreemption is exercised by unit tests against the in-processTranscriptionEngine).