18 KiB
Model watch
Weekly running log of local ASR models and runtimes relevant to Hex's compiled catalog; newest entry first, each dated, each ranked against what Hex ships today.
2026-10-22: streaming Parakeet vs Moonshine, and what to add
Question (Kit): do we still need Moonshine if Parakeet can stream? Is always-on streaming Parakeet better? Is there a better small model than Moonshine?
Verdict
Moonshine?
├── Drop the ONNX Moonshine stack (moonshine-voice C lib, ORT, download.moonshine.ai)
│ └── Replace with moonshine-streaming-small Q8 GGUF through transcribe-cpp (already in the pinned 0.2.4)
├── Keep a small streaming model for commands and the live draft
│ └── Always-on streaming Parakeet is not better: same WER at command latency, 4-5x the parameters,
│ a second 0.7 GB model copy, and (buffered Unified) a recomputed 5.6 s left context per chunk
└── Meetings only: Apple SpeechTranscriber beats both on cost (Hex measured ~2% CPU, no model memory);
plans/meeting-live-transcription.md already recommends it
Confidence: moderate. Everything below is vendor-reported or Hex's own measurements; nothing here was re-run on Kit's hardware. Unmeasured and decisive: command-recognition latency and miss rate of the GGUF Moonshine behind Hex's line segmentation, and Metal power for any always-on stream.
1. Streaming Parakeet vs Moonshine
What Hex runs today: moonshine.rs loads Moonshine Streaming Small (the
small-streaming-en quantized ONNX bundle, about 250 MB on disk, 140M-class)
through the moonshine-voice C library, single-threaded. Hex's own numbers
(docs/features/meetings.md, Mac Studio/Pro, 2026-10-22): two meeting-draft
streams 31% of a core, command recognizer 22% of a core, about 0.4 GB resident
per Moonshine, and a second Parakeet copy costs about +0.74 GB at meeting peak
(2808 vs 2072 MB).
Accuracy by latency. NVIDIA's own table on the eight Open ASR datasets (AMI, Earnings22, GigaSpeech, LibriSpeech clean/other, SPGI, TED-LIUM, VoxPopuli), average WER %. Moonshine is from its paper, same datasets.
latency (chunk+right) avg WER
Moonshine v2 small (123M) 80 ms lookahead 7.84 // paper Table 3
Moonshine v2 medium (245M) 80 ms lookahead 6.65
Parakeet Unified 0.6B offline 5.91
2.08 s (Hex default 5.6/1.04/1.04) 6.14
1.12 s 6.29
0.56 s 6.52
0.32 s 6.92
0.24 s 7.35
0.16 s 8.44
0.08 s 15.63 // collapses
Nemotron Speech Streaming EN 0.6B (cache-aware)
offline 6.92
1.12 s 6.92
0.56 s 7.09
0.16 s 7.84
0.08 s 8.70
Parakeet TDT v2 0.6B buffered at 1.12 s 22.83 // not trained for it
Reading it:
- At command latency (under 0.3 s) Moonshine small (7.84) already matches Nemotron at 0.16 s (7.84) and beats Unified at 0.16 s (8.44). Moonshine medium (6.65) matches Nemotron at 1.12 s (6.92) at 40% of the size.
- Unified wins only when you accept 0.56 to 2 s of delay (6.14 to 6.52), which is fine for a caption draft and irrelevant to a spoken command.
- The Moonshine paper times end-of-utterance latency on an M3 MacBook: small 148 ms and 18% compute load, medium 258 ms and 29% (Table 2). It does not report power. Nobody I found publishes Metal power for Parakeet or Nemotron streaming.
Compute shape. Unified is buffered streaming: NVIDIA's card says "left
context is recomputed for each chunk", and Hex's transcription/gguf_session.rs uses 5.6 s
left, 1.04 s chunk, 1.04 s right. Each step encodes about 7.7 s of audio for
1.04 s of new audio, roughly 7.4x redundancy (12x at the 0.56 s setting). At the
cards' M4 Max Metal throughput of about 190 to 230x realtime offline, that is
my derivation of a few percent of the GPU per continuous stream, ignoring
per-chunk overheads; it is not measured. Nemotron is cache-aware (constant
caches, no overlap), so a stream costs about one offline pass. Moonshine's
encoder is also incremental (sliding window, 50 Hz frontend).
Throughput on Apple Silicon (transcribe.cpp cards, M4 Max, Q8_0, Metal, offline compute, x realtime on an 11 s clip / 35 s clip):
Parakeet Unified 188 / 228
Nemotron EN 207 / 225
Nemotron 3.5 (multiling.) 144 / 138
Moonshine streaming tiny 239 / 112 (CPU 271 / 169)
Moonshine streaming small 129 / 59 (CPU 64 / 49)
Moonshine streaming med 87 / 36 (CPU 47 / 32)
Voxtral Realtime 4B 6 / 7
Moonshine's autoregressive decoder dominates long transcripts, and its CPU path is close to its Metal path, so it does not need the GPU. Nemotron/Unified are GPU models: CPU is 32 to 39x on the same chip.
Can one loaded Parakeet serve several streams? Not in Hex's binding.
Checked against transcribe-cpp 0.2.4 (Hex's pin) and upstream:
transcribe.h(upstream main): "KNOWN 0.x LIMITATION — concurrent COMPUTE is not yet supported: at most one transcribe_run / transcribe_run_batch / active stream may be in flight across ALL sessions of a given model." Callers wanting parallel transcription "should load one model per worker"; serialized use of many sessions on one model is supported.docs/bindings.md: the C header would let several sessions each hold an active stream and interleave feeds under caller serialization, but "a binding allows one active stream per model".src/session.rs(0.2.4)Session::streamtakes a per-model lease for the stream's whole life; a second stream or anyrunreturnsError::Busy. That is the lease Hex's engine thread holds during every dictation capture.run_batchbatches offline utterances only. There is no batched streaming.
So: each concurrent stream needs its own Model (Hex measured about 0.7 GB for
a second Parakeet), and an always-on stream on the dictation model would block
dictation (or be blocked by it). transcribe.cpp's README says CPU and Metal
weights are memory-mapped, so two loads of one GGUF may share pages; Hex's
+0.74 GB measurement says do not assume it without measuring. A "per-session
backend architecture" lifting the limit is described as planned, with no date.
(a) Two-track live captions. Moonshine remains the cheapest correct fit, but both its current runtime and Parakeet share the same constraint: one stream per model object, so two tracks mean two models. With 199 MB Q8 GGUFs (mmap-able, Metal or CPU) that is about 0.4 GB, the same as today, minus ORT and its "one streaming state per transcriber" problem that dropped 553 to 631 packets per run in Hex's measurement. A Parakeet pair is about 1.5 GB for a draft that rolling Parakeet replaces within a chunk anyway. Apple SpeechTranscriber (Hex measured ~2% of a core for both tracks, finals about 2.3 s p50 after the segment ends, 3.0% vs Moonshine's 1.7% WER on a synthetic call) is the cheapest by an order of magnitude and needs no model memory.
(b) Always-on voice commands. Ordinary resolver commands run only from completed lines (AGENTS invariant), so partial-hypothesis latency matters only for the voice-delimited exceptions. Candidates:
- Moonshine small GGUF behind a VAD line segmenter: 199 MB, CPU or Metal, 2.54% LibriSpeech clean (card), about the same as the ONNX build.
parakeet-tdt_ctc-110moffline on VAD-segmented utterances: 135 MB Q8, 2.43% LibriSpeech clean, English, CC-BY-4.0. Offline decode of a one-second command is tens of milliseconds, so the cost is the endpoint hangover, not the model. Untested here.- Always-on Nemotron or Unified: needs a dedicated 0.7 GB model, a 12 Hz (80 ms chunk) wake-up loop on the GPU, and gives no accuracy gain at these latencies. Only worth it if Kit wants dictation-grade punctuation in the live draft.
Moonshine's one structural advantage over the transcribe.cpp route is that
moonshine-voice ships its own VAD and line events. transcribe.cpp's Moonshine
stream returns committed/tentative text, caps one stream at 81.9 s, and has no
line segmentation; Hex would drive it with Silero VAD v6.2 (shipped in
transcribe.cpp) or transcription/pause_segmenter and start a new stream per line.
2. Landscape (as of 2026-10-22)
Two leaderboards are in play and must not be compared across:
- Eight-set average (older; NVIDIA and Moonshine report on it).
- Current Open ASR Leaderboard (results commits through 2026-10-19): nine public sets, including Voice Arena Monsoon and URGENT24 noisy/reverberant speech. Scores are higher than on the eight-set average for the same model. Moonshine, Parakeet Unified and Kyutai are not in the current results file.
Current leaderboard, english_short_latest.csv (dataset
hf-audio/open-asr-leaderboard-results, commit 2026-10-19), open models only.
RTFx is on an A100 and says nothing about a Mac:
model params avg WER RTFx license GGUF / transcribe.cpp
Granite Speech 5.0 2B 2.3B 5.06 545 Apache-2.0 not yet (4.0/4.1 and 5.0 TurboCTC only)
Qwen3-ASR 1.7B 2.0B 5.75 836 Apache-2.0 yes (qwen3-asr-1.7b)
Cohere Transcribe 03-2026 2B 6.26 924 Apache-2.0 yes (already in Hex)
Canary-Qwen 2.5B 2.5B 6.37 882 CC-BY-4.0 yes
Parakeet TDT 0.6B v2 0.6B 6.43 6148 CC-BY-4.0 yes (already in Hex)
Parakeet TDT 0.6B v3 0.6B 6.51 6228 CC-BY-4.0 yes (already in Hex)
Qwen3-ASR 0.6B 0.8B 6.60 750 Apache-2.0 yes (already in Hex)
Granite 5.0 TurboCTC (new rev) 0.47B 6.84 21134 Apache-2.0 old English-only rev only
Nemotron Speech Streaming EN 0.62B 6.89 1114 NVIDIA OML yes
Distil-Whisper large-v3.5 0.8B 7.02 898 MIT not in the model list
Whisper large-v3-turbo 0.8B 8.39 811 MIT yes (already in Hex)
Voxtral Mini 4B Realtime 4B 9.01 103 Apache-2.0 yes, 4.7 GB Q8, 6-7x realtime on M4 Max
Nemotron 3.5 ASR Streaming 0.64B 10.51 1348 OpenMDW-1.1 yes (multilingual; English worse)
Streaming-capable and small, with how Hex would run each:
Moonshine v2 Streaming tiny/small/med MIT; 44M/140M/266M (GGUF params); English only
│ streaming yes (50 Hz frontend, 80 ms feed); 8-set WER 12.01/7.84/6.65; LS clean Q8 4.52/2.54/2.16
│ GGUF Q8 50/199/296 MB in transcribe.cpp. No multilingual in this family; other-language Moonshine is non-streaming
├── Parakeet Unified EN 0.6B NVIDIA OML; buffered streaming, 160 ms to 2.08 s; Hex default; 731 MB Q8
├── Nemotron Speech Streaming EN 0.6B NVIDIA OML; cache-aware, 80 ms to 1.12 s; 730 MB Q8
│ LibriSpeech clean streaming = offline = 2.31% Q8 (byte-equal to NeMo at R=13); supported by 0.2.4
├── Nemotron 3.5 ASR Streaming 0.6B OpenMDW-1.1; 32 supported locales, prompt-conditioned; 751 MB Q8
│ LS clean 3.05% (vs 2.31 EN-only); English regressed on the current leaderboard (10.51 vs 6.89)
├── Multitalker Parakeet Streaming 0.6B NVIDIA OML; Nemotron fine-tune with an embedded Sortformer diarizer
│ (up to 4 speakers); the only local route to "who said it" in a stream; unmeasured by Hex
├── Voxtral Mini 4B Realtime Apache-2.0; 480 ms default delay; auto language detection; too heavy (4.7 GB, 6-7x realtime)
├── Kyutai STT 2.6B EN / 1B EN-FR CC-BY-4.0; 2.5 s / 0.5 s delay; PyTorch/MLX, no GGUF
│ old 8-set avg 6.4; 2.6B is too large for an always-on slot
├── Apple SpeechAnalyzer / SpeechTranscriber system model, macOS 26+, no download; Hex already ships
│ it as a batch model. Independent tests: 2.12% clean / 4.56% other (M2 Pro, 5,559 LibriSpeech
│ utterances, Lyonesse, July 2026), 1.98% on a 40-clip snapshot (IraVoice, M5 Max, p50 about 130 ms
│ post-speech). Parakeet v2 CoreML int8 2.01 / 3.40 in the same Lyonesse round
└── Granite Speech 5.0 TurboCTC Apache-2.0; offline only despite the "streaming-style" encoder;
21k RTFx on A100; the Oct 21 revision adds PnC and 7 languages but is not in transcribe.cpp yet
Offline accuracy tier, for reference (not streaming): Granite Speech 5.0 2B is
#2 open on the current leaderboard (5.06, released 2026-10-20, Apache-2.0,
translation and 2,000-term keyword biasing, 6 languages); Qwen3-ASR 1.7B 5.75
(52 languages); parakeet-ultra (Moondream's post-trained v3, CC-BY-4.0, 25
languages) lists 1.80% vs v3's 1.94% on LibriSpeech clean in transcribe.cpp,
and was 5.32 vs 6.32 on the earlier leaderboard average per the HF dataset
listing. TheWhisper large-v3-turbo (TheStage, CC-BY-4.0) is 6.07 on the current
leaderboard with CoreML and a streaming tag, but ships no GGUF.
3. Catalog additions, ranked
Hex's catalog today (transcription_models.rs): Parakeet Unified EN, v2, v3,
Whisper large-v3-turbo, Qwen3-ASR 0.6B, SenseVoice Small, Cohere Transcribe,
Apple Speech. Moonshine is a separate ONNX path outside the catalog.
1. Moonshine Streaming small (and medium as an option), GGUF // replaces the ONNX Moonshine
│ Gain: removes ORT, moonshine-voice (header 20000 now, 30000 breaking), the download.moonshine.ai
│ host and its failure mode; one runtime, one downloader, Metal or CPU; weights 199 MB vs ~250 MB;
│ same accuracy (card LS clean 2.54% vs Useful Sensors' 2.49%); medium is 6.65 avg for 296 MB.
│ Cost: medium. Rewrite the thin layer behind moonshine.rs: VAD line segmentation (Silero VAD is in
│ transcribe.cpp), a stream per line (81.9 s cap), committed vs tentative text to LineStarted/
│ Updated/Completed, one Model per meeting track and one for commands.
│ Needs no crate bump (0.2.4 has moonshine_streaming). Gate: replay Hex's command fixtures
│ (moonshine_lab) and the meeting_echo_replay call against both backends before deleting ONNX.
│
2. parakeet-ultra (GGUF, offline) // replaces Parakeet v3
│ Gain: same size and API as v3 (740 MB Q8), lower error on English and every FLEURS language,
│ cuts long audio at pauses by itself (30 s segments), which suits meeting chunks.
│ Cost: low. Catalog entry plus a crate bump: the port is pinned 2026-10-04, newer than 0.2.4
│ (2026-09-25); 0.4.1 is current (2026-10-22). Q8_0 (740 MB, 1.80%) and smaller quants are published.
│
3. Nemotron Speech Streaming EN 0.6B (GGUF, cache-aware) // new "Live" tier, not a dictation default
│ Gain: streaming WER equals offline (2.31% LS clean), 80 to 1120 ms chunks, constant memory, no 5.6 s
│ recompute, native PnC. Right choice if the live meeting draft should read like final text, or if
│ Kit wants always-on streaming with a Parakeet-family model.
│ Cost: medium. transcription/gguf_session.rs only sets the buffered Parakeet extension; Nemotron needs
│ StreamExtension::ParakeetStream{att_context_right}. Runs in 0.2.4. License is the NVIDIA Open
│ Model License. Do not make it the dictation default: Unified is more accurate offline (5.91 vs 6.92).
│
4. Granite Speech 5.0 2B and TurboCTC // wait for the port
│ 2B is the accuracy leader (5.06) but needs a 2B LLM decoder; TurboCTC is the fastest and Apache but
│ its revised multilingual/PnC weights are not in transcribe.cpp (README lists the August revision).
│
5. Qwen3-ASR 1.7B // quality tier for final meeting passes
│ 5.75 current, 52 languages, Apache-2.0, already in transcribe.cpp. 2B-class decode cost, so
│ final-pass only. Hex already carries the 0.6B.
│
Not recommended: Voxtral Realtime (4.7 GB, 6-7x realtime, 9.01 avg), Nemotron 3.5 for English
(10.51 current vs 6.89), Kyutai (no GGUF, 2.6B), Moonshine tiny (12.01 avg).
Dropping Moonshine entirely is possible for the meeting draft only: use Apple SpeechTranscriber on macOS 26+ and Nemotron or nothing elsewhere. For commands, a small streaming or VAD-endpointed model stays, and Moonshine small is the smallest one with a published streaming port at command-grade WER.
Next measurements
- GGUF Moonshine small vs the ONNX build on
moonshine_labfixtures: miss rate, update p50/p95, CPU, resident memory (M5 Max and a base M-series). - A second
Modelof the same Parakeet GGUF in-process: real resident delta with mmap on Metal. - Nemotron EN at 160 and 560 ms against Unified (5.6/0.56/0.56) on Hex's call
replay: caption latency, CPU/GPU time per audio minute,
powermetricsenergy. - Whether the HEAD of transcribe.cpp has lifted the one-compute-per-model rule
(watch
docs/bindings.mdand the 0.4.x notes).
Sources
- Moonshine v2 paper (Kudlur et al., 2026-02-12), arXiv 2602.12241: Tables 2 and 3, encoder window (16,4)/(16,0), 80 ms lookahead.
- UsefulSensors/moonshine-streaming-small model card (MIT); moonshine-ai/moonshine releases (v0.1.5, 2026-08-24).
- transcribe.cpp (MIT)
docs/models/:moonshine-streaming*.md,parakeet.md,parakeet-unified-en-0.6b.md,parakeet-ultra.md,nemotron-speech-streaming-en-0.6b.md,nemotron-3.5-asr-streaming-0.6b.md,voxtral-realtime.md,granite-speech-5.0-turboctc.md,bindings.md;include/transcribe.h; Hex's pinnedtranscribe-cpp/-sys0.2.4 sources (session.rs,lib.rs,family.rs,arch/). crates.io: 0.2.4 on 2026-09-25, 0.4.1 on 2026-10-22. - nvidia/parakeet-unified-en-0.6b (2026-04-07) and nvidia/nemotron-speech- streaming-en-0.6b (2026-03-13) model cards; arXiv 2604.19079 and 2312.17279.
- Open ASR Leaderboard results file
english_short_latest.csvand its commit log (2026-10-19), arXiv 2510.06961; IBM Granite Speech 5.0 blog (2026-10-21). - kyutai/stt-2.6b-en model card; Mistral Voxtral Mini 4B Realtime via the transcribe.cpp card above.
- Apple on-device comparisons are third-party and from interested publishers:
Lyonesse (2026-07-13/15), IraVoice
mac-asr-benchmark(2026-08-01), Mumr meeting benchmark (2026-09-30). Hex's own measurements:docs/features/meetings.mdanddocs/plans/meeting-live-transcription.md.