28 KiB
Per-Language Model Recommendations Plan
Status: Approved 2026-10-22; Phase A implemented, Phase B waits for the transcribe-cpp bump. Prompting existing users about a better model is deferred until the Rune port is done (see ROADMAP.md "Recommend Better Models To Existing Users"); Phase A adds no new UI. Synthesizes docs/research/language-recommendations-european.md (ebd9fc8), docs/research/language-recommendations-other.md (fde2bb3), and docs/research/model-watch.md, and integrates with docs/plans/transcription-quality.md (which owns the transcribe-cpp 0.2.4 -> 0.4.1 bump, shared vocabulary, and parakeet-ultra runtime work).
Summary
Hex's current choices_for_runtime (src/transcription_models.rs:494-567) routes 40 language entries (auto plus 39 languages) through four coarse buckets. Apples-to-apples FLEURS test evaluations on the exact Q8_0 GGUF artifacts in handy-computer/transcribe.cpp expose three classes of fixes:
Per-language recommendation updates
├── Phase A: Ship now on pinned transcribe-cpp 0.2.4 (0.5d, zero new models)
│ ├── Greek (el): Parakeet v3 gets 1 in 3 words wrong (35.33% WER) -> route to shipped Cohere (8.96%) + Whisper (13.26%)
│ ├── Qwen 0.6B tail failures: Whisper turbo replaces Qwen 0.6B as Recommended on tr, mk, ar, ms, fil, fa, ja (-3.8 to -23.4 pts)
│ ├── Nordic Parakeet gap: Whisper turbo becomes Recommended on fi (8.29 vs 13.30) and sv (8.72 vs 15.25), Parakeet as Fastest
│ ├── Auto & CJK cleanup: drop 5-language SenseVoice from Auto, yue (37.44% CER), and ko (8.27%); Qwen 0.6B becomes Auto Fastest
│ └── Maltese (mt): drop Whisper turbo (70.92% WER) from Recognition hints
└── Phase B: Ship with transcribe-cpp 0.4.1 in transcription-quality.md Phase 3 (+1d)
├── ParakeetUltra (740 MB): replaces Parakeet v3 across 24 EU languages (-0.6 to -6.2 pts everywhere, 30 s pause cuts)
├── Canary1BV2 (1.14 GB): Recommended on lv, sl (-6.2 to -7.6 pts vs ultra); Most accurate on bg, cs, da, et, hr, lt, ro (-1.7 to -4.6 pts)
└── Qwen3Asr17B (2.19 GB): Most accurate on auto, yue, ko, hi, vi, th, id, fa (and en pending ses_e98f7b094ffe1Q3KtZFS6Np4KL)
1. Final Per-Language Table
Rules applied across all 40 entries in LANGUAGES (src/transcription_models.rs:437-478):
- Noise floor (
~1.5 points): Differences under ~1.5 WER/CER points on FLEURS test (~860–900 read utterances per language, bootstrap 95% CI ~±0.9 pts) are treated as ties. We do not churn an existing recommendation or add aMost accuratecard for a<1.5 ptedge (e.g.,zhQwen 0.6B 7.57% vs 1.7B 7.14%;koQwen 0.6B 5.82% vs Whisper 5.24%;idQwen 0.6B 8.49% vs Whisper 7.20%;skParakeet Ultra 7.83% vs Canary v2 6.84%;de/es/fr/hu/it/nl/pl/pt/ru/ukUltra vs Canary/Whisper/Voxtral). - Roles:
Rec=Recommended(first entry is alsorecommended_selection(lang)for first-run setup),Fast=Fastest,Acc=Most accurate,Hint=Recognition hints. WhenWhisper turboisRec, no duplicateHintrow is shown because it already supports hints. - Metrics: CER % for
zh,yue,ja,ko,th; WER % for all other languages (Q8_0 GGUF on FLEURS test viatranscribe.cpp).
| Code | Language | Current (0.2.4) |
Proposed (Phase A: 0.2.4 → Phase B: 0.4.1) |
Evidence (FLEURS Q8_0 WER/CER %) | Conf. | Runtime |
|---|---|---|---|---|---|---|
auto |
Auto | Rec: Whisper turbo Fast: SenseVoice Rec: Qwen 0.6B |
A: Rec: Whisper turbo; Fast: Qwen 0.6B (drop SenseVoice) B: + Acc: Qwen 1.7B |
30-lang median: Qwen 1.7B 7.6, Whisper turbo 8.6, Qwen 0.6B 13.3 (mean 21.4). SenseVoice covers only 5 langs. | Med | 0.2.4 (1.7B in B) |
en |
English | Rec: Parakeet Unified Acc: Cohere Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Unified; Acc: Qwen 1.7B (or Cohere, gated on audit); Hint: Whisper turbo |
FLEURS: Qwen 1.7B 3.23, Unified 3.99, Ultra 4.26, Whisper 4.38, Canary 4.47, v3 4.83, Cohere 5.08. Open ASR avg: Qwen 1.7B 5.75, Cohere 6.26, v3 6.51. | High (Rec) Med (Acc) |
0.2.4 |
zh |
Mandarin Chinese | Rec: Qwen 0.6B Fast: SenseVoice Hint: Whisper turbo |
Keep current (no churn; top models within 0.43 pt noise band) | Qwen 1.7B 7.14, Fun-ASR-MLT 7.33, Qwen 0.6B 7.57, Whisper turbo 8.50, SenseVoice 10.12 (245x). | Med | 0.2.4 |
yue |
Cantonese | Rec: Qwen 0.6B Fast: SenseVoice Hint: Whisper turbo |
A: Rec: Qwen 0.6B; Hint: Whisper turbo (drop SenseVoice Fast) B: + Acc: Qwen 1.7B (Qwen hints enabled in 0.4.1) |
Qwen 1.7B 6.13, Qwen 0.6B 7.91 vs Whisper turbo 34.62, SenseVoice 37.44 (+29.5 pts; verify script norm). | High | 0.2.4 (1.7B in B) |
ja |
Japanese | Rec: Qwen 0.6B Fast: SenseVoice Hint: Whisper turbo |
A/B: Rec: Whisper turbo; Fast: SenseVoice | Whisper turbo 4.82, Cohere 5.13, Qwen 1.7B 5.29, SenseVoice 7.63 (245x), Qwen 0.6B 8.61 (-3.79 pts). (Fun-ASR-MLT 2.10 outlier deferred). | Med-High | 0.2.4 |
ko |
Korean | Rec: Qwen 0.6B Fast: SenseVoice Hint: Whisper turbo |
A: Rec: Qwen 0.6B; Hint: Whisper turbo (drop SenseVoice Fast) B: + Acc: Qwen 1.7B |
Qwen 1.7B 4.60, Whisper turbo 5.24, Qwen 0.6B 5.82 (0.58 pt gap vs Whisper is noise; Qwen 2x faster), SenseVoice 8.27 (+2.45 pts worse). | Med | 0.2.4 (1.7B in B) |
es |
Spanish | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Ultra 3.06, Canary v2 3.10, Whisper 3.12, Qwen 1.7B 3.31, v3 3.65 (all top models within 0.25 pt). | High | 0.4.1 (Ultra) |
fr |
French | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Ultra 4.66, Canary v2 5.09, v3 5.30, Whisper 5.51 (Voxtral 3B 4.51 is 0.15 pt noise at 13x speed). | High | 0.4.1 (Ultra) |
de |
German | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Qwen 1.7B 4.25, Canary v2 4.46, Whisper 4.54, Ultra 4.58, v3 5.24 (all within 0.33 pt noise). | High | 0.4.1 (Ultra) |
pt |
Portuguese | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Ultra 4.16, Whisper 4.17, Canary v2 4.50, v3 4.96 (Voxtral 3B 3.84 is 0.32 pt noise). | High | 0.4.1 (Ultra) |
it |
Italian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Ultra 2.34, Whisper 2.77, v3 3.02, Canary v2 3.10. | High | 0.4.1 (Ultra) |
ar |
Arabic | Rec: Qwen 0.6B Hint: Whisper turbo |
A/B: Rec: Whisper turbo; Acc: Cohere Transcribe | Cohere (shipped) 13.60, Qwen 1.7B 14.91, Whisper turbo 15.48 vs Qwen 0.6B 24.51 (-9.03 to -10.91 pts). | High | 0.2.4 |
hi |
Hindi | Rec: Qwen 0.6B Hint: Whisper turbo |
A: Keep current B: Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo |
Qwen 1.7B 7.84 (-4.84 pts), Qwen 0.6B 12.68, Whisper turbo 18.85. | High | 0.2.4 (1.7B in B) |
vi |
Vietnamese | Rec: Qwen 0.6B Hint: Whisper turbo |
A: Rec: Qwen 0.6B; Acc: Cohere; Hint: Whisper turbo B: Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo |
Qwen 1.7B 6.15 (-3.17 pts), Cohere (shipped) 7.39 (-1.93 pts), Qwen 0.6B 9.32, Whisper turbo 9.48. | Med | 0.2.4 (1.7B in B) |
ru |
Russian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Ultra 5.63, Whisper 5.93, Qwen 1.7B 6.25, v3 6.54, Canary v2 7.83. | High | 0.4.1 (Ultra) |
uk |
Ukrainian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Ultra 5.71, v3 6.84, Whisper 7.31, Canary v2 10.58. | High | 0.4.1 (Ultra) |
nl |
Dutch | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Whisper 5.98, Canary v2 6.28, Ultra 6.70, v3 7.66 (0.72 pt Whisper edge is within 1.5 pt noise; Whisper already shown as Hint). | Med | 0.4.1 (Ultra) |
pl |
Polish | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Whisper 5.81, Cohere 6.15, Ultra 6.16, Canary v2 6.88, v3 7.37 (0.35 pt gap is noise). | Med | 0.4.1 (Ultra) |
tr |
Turkish | Rec: Qwen 0.6B Hint: Whisper turbo |
A/B: Rec: Whisper turbo | Whisper turbo 6.97, Qwen 1.7B 9.46 vs Qwen 0.6B 16.74 (-9.77 pts). | High | 0.2.4 |
sv |
Swedish | Rec: Parakeet v3 Hint: Whisper turbo |
A: Rec: Whisper turbo; Fast: Parakeet v3 B: Rec: Whisper turbo; Fast: Parakeet Ultra |
Whisper turbo 8.72, Canary v2 9.74 (within 1.02 pt of Whisper), Ultra 12.83, v3 15.25 (-6.53 pts vs v3, -4.11 pts vs Ultra). | Med-High | 0.2.4 (Ultra in B) |
da |
Danish | Rec: Parakeet v3 Hint: Whisper turbo |
A: Rec: Whisper turbo; Fast: Parakeet v3 B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 11.34 (-4.64 pts vs Ultra), Whisper 13.60 (-5.04 pts vs v3), Ultra 15.98, v3 18.64. | High | 0.2.4 / 0.4.1 |
fi |
Finnish | Rec: Parakeet v3 Hint: Whisper turbo |
A: Rec: Whisper turbo; Fast: Parakeet v3 B: Rec: Whisper turbo; Fast: Parakeet Ultra |
Whisper turbo 8.29, Canary v2 8.86 (within 0.57 pt of Whisper), Ultra 11.14, v3 13.30 (-5.01 pts vs v3, -2.85 pts vs Ultra). | Med-High | 0.2.4 (Ultra in B) |
cs |
Czech | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 8.56 (-2.48 pts vs Ultra), Ultra 11.04, Whisper 11.81, v3 12.31. | Med | 0.4.1 |
el |
Greek | Rec: Parakeet v3 Hint: Whisper turbo |
A/B: Rec: Cohere Transcribe; Hint: Whisper turbo | Cohere (shipped) 8.96, Whisper turbo 13.26 vs Canary v2 26.02, Ultra 32.25, v3 35.33 (-26.37 pts). | High | 0.2.4 |
ro |
Romanian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 6.87 (-3.25 pts vs Ultra), Ultra 10.12, Whisper 10.90, v3 12.62. | High | 0.4.1 |
hu |
Hungarian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Canary v2 13.06, Ultra 13.09 (0.03 pt tie), Whisper 15.07, v3 16.06. | Med | 0.4.1 (Ultra) |
id |
Indonesian | Rec: Qwen 0.6B Hint: Whisper turbo |
A: Keep current (1.29 pt gap is <1.5 pt noise) B: Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo |
Qwen 1.7B 5.37 (-3.12 pts vs 0.6B), Whisper turbo 7.20, Qwen 0.6B 8.49. | Med | 0.2.4 (1.7B in B) |
ms |
Malay | Rec: Qwen 0.6B Hint: Whisper turbo |
A/B: Rec: Whisper turbo | Whisper turbo 8.64, Qwen 1.7B 10.42 vs Qwen 0.6B 17.18 (-8.54 pts). | Med-High | 0.2.4 |
th |
Thai | Rec: Qwen 0.6B Hint: Whisper turbo |
A: Keep current B: Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo |
Qwen 1.7B 6.89 (-1.92 pts vs 0.6B), Qwen 0.6B 8.81, Whisper turbo 13.15. | Med | 0.2.4 (1.7B in B) |
fa |
Persian | Rec: Qwen 0.6B Hint: Whisper turbo |
A: Rec: Whisper turbo B: Rec: Whisper turbo; Acc: Qwen 1.7B |
Qwen 1.7B 28.29 (-2.27 pts vs Whisper), Whisper turbo 30.56 vs Qwen 0.6B 50.30 (-19.74 pts). | High | 0.2.4 (1.7B in B) |
bg |
Bulgarian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 9.22 (-1.82 pts vs Ultra), Ultra 11.04, v3 12.81, Whisper 13.58. | Med | 0.4.1 |
hr |
Croatian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 8.40 (-2.90 pts vs Ultra), Ultra 11.30, Whisper 12.54, v3 12.59. | Med | 0.4.1 |
et |
Estonian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 12.72 (-1.72 pts vs Ultra), Ultra 14.44, v3 17.96, Whisper 18.44. | Med | 0.4.1 |
lv |
Latvian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Canary 1B v2; Fast: Parakeet Ultra; Hint: Whisper turbo |
Canary v2 10.41 (-7.63 pts vs Ultra, -13.36 pts vs v3), Ultra 18.04, Whisper 19.53, v3 23.77. | High | 0.4.1 |
lt |
Lithuanian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo |
Canary v2 13.45 (-3.61 pts vs Ultra), Ultra 17.06, v3 22.20, Whisper 25.11. | Med-High | 0.4.1 |
mt |
Maltese | Rec: Parakeet v3 Hint: Whisper turbo |
A: Rec: Parakeet v3 (drop Whisper Hint) B: Rec: Parakeet Ultra |
Ultra 17.01, Canary v2 19.75, v3 20.63 vs Whisper turbo 70.92 (unusable on Maltese). | Med | 0.2.4 / 0.4.1 |
sk |
Slovak | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Parakeet Ultra; Hint: Whisper turbo |
Canary v2 6.84 vs Ultra 7.83 (-0.99 pt is inside 1.5 pt noise floor; omit Canary v2), v3 10.19, Whisper 10.21. | Med | 0.4.1 (Ultra) |
sl |
Slovenian | Rec: Parakeet v3 Hint: Whisper turbo |
A: Keep current B: Rec: Canary 1B v2; Fast: Parakeet Ultra; Hint: Whisper turbo |
Canary v2 12.76 (-6.22 pts vs Ultra, -11.54 pts vs v3), Ultra 18.98, Whisper 20.56, v3 24.30. | High | 0.4.1 |
mk |
Macedonian | Rec: Qwen 0.6B Hint: Whisper turbo |
A/B: Rec: Whisper turbo | Whisper turbo 17.85, Qwen 1.7B 18.22 vs Qwen 0.6B 35.09 (-17.24 pts). | High | 0.2.4 |
fil |
Filipino | Rec: Qwen 0.6B Hint: Whisper turbo |
A/B: Rec: Whisper turbo | Whisper turbo 12.08 (tl runtime hint) vs Qwen 1.7B 24.29, Qwen 0.6B 35.43 (-23.35 pts). |
High | 0.2.4 |
2. New Catalog Entries (Small-Catalog Rule)
A model earns a slot in MODELS (src/transcription_models.rs) only if it is Recommended or Most accurate across several languages by >1.5 points or materially better for a major language. Three models qualify; seven candidates are rejected.
Qualifying additions (Phase B, transcribe-cpp 0.4.1)
TranscriptionModelId |
Wire name | Display name | Size (Q8_0) | Speed (M4 Max) | License | Pinned artifact repository / filename | Architecture / Variant | Role |
|---|---|---|---|---|---|---|---|---|
ParakeetUltra |
parakeet_ultra |
Parakeet Ultra |
~740 MB | ~207x | CC-BY-4.0 | handy-computer/parakeet-ultra-ggufparakeet-ultra-Q8_0.gguf |
parakeet / ultra-0.6b |
Replaces ParakeetV3 in picker across 24 EU languages (Rec for 20, Fast for 4); 30 s pause-aware VAD cuts (0.4.1). |
Canary1BV2 |
canary_1b_v2 |
Canary 1B v2 |
~1.14 GB | ~105x | CC-BY-4.0 | handy-computer/canary-1b-v2-ggufcanary-1b-v2-Q8_0.gguf |
canary / canary-1b-v2 |
Rec for lv, sl (-6.2 to -7.6 pts vs Ultra); Acc for bg, cs, da, et, hr, lt, ro (-1.7 to -4.6 pts). Explicit language required (supports_language_detection: false). |
Qwen3Asr17B |
qwen3_asr17_b |
Qwen3-ASR 1.7B |
2.19 GB | ~45x | Apache-2.0 | handy-computer/Qwen3-ASR-1.7B-ggufQwen3-ASR-1.7B-Q8_0.gguf |
qwen3_asr / qwen3-asr-1.7b |
Acc for auto, yue, ko, hi, vi, th, id, fa (and en pending audit); supports vocabulary hints in 0.4.1 without 0.6B's dictionary echo. |
(Note: MoonshineStreamingSmall is tracked separately in Phase 6 of docs/plans/transcription-quality.md as the internal voice-command recognizer replacing libmoonshine, not a dictation-picker entry.)
Already-shipped model unlocked in Phase A (0.2.4)
CohereTranscribe(cohere_transcribe, 2.41 GB, Apache-2.0): Already compiled intoMODELSwith"el","ar", and"vi"inCOHERE_LANGUAGES(src/transcription_models.rs:243-245), but currently offered only for"en". Exposing it asRecommendedfor Greek (el, 8.96% vs Parakeet v3 35.33%) andMost accuratefor Arabic (ar, 13.60% vs Qwen 0.6B 24.51%) requires zero new catalog entries.
Rejected candidates
Fun-ASR-MLT-Nano-2512(891 MB, FunASR Model License v1.1): Only stands out on Japanese (2.10%CER vs Whisper turbo4.82%), which looks like FLEURS training-set overlap, carries a custom license with a non-disparagement termination clause, and enables ITN by default. Deferred unless non-FLEURS Japanese benchmarks confirm a real gain.Fun-ASR-Nano-2512(891 MB, FunASR license): TrailsQwen 0.6Bonzh(8.59% vs 7.57%) andWhisper turboonja(8.50% vs 4.82%).Cohere Transcribe Arabic(cohere-transcribe-arabic-07-2026, 2.41 GB, Apache-2.0): Single-language 2.41 GB download from a gated upstream repo for a 2.54 pt gain over the already-shipped 14-languageCohereTranscribe(11.06%vs13.60%onar).Whisper large-v3(1.67 GB, Apache-2.0): 2x the size and slower thanWhisper turbo; only materially beats turbo whereQwen3-ASRalready wins (yue,th) or ties (mk).Voxtral Mini 3B 2507(Apache-2.0, 13x Metal): Within 0.15–0.32 pts ofParakeetUltraonfr/it/nl/ptat 1/15th the speed.Nemotron 3.5 ASR Streaming 0.6B(751 MB, OpenMDW-1.1): Regressed accuracy on English (10.51%) and European languages (de10.33%,fr10.78%).Granite Speech 4.1 2B/5.0 2B(Apache-2.0):4.1does not beat Unified/Whisper;5.0 2B(5.06% Open ASR Leaderboard) is not yet ported totranscribe.cpp.
3. The English Decision
ParakeetUnifiedEnglish (parakeet_unified_en, 731 MB, ~188x Metal, 3.99% FLEURS en) stays the default (Recommended) model for English. It is the only model in Hex's catalog that streams on GPU during hotkey capture (streams: true), so switching the default to an offline-only model (parakeet-ultra or Qwen3-ASR 1.7B) would sacrifice release-to-paste latency on every dictation clip.
Two decisions depend on Kit's in-progress own-recordings audit (ses_e98f7b094ffe1Q3KtZFS6Np4KL, comparing ParakeetUnifiedEnglish, Qwen3-ASR 1.7B, and parakeet-ultra):
- Default (
Recommended) threshold: KeepParakeetUnifiedEnglishasRecommendedunlessparakeet-ultraorQwen3-ASR 1.7BbeatsParakeetUnifiedEnglishby >= 1.5 WER points on Kit's real dictation corpus after accounting for shared vocabulary (Phase 2/4 oftranscription-quality.md) and keeps p95 post-release transcription latency <= 150 ms on a 10 s clip. Most accurateslot (Qwen3-ASR 1.7BvsCohereTranscribe):CohereTranscribeleads only on read audiobooks (LibriSpeech clean 1.27%), whileQwen3-ASR 1.7Bbeats it on both FLEURSen(3.23% vs 5.08%, -1.85 pts) and the 9-dataset Open ASR Leaderboard (5.75% vs 6.26%) at a slightly smaller size (2.19 GB vs 2.41 GB) and adds vocabulary hint support in0.4.1. ReplaceCohereTranscribewithQwen3-ASR 1.7Bas EnglishMost accuratein Phase B ifQwen3-ASR 1.7Bties or beatsCohereTranscribeonses_e98f7b094ffe1Q3KtZFS6Np4KL.
4. Verification Before Shipping
Run targeted local checks via cargo run --release -- benchmark-transcription on 30–50 clip subsets before locking disputed rows:
- Cantonese (
yue) script normalization (0.2.4, before Phase A):- Claim to check: Whether
SenseVoice Small(37.44% CER) andWhisper turbo(34.62% CER) fail on Cantonese acoustics or merely emit Simplified characters against Traditional FLEURS references. - Check: Run 30
yueclips throughQwen3Asr06B,SenseVoiceSmall, andWhisperLargeV3Turboand score both raw and Traditional/Simplified-normalized CER. Even if script-normalized CER improves, check whether users dictating Cantonese receive the expected Traditional script.
- Claim to check: Whether
- Greek (
el) and Arabic (ar) with shippedCohereTranscribe(0.2.4, before Phase A):- Claim to check: Confirm
CohereTranscribeloads and decodeselandarcleanly under Hex's 35 s chunk cap (max_audio_chunk_samplesinsrc/transcription_models.rs:597-609) and matches the ~9% (el) / ~13.6% (ar) FLEURS WER on a 30-clip subset.
- Claim to check: Confirm
Canary 1B v2language conditioning (0.4.1, before Phase B):- Claim to check: Verify
canary-1b-v2-Q8_0.ggufhonorsRunOptionslanguage hints forlv,sl,da,ro,cs,hr,lt,bg,etwithout requiring a--prefixworkaround, and measure M-series Metal load/prewarm time and RSS.
- Claim to check: Verify
Qwen3-ASR 0.6Bvs1.7Bvocabulary hints (0.4.1, Phase B /transcription-quality.mdPhase 3):- Claim to check:
transcribe.cppnotes that0.6Bechoes--vocabularydictionary terms into the output more often than1.7B. Run the Phase 1 term-free guard set (transcription-quality.md) on both models before settingsupports_recognition_hints: trueon0.6Bvs1.7Bonly.
- Claim to check:
Fun-ASR-MLT-Nano-2512Japanese outlier (optional, only if considering adding it later):- Claim to check: Test
Fun-ASR-MLTvsWhisper turboon 30 non-FLEURS Japanese clips (Common Voiceja) to see if2.10%CER is real or FLEURS training contamination.
- Claim to check: Test
5. Migration
Changing choices_for_runtime and adding catalog entries must never disrupt existing installs:
- Never auto-switch or auto-download:
AppSettings::load_from(src/app_settings.rs:1086) validates saved selections throughtranscription_models::validate(&settings.transcription)(src/transcription_models.rs:569-591), which checksmodel.available()andmodel.supports_language(&selection.language)— notchoices_for_runtime().- A user whose saved selection is
(ParakeetV3, "de"),(ParakeetV3, "el"), or(Qwen3Asr06B, "tr")keeps their exact model and language on upgrade with zero background downloads or migrations. ParakeetV3("parakeet_v3"),CohereTranscribe("cohere_transcribe"),Qwen3Asr06B("qwen3_asr06_b"), andSenseVoiceSmall("sense_voice_small") remainavailable() == trueinMODELSwith stable wire names.
- Menu-bar picker stays intact:
src/macos_rune/status_item.rs:340lists allMODELSthat areavailable()andis_installed(), remembering each model's last language viatranscription_recents. Users withParakeetV3installed can still switch to and from it in the menu bar.
- Keep active/installed legacy models visible and deletable in the Settings picker (
src/rune_ui/picker.rs):- Today
picker::modal(src/rune_ui/picker.rs:170) only renders cards fromchoices_for_runtime(&language). If a user upgrades withParakeetV3active on French (or installed on disk using 740 MB) afterParakeetUltrareplacesParakeetV3inchoices_for_runtime("fr"),ParakeetV3would disappear from the French picker modal, hiding both itsActiveindicator and itsDeletebutton. - Fix in
picker.rs: MakeModelChoice::recommendationanOption<Recommendation>(or append a tail iterator inpicker::modal) for any model inMODELSthat supportslanguage, is not already inchoices_for_runtime(&language), and is either the user's active selection forlanguageoris_installed(). Render its card without a recommendation badge so the user seesActiveand can clickDeleteto reclaim the 740 MB / 850 MB after switching to the newRecommendedmodel.
- Today
6. Phasing and Integration with transcription-quality.md
Rollout timeline
├── Phase A (now, on transcribe-cpp 0.2.4, 0.5d)
│ └── Pure routing & picker cleanup in src/transcription_models.rs + src/rune_ui/picker.rs
└── Phase B (with transcription-quality.md Phase 3, transcribe-cpp 0.4.1, +1d)
└── Add ParakeetUltra, Canary1BV2, Qwen3Asr17B + full European/Asian table
Phase A: Immediate Routing Fixes on 0.2.4 (Effort: 0.5 days)
Ships immediately before any crate bump or new model downloads:
src/transcription_models.rs(choices_for_runtime):- Route
"el"(Greek) to[Rec: CohereTranscribe, Hint: WhisperLargeV3Turbo]before thePARAKEET_V3_LANGUAGESarm. - Route
"fi"and"sv"to[Rec: WhisperLargeV3Turbo, Fast: ParakeetV3]. - Route
"da"to[Rec: WhisperLargeV3Turbo, Fast: ParakeetV3](untilParakeetUltra+Canary1BV2land in Phase B). - Route
"ja"to[Rec: WhisperLargeV3Turbo, Fast: SenseVoiceSmall]. - Drop
SenseVoiceSmall(Fastest) from"yue"and"ko". - Split the
QWEN_LANGUAGESarm: keepQwen3Asr06BRecommendedfor"hi" | "vi" | "th" | "id"(addingAcc: CohereTranscribeon"vi"), and route"ar" | "tr" | "ms" | "fil" | "fa" | "mk"toRec: WhisperLargeV3Turbo(addingAcc: CohereTranscribeon"ar"). - In
AUTO_LANGUAGE: dropSenseVoiceSmalland changeQwen3Asr06Bfrom a secondRecommendedtoFastest([Rec: WhisperLargeV3Turbo, Fast: Qwen3Asr06B]). - In
"mt"(Maltese): omitWhisperLargeV3Turbo(RecognitionHints) where it gets 70.92% WER.
- Route
src/rune_ui/picker.rs:- Show installed or active models supporting
languagethat are no longer inchoices_for_runtime(&language)without a recommendation badge so users can see their active model and delete old weights. - Update unit tests in
src/transcription_models.rs.
- Show installed or active models supporting
Phase B: New Models with transcribe-cpp 0.4.1 (Effort: +1 day inside docs/plans/transcription-quality.md Phase 3)
Integrates directly into Phase 3 of docs/plans/transcription-quality.md (which already budgets 2 days for the 0.2.4 -> 0.4.1 bump and ParakeetUltra):
- Pin
ParakeetUltra,Canary1BV2, andQwen3Asr17BinMODELSwith verified byte counts and SHA-256 checksums. - Swap the
PARAKEET_V3_LANGUAGESbranches fromParakeetV3toParakeetUltra, addCanary1BV2asRecommendedonlv,sl(withParakeetUltraasFastest) and asMost accurateonbg,cs,da,et,hr,lt,ro. - Add
Qwen3Asr17BasMost accurateonauto,yue,ko,hi,vi,th,id,fa, and (if confirmed byses_e98f7b094ffe1Q3KtZFS6Np4KL)en. - Enable
supports_recognition_hints: trueonQwen3Asr17B(andQwen3Asr06Bif it passes the Phase 4 dictionary-echo check).
Decisions for Kit
- Ship Phase A (
el,tr,mk,ar,ms,fil,fa,ja,fi,sv,autofixes on0.2.4) now, or batch everything into the0.4.1bump?- Recommendation: Ship Phase A now (0.5d) — it fixes broken defaults like Greek (35.3% -> 9.0% WER), Filipino (35.4% -> 12.1%), Persian (50.3% -> 30.6%), Macedonian (35.1% -> 17.9%), and Turkish (16.7% -> 7.0%) using models Hex already ships.
- Which new models earn a catalog slot in Phase B alongside
ParakeetUltra: add bothCanary 1B v2(1.14 GB) andQwen3-ASR 1.7B(2.19 GB), or only one?- Recommendation: Add both
Canary 1B v2andQwen3-ASR 1.7B(3 new models total withParakeetUltra), and reject the other 7 candidates —Canary 1B v2cuts WER by 1.7–7.6 points across 9 European languages, whileQwen3-ASR 1.7Bis theMost accuratetier forAuto, Asian/Middle Eastern languages, and potentially English.
- Recommendation: Add both
- On Latvian (
lv) and Slovenian (sl), whereParakeetUltrais 18–19% WER andCanary 1B v2is 10.4–12.8% WER, shouldCanary 1B v2beRecommended(1.14 GB default download) orMost accurate?- Recommendation: Make
Canary 1B v2RecommendedandParakeetUltraFastestonlvandsl— a 6.2 to 7.6 point WER drop (nearly cutting errors in half at 105x realtime) is worth the extra 400 MB on first download.
- Recommendation: Make