iOS: the multilingual model experiment never takes dictation away; its download says what it's doing and recovers (#2337)

A TestFlight tester (build 120, iPhone 15 Pro Max, iOS 26.6) had Experiments ›
Multilingual model (Parakeet) on: voice input "started demanding Set up", then
sat at "Downloading 632.2 MB · Wi-Fi recommended" with no movement.

Why it blocked: with the experiment on and the dictation language one of
Parakeet Ultra's 25 (English included), Ultra became Hex's engine at once,
before it was downloaded. Hex's phase stayed on preparing (or failed, which
shows Set up) for the whole 632 MB download and the first Core ML compile, and
the keyboard's Dictate and Hex key opened Hex for it whether or not the model
was there, so nothing dictated anywhere until it finished.

Why it sat still: the download ran in prepare() on a foreground URLSession,
one 594 MB encoder file, shown only as a bar at 70% scale under a label that
never changed. Leaving Hex (the keyboard's handoff sends you back to the app you
type in) or Auto-Lock suspends it; on return a dead connection showed nothing
for up to 60-90 s, then failed to Set up.

What changed:
- Until Ultra is ready, Hex and the keyboard dictate with the engine they use
  with the experiment off. ParakeetUltraDownload (new) downloads beside it and
  ParakeetUltraTranscriber.prepare only loads (throws notDownloaded otherwise).
  A first load (Core ML's compile) runs beside the usual engine
  (DictationController.loadMultilingualParakeetBeside) and switches to Ultra
  when loaded; launches load Ultra directly only once downloaded and compiled
  for this iOS (multilingualStep). Where V2 dictates and leaves no room
  (English before iOS 26), Ultra takes V2's place for that load.
- The keyboard opens Hex for the experiment only once Hex has loaded it
  (MultilingualParakeetSettings.isModelReady in the App Group, set after a
  load, cleared while missing, set aside or off).
- The download shows bytes ("Downloading the multilingual model: 123.4 MB of
  632.2 MB" with a bar) on home and under the switch; resumes every file from its
  .partial (launch, retry, return to the front); a drop or 20 s without bytes
  tries again after 2, 4, 8, 16, 30 s, starting over whenever it got further; in
  the background it waits and picks up at once on return (fresh connection after
  25 s away). After 6 fruitless tries, a refused request or a full iPhone it says
  what happened in plain words with Try again and Turn off.
- A failed Ultra load or a load that ended Hex (ParakeetLoadGuard) falls back
  to the usual engine and says so with Try again (no longer only by turning the
  switch off, which deletes the 632 MB).
- A launch deletes a leftover download only when the switch is stored as off
  (isTurnedOff), not when the App Group reads empty (a launch before the first
  unlock would have deleted a good model). Turning it off now stops the download
  and waits for it before deleting.
- Removed: ParakeetUltraTranscriber.downloadDetail, its .downloadFailed error
  and the download/StallWatch inside its prepare (replaced by the above);
  strings "Downloading %@ · Wi-Fi recommended" and the old switch detail.
  New strings have zh-Hans. README describes the experiment's new behaviour.
- DEBUG builds read HEX_ULTRA_MODEL_BASE to download from a stand-in server.

Verified:
- xcodebuild test, iPhone 17 Pro Max iOS 26.5 simulator on the Mac Studio:
  753 tests, 0 failures, 3 skipped (new cases in DictationControllerTests:
  resume from .partial with Range, drop + stall retries, a moving download never
  running out of tries, 503 x6 / 404 failures with their words, stop before
  delete, keyboard flag, stored-off rule, multilingualStep).
- iPhone 15 Pro Max iOS 26.5 simulator, experiment on, a local stand-in server
  (Range, throttled, drop/stall switches): Dictate stayed available with the
  usual engine (V2 here; the sim has no Apple Speech) while the status counted
  MB; a dropped connection said "Trying again in 1 s" and resumed with Range; a
  20 s stall said so and resumed; Hex killed mid-download resumed from the
  partial at relaunch; backgrounded with the connection cut, it resumed at once
  on return; after a kill the resumed download from real Hugging Face finished
  from a partial and passed the hashes; Ultra loaded beside and took over (setup
  sheet "English · Parakeet Ultra", keyboard flag set); a forced load guard
  showed the failure with Try again, which loaded Ultra; deleting the model with
  the flag set cleared it at launch and downloaded again; the keyboard logged
  "multilingual model on, Dictate opens Hex" only when ready (else the sim's own
  "speech model missing" reason) and its Hex key went straight to recording
  instead of the download screen; turning it off deleted the folder and cleared
  the flag; with it off nothing shows or downloads.
- Not verified: a real iPhone, a real slow or flaky network, iOS's own
  suspension timing, the V2-replacement path (sims report no memory limit), and
  real audio (the Studio has no microphone).
This commit is contained in:
Ryan Vogel
2026-11-22 02:25:14 -05:00
committed by GitHub
parent 69f94b242a
commit 8583160e68
12 changed files with 1221 additions and 132 deletions
+3 -2
View File
@@ -1081,7 +1081,8 @@ final class KeyboardViewController: UIInputViewController {
/// English."): Hex downloads it, and the keyboard dictates again once it installs.
/// Without Full Access it is nil too, and the Hex key says dictation needs Full Access.
/// The same while Experiments › Multilingual model (Parakeet) dictates the language: only
/// Hex can load that model, so Dictate and the Hex key open Hex.
/// Hex can load that model, so Dictate and the Hex key open Hex, once Hex has it ready.
/// Until then the keyboard dictates as it would with the experiment off.
private func syncLocalDictationAvailability(access: Bool) {
guard KeyboardLocalDictation.isSupported else { return }
guard access || localDictation.isActive else {
@@ -1090,7 +1091,7 @@ final class KeyboardViewController: UIInputViewController {
return
}
let multilingual = MultilingualParakeetSettings()
let multilingualOpensHex = multilingual.isEnabled && multilingual.language() != nil
let multilingualOpensHex = multilingual.keyboardOpensHex(language: multilingual.language())
let needsHex = !localDictation.isActive && (LanguageSettings().keyboardNeedsHex() || multilingualOpensHex)
let available = localDictation.isActive || (!needsHex && localDictation.hasMemoryToStart)
guard available != localDictationAvailable else { return }
+7
View File
@@ -200,6 +200,13 @@ struct ContentView: View {
.transition(.opacity.combined(with: .scale(scale: 0.98)))
}
// Experiments › Multilingual model: its download or load, beside dictation.
MultilingualModelStatus(
download: dictation.multilingualDownload,
retry: { dictation.retryMultilingualParakeet() },
turnOff: { dictation.turnOffMultilingualParakeet() }
)
SettingsCard {
ForEach(Array(SettingsPage.allCases.enumerated()), id: \.element) { index, page in
if index > 0 {
+141 -8
View File
@@ -37,20 +37,147 @@ extension DictationController {
}
/// Experiments › Multilingual model (Parakeet) at launch: true when Parakeet Ultra takes
/// the dictation language and is getting ready. With the experiment off, it only deletes a
/// download turning it off left mid-way.
/// the dictation language and loads now, downloaded and compiled for this iOS (seconds).
/// Otherwise Hex gets ready with the engine it uses without it, and the download or a
/// first load runs beside it (`syncMultilingualParakeet`). With the experiment turned
/// off, it only deletes a download turning it off left mid-way.
private func prepareMultilingualParakeetAtLaunch() -> Bool {
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).noteLaunch()
guard MultilingualParakeetSettings().isEnabled else {
ParakeetUltraModel.remove()
let loads = ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel)
loads.noteLaunch()
let settings = MultilingualParakeetSettings()
guard settings.isEnabled else {
if settings.isTurnedOff {
ParakeetUltraModel.remove()
}
return false
}
if !ParakeetUltraModel.isDownloaded() {
settings.isModelReady = false
}
if loads.lastLoadWasInterrupted, settings.language() != nil {
settings.isModelReady = false
multilingualDownload.fail(ParakeetUltraTranscriber.setAsideMessage)
}
guard Self.multilingualStep(
language: ParakeetUltraTranscriber.chosenLanguage(),
downloaded: ParakeetUltraModel.isDownloaded(),
compiled: ParakeetUltraTranscriber.isCompiled
) == .dictate else {
// A first load waits for the engine Hex gets ready meanwhile (its end syncs).
startMultilingualDownloadIfWanted()
return false
}
guard ParakeetUltraTranscriber.chosenLanguage() != nil else { return false }
engineName = ParakeetUltraTranscriber.name
prepareModel()
return true
}
/// What Hex does about Parakeet Ultra: nothing (off, a language it doesn't take, set
/// aside), download it, load it beside the engine that dictates meanwhile (a first load
/// compiles for a minute or more), or dictate with it.
enum MultilingualStep: Equatable {
case none
case download
case loadBeside
case dictate
}
nonisolated static func multilingualStep(language: String?, downloaded: Bool, compiled: Bool) -> MultilingualStep {
guard language != nil else { return .none }
guard downloaded else { return .download }
return compiled ? .dictate : .loadBeside
}
/// Starts Parakeet Ultra's download when the experiment wants it and it isn't on the
/// iPhone, whatever engine is getting ready meanwhile.
func startMultilingualDownloadIfWanted() {
// Memory left only matters for a load, not a download.
guard ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0) != nil,
!ParakeetUltraModel.isDownloaded() else { return }
MultilingualParakeetSettings().isModelReady = false
multilingualDownload.start()
}
/// Once an engine is ready, and once the download finishes: downloads Parakeet Ultra, or
/// loads it beside the engine that dictates, when the experiment wants it and it isn't the
/// one dictating already.
func syncMultilingualParakeet() {
guard MultilingualParakeetSettings().isEnabled else {
multilingualDownload.settle()
return
}
let language = ParakeetUltraTranscriber.chosenLanguage()
let wanted = ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0)
switch Self.multilingualStep(
language: wanted,
downloaded: ParakeetUltraModel.isDownloaded(),
compiled: true
) {
case .none:
if !multilingualDownload.isFailed { multilingualDownload.settle() }
case .download:
startMultilingualDownloadIfWanted()
case .loadBeside, .dictate:
if engineName == ParakeetUltraTranscriber.name, isModelReady {
multilingualDownload.settle()
} else if let language {
loadMultilingualParakeetBeside(language: language)
} else if let wanted, replaceParakeetWithMultilingual(language: wanted) {
multilingualDownload.loading()
} else {
// Too little memory left to load it now: said, with Try again.
multilingualDownload.fail(
String(localized: "Parakeet Ultra needs more memory than Hex has here. Hex uses its usual speech model instead.")
)
}
}
}
/// Parakeet Ultra loaded: the keyboard opens Hex for its languages from now on.
func multilingualParakeetLoaded() {
MultilingualParakeetSettings().isModelReady = true
multilingualDownload.settle()
}
/// Parakeet Ultra didn't load. Missing files download again; anything else sets it aside
/// (`ParakeetLoadGuard`) until Try again, so Hex doesn't try at every launch.
func multilingualParakeetFailedToLoad(_ error: Error) {
let settings = MultilingualParakeetSettings()
settings.isModelReady = false
guard settings.isEnabled, !(error is CancellationError) else {
multilingualDownload.settle()
return
}
if (error as? ParakeetUltraTranscriber.TranscriberError) == .notDownloaded {
multilingualDownload.settle()
startMultilingualDownloadIfWanted()
return
}
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).setAside()
multilingualDownload.fail(
Self.switchNotice(for: error) ?? UserFacingError.message(for: error, context: .model)
)
}
/// Try again beside the multilingual model's status: lets a set-aside load try again, and
/// picks up the download from what's on disk.
func retryMultilingualParakeet() {
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
multilingualDownload.settle()
if ParakeetUltraModel.isDownloaded() {
syncMultilingualParakeet()
} else if ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0) != nil {
MultilingualParakeetSettings().isModelReady = false
multilingualDownload.restart()
}
}
/// Turn off beside the multilingual model's status, as the switch in Experiments does.
func turnOffMultilingualParakeet() {
MultilingualParakeetSettings().isEnabled = false
multilingualParakeetChanged()
}
/// Before iOS 26: Parakeet loads at launch once downloaded, as before. The recognizer gets
/// ready at launch only once Speech Recognition is allowed; otherwise setup asks for it
/// (onboarding's first page, or Set up on home), never a launch by itself.
@@ -86,8 +213,14 @@ extension DictationController {
}
static func defaultTranscriber() async -> any LocalTranscribing {
// Experiments › Multilingual model (Parakeet), for its languages; off, nothing changes.
if let language = ParakeetUltraTranscriber.chosenLanguage() {
// Experiments › Multilingual model (Parakeet), for its languages once it's downloaded and
// compiled; until then (and off) the engine Hex uses without it.
if let language = ParakeetUltraTranscriber.chosenLanguage(),
multilingualStep(
language: language,
downloaded: ParakeetUltraModel.isDownloaded(),
compiled: ParakeetUltraTranscriber.isCompiled
) == .dictate {
return ParakeetUltraTranscriber(language: language)
}
let choice = SpeechEngineChoice.current
+112 -7
View File
@@ -78,6 +78,9 @@ final class DictationController {
private var transcriber: any LocalTranscribing
private let transcriberFactory: () async -> any LocalTranscribing
/// Experiments › Multilingual model (Parakeet): its download, beside the engine Hex
/// dictates with until it's ready.
@ObservationIgnored let multilingualDownload: ParakeetUltraDownload
private let cleaner: (any TranscriptCleaning)?
private let cleanupSettings: AppleIntelligenceCleanupSettings
private let customVocabularySettings: CustomVocabularySettings
@@ -228,7 +231,8 @@ final class DictationController {
hinglishSettings: HinglishSettings = HinglishSettings(),
typingJournalSettings: TypingJournalSettings = TypingJournalSettings(),
learnedDictationCorrections: (@MainActor (String?) -> [DictationCorrection])? = nil,
languageSettings: LanguageSettings = LanguageSettings()
languageSettings: LanguageSettings = LanguageSettings(),
multilingualDownload: ParakeetUltraDownload? = nil
) {
let resolvedBridge = keyboardBridge ?? KeyboardBridge()
let resolvedAudio = audio ?? SystemAudioManager()
@@ -238,6 +242,7 @@ final class DictationController {
self.keyboardBridge = resolvedBridge
self.transcriber = initialTranscriber
self.transcriberFactory = transcriberFactory ?? Self.defaultTranscriber
self.multilingualDownload = multilingualDownload ?? ParakeetUltraDownload()
self.cleaner = cleaner
self.cleanupSettings = cleanupSettings
self.customVocabularySettings = customVocabularySettings
@@ -265,6 +270,9 @@ final class DictationController {
wireAudioCallbacks()
wireSystemIntentBus()
self.multilingualDownload.onDownloaded = { [weak self] in
self?.syncMultilingualParakeet()
}
if autoPrepareOnInit {
// At launch no dictation is running, so any Hex Live Activity is left over.
liveActivity.endImmediately()
@@ -373,9 +381,34 @@ final class DictationController {
}
}
/// Experiments › Multilingual model (Parakeet) turned on or off: prepares the engine Hex
/// picks now, or once the dictation in flight is done.
/// Experiments › Multilingual model (Parakeet) turned on or off. On, the engine Hex
/// dictates with stays while Parakeet Ultra downloads and loads beside it; a load that
/// ended Hex gets another try. Off, the download stops and is deleted, and Hex prepares
/// the engine it picks without it if Ultra was the one dictating (now, or once the
/// dictation in flight is done).
func multilingualParakeetChanged() {
let settings = MultilingualParakeetSettings()
if settings.isEnabled {
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
multilingualDownload.settle()
if isModelReady || phase == .preparingModel {
syncMultilingualParakeet()
return
}
} else {
settings.isModelReady = false
let download = multilingualDownload
Task {
// Nothing writes to the folder any more once `stop` returns.
await download.stop()
ParakeetUltraModel.remove()
}
if engineName != ParakeetUltraTranscriber.name, isModelReady, phase != .preparingModel {
// Ultra wasn't dictating: drop a load of it beside the engine that is.
modelPrepGeneration &+= 1
return
}
}
switch phase {
case .recording, .transcribing, .requestingPermission:
pendingLanguagePreparation = true
@@ -701,9 +734,12 @@ final class DictationController {
/// `fallbacksLeft` bounds how often Automatic moves on to another language after one
/// whose model won't install here; each move sets a language aside, so it can't loop.
func prepareModel(fallbacksLeft: Int = 2) {
func prepareModel(fallbacksLeft: Int = 2, candidate preset: (any LocalTranscribing)? = nil) {
modelPrepGeneration &+= 1
let prepGen = modelPrepGeneration
// A load beside the engine this replaces is dropped; the end of this one decides again.
multilingualDownload.dropLoad()
startMultilingualDownloadIfWanted()
phase = .preparingModel
copied = false
modelPreparation = ModelPreparationProgress(
@@ -716,10 +752,17 @@ final class DictationController {
Task {
defer { backgroundTime.end() }
var candidateIsUltra = false
do {
let candidate = await transcriberFactory()
let candidate: any LocalTranscribing
if let preset {
candidate = preset
} else {
candidate = await transcriberFactory()
}
guard self.modelPrepGeneration == prepGen else { return }
engineName = candidate.displayName
candidateIsUltra = candidate is ParakeetUltraTranscriber
try await candidate.prepare { [weak self] progress in
Task { @MainActor [weak self] in
guard let self, self.modelPrepGeneration == prepGen else { return }
@@ -734,12 +777,23 @@ final class DictationController {
self.keyboardBridge.setModelInstalled(true)
self.isModelReady = true
self.phase = .ready
if candidateIsUltra {
self.multilingualParakeetLoaded()
}
self.publishKeyboardSnapshot()
if self.pendingHandoff != nil {
self.activateWarmSessionAndRunHandoff()
}
self.syncMultilingualParakeet()
} catch {
guard self.modelPrepGeneration == prepGen else { return }
if candidateIsUltra, fallbacksLeft > 0, !(error is CancellationError) {
// Parakeet Ultra didn't load: the engine Hex uses without it dictates, and
// Hex says why, with Try again.
self.multilingualParakeetFailedToLoad(error)
self.prepareModel(fallbacksLeft: fallbacksLeft - 1)
return
}
let unavailable = error as? SpeechModelUnavailable
if fallbacksLeft > 0, let notice = Self.switchNotice(for: error) {
// Automatic set the language aside, or Parakeet had no room: prepare what
@@ -754,18 +808,70 @@ final class DictationController {
self.pendingHandoff = nil
self.phase = .failed(UserFacingError.message(for: error, context: .model))
self.publishKeyboardSnapshot()
// A downloaded Parakeet Ultra can still dictate.
self.syncMultilingualParakeet()
}
}
}
/// Parakeet V2 dictates and leaves no room to load Parakeet Ultra beside it (English
/// before iOS 26): Ultra takes its place and loads as it would at launch, rather than
/// never getting its first load. False when V2 isn't the engine.
func replaceParakeetWithMultilingual(language: String) -> Bool {
guard transcriber is ParakeetTranscriber, phase != .preparingModel else { return false }
transcriber = ParakeetTranscriber()
isModelReady = false
prepareModel(candidate: ParakeetUltraTranscriber(language: language))
return true
}
/// Loads Parakeet Ultra while the engine Hex has keeps dictating, and switches to it once
/// it's loaded; a dictation in flight isn't interrupted. A failure keeps that engine and
/// says why. Only while no other preparation runs.
func loadMultilingualParakeetBeside(language: String) {
guard phase != .preparingModel else { return }
modelPrepGeneration &+= 1
let prepGen = modelPrepGeneration
multilingualDownload.loading()
let backgroundTime = BackgroundTime("Prepare the multilingual model")
Task {
defer { backgroundTime.end() }
let candidate = ParakeetUltraTranscriber(language: language)
do {
try await candidate.prepare { _ in }
guard self.modelPrepGeneration == prepGen else { return }
self.transcriber = candidate
self.engineName = candidate.displayName
self.transcriptionLanguage = await candidate.languageName
self.transcriptionLanguageID = await candidate.languageIdentifier
self.languageOffer = nil
self.isModelReady = true
self.keyboardBridge.setModelInstalled(true)
switch self.phase {
case .modelRequired, .failed:
self.phase = .ready
default:
break
}
self.multilingualParakeetLoaded()
self.publishKeyboardSnapshot()
} catch {
guard self.modelPrepGeneration == prepGen else { return }
self.multilingualParakeetFailedToLoad(error)
}
}
}
func appMovedToBackground() {
isInBackground = true
multilingualDownload.appMovedToBackground()
}
/// Picks up a download that iOS cut off while Hex was in the background; FluidAudio
/// resumes each file from the bytes it has.
func appBecameActive() {
isInBackground = false
multilingualDownload.appBecameActive()
guard preparationFailedInBackground else { return }
preparationFailedInBackground = false
if case .failed = phase, !isModelReady {
@@ -775,7 +881,6 @@ final class DictationController {
nonisolated static func isDownloadInterruption(_ error: Error) -> Bool {
if error is StallWatch.Stalled || error is URLError { return true }
if (error as? ParakeetUltraTranscriber.TranscriberError) == .downloadFailed { return true }
return (error as? ParakeetTranscriber.TranscriberError) == .downloadFailed
}
@@ -1469,7 +1574,7 @@ final class DictationController {
/// Background time from iOS while Hex finishes a keyboard's dictation, ended when it's done
/// or when iOS asks for it back.
@MainActor
private final class BackgroundTime {
final class BackgroundTime {
private var id = UIBackgroundTaskIdentifier.invalid
init(_ name: String) {
+150 -30
View File
@@ -3553,26 +3553,6 @@
}
}
},
"Parakeet Ultra didn't download. Check your connection and try again." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "Parakeet Ultra 未能下载。请检查网络连接后重试。"
}
}
}
},
"Downloading %@ · Wi-Fi recommended" : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "正在下载 %@ · 建议使用 Wi-Fi"
}
}
}
},
"Multilingual model (Parakeet)" : {
"localizations" : {
"zh-Hans" : {
@@ -3583,16 +3563,6 @@
}
}
},
"25 languages, even mixed in a sentence. %@ download. Keyboard dictation opens Hex." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "支持 25 种语言,一句话里混着说也行。需下载 %@。键盘听写会打开 Hex。"
}
}
}
},
"Needs an iPhone with 6 GB of memory or more" : {
"localizations" : {
"zh-Hans" : {
@@ -3862,6 +3832,156 @@
}
}
}
},
"Parakeet Ultra isn't downloaded yet." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "Parakeet Ultra 尚未下载。"
}
}
}
},
"Hex closed while getting the multilingual model ready, likely short of memory." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "Hex 在准备多语言模型时关闭了,可能是内存不足。"
}
}
}
},
"The multilingual model needs %@ of free space. Free some up, then try again." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "多语言模型需要 %@ 的可用空间。请先释放一些空间,然后重试。"
}
}
}
},
"The multilingual model's download stopped. Check your connection. It picks up where it left off." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "多语言模型的下载已停止。请检查网络连接。下载会从中断处继续。"
}
}
}
},
"Hugging Face is limiting downloads right now. Try again in a few minutes." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "Hugging Face 目前限制了下载。请过几分钟再试。"
}
}
}
},
"Hugging Face didn't send the multilingual model (error %lld). Try again later." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "Hugging Face 没有发送多语言模型(错误 %lld)。请稍后再试。"
}
}
}
},
"A file of the multilingual model arrived damaged. Try again to download it again." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "多语言模型的一个文件下载后已损坏。请重试以重新下载。"
}
}
}
},
"The multilingual model didn't download. Try again." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "多语言模型未能下载。请重试。"
}
}
}
},
"Downloading the multilingual model: %@ of %@" : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "正在下载多语言模型:已下载 %1$@,共 %2$@"
}
}
}
},
"The multilingual model's download stopped. Trying again in %lld s." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "多语言模型的下载已停止。%lld 秒后重试。"
}
}
}
},
"The multilingual model's download is paused. It carries on when you open Hex." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "多语言模型的下载已暂停。打开 Hex 后会继续。"
}
}
}
},
"Getting the multilingual model ready" : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "正在准备多语言模型"
}
}
}
},
"Turn off" : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "关闭"
}
}
}
},
"25 languages, even mixed in a sentence. %@ download, Wi-Fi recommended. Once it's ready, keyboard dictation opens Hex." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "支持 25 种语言,一句话里混着说也行。需下载 %@,建议使用 Wi-Fi。准备好后,键盘听写会打开 Hex。"
}
}
}
},
"Hex dictates with its usual model meanwhile." : {
"localizations" : {
"zh-Hans" : {
"stringUnit" : {
"state" : "translated",
"value" : "在此期间,Hex 会用平时的模型听写。"
}
}
}
}
},
"version" : "1.0"
+422
View File
@@ -0,0 +1,422 @@
import Foundation
import Observation
import SwiftUI
/// Experiments › Multilingual model (Parakeet): gets Parakeet Ultra onto the iPhone beside the
/// engine Hex dictates with meanwhile, so turning the experiment on never takes dictation away.
/// Each file resumes from its `.partial`, at the next launch too. A dropped connection or a
/// download that stops moving for `stallLimit` tries again by itself (after 2, 4, 8, 16, 30 s),
/// and at once when Hex comes back to the front; after `automaticTries` in a row that got nothing
/// further, it says what went wrong and waits for Try again (or, for a connection, the next
/// time Hex comes to the front).
@MainActor
@Observable
final class ParakeetUltraDownload {
enum State: Equatable {
/// Nothing to say: the experiment off, not wanted for this language, or Ultra dictates.
case idle
/// Bytes on disk.
case downloading(Int64)
/// Stopped, and tries again by itself in `retryIn` seconds, or once Hex is in front (nil).
case waiting(Int64, retryIn: Int?)
/// On disk; Hex gets it ready beside the engine it dictates with.
case loading
/// What went wrong, in plain words.
case failed(String)
}
private(set) var state = State.idle
/// Called once every file is on disk.
@ObservationIgnored var onDownloaded: (@MainActor () -> Void)?
nonisolated static let stallLimit: Duration = .seconds(20)
static let automaticTries = 6
@ObservationIgnored private var task: Task<Void, Never>?
@ObservationIgnored private var tries = 0
/// What was on disk at the last stop: a download that moved since starts its tries over.
@ObservationIgnored private var bytesAtLastStop: Int64 = 0
@ObservationIgnored private var failedForConnection = false
@ObservationIgnored private var backgroundedAt: Date?
@ObservationIgnored private var backgroundTime: BackgroundTime?
@ObservationIgnored private let directory: URL
@ObservationIgnored private let files: [ParakeetUltraModel.File]
@ObservationIgnored private let base: String
@ObservationIgnored private let session: URLSession
@ObservationIgnored private let stallLimit: Duration
@ObservationIgnored private let backoff: (Int) -> Int
init(
directory: URL = ParakeetUltraModel.directory,
files: [ParakeetUltraModel.File] = ParakeetUltraModel.files,
base: String = ParakeetUltraModel.baseURL,
session: URLSession = ParakeetUltraDownload.makeSession(),
stallLimit: Duration = ParakeetUltraDownload.stallLimit,
backoff: @escaping (Int) -> Int = ParakeetUltraDownload.backoff
) {
self.directory = directory
self.files = files
self.base = base
self.session = session
self.stallLimit = stallLimit
self.backoff = backoff
}
/// A connection that sends nothing for 30 s ends, rather than after the default minute.
nonisolated static func makeSession() -> URLSession {
let configuration = URLSessionConfiguration.default
configuration.timeoutIntervalForRequest = 30
return URLSession(configuration: configuration)
}
/// Seconds before automatic try `n` (1, 2, …): 2, 4, 8, 16, then 30.
nonisolated static func backoff(_ tries: Int) -> Int {
min(30, 1 << max(1, min(tries, 5)))
}
var isRunning: Bool { task != nil }
var total: Int64 { files.reduce(0) { $0 + $1.size } }
/// Starts, or resumes from what's on disk; does nothing while it runs.
func start() {
guard task == nil else { return }
tries = 0
run()
}
/// Starts over from what's on disk, with all the automatic tries again.
func restart() {
task?.cancel()
task = nil
tries = 0
run()
}
/// Stops it and waits until nothing writes to the folder any more, so it can be deleted.
func stop() async {
let running = task
task = nil
running?.cancel()
await running?.value
endBackgroundTime()
state = .idle
}
func loading() {
state = .loading
}
/// A load beside the usual engine that another preparation replaced.
func dropLoad() {
if state == .loading { state = .idle }
}
/// Ultra dictates, or isn't wanted any more.
func settle() {
guard task == nil else { return }
state = .idle
}
/// A load that failed, or one that ended Hex: said with Try again beside it.
func fail(_ message: String) {
task?.cancel()
task = nil
failedForConnection = false
state = .failed(message)
}
func appMovedToBackground() {
backgroundedAt = Date()
// A few more seconds from iOS, so a quick look at another app doesn't stop it.
if task != nil, backgroundTime == nil {
backgroundTime = BackgroundTime("Download the multilingual model")
}
}
/// Back in front: a download that stopped tries again at once, and one suspended long
/// enough for iOS to drop its connection starts a fresh one from where it is.
func appBecameActive() {
let away = backgroundedAt.map { Date().timeIntervalSince($0) } ?? 0
backgroundedAt = nil
endBackgroundTime()
switch state {
case .waiting:
restart()
case .failed where failedForConnection:
restart()
case .downloading where away > 25:
restart()
default:
break
}
}
private func endBackgroundTime() {
backgroundTime?.end()
backgroundTime = nil
}
private func run() {
failedForConnection = false
bytesAtLastStop = ParakeetUltraModel.bytesOnDisk(in: directory, files: files)
state = .downloading(bytesAtLastStop)
task = Task { [weak self] in
await self?.loop()
}
}
private func loop() async {
while !Task.isCancelled {
do {
try await Self.fetch(
directory: directory,
files: files,
base: base,
session: session,
stallLimit: stallLimit
) { [weak self] bytes in
await MainActor.run {
guard let self, self.task != nil, case .downloading = self.state else { return }
self.state = .downloading(bytes)
}
}
guard !Task.isCancelled else { return }
task = nil
endBackgroundTime()
state = .loading
onDownloaded?()
return
} catch {
guard !Task.isCancelled else { return }
let onDisk = ParakeetUltraModel.bytesOnDisk(in: directory, files: files)
if onDisk > bytesAtLastStop {
tries = 0
}
bytesAtLastStop = onDisk
tries += 1
print("multilingual model: download try \(tries) stopped at \(onDisk) bytes: \(error)")
guard Self.triesAgain(after: error), tries < Self.automaticTries else {
task = nil
endBackgroundTime()
failedForConnection = Self.triesAgain(after: error)
state = .failed(Self.message(for: error))
return
}
if backgroundedAt != nil {
// Suspended soon anyway: tries again once Hex is back in front.
task = nil
endBackgroundTime()
state = .waiting(onDisk, retryIn: nil)
return
}
for remaining in stride(from: backoff(tries), to: 0, by: -1) {
state = .waiting(onDisk, retryIn: remaining)
try? await Task.sleep(for: .seconds(1))
guard !Task.isCancelled else { return }
}
state = .downloading(onDisk)
}
}
}
/// One try: the download beside a watch that publishes its bytes twice a second and ends
/// it once they stop moving for `stallLimit` (time while iOS had Hex suspended doesn't
/// count). Returns only once nothing writes to the folder any more.
nonisolated static func fetch(
directory: URL,
files: [ParakeetUltraModel.File],
base: String,
session: URLSession,
stallLimit: Duration,
publish: @escaping @Sendable (Int64) async -> Void
) async throws {
let bytes = ByteReading(ParakeetUltraModel.bytesOnDisk(in: directory, files: files))
try await withThrowingTaskGroup(of: Void.self) { group in
group.addTask {
try await ParakeetUltraModel.download(to: directory, files: files, base: base, session: session) {
bytes.value = $0
}
}
group.addTask {
let clock = ContinuousClock()
let poll = Duration.milliseconds(500)
var last = bytes.value
var since = clock.now
var polled = since
while true {
try await Task.sleep(for: poll)
let now = clock.now
let resumed = now - polled > poll + StallWatch.suspensionGap
polled = now
let current = bytes.value
await publish(current)
if current != last || resumed {
last = current
since = now
} else if now - since >= stallLimit {
throw StallWatch.Stalled()
}
}
}
// The download finishing, or either one throwing, ends the other.
_ = try await group.next()
group.cancelAll()
}
}
/// A dropped connection, a download that stopped moving, a busy server or a damaged file
/// is tried again; a refused request, a full iPhone or anything else waits for the person.
nonisolated static func triesAgain(after error: Error) -> Bool {
if error is StallWatch.Stalled || error is URLError { return true }
switch error as? ParakeetUltraModel.DownloadError {
case .badStatus(let status):
return status == 0 || status == 408 || status == 429 || status >= 500
case .corrupt:
return true
case nil:
return false
}
}
nonisolated static func isOutOfSpace(_ error: Error) -> Bool {
let ns = error as NSError
return (ns.domain == NSCocoaErrorDomain && ns.code == NSFileWriteOutOfSpaceError)
|| (ns.domain == NSPOSIXErrorDomain && ns.code == Int(ENOSPC))
}
nonisolated static func message(for error: Error) -> String {
if isOutOfSpace(error) {
return String(localized: "The multilingual model needs \(ParakeetUltraTranscriber.downloadSize) of free space. Free some up, then try again.")
}
if error is StallWatch.Stalled || error is URLError {
return String(localized: "The multilingual model's download stopped. Check your connection. It picks up where it left off.")
}
switch error as? ParakeetUltraModel.DownloadError {
case .badStatus(429):
return String(localized: "Hugging Face is limiting downloads right now. Try again in a few minutes.")
case .badStatus(let status):
return String(localized: "Hugging Face didn't send the multilingual model (error \(status)). Try again later.")
case .corrupt:
return String(localized: "A file of the multilingual model arrived damaged. Try again to download it again.")
case nil:
return String(localized: "The multilingual model didn't download. Try again.")
}
}
/// What Hex says about it, or nil while there's nothing to say.
var status: String? {
switch state {
case .idle:
nil
case .downloading(let bytes):
String(localized: "Downloading the multilingual model: \(Self.size(bytes)) of \(Self.size(total))")
case .waiting(_, let seconds?):
String(localized: "The multilingual model's download stopped. Trying again in \(seconds) s.")
case .waiting(_, nil):
String(localized: "The multilingual model's download is paused. It carries on when you open Hex.")
case .loading:
String(localized: "Getting the multilingual model ready")
case .failed(let message):
message
}
}
/// The share on disk, while it downloads or waits.
var fraction: Double? {
switch state {
case .downloading(let bytes), .waiting(let bytes, _):
total > 0 ? min(1, Double(bytes) / Double(total)) : 0
default:
nil
}
}
var isFailed: Bool {
if case .failed = state { return true }
return false
}
nonisolated static func size(_ bytes: Int64) -> String {
ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
.replacingOccurrences(of: " ", with: "\u{00A0}")
}
}
/// Bytes written by the download and read by its watch.
private nonisolated final class ByteReading: @unchecked Sendable {
private let lock = NSLock()
private var current: Int64
init(_ value: Int64) {
current = value
}
var value: Int64 {
get { lock.withLock { current } }
set { lock.withLock { current = newValue } }
}
}
/// The multilingual model's download or load, on home and under its switch in Experiments:
/// what it's doing, that Hex dictates with its usual model meanwhile, and after a failure
/// Try again and Turn off.
struct MultilingualModelStatus: View {
let download: ParakeetUltraDownload
let retry: () -> Void
let turnOff: () -> Void
var body: some View {
if let status = download.status {
VStack(alignment: .leading, spacing: 10) {
HStack(alignment: .firstTextBaseline, spacing: 10) {
Image(systemName: download.isFailed ? "exclamationmark.circle.fill" : "arrow.down.circle")
.font(.system(size: 15))
.foregroundStyle(download.isFailed ? HexTheme.danger : HexTheme.textMuted)
VStack(alignment: .leading, spacing: 3) {
Text(status)
.font(.system(size: 13))
.foregroundStyle(HexTheme.textBase)
.monospacedDigit()
.fixedSize(horizontal: false, vertical: true)
.accessibilityIdentifier("multilingual-model-status")
Text("Hex dictates with its usual model meanwhile.")
.font(.system(size: 12))
.foregroundStyle(HexTheme.textMuted)
.fixedSize(horizontal: false, vertical: true)
}
Spacer(minLength: 0)
}
if let fraction = download.fraction {
ProgressView(value: fraction)
.tint(HexTheme.textBase)
}
if download.isFailed {
HStack(spacing: 8) {
Button("Try again", action: retry)
.accessibilityIdentifier("multilingual-model-try-again")
Button("Turn off", action: turnOff)
.accessibilityIdentifier("multilingual-model-turn-off")
}
.font(.system(size: 13, weight: .medium))
.foregroundStyle(HexTheme.textBase)
.buttonStyle(MultilingualStatusButtonStyle())
}
}
.padding(14)
.background(
download.isFailed ? HexTheme.dangerSurface : Color.white.opacity(0.055),
in: RoundedRectangle(cornerRadius: 14, style: .continuous)
)
}
}
}
private struct MultilingualStatusButtonStyle: ButtonStyle {
func makeBody(configuration: Configuration) -> some View {
configuration.label
.padding(.horizontal, 12)
.frame(height: 28)
.background(Color.white.opacity(configuration.isPressed ? 0.15 : 0.09), in: Capsule())
}
}
+35 -9
View File
@@ -21,6 +21,17 @@ nonisolated enum ParakeetUltraModel {
case corrupt(String)
}
/// The stand-in server simulator runs download from (`HEX_ULTRA_MODEL_BASE`), in debug
/// builds only: it serves the same files at the same paths.
static var baseURL: String {
#if DEBUG
if let base = ProcessInfo.processInfo.environment["HEX_ULTRA_MODEL_BASE"], !base.isEmpty {
return base
}
#endif
return "https://huggingface.co/\(repository)/resolve/\(revision)"
}
static let repository = "FluidInference/parakeet-ultra-coreml"
static let revision = "95eaa59a39d4394f047a4dc5cce480388a60d1b6"
@@ -76,20 +87,32 @@ nonisolated enum ParakeetUltraModel {
}
/// Every file at its full size; each was checked against its digest before it got its name.
static func isDownloaded(in directory: URL = directory) -> Bool {
static func isDownloaded(in directory: URL = directory, files: [File] = files) -> Bool {
files.allSatisfy { size(of: directory.appendingPathComponent($0.path)) == $0.size }
}
static func url(for file: File) -> URL {
URL(string: "https://huggingface.co/\(repository)/resolve/\(revision)/\(file.path)")!
/// What's on disk so far, finished files and `.partial` ones: where a download resumes.
static func bytesOnDisk(in directory: URL = directory, files: [File] = files) -> Int64 {
files.reduce(0) { total, file in
let destination = directory.appendingPathComponent(file.path)
if let size = size(of: destination), size == file.size { return total + size }
let partial = size(of: destination.appendingPathExtension("partial")) ?? 0
return total + min(partial, file.size)
}
}
static func url(for file: File, base: String = baseURL) -> URL {
URL(string: "\(base)/\(file.path)")!
}
/// Downloads what's missing, resuming a file from its `.partial`. `progress` gets the
/// fraction of `totalBytes` on disk.
/// bytes on disk, first what an earlier download left.
static func download(
to directory: URL = directory,
files: [File] = files,
base: String = baseURL,
session: URLSession = .shared,
progress: @escaping @Sendable (Double) -> Void
progress: @escaping @Sendable (Int64) -> Void
) async throws {
try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
var folder = directory
@@ -97,19 +120,21 @@ nonisolated enum ParakeetUltraModel {
values.isExcludedFromBackup = true
try? folder.setResourceValues(values)
progress(bytesOnDisk(in: directory, files: files))
var done: Int64 = 0
for file in files {
try Task.checkCancellation()
let destination = directory.appendingPathComponent(file.path)
if size(of: destination) == file.size {
done += file.size
progress(Double(done) / Double(totalBytes))
continue
}
let before = done
try await fetch(file, to: destination, session: session) { written in
progress(Double(before + written) / Double(totalBytes))
try await fetch(file, to: destination, base: base, session: session) { written in
progress(before + written)
}
done += file.size
progress(done)
}
}
@@ -167,6 +192,7 @@ nonisolated enum ParakeetUltraModel {
private static func fetch(
_ file: File,
to destination: URL,
base: String,
session: URLSession,
onBytes: @escaping @Sendable (Int64) -> Void
) async throws {
@@ -179,7 +205,7 @@ nonisolated enum ParakeetUltraModel {
offset = 0
}
if offset < file.size {
var request = URLRequest(url: url(for: file))
var request = URLRequest(url: url(for: file, base: base))
if offset > 0 {
request.setValue("bytes=\(offset)-", forHTTPHeaderField: "Range")
}
+16 -45
View File
@@ -10,8 +10,9 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
enum TranscriberError: HexDescribedError {
case notPrepared
case emptyTranscript
case downloadFailed
case notEnoughMemory
/// Not on the iPhone yet: `ParakeetUltraDownload` gets it beside the usual engine.
case notDownloaded
var errorDescription: String? {
switch self {
@@ -19,15 +20,19 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
String(localized: "Parakeet Ultra is not ready.")
case .emptyTranscript:
String(localized: "Parakeet did not detect any speech.")
case .downloadFailed:
String(localized: "Parakeet Ultra didn't download. Check your connection and try again.")
case .notEnoughMemory:
String(localized: "Parakeet Ultra needs more memory than Hex has here.")
case .notDownloaded:
String(localized: "Parakeet Ultra isn't downloaded yet.")
}
}
}
static let name = "Parakeet Ultra"
/// After a load that ended Hex (`ParakeetLoadGuard`), beside Try again.
nonisolated static var setAsideMessage: String {
String(localized: "Hex closed while getting the multilingual model ready, likely short of memory.")
}
/// `ParakeetLoadGuard`'s and `ParakeetCompileRecord`'s keys, apart from V2's.
static let loadGuardModel = "parakeetUltra"
static let compileRecordKey = "parakeetUltra.compiledForSystem"
@@ -74,47 +79,33 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
.replacingOccurrences(of: " ", with: "\u{00A0}")
}
/// What the bar says while it downloads: the size, and that Wi-Fi is better for it.
nonisolated static var downloadDetail: String {
String(localized: "Downloading \(downloadSize) · Wi-Fi recommended")
/// Whether Core ML compiled it for this iOS already, so loading it takes seconds rather
/// than the minute or more of a first load.
nonisolated static var isCompiled: Bool {
ParakeetCompileRecord(key: compileRecordKey).isCurrent
}
/// Loads the downloaded model; the download itself is `ParakeetUltraDownload`'s, beside
/// the engine Hex dictates with meanwhile.
func prepare(
progressHandler: @escaping @Sendable (ModelPreparationProgress) -> Void
) async throws {
guard manager == nil else { return }
guard ParakeetUltraModel.isDownloaded() else { throw TranscriberError.notDownloaded }
let compiled = ParakeetCompileRecord(key: Self.compileRecordKey)
let downloads = !ParakeetUltraModel.isDownloaded()
let progress = ParakeetProgressFeed(
ParakeetPreparationProgress(
downloads: downloads,
compiledBefore: compiled.isCurrent,
downloadDetail: Self.downloadDetail
),
ParakeetPreparationProgress(downloads: false, compiledBefore: compiled.isCurrent),
emit: progressHandler
)
let download: Task<Void, Error>? = downloads ? Task { try await Self.download(progress) } : nil
let ticker = Task {
while !Task.isCancelled {
progress.tick()
// Turned off mid-download: stop rather than finish a download it then deletes.
if download != nil, !MultilingualParakeetSettings().isEnabled {
download?.cancel()
}
try? await Task.sleep(for: .milliseconds(250))
}
}
defer { ticker.cancel() }
if let download {
try await withTaskCancellationHandler {
try await download.value
} onCancel: {
download.cancel()
}
}
// As V2: only the load, where iOS can end Hex short of memory, counts for the guard,
// and too little left to load it sets it aside for the engine Hex uses without it.
let loads = ParakeetLoadGuard(model: Self.loadGuardModel)
@@ -138,26 +129,6 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
progress.update { $0.finish() }
}
/// Resumes each file from its `.partial`; a download that stops moving for
/// `ParakeetTranscriber.downloadStallLimit` ends, as V2's does.
private static func download(_ progress: ParakeetProgressFeed) async throws {
let fraction = ProgressReading()
do {
try await StallWatch.run(
limit: ParakeetTranscriber.downloadStallLimit,
progress: { fraction.value },
operation: {
try await ParakeetUltraModel.download { value in
fraction.value = value
progress.update { $0.download(value) }
}
}
)
} catch is ParakeetUltraModel.DownloadError {
throw TranscriberError.downloadFailed
}
}
func transcribe(_ audioURL: URL) async throws -> String {
guard let manager else {
throw TranscriberError.notPrepared
+12 -7
View File
@@ -750,11 +750,21 @@ struct ExperimentsSettingsPage: View {
SettingsToggleRow(
title: String(localized: "Multilingual model (Parakeet)"),
detail: multilingualParakeetFits
? String(localized: "25 languages, even mixed in a sentence. \(Self.multilingualParakeetSize) download. Keyboard dictation opens Hex.")
? String(localized: "25 languages, even mixed in a sentence. \(Self.multilingualParakeetSize) download, Wi-Fi recommended. Once it's ready, keyboard dictation opens Hex.")
: String(localized: "Needs an iPhone with 6 GB of memory or more"),
isOn: $multilingualParakeet
)
.disabled(!multilingualParakeetFits)
if multilingualParakeet {
MultilingualModelStatus(
download: DictationController.shared.multilingualDownload,
retry: { DictationController.shared.retryMultilingualParakeet() },
turnOff: { multilingualParakeet = false }
)
.padding(.horizontal, 12)
.padding(.bottom, 12)
}
}
}
.animation(HexMotion.settleAnimation(reduceMotion: reduceMotion), value: adaptiveTouch)
@@ -789,12 +799,7 @@ struct ExperimentsSettingsPage: View {
.onChange(of: multilingualParakeet) { _, enabled in
guard enabled != MultilingualParakeetSettings().isEnabled else { return }
MultilingualParakeetSettings().isEnabled = enabled
if enabled {
// Turning it on is the way to try again after a load that ended Hex.
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
} else {
ParakeetUltraModel.remove()
}
// On: downloads beside the engine that dictates; off: stops and deletes it.
DictationController.shared.multilingualParakeetChanged()
}
.onAppear {
+274 -12
View File
@@ -1569,23 +1569,136 @@ final class DictationControllerTests: XCTestCase {
XCTAssertFalse(ParakeetCompileRecord(key: ParakeetUltraTranscriber.compileRecordKey, defaults: defaults).isCurrent)
}
func testMultilingualParakeetsDownloadSaysItsSizeAndWiFi() {
var progress = ParakeetPreparationProgress(
downloads: true,
compiledBefore: false,
downloadDetail: ParakeetUltraTranscriber.downloadDetail
)
progress.download(0.5)
let detail = progress.snapshot(at: 1).detail
XCTAssertEqual(detail, ParakeetUltraTranscriber.downloadDetail)
XCTAssertTrue(detail.contains("632"), detail)
XCTAssertTrue(detail.contains("Wi-Fi"), detail)
func testMultilingualParakeetNeverTakesDictationUntilItIsReady() {
let step = DictationController.multilingualStep
XCTAssertEqual(step(nil, true, true), .none, "off, another language, or set aside")
XCTAssertEqual(step("nl-NL", false, false), .download, "the usual engine dictates while it downloads")
XCTAssertEqual(step("nl-NL", false, true), .download)
XCTAssertEqual(step("nl-NL", true, false), .loadBeside, "a first load compiles beside the usual engine")
XCTAssertEqual(step("nl-NL", true, true), .dictate)
XCTAssertEqual(
ParakeetPreparationProgress(downloads: true, compiledBefore: false).downloadDetail,
"Downloading the speech model",
"V2's words are unchanged"
)
XCTAssertTrue(DictationController.isDownloadInterruption(ParakeetUltraTranscriber.TranscriberError.downloadFailed))
}
func testTheKeyboardOpensHexForMultilingualParakeetOnlyOnceItIsReady() {
let settings = MultilingualParakeetSettings(defaults: defaults)
XCTAssertFalse(settings.keyboardOpensHex(language: "nl-NL"), "off")
settings.isEnabled = true
XCTAssertFalse(settings.keyboardOpensHex(language: "nl-NL"), "on, still downloading: the keyboard dictates itself")
settings.isModelReady = true
XCTAssertTrue(settings.keyboardOpensHex(language: "nl-NL"))
XCTAssertFalse(settings.keyboardOpensHex(language: nil), "a language it doesn't take")
settings.isEnabled = false
XCTAssertFalse(settings.keyboardOpensHex(language: "nl-NL"))
}
func testOnlyASwitchTurnedOffDeletesTheMultilingualModelAtLaunch() {
let settings = MultilingualParakeetSettings(defaults: defaults)
XCTAssertFalse(settings.isTurnedOff, "never set: nothing to delete")
settings.isEnabled = true
XCTAssertFalse(settings.isTurnedOff)
settings.isEnabled = false
XCTAssertTrue(settings.isTurnedOff)
XCTAssertFalse(
MultilingualParakeetSettings(defaults: nil).isTurnedOff,
"an App Group Hex can't read says nothing about the switch"
)
}
func testTheMultilingualDownloadResumesFromItsPartialFileAndSaysItsBytes() async throws {
let model = try UltraStubModel()
defer { model.remove() }
let first = model.files[0]
let partial = model.folder.appendingPathComponent(first.path).appendingPathExtension("partial")
try FileManager.default.createDirectory(at: partial.deletingLastPathComponent(), withIntermediateDirectories: true)
try model.bodies[first.path]!.prefix(1000).write(to: partial)
XCTAssertEqual(ParakeetUltraModel.bytesOnDisk(in: model.folder, files: model.files), 1000)
let download = model.download()
download.start()
if case .downloading(let bytes) = download.state {
XCTAssertEqual(bytes, 1000, "picks up from what's on disk")
} else {
XCTFail("\(download.state)")
}
XCTAssertEqual(download.status, "Downloading the multilingual model: 1\u{00A0}KB of 5\u{00A0}KB")
try await waitUntil { download.state == .loading }
XCTAssertTrue(ParakeetUltraModel.isDownloaded(in: model.folder, files: model.files))
XCTAssertEqual(UltraStubProtocol.requested.first ?? nil, "bytes=1000-", "resumed with a range")
}
func testAMultilingualDownloadThatDropsOrStallsTriesAgainFromWhereItWas() async throws {
let model = try UltraStubModel()
defer { model.remove() }
UltraStubProtocol.script([.cut(1200), .hang(800), .serve])
let download = model.download()
var finished = 0
download.onDownloaded = { finished += 1 }
download.start()
try await waitUntil(seconds: 10) { download.state == .loading }
XCTAssertEqual(finished, 1)
XCTAssertTrue(ParakeetUltraModel.isDownloaded(in: model.folder, files: model.files))
XCTAssertEqual(
Array(UltraStubProtocol.requested.prefix(3)),
[nil, "bytes=1200-", "bytes=2000-"],
"a dropped connection and a stalled one each pick up from the partial file"
)
}
func testAMultilingualDownloadThatKeepsMovingNeverRunsOutOfTries() async throws {
let model = try UltraStubModel()
defer { model.remove() }
UltraStubProtocol.script(Array(repeating: .cut(400), count: ParakeetUltraDownload.automaticTries + 1))
let download = model.download()
download.start()
try await waitUntil(seconds: 10) { download.state == .loading || download.isFailed }
XCTAssertEqual(download.state, .loading, "each drop came after more bytes, so it kept going")
XCTAssertTrue(ParakeetUltraModel.isDownloaded(in: model.folder, files: model.files))
}
func testAMultilingualDownloadThatKeepsFailingSaysWhatHappened() async throws {
let model = try UltraStubModel()
defer { model.remove() }
UltraStubProtocol.script(Array(repeating: .status(503), count: 10))
let download = model.download()
download.start()
try await waitUntil { download.isFailed }
XCTAssertEqual(UltraStubProtocol.requested.count, ParakeetUltraDownload.automaticTries)
XCTAssertEqual(download.status, "Hugging Face didn't send the multilingual model (error 503). Try again later.")
UltraStubProtocol.script([.status(404)])
download.restart()
try await waitUntil { download.isFailed }
XCTAssertEqual(UltraStubProtocol.requested.count, 1, "a refused request isn't tried again by itself")
XCTAssertEqual(
ParakeetUltraDownload.message(for: URLError(.notConnectedToInternet)),
"The multilingual model's download stopped. Check your connection. It picks up where it left off."
)
XCTAssertTrue(ParakeetUltraDownload.message(for: CocoaError(.fileWriteOutOfSpace)).contains("free space"))
XCTAssertEqual(ParakeetUltraDownload.backoff(1), 2)
XCTAssertEqual(ParakeetUltraDownload.backoff(4), 16)
XCTAssertEqual(ParakeetUltraDownload.backoff(9), 30)
}
func testTurningTheMultilingualModelOffStopsItsDownloadBeforeItIsDeleted() async throws {
let model = try UltraStubModel()
defer { model.remove() }
UltraStubProtocol.script([.hang(800)])
let download = model.download(stallLimit: .seconds(30))
download.start()
try await waitUntil { ParakeetUltraModel.bytesOnDisk(in: model.folder, files: model.files) == 800 }
await download.stop()
XCTAssertEqual(download.state, .idle)
ParakeetUltraModel.remove(from: model.folder)
try await Task.sleep(for: .milliseconds(300))
XCTAssertFalse(FileManager.default.fileExists(atPath: model.folder.path), "nothing wrote after stop")
}
func testParakeetUltrasFilesArePinnedAndComplete() throws {
@@ -1631,6 +1744,15 @@ final class DictationControllerTests: XCTestCase {
try await settle()
XCTAssertEqual(made, 1)
// Turned off (the test host's App Group has it off) while the usual engine dictates:
// nothing to prepare again.
controller.multilingualParakeetChanged()
try await settle()
XCTAssertEqual(made, 1)
XCTAssertEqual(controller.phase, .ready)
// Turned off while Parakeet Ultra dictated: Hex prepares the engine it picks without it.
controller.engineName = ParakeetUltraTranscriber.name
controller.multilingualParakeetChanged()
try await settle()
XCTAssertEqual(made, 2)
@@ -2155,6 +2277,146 @@ final class DictationControllerTests: XCTestCase {
try await Task.sleep(for: .milliseconds(25))
}
}
private func waitUntil(seconds: Double = 5, _ condition: () -> Bool) async throws {
let deadline = Date().addingTimeInterval(seconds)
while !condition() {
guard Date() < deadline else {
XCTFail("timed out")
return
}
try await Task.sleep(for: .milliseconds(20))
}
}
}
/// Two small files standing in for Parakeet Ultra's, served by `UltraStubProtocol`.
@MainActor
private struct UltraStubModel {
let folder = FileManager.default.temporaryDirectory.appendingPathComponent("ultra-\(UUID().uuidString)")
let bodies: [String: Data]
let files: [ParakeetUltraModel.File]
init() throws {
var bodies: [String: Data] = [:]
var files: [ParakeetUltraModel.File] = []
let scratch = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
defer { try? FileManager.default.removeItem(at: scratch) }
for (path, size) in [("Encoder.mlmodelc/weights/weight.bin", 3000), ("parakeet_vocab.json", 2000)] {
let body = Data((0..<size).map { UInt8(truncatingIfNeeded: $0 &* 31 &+ path.count) })
try body.write(to: scratch)
bodies[path] = body
files.append(.init(path: path, size: Int64(size), sha256: try ParakeetUltraModel.sha256(of: scratch)))
}
self.bodies = bodies
self.files = files
UltraStubProtocol.reset(bodies: bodies)
}
func download(stallLimit: Duration = .seconds(1)) -> ParakeetUltraDownload {
let configuration = URLSessionConfiguration.ephemeral
configuration.protocolClasses = [UltraStubProtocol.self]
return ParakeetUltraDownload(
directory: folder,
files: files,
base: "https://stub.invalid/model",
session: URLSession(configuration: configuration),
stallLimit: stallLimit,
backoff: { _ in 0 }
)
}
func remove() {
try? FileManager.default.removeItem(at: folder)
}
}
/// Serves `UltraStubModel`'s files with ranges, following a script for its next requests:
/// cut the connection after some bytes, send some and then nothing, or answer a status.
private final class UltraStubProtocol: URLProtocol, @unchecked Sendable {
enum Step: Sendable {
case serve
case cut(Int)
case hang(Int)
case status(Int)
}
private static let lock = NSLock()
nonisolated(unsafe) private static var bodies: [String: Data] = [:]
nonisolated(unsafe) private static var steps: [Step] = []
nonisolated(unsafe) private static var ranges: [String?] = []
static func reset(bodies: [String: Data]) {
lock.withLock {
self.bodies = bodies
steps = []
ranges = []
}
}
static func script(_ steps: [Step]) {
lock.withLock {
self.steps = steps
ranges = []
}
}
/// Each request's Range header, in order.
static var requested: [String?] {
lock.withLock { ranges }
}
override class func canInit(with request: URLRequest) -> Bool { true }
override class func canonicalRequest(for request: URLRequest) -> URLRequest { request }
override func startLoading() {
let url = request.url!
let range = request.value(forHTTPHeaderField: "Range")
let path = url.path.replacingOccurrences(of: "/model/", with: "")
let (step, body): (Step, Data?) = Self.lock.withLock {
Self.ranges.append(range)
let step = Self.steps.isEmpty ? .serve : Self.steps.removeFirst()
return (step, Self.bodies[path])
}
guard let body else {
respond(404, url: url)
client?.urlProtocolDidFinishLoading(self)
return
}
if case .status(let status) = step {
respond(status, url: url)
client?.urlProtocolDidFinishLoading(self)
return
}
let offset = range.flatMap { Int($0.dropFirst("bytes=".count).dropLast()) } ?? 0
respond(offset > 0 ? 206 : 200, url: url)
let rest = body.dropFirst(offset)
switch step {
case .cut(let count):
client?.urlProtocol(self, didLoad: Data(rest.prefix(count)))
// The connection drops a moment after the bytes arrive, as on a network.
DispatchQueue.global().asyncAfter(deadline: .now() + 0.2) {
self.client?.urlProtocol(self, didFailWithError: URLError(.networkConnectionLost))
}
case .hang(let count):
client?.urlProtocol(self, didLoad: Data(rest.prefix(count)))
case .serve, .status:
client?.urlProtocol(self, didLoad: Data(rest))
client?.urlProtocolDidFinishLoading(self)
}
}
override func stopLoading() {}
private func respond(_ status: Int, url: URL) {
let response = HTTPURLResponse(
url: url,
statusCode: status,
httpVersion: "HTTP/1.1",
headerFields: ["Content-Type": "application/octet-stream"]
)!
client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed)
}
}
/// The dictation language rule: an explicit choice, else the first preferred language either
@@ -1,12 +1,15 @@
import Foundation
/// Experiments › Multilingual model (Parakeet), in the shared App Group so the keyboard reads
/// it too. Off by default. While it's on, Hex dictates its 25 languages with Parakeet Ultra
/// (`ParakeetUltraTranscriber`), which hears them mixed within a sentence (Dutch with English
/// words), and the Hex key hands its dictations to Hex, as before iOS 26: the model can't load
/// within a keyboard's memory limit.
/// it too. Off by default. While it's on and Hex has the model ready, Hex dictates its 25
/// languages with Parakeet Ultra (`ParakeetUltraTranscriber`), which hears them mixed within a
/// sentence (Dutch with English words), and the Hex key hands its dictations to Hex, as before
/// iOS 26: the model can't load within a keyboard's memory limit. Until it's ready, both
/// dictate as they would with the experiment off.
nonisolated struct MultilingualParakeetSettings {
static let enabledKey = "experiments.multilingualParakeet"
/// Set by Hex once Parakeet Ultra loaded; cleared while it's missing, set aside or off.
static let modelReadyKey = "experiments.multilingualParakeet.ready"
static let launchArgument = "--experiment-multilingual-parakeet"
/// Parakeet Ultra's languages (ISO 639-1), those of NVIDIA Parakeet TDT 0.6B v3.
@@ -29,6 +32,25 @@ nonisolated struct MultilingualParakeetSettings {
nonmutating set { defaults?.set(newValue, forKey: Self.enabledKey) }
}
/// Turned off by the person, as stored: never set reads false here, and so does an App
/// Group iOS doesn't let Hex read yet (a launch before the iPhone's first unlock), which
/// must not delete a model that is on.
var isTurnedOff: Bool {
defaults?.object(forKey: Self.enabledKey) as? Bool == false
}
/// Whether Hex has Parakeet Ultra loaded at least once since it last downloaded or was let
/// back in: until then, the keyboard dictates as it would with the experiment off.
var isModelReady: Bool {
get { defaults?.bool(forKey: Self.modelReadyKey) ?? false }
nonmutating set { defaults?.set(newValue, forKey: Self.modelReadyKey) }
}
/// The keyboard's Dictate and Hex keys open Hex for it: on, ready, and the language its.
func keyboardOpensHex(language: String?) -> Bool {
isEnabled && isModelReady && language != nil
}
/// Whether this iPhone has the memory to load it at all.
static func fits(physicalMemory: UInt64 = ProcessInfo.processInfo.physicalMemory) -> Bool {
physicalMemory >= minimumPhysicalMemory
+23 -8
View File
@@ -310,20 +310,35 @@ never at a launch by itself.
0.6B v3) with it, which hears them mixed within a sentence (Dutch with English
words, a TestFlight request); any other dictation language keeps the engine Hex
picks without it. The language Ultra takes is the one set, else the one Automatic
follows on the keyboard, else the iPhone's first. Turning it on downloads
follows on the keyboard, else the iPhone's first. The experiment never takes
dictation away: until Ultra is ready, Hex and the keyboard dictate as they would
with it off, and Hex switches to Ultra once it has loaded (a dictation in flight
isn't interrupted). Turning it on downloads
FluidInference's Core ML build (632 MB, pinned to revision `95eaa59`, each file
checked against its SHA-256 and resumed from its `.partial`) into Application
Support, out of backups, through the usual preparation bar ("Downloading 632.2 MB ·
Wi-Fi recommended", then Core ML's compile); turning it off deletes it, as does a
launch that finds a download left with it off. FluidAudio 0.15.5 has no
Support, out of backups, beside that engine (`ParakeetUltraDownload`): home and the
switch say "Downloading the multilingual model: 123.4 MB of 632.2 MB" with a bar. A
dropped connection, or bytes that stop for 20 s, tries again by itself after 2, 4, 8,
16, then 30 s, starting over whenever it got further; in the background it waits for
Hex to come back to the front, which picks it up at once (and starts a fresh
connection after 25 s away). After six tries in a row that got nothing further, or a
refused request or a full iPhone, it says what happened in plain words with Try again
and Turn off. Once downloaded, Ultra's first load (Core ML's compile, a minute or
more) runs beside the usual engine too, except where Parakeet V2 dictates and leaves
no room for it (English before iOS 26), where Ultra takes V2's place for that load;
later launches load it directly when Core ML has compiled it for this iOS. Turning it off stops the download, then deletes it;
a launch deletes a leftover only when the switch reads off as stored, never when the
App Group can't be read (a launch before the first unlock). FluidAudio 0.15.5 has no
`.ultra`, so `ParakeetUltraModel` loads the four models itself and hands
FluidAudio v3 `AsrModels` (its own loader would replace a folder it can't load
with v3's download). It is chosen only with 1.5 GB left to load it
(`os_proc_available_memory`), and never again after a load that ended Hex
(`ParakeetLoadGuard`, its own keys) until the switch is turned off and on. The
keyboard can't load it, so while it takes the dictation language, Dictate and the
Hex key open Hex, as before iOS 26 (a warm session still dictates without
leaving the app). Measured on a Mac (`~/dev/experiments/2026-11-21-hex-multilingual-parakeet`):
(`ParakeetLoadGuard`, its own keys) until Try again beside the status, or the switch
turned off and on. The keyboard can't load it, so once Hex has it ready
(`MultilingualParakeetSettings.isModelReady`, set after a load, cleared while it's
missing, set aside or off) and it takes the dictation language, Dictate and the Hex
key open Hex, as before iOS 26 (a warm session still dictates without leaving the
app); until then the keyboard dictates itself. Measured on a Mac (`~/dev/experiments/2026-11-21-hex-multilingual-parakeet`):
mixed Dutch-English test sentences 7.5% WER against Apple Speech nl-NL's 26.9%,
pure Dutch 0.9% (0.4%), pure English 0% (0%). Load memory, compile time and speed
on an iPhone are unmeasured.