mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-23 01:58:23 +08:00
iOS: the multilingual model experiment never takes dictation away; its download says what it's doing and recovers (#2337)
A TestFlight tester (build 120, iPhone 15 Pro Max, iOS 26.6) had Experiments ›
Multilingual model (Parakeet) on: voice input "started demanding Set up", then
sat at "Downloading 632.2 MB · Wi-Fi recommended" with no movement.
Why it blocked: with the experiment on and the dictation language one of
Parakeet Ultra's 25 (English included), Ultra became Hex's engine at once,
before it was downloaded. Hex's phase stayed on preparing (or failed, which
shows Set up) for the whole 632 MB download and the first Core ML compile, and
the keyboard's Dictate and Hex key opened Hex for it whether or not the model
was there, so nothing dictated anywhere until it finished.
Why it sat still: the download ran in prepare() on a foreground URLSession,
one 594 MB encoder file, shown only as a bar at 70% scale under a label that
never changed. Leaving Hex (the keyboard's handoff sends you back to the app you
type in) or Auto-Lock suspends it; on return a dead connection showed nothing
for up to 60-90 s, then failed to Set up.
What changed:
- Until Ultra is ready, Hex and the keyboard dictate with the engine they use
with the experiment off. ParakeetUltraDownload (new) downloads beside it and
ParakeetUltraTranscriber.prepare only loads (throws notDownloaded otherwise).
A first load (Core ML's compile) runs beside the usual engine
(DictationController.loadMultilingualParakeetBeside) and switches to Ultra
when loaded; launches load Ultra directly only once downloaded and compiled
for this iOS (multilingualStep). Where V2 dictates and leaves no room
(English before iOS 26), Ultra takes V2's place for that load.
- The keyboard opens Hex for the experiment only once Hex has loaded it
(MultilingualParakeetSettings.isModelReady in the App Group, set after a
load, cleared while missing, set aside or off).
- The download shows bytes ("Downloading the multilingual model: 123.4 MB of
632.2 MB" with a bar) on home and under the switch; resumes every file from its
.partial (launch, retry, return to the front); a drop or 20 s without bytes
tries again after 2, 4, 8, 16, 30 s, starting over whenever it got further; in
the background it waits and picks up at once on return (fresh connection after
25 s away). After 6 fruitless tries, a refused request or a full iPhone it says
what happened in plain words with Try again and Turn off.
- A failed Ultra load or a load that ended Hex (ParakeetLoadGuard) falls back
to the usual engine and says so with Try again (no longer only by turning the
switch off, which deletes the 632 MB).
- A launch deletes a leftover download only when the switch is stored as off
(isTurnedOff), not when the App Group reads empty (a launch before the first
unlock would have deleted a good model). Turning it off now stops the download
and waits for it before deleting.
- Removed: ParakeetUltraTranscriber.downloadDetail, its .downloadFailed error
and the download/StallWatch inside its prepare (replaced by the above);
strings "Downloading %@ · Wi-Fi recommended" and the old switch detail.
New strings have zh-Hans. README describes the experiment's new behaviour.
- DEBUG builds read HEX_ULTRA_MODEL_BASE to download from a stand-in server.
Verified:
- xcodebuild test, iPhone 17 Pro Max iOS 26.5 simulator on the Mac Studio:
753 tests, 0 failures, 3 skipped (new cases in DictationControllerTests:
resume from .partial with Range, drop + stall retries, a moving download never
running out of tries, 503 x6 / 404 failures with their words, stop before
delete, keyboard flag, stored-off rule, multilingualStep).
- iPhone 15 Pro Max iOS 26.5 simulator, experiment on, a local stand-in server
(Range, throttled, drop/stall switches): Dictate stayed available with the
usual engine (V2 here; the sim has no Apple Speech) while the status counted
MB; a dropped connection said "Trying again in 1 s" and resumed with Range; a
20 s stall said so and resumed; Hex killed mid-download resumed from the
partial at relaunch; backgrounded with the connection cut, it resumed at once
on return; after a kill the resumed download from real Hugging Face finished
from a partial and passed the hashes; Ultra loaded beside and took over (setup
sheet "English · Parakeet Ultra", keyboard flag set); a forced load guard
showed the failure with Try again, which loaded Ultra; deleting the model with
the flag set cleared it at launch and downloaded again; the keyboard logged
"multilingual model on, Dictate opens Hex" only when ready (else the sim's own
"speech model missing" reason) and its Hex key went straight to recording
instead of the download screen; turning it off deleted the folder and cleared
the flag; with it off nothing shows or downloads.
- Not verified: a real iPhone, a real slow or flaky network, iOS's own
suspension timing, the V2-replacement path (sims report no memory limit), and
real audio (the Studio has no microphone).
This commit is contained in:
@@ -1081,7 +1081,8 @@ final class KeyboardViewController: UIInputViewController {
|
||||
/// English."): Hex downloads it, and the keyboard dictates again once it installs.
|
||||
/// Without Full Access it is nil too, and the Hex key says dictation needs Full Access.
|
||||
/// The same while Experiments › Multilingual model (Parakeet) dictates the language: only
|
||||
/// Hex can load that model, so Dictate and the Hex key open Hex.
|
||||
/// Hex can load that model, so Dictate and the Hex key open Hex, once Hex has it ready.
|
||||
/// Until then the keyboard dictates as it would with the experiment off.
|
||||
private func syncLocalDictationAvailability(access: Bool) {
|
||||
guard KeyboardLocalDictation.isSupported else { return }
|
||||
guard access || localDictation.isActive else {
|
||||
@@ -1090,7 +1091,7 @@ final class KeyboardViewController: UIInputViewController {
|
||||
return
|
||||
}
|
||||
let multilingual = MultilingualParakeetSettings()
|
||||
let multilingualOpensHex = multilingual.isEnabled && multilingual.language() != nil
|
||||
let multilingualOpensHex = multilingual.keyboardOpensHex(language: multilingual.language())
|
||||
let needsHex = !localDictation.isActive && (LanguageSettings().keyboardNeedsHex() || multilingualOpensHex)
|
||||
let available = localDictation.isActive || (!needsHex && localDictation.hasMemoryToStart)
|
||||
guard available != localDictationAvailable else { return }
|
||||
|
||||
@@ -200,6 +200,13 @@ struct ContentView: View {
|
||||
.transition(.opacity.combined(with: .scale(scale: 0.98)))
|
||||
}
|
||||
|
||||
// Experiments › Multilingual model: its download or load, beside dictation.
|
||||
MultilingualModelStatus(
|
||||
download: dictation.multilingualDownload,
|
||||
retry: { dictation.retryMultilingualParakeet() },
|
||||
turnOff: { dictation.turnOffMultilingualParakeet() }
|
||||
)
|
||||
|
||||
SettingsCard {
|
||||
ForEach(Array(SettingsPage.allCases.enumerated()), id: \.element) { index, page in
|
||||
if index > 0 {
|
||||
|
||||
@@ -37,20 +37,147 @@ extension DictationController {
|
||||
}
|
||||
|
||||
/// Experiments › Multilingual model (Parakeet) at launch: true when Parakeet Ultra takes
|
||||
/// the dictation language and is getting ready. With the experiment off, it only deletes a
|
||||
/// download turning it off left mid-way.
|
||||
/// the dictation language and loads now, downloaded and compiled for this iOS (seconds).
|
||||
/// Otherwise Hex gets ready with the engine it uses without it, and the download or a
|
||||
/// first load runs beside it (`syncMultilingualParakeet`). With the experiment turned
|
||||
/// off, it only deletes a download turning it off left mid-way.
|
||||
private func prepareMultilingualParakeetAtLaunch() -> Bool {
|
||||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).noteLaunch()
|
||||
guard MultilingualParakeetSettings().isEnabled else {
|
||||
ParakeetUltraModel.remove()
|
||||
let loads = ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel)
|
||||
loads.noteLaunch()
|
||||
let settings = MultilingualParakeetSettings()
|
||||
guard settings.isEnabled else {
|
||||
if settings.isTurnedOff {
|
||||
ParakeetUltraModel.remove()
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !ParakeetUltraModel.isDownloaded() {
|
||||
settings.isModelReady = false
|
||||
}
|
||||
if loads.lastLoadWasInterrupted, settings.language() != nil {
|
||||
settings.isModelReady = false
|
||||
multilingualDownload.fail(ParakeetUltraTranscriber.setAsideMessage)
|
||||
}
|
||||
guard Self.multilingualStep(
|
||||
language: ParakeetUltraTranscriber.chosenLanguage(),
|
||||
downloaded: ParakeetUltraModel.isDownloaded(),
|
||||
compiled: ParakeetUltraTranscriber.isCompiled
|
||||
) == .dictate else {
|
||||
// A first load waits for the engine Hex gets ready meanwhile (its end syncs).
|
||||
startMultilingualDownloadIfWanted()
|
||||
return false
|
||||
}
|
||||
guard ParakeetUltraTranscriber.chosenLanguage() != nil else { return false }
|
||||
engineName = ParakeetUltraTranscriber.name
|
||||
prepareModel()
|
||||
return true
|
||||
}
|
||||
|
||||
/// What Hex does about Parakeet Ultra: nothing (off, a language it doesn't take, set
|
||||
/// aside), download it, load it beside the engine that dictates meanwhile (a first load
|
||||
/// compiles for a minute or more), or dictate with it.
|
||||
enum MultilingualStep: Equatable {
|
||||
case none
|
||||
case download
|
||||
case loadBeside
|
||||
case dictate
|
||||
}
|
||||
|
||||
nonisolated static func multilingualStep(language: String?, downloaded: Bool, compiled: Bool) -> MultilingualStep {
|
||||
guard language != nil else { return .none }
|
||||
guard downloaded else { return .download }
|
||||
return compiled ? .dictate : .loadBeside
|
||||
}
|
||||
|
||||
/// Starts Parakeet Ultra's download when the experiment wants it and it isn't on the
|
||||
/// iPhone, whatever engine is getting ready meanwhile.
|
||||
func startMultilingualDownloadIfWanted() {
|
||||
// Memory left only matters for a load, not a download.
|
||||
guard ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0) != nil,
|
||||
!ParakeetUltraModel.isDownloaded() else { return }
|
||||
MultilingualParakeetSettings().isModelReady = false
|
||||
multilingualDownload.start()
|
||||
}
|
||||
|
||||
/// Once an engine is ready, and once the download finishes: downloads Parakeet Ultra, or
|
||||
/// loads it beside the engine that dictates, when the experiment wants it and it isn't the
|
||||
/// one dictating already.
|
||||
func syncMultilingualParakeet() {
|
||||
guard MultilingualParakeetSettings().isEnabled else {
|
||||
multilingualDownload.settle()
|
||||
return
|
||||
}
|
||||
let language = ParakeetUltraTranscriber.chosenLanguage()
|
||||
let wanted = ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0)
|
||||
switch Self.multilingualStep(
|
||||
language: wanted,
|
||||
downloaded: ParakeetUltraModel.isDownloaded(),
|
||||
compiled: true
|
||||
) {
|
||||
case .none:
|
||||
if !multilingualDownload.isFailed { multilingualDownload.settle() }
|
||||
case .download:
|
||||
startMultilingualDownloadIfWanted()
|
||||
case .loadBeside, .dictate:
|
||||
if engineName == ParakeetUltraTranscriber.name, isModelReady {
|
||||
multilingualDownload.settle()
|
||||
} else if let language {
|
||||
loadMultilingualParakeetBeside(language: language)
|
||||
} else if let wanted, replaceParakeetWithMultilingual(language: wanted) {
|
||||
multilingualDownload.loading()
|
||||
} else {
|
||||
// Too little memory left to load it now: said, with Try again.
|
||||
multilingualDownload.fail(
|
||||
String(localized: "Parakeet Ultra needs more memory than Hex has here. Hex uses its usual speech model instead.")
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parakeet Ultra loaded: the keyboard opens Hex for its languages from now on.
|
||||
func multilingualParakeetLoaded() {
|
||||
MultilingualParakeetSettings().isModelReady = true
|
||||
multilingualDownload.settle()
|
||||
}
|
||||
|
||||
/// Parakeet Ultra didn't load. Missing files download again; anything else sets it aside
|
||||
/// (`ParakeetLoadGuard`) until Try again, so Hex doesn't try at every launch.
|
||||
func multilingualParakeetFailedToLoad(_ error: Error) {
|
||||
let settings = MultilingualParakeetSettings()
|
||||
settings.isModelReady = false
|
||||
guard settings.isEnabled, !(error is CancellationError) else {
|
||||
multilingualDownload.settle()
|
||||
return
|
||||
}
|
||||
if (error as? ParakeetUltraTranscriber.TranscriberError) == .notDownloaded {
|
||||
multilingualDownload.settle()
|
||||
startMultilingualDownloadIfWanted()
|
||||
return
|
||||
}
|
||||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).setAside()
|
||||
multilingualDownload.fail(
|
||||
Self.switchNotice(for: error) ?? UserFacingError.message(for: error, context: .model)
|
||||
)
|
||||
}
|
||||
|
||||
/// Try again beside the multilingual model's status: lets a set-aside load try again, and
|
||||
/// picks up the download from what's on disk.
|
||||
func retryMultilingualParakeet() {
|
||||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
|
||||
multilingualDownload.settle()
|
||||
if ParakeetUltraModel.isDownloaded() {
|
||||
syncMultilingualParakeet()
|
||||
} else if ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0) != nil {
|
||||
MultilingualParakeetSettings().isModelReady = false
|
||||
multilingualDownload.restart()
|
||||
}
|
||||
}
|
||||
|
||||
/// Turn off beside the multilingual model's status, as the switch in Experiments does.
|
||||
func turnOffMultilingualParakeet() {
|
||||
MultilingualParakeetSettings().isEnabled = false
|
||||
multilingualParakeetChanged()
|
||||
}
|
||||
|
||||
/// Before iOS 26: Parakeet loads at launch once downloaded, as before. The recognizer gets
|
||||
/// ready at launch only once Speech Recognition is allowed; otherwise setup asks for it
|
||||
/// (onboarding's first page, or Set up on home), never a launch by itself.
|
||||
@@ -86,8 +213,14 @@ extension DictationController {
|
||||
}
|
||||
|
||||
static func defaultTranscriber() async -> any LocalTranscribing {
|
||||
// Experiments › Multilingual model (Parakeet), for its languages; off, nothing changes.
|
||||
if let language = ParakeetUltraTranscriber.chosenLanguage() {
|
||||
// Experiments › Multilingual model (Parakeet), for its languages once it's downloaded and
|
||||
// compiled; until then (and off) the engine Hex uses without it.
|
||||
if let language = ParakeetUltraTranscriber.chosenLanguage(),
|
||||
multilingualStep(
|
||||
language: language,
|
||||
downloaded: ParakeetUltraModel.isDownloaded(),
|
||||
compiled: ParakeetUltraTranscriber.isCompiled
|
||||
) == .dictate {
|
||||
return ParakeetUltraTranscriber(language: language)
|
||||
}
|
||||
let choice = SpeechEngineChoice.current
|
||||
|
||||
@@ -78,6 +78,9 @@ final class DictationController {
|
||||
|
||||
private var transcriber: any LocalTranscribing
|
||||
private let transcriberFactory: () async -> any LocalTranscribing
|
||||
/// Experiments › Multilingual model (Parakeet): its download, beside the engine Hex
|
||||
/// dictates with until it's ready.
|
||||
@ObservationIgnored let multilingualDownload: ParakeetUltraDownload
|
||||
private let cleaner: (any TranscriptCleaning)?
|
||||
private let cleanupSettings: AppleIntelligenceCleanupSettings
|
||||
private let customVocabularySettings: CustomVocabularySettings
|
||||
@@ -228,7 +231,8 @@ final class DictationController {
|
||||
hinglishSettings: HinglishSettings = HinglishSettings(),
|
||||
typingJournalSettings: TypingJournalSettings = TypingJournalSettings(),
|
||||
learnedDictationCorrections: (@MainActor (String?) -> [DictationCorrection])? = nil,
|
||||
languageSettings: LanguageSettings = LanguageSettings()
|
||||
languageSettings: LanguageSettings = LanguageSettings(),
|
||||
multilingualDownload: ParakeetUltraDownload? = nil
|
||||
) {
|
||||
let resolvedBridge = keyboardBridge ?? KeyboardBridge()
|
||||
let resolvedAudio = audio ?? SystemAudioManager()
|
||||
@@ -238,6 +242,7 @@ final class DictationController {
|
||||
self.keyboardBridge = resolvedBridge
|
||||
self.transcriber = initialTranscriber
|
||||
self.transcriberFactory = transcriberFactory ?? Self.defaultTranscriber
|
||||
self.multilingualDownload = multilingualDownload ?? ParakeetUltraDownload()
|
||||
self.cleaner = cleaner
|
||||
self.cleanupSettings = cleanupSettings
|
||||
self.customVocabularySettings = customVocabularySettings
|
||||
@@ -265,6 +270,9 @@ final class DictationController {
|
||||
|
||||
wireAudioCallbacks()
|
||||
wireSystemIntentBus()
|
||||
self.multilingualDownload.onDownloaded = { [weak self] in
|
||||
self?.syncMultilingualParakeet()
|
||||
}
|
||||
if autoPrepareOnInit {
|
||||
// At launch no dictation is running, so any Hex Live Activity is left over.
|
||||
liveActivity.endImmediately()
|
||||
@@ -373,9 +381,34 @@ final class DictationController {
|
||||
}
|
||||
}
|
||||
|
||||
/// Experiments › Multilingual model (Parakeet) turned on or off: prepares the engine Hex
|
||||
/// picks now, or once the dictation in flight is done.
|
||||
/// Experiments › Multilingual model (Parakeet) turned on or off. On, the engine Hex
|
||||
/// dictates with stays while Parakeet Ultra downloads and loads beside it; a load that
|
||||
/// ended Hex gets another try. Off, the download stops and is deleted, and Hex prepares
|
||||
/// the engine it picks without it if Ultra was the one dictating (now, or once the
|
||||
/// dictation in flight is done).
|
||||
func multilingualParakeetChanged() {
|
||||
let settings = MultilingualParakeetSettings()
|
||||
if settings.isEnabled {
|
||||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
|
||||
multilingualDownload.settle()
|
||||
if isModelReady || phase == .preparingModel {
|
||||
syncMultilingualParakeet()
|
||||
return
|
||||
}
|
||||
} else {
|
||||
settings.isModelReady = false
|
||||
let download = multilingualDownload
|
||||
Task {
|
||||
// Nothing writes to the folder any more once `stop` returns.
|
||||
await download.stop()
|
||||
ParakeetUltraModel.remove()
|
||||
}
|
||||
if engineName != ParakeetUltraTranscriber.name, isModelReady, phase != .preparingModel {
|
||||
// Ultra wasn't dictating: drop a load of it beside the engine that is.
|
||||
modelPrepGeneration &+= 1
|
||||
return
|
||||
}
|
||||
}
|
||||
switch phase {
|
||||
case .recording, .transcribing, .requestingPermission:
|
||||
pendingLanguagePreparation = true
|
||||
@@ -701,9 +734,12 @@ final class DictationController {
|
||||
|
||||
/// `fallbacksLeft` bounds how often Automatic moves on to another language after one
|
||||
/// whose model won't install here; each move sets a language aside, so it can't loop.
|
||||
func prepareModel(fallbacksLeft: Int = 2) {
|
||||
func prepareModel(fallbacksLeft: Int = 2, candidate preset: (any LocalTranscribing)? = nil) {
|
||||
modelPrepGeneration &+= 1
|
||||
let prepGen = modelPrepGeneration
|
||||
// A load beside the engine this replaces is dropped; the end of this one decides again.
|
||||
multilingualDownload.dropLoad()
|
||||
startMultilingualDownloadIfWanted()
|
||||
phase = .preparingModel
|
||||
copied = false
|
||||
modelPreparation = ModelPreparationProgress(
|
||||
@@ -716,10 +752,17 @@ final class DictationController {
|
||||
|
||||
Task {
|
||||
defer { backgroundTime.end() }
|
||||
var candidateIsUltra = false
|
||||
do {
|
||||
let candidate = await transcriberFactory()
|
||||
let candidate: any LocalTranscribing
|
||||
if let preset {
|
||||
candidate = preset
|
||||
} else {
|
||||
candidate = await transcriberFactory()
|
||||
}
|
||||
guard self.modelPrepGeneration == prepGen else { return }
|
||||
engineName = candidate.displayName
|
||||
candidateIsUltra = candidate is ParakeetUltraTranscriber
|
||||
try await candidate.prepare { [weak self] progress in
|
||||
Task { @MainActor [weak self] in
|
||||
guard let self, self.modelPrepGeneration == prepGen else { return }
|
||||
@@ -734,12 +777,23 @@ final class DictationController {
|
||||
self.keyboardBridge.setModelInstalled(true)
|
||||
self.isModelReady = true
|
||||
self.phase = .ready
|
||||
if candidateIsUltra {
|
||||
self.multilingualParakeetLoaded()
|
||||
}
|
||||
self.publishKeyboardSnapshot()
|
||||
if self.pendingHandoff != nil {
|
||||
self.activateWarmSessionAndRunHandoff()
|
||||
}
|
||||
self.syncMultilingualParakeet()
|
||||
} catch {
|
||||
guard self.modelPrepGeneration == prepGen else { return }
|
||||
if candidateIsUltra, fallbacksLeft > 0, !(error is CancellationError) {
|
||||
// Parakeet Ultra didn't load: the engine Hex uses without it dictates, and
|
||||
// Hex says why, with Try again.
|
||||
self.multilingualParakeetFailedToLoad(error)
|
||||
self.prepareModel(fallbacksLeft: fallbacksLeft - 1)
|
||||
return
|
||||
}
|
||||
let unavailable = error as? SpeechModelUnavailable
|
||||
if fallbacksLeft > 0, let notice = Self.switchNotice(for: error) {
|
||||
// Automatic set the language aside, or Parakeet had no room: prepare what
|
||||
@@ -754,18 +808,70 @@ final class DictationController {
|
||||
self.pendingHandoff = nil
|
||||
self.phase = .failed(UserFacingError.message(for: error, context: .model))
|
||||
self.publishKeyboardSnapshot()
|
||||
// A downloaded Parakeet Ultra can still dictate.
|
||||
self.syncMultilingualParakeet()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parakeet V2 dictates and leaves no room to load Parakeet Ultra beside it (English
|
||||
/// before iOS 26): Ultra takes its place and loads as it would at launch, rather than
|
||||
/// never getting its first load. False when V2 isn't the engine.
|
||||
func replaceParakeetWithMultilingual(language: String) -> Bool {
|
||||
guard transcriber is ParakeetTranscriber, phase != .preparingModel else { return false }
|
||||
transcriber = ParakeetTranscriber()
|
||||
isModelReady = false
|
||||
prepareModel(candidate: ParakeetUltraTranscriber(language: language))
|
||||
return true
|
||||
}
|
||||
|
||||
/// Loads Parakeet Ultra while the engine Hex has keeps dictating, and switches to it once
|
||||
/// it's loaded; a dictation in flight isn't interrupted. A failure keeps that engine and
|
||||
/// says why. Only while no other preparation runs.
|
||||
func loadMultilingualParakeetBeside(language: String) {
|
||||
guard phase != .preparingModel else { return }
|
||||
modelPrepGeneration &+= 1
|
||||
let prepGen = modelPrepGeneration
|
||||
multilingualDownload.loading()
|
||||
let backgroundTime = BackgroundTime("Prepare the multilingual model")
|
||||
Task {
|
||||
defer { backgroundTime.end() }
|
||||
let candidate = ParakeetUltraTranscriber(language: language)
|
||||
do {
|
||||
try await candidate.prepare { _ in }
|
||||
guard self.modelPrepGeneration == prepGen else { return }
|
||||
self.transcriber = candidate
|
||||
self.engineName = candidate.displayName
|
||||
self.transcriptionLanguage = await candidate.languageName
|
||||
self.transcriptionLanguageID = await candidate.languageIdentifier
|
||||
self.languageOffer = nil
|
||||
self.isModelReady = true
|
||||
self.keyboardBridge.setModelInstalled(true)
|
||||
switch self.phase {
|
||||
case .modelRequired, .failed:
|
||||
self.phase = .ready
|
||||
default:
|
||||
break
|
||||
}
|
||||
self.multilingualParakeetLoaded()
|
||||
self.publishKeyboardSnapshot()
|
||||
} catch {
|
||||
guard self.modelPrepGeneration == prepGen else { return }
|
||||
self.multilingualParakeetFailedToLoad(error)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func appMovedToBackground() {
|
||||
isInBackground = true
|
||||
multilingualDownload.appMovedToBackground()
|
||||
}
|
||||
|
||||
/// Picks up a download that iOS cut off while Hex was in the background; FluidAudio
|
||||
/// resumes each file from the bytes it has.
|
||||
func appBecameActive() {
|
||||
isInBackground = false
|
||||
multilingualDownload.appBecameActive()
|
||||
guard preparationFailedInBackground else { return }
|
||||
preparationFailedInBackground = false
|
||||
if case .failed = phase, !isModelReady {
|
||||
@@ -775,7 +881,6 @@ final class DictationController {
|
||||
|
||||
nonisolated static func isDownloadInterruption(_ error: Error) -> Bool {
|
||||
if error is StallWatch.Stalled || error is URLError { return true }
|
||||
if (error as? ParakeetUltraTranscriber.TranscriberError) == .downloadFailed { return true }
|
||||
return (error as? ParakeetTranscriber.TranscriberError) == .downloadFailed
|
||||
}
|
||||
|
||||
@@ -1469,7 +1574,7 @@ final class DictationController {
|
||||
/// Background time from iOS while Hex finishes a keyboard's dictation, ended when it's done
|
||||
/// or when iOS asks for it back.
|
||||
@MainActor
|
||||
private final class BackgroundTime {
|
||||
final class BackgroundTime {
|
||||
private var id = UIBackgroundTaskIdentifier.invalid
|
||||
|
||||
init(_ name: String) {
|
||||
|
||||
@@ -3553,26 +3553,6 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"Parakeet Ultra didn't download. Check your connection and try again." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "Parakeet Ultra 未能下载。请检查网络连接后重试。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Downloading %@ · Wi-Fi recommended" : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "正在下载 %@ · 建议使用 Wi-Fi"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Multilingual model (Parakeet)" : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
@@ -3583,16 +3563,6 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"25 languages, even mixed in a sentence. %@ download. Keyboard dictation opens Hex." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "支持 25 种语言,一句话里混着说也行。需下载 %@。键盘听写会打开 Hex。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Needs an iPhone with 6 GB of memory or more" : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
@@ -3862,6 +3832,156 @@
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Parakeet Ultra isn't downloaded yet." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "Parakeet Ultra 尚未下载。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Hex closed while getting the multilingual model ready, likely short of memory." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "Hex 在准备多语言模型时关闭了,可能是内存不足。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"The multilingual model needs %@ of free space. Free some up, then try again." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "多语言模型需要 %@ 的可用空间。请先释放一些空间,然后重试。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"The multilingual model's download stopped. Check your connection. It picks up where it left off." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "多语言模型的下载已停止。请检查网络连接。下载会从中断处继续。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Hugging Face is limiting downloads right now. Try again in a few minutes." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "Hugging Face 目前限制了下载。请过几分钟再试。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Hugging Face didn't send the multilingual model (error %lld). Try again later." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "Hugging Face 没有发送多语言模型(错误 %lld)。请稍后再试。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"A file of the multilingual model arrived damaged. Try again to download it again." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "多语言模型的一个文件下载后已损坏。请重试以重新下载。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"The multilingual model didn't download. Try again." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "多语言模型未能下载。请重试。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Downloading the multilingual model: %@ of %@" : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "正在下载多语言模型:已下载 %1$@,共 %2$@"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"The multilingual model's download stopped. Trying again in %lld s." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "多语言模型的下载已停止。%lld 秒后重试。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"The multilingual model's download is paused. It carries on when you open Hex." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "多语言模型的下载已暂停。打开 Hex 后会继续。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Getting the multilingual model ready" : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "正在准备多语言模型"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Turn off" : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "关闭"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"25 languages, even mixed in a sentence. %@ download, Wi-Fi recommended. Once it's ready, keyboard dictation opens Hex." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "支持 25 种语言,一句话里混着说也行。需下载 %@,建议使用 Wi-Fi。准备好后,键盘听写会打开 Hex。"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Hex dictates with its usual model meanwhile." : {
|
||||
"localizations" : {
|
||||
"zh-Hans" : {
|
||||
"stringUnit" : {
|
||||
"state" : "translated",
|
||||
"value" : "在此期间,Hex 会用平时的模型听写。"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"version" : "1.0"
|
||||
|
||||
@@ -0,0 +1,422 @@
|
||||
import Foundation
|
||||
import Observation
|
||||
import SwiftUI
|
||||
|
||||
/// Experiments › Multilingual model (Parakeet): gets Parakeet Ultra onto the iPhone beside the
|
||||
/// engine Hex dictates with meanwhile, so turning the experiment on never takes dictation away.
|
||||
/// Each file resumes from its `.partial`, at the next launch too. A dropped connection or a
|
||||
/// download that stops moving for `stallLimit` tries again by itself (after 2, 4, 8, 16, 30 s),
|
||||
/// and at once when Hex comes back to the front; after `automaticTries` in a row that got nothing
|
||||
/// further, it says what went wrong and waits for Try again (or, for a connection, the next
|
||||
/// time Hex comes to the front).
|
||||
@MainActor
|
||||
@Observable
|
||||
final class ParakeetUltraDownload {
|
||||
enum State: Equatable {
|
||||
/// Nothing to say: the experiment off, not wanted for this language, or Ultra dictates.
|
||||
case idle
|
||||
/// Bytes on disk.
|
||||
case downloading(Int64)
|
||||
/// Stopped, and tries again by itself in `retryIn` seconds, or once Hex is in front (nil).
|
||||
case waiting(Int64, retryIn: Int?)
|
||||
/// On disk; Hex gets it ready beside the engine it dictates with.
|
||||
case loading
|
||||
/// What went wrong, in plain words.
|
||||
case failed(String)
|
||||
}
|
||||
|
||||
private(set) var state = State.idle
|
||||
/// Called once every file is on disk.
|
||||
@ObservationIgnored var onDownloaded: (@MainActor () -> Void)?
|
||||
|
||||
nonisolated static let stallLimit: Duration = .seconds(20)
|
||||
static let automaticTries = 6
|
||||
|
||||
@ObservationIgnored private var task: Task<Void, Never>?
|
||||
@ObservationIgnored private var tries = 0
|
||||
/// What was on disk at the last stop: a download that moved since starts its tries over.
|
||||
@ObservationIgnored private var bytesAtLastStop: Int64 = 0
|
||||
@ObservationIgnored private var failedForConnection = false
|
||||
@ObservationIgnored private var backgroundedAt: Date?
|
||||
@ObservationIgnored private var backgroundTime: BackgroundTime?
|
||||
@ObservationIgnored private let directory: URL
|
||||
@ObservationIgnored private let files: [ParakeetUltraModel.File]
|
||||
@ObservationIgnored private let base: String
|
||||
@ObservationIgnored private let session: URLSession
|
||||
@ObservationIgnored private let stallLimit: Duration
|
||||
@ObservationIgnored private let backoff: (Int) -> Int
|
||||
|
||||
init(
|
||||
directory: URL = ParakeetUltraModel.directory,
|
||||
files: [ParakeetUltraModel.File] = ParakeetUltraModel.files,
|
||||
base: String = ParakeetUltraModel.baseURL,
|
||||
session: URLSession = ParakeetUltraDownload.makeSession(),
|
||||
stallLimit: Duration = ParakeetUltraDownload.stallLimit,
|
||||
backoff: @escaping (Int) -> Int = ParakeetUltraDownload.backoff
|
||||
) {
|
||||
self.directory = directory
|
||||
self.files = files
|
||||
self.base = base
|
||||
self.session = session
|
||||
self.stallLimit = stallLimit
|
||||
self.backoff = backoff
|
||||
}
|
||||
|
||||
/// A connection that sends nothing for 30 s ends, rather than after the default minute.
|
||||
nonisolated static func makeSession() -> URLSession {
|
||||
let configuration = URLSessionConfiguration.default
|
||||
configuration.timeoutIntervalForRequest = 30
|
||||
return URLSession(configuration: configuration)
|
||||
}
|
||||
|
||||
/// Seconds before automatic try `n` (1, 2, …): 2, 4, 8, 16, then 30.
|
||||
nonisolated static func backoff(_ tries: Int) -> Int {
|
||||
min(30, 1 << max(1, min(tries, 5)))
|
||||
}
|
||||
|
||||
var isRunning: Bool { task != nil }
|
||||
|
||||
var total: Int64 { files.reduce(0) { $0 + $1.size } }
|
||||
|
||||
/// Starts, or resumes from what's on disk; does nothing while it runs.
|
||||
func start() {
|
||||
guard task == nil else { return }
|
||||
tries = 0
|
||||
run()
|
||||
}
|
||||
|
||||
/// Starts over from what's on disk, with all the automatic tries again.
|
||||
func restart() {
|
||||
task?.cancel()
|
||||
task = nil
|
||||
tries = 0
|
||||
run()
|
||||
}
|
||||
|
||||
/// Stops it and waits until nothing writes to the folder any more, so it can be deleted.
|
||||
func stop() async {
|
||||
let running = task
|
||||
task = nil
|
||||
running?.cancel()
|
||||
await running?.value
|
||||
endBackgroundTime()
|
||||
state = .idle
|
||||
}
|
||||
|
||||
func loading() {
|
||||
state = .loading
|
||||
}
|
||||
|
||||
/// A load beside the usual engine that another preparation replaced.
|
||||
func dropLoad() {
|
||||
if state == .loading { state = .idle }
|
||||
}
|
||||
|
||||
/// Ultra dictates, or isn't wanted any more.
|
||||
func settle() {
|
||||
guard task == nil else { return }
|
||||
state = .idle
|
||||
}
|
||||
|
||||
/// A load that failed, or one that ended Hex: said with Try again beside it.
|
||||
func fail(_ message: String) {
|
||||
task?.cancel()
|
||||
task = nil
|
||||
failedForConnection = false
|
||||
state = .failed(message)
|
||||
}
|
||||
|
||||
func appMovedToBackground() {
|
||||
backgroundedAt = Date()
|
||||
// A few more seconds from iOS, so a quick look at another app doesn't stop it.
|
||||
if task != nil, backgroundTime == nil {
|
||||
backgroundTime = BackgroundTime("Download the multilingual model")
|
||||
}
|
||||
}
|
||||
|
||||
/// Back in front: a download that stopped tries again at once, and one suspended long
|
||||
/// enough for iOS to drop its connection starts a fresh one from where it is.
|
||||
func appBecameActive() {
|
||||
let away = backgroundedAt.map { Date().timeIntervalSince($0) } ?? 0
|
||||
backgroundedAt = nil
|
||||
endBackgroundTime()
|
||||
switch state {
|
||||
case .waiting:
|
||||
restart()
|
||||
case .failed where failedForConnection:
|
||||
restart()
|
||||
case .downloading where away > 25:
|
||||
restart()
|
||||
default:
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
private func endBackgroundTime() {
|
||||
backgroundTime?.end()
|
||||
backgroundTime = nil
|
||||
}
|
||||
|
||||
private func run() {
|
||||
failedForConnection = false
|
||||
bytesAtLastStop = ParakeetUltraModel.bytesOnDisk(in: directory, files: files)
|
||||
state = .downloading(bytesAtLastStop)
|
||||
task = Task { [weak self] in
|
||||
await self?.loop()
|
||||
}
|
||||
}
|
||||
|
||||
private func loop() async {
|
||||
while !Task.isCancelled {
|
||||
do {
|
||||
try await Self.fetch(
|
||||
directory: directory,
|
||||
files: files,
|
||||
base: base,
|
||||
session: session,
|
||||
stallLimit: stallLimit
|
||||
) { [weak self] bytes in
|
||||
await MainActor.run {
|
||||
guard let self, self.task != nil, case .downloading = self.state else { return }
|
||||
self.state = .downloading(bytes)
|
||||
}
|
||||
}
|
||||
guard !Task.isCancelled else { return }
|
||||
task = nil
|
||||
endBackgroundTime()
|
||||
state = .loading
|
||||
onDownloaded?()
|
||||
return
|
||||
} catch {
|
||||
guard !Task.isCancelled else { return }
|
||||
let onDisk = ParakeetUltraModel.bytesOnDisk(in: directory, files: files)
|
||||
if onDisk > bytesAtLastStop {
|
||||
tries = 0
|
||||
}
|
||||
bytesAtLastStop = onDisk
|
||||
tries += 1
|
||||
print("multilingual model: download try \(tries) stopped at \(onDisk) bytes: \(error)")
|
||||
guard Self.triesAgain(after: error), tries < Self.automaticTries else {
|
||||
task = nil
|
||||
endBackgroundTime()
|
||||
failedForConnection = Self.triesAgain(after: error)
|
||||
state = .failed(Self.message(for: error))
|
||||
return
|
||||
}
|
||||
if backgroundedAt != nil {
|
||||
// Suspended soon anyway: tries again once Hex is back in front.
|
||||
task = nil
|
||||
endBackgroundTime()
|
||||
state = .waiting(onDisk, retryIn: nil)
|
||||
return
|
||||
}
|
||||
for remaining in stride(from: backoff(tries), to: 0, by: -1) {
|
||||
state = .waiting(onDisk, retryIn: remaining)
|
||||
try? await Task.sleep(for: .seconds(1))
|
||||
guard !Task.isCancelled else { return }
|
||||
}
|
||||
state = .downloading(onDisk)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One try: the download beside a watch that publishes its bytes twice a second and ends
|
||||
/// it once they stop moving for `stallLimit` (time while iOS had Hex suspended doesn't
|
||||
/// count). Returns only once nothing writes to the folder any more.
|
||||
nonisolated static func fetch(
|
||||
directory: URL,
|
||||
files: [ParakeetUltraModel.File],
|
||||
base: String,
|
||||
session: URLSession,
|
||||
stallLimit: Duration,
|
||||
publish: @escaping @Sendable (Int64) async -> Void
|
||||
) async throws {
|
||||
let bytes = ByteReading(ParakeetUltraModel.bytesOnDisk(in: directory, files: files))
|
||||
try await withThrowingTaskGroup(of: Void.self) { group in
|
||||
group.addTask {
|
||||
try await ParakeetUltraModel.download(to: directory, files: files, base: base, session: session) {
|
||||
bytes.value = $0
|
||||
}
|
||||
}
|
||||
group.addTask {
|
||||
let clock = ContinuousClock()
|
||||
let poll = Duration.milliseconds(500)
|
||||
var last = bytes.value
|
||||
var since = clock.now
|
||||
var polled = since
|
||||
while true {
|
||||
try await Task.sleep(for: poll)
|
||||
let now = clock.now
|
||||
let resumed = now - polled > poll + StallWatch.suspensionGap
|
||||
polled = now
|
||||
let current = bytes.value
|
||||
await publish(current)
|
||||
if current != last || resumed {
|
||||
last = current
|
||||
since = now
|
||||
} else if now - since >= stallLimit {
|
||||
throw StallWatch.Stalled()
|
||||
}
|
||||
}
|
||||
}
|
||||
// The download finishing, or either one throwing, ends the other.
|
||||
_ = try await group.next()
|
||||
group.cancelAll()
|
||||
}
|
||||
}
|
||||
|
||||
/// A dropped connection, a download that stopped moving, a busy server or a damaged file
|
||||
/// is tried again; a refused request, a full iPhone or anything else waits for the person.
|
||||
nonisolated static func triesAgain(after error: Error) -> Bool {
|
||||
if error is StallWatch.Stalled || error is URLError { return true }
|
||||
switch error as? ParakeetUltraModel.DownloadError {
|
||||
case .badStatus(let status):
|
||||
return status == 0 || status == 408 || status == 429 || status >= 500
|
||||
case .corrupt:
|
||||
return true
|
||||
case nil:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
nonisolated static func isOutOfSpace(_ error: Error) -> Bool {
|
||||
let ns = error as NSError
|
||||
return (ns.domain == NSCocoaErrorDomain && ns.code == NSFileWriteOutOfSpaceError)
|
||||
|| (ns.domain == NSPOSIXErrorDomain && ns.code == Int(ENOSPC))
|
||||
}
|
||||
|
||||
nonisolated static func message(for error: Error) -> String {
|
||||
if isOutOfSpace(error) {
|
||||
return String(localized: "The multilingual model needs \(ParakeetUltraTranscriber.downloadSize) of free space. Free some up, then try again.")
|
||||
}
|
||||
if error is StallWatch.Stalled || error is URLError {
|
||||
return String(localized: "The multilingual model's download stopped. Check your connection. It picks up where it left off.")
|
||||
}
|
||||
switch error as? ParakeetUltraModel.DownloadError {
|
||||
case .badStatus(429):
|
||||
return String(localized: "Hugging Face is limiting downloads right now. Try again in a few minutes.")
|
||||
case .badStatus(let status):
|
||||
return String(localized: "Hugging Face didn't send the multilingual model (error \(status)). Try again later.")
|
||||
case .corrupt:
|
||||
return String(localized: "A file of the multilingual model arrived damaged. Try again to download it again.")
|
||||
case nil:
|
||||
return String(localized: "The multilingual model didn't download. Try again.")
|
||||
}
|
||||
}
|
||||
|
||||
/// What Hex says about it, or nil while there's nothing to say.
|
||||
var status: String? {
|
||||
switch state {
|
||||
case .idle:
|
||||
nil
|
||||
case .downloading(let bytes):
|
||||
String(localized: "Downloading the multilingual model: \(Self.size(bytes)) of \(Self.size(total))")
|
||||
case .waiting(_, let seconds?):
|
||||
String(localized: "The multilingual model's download stopped. Trying again in \(seconds) s.")
|
||||
case .waiting(_, nil):
|
||||
String(localized: "The multilingual model's download is paused. It carries on when you open Hex.")
|
||||
case .loading:
|
||||
String(localized: "Getting the multilingual model ready")
|
||||
case .failed(let message):
|
||||
message
|
||||
}
|
||||
}
|
||||
|
||||
/// The share on disk, while it downloads or waits.
|
||||
var fraction: Double? {
|
||||
switch state {
|
||||
case .downloading(let bytes), .waiting(let bytes, _):
|
||||
total > 0 ? min(1, Double(bytes) / Double(total)) : 0
|
||||
default:
|
||||
nil
|
||||
}
|
||||
}
|
||||
|
||||
var isFailed: Bool {
|
||||
if case .failed = state { return true }
|
||||
return false
|
||||
}
|
||||
|
||||
nonisolated static func size(_ bytes: Int64) -> String {
|
||||
ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
|
||||
.replacingOccurrences(of: " ", with: "\u{00A0}")
|
||||
}
|
||||
}
|
||||
|
||||
/// Bytes written by the download and read by its watch.
|
||||
private nonisolated final class ByteReading: @unchecked Sendable {
|
||||
private let lock = NSLock()
|
||||
private var current: Int64
|
||||
|
||||
init(_ value: Int64) {
|
||||
current = value
|
||||
}
|
||||
|
||||
var value: Int64 {
|
||||
get { lock.withLock { current } }
|
||||
set { lock.withLock { current = newValue } }
|
||||
}
|
||||
}
|
||||
|
||||
/// The multilingual model's download or load, on home and under its switch in Experiments:
|
||||
/// what it's doing, that Hex dictates with its usual model meanwhile, and after a failure
|
||||
/// Try again and Turn off.
|
||||
struct MultilingualModelStatus: View {
|
||||
let download: ParakeetUltraDownload
|
||||
let retry: () -> Void
|
||||
let turnOff: () -> Void
|
||||
|
||||
var body: some View {
|
||||
if let status = download.status {
|
||||
VStack(alignment: .leading, spacing: 10) {
|
||||
HStack(alignment: .firstTextBaseline, spacing: 10) {
|
||||
Image(systemName: download.isFailed ? "exclamationmark.circle.fill" : "arrow.down.circle")
|
||||
.font(.system(size: 15))
|
||||
.foregroundStyle(download.isFailed ? HexTheme.danger : HexTheme.textMuted)
|
||||
VStack(alignment: .leading, spacing: 3) {
|
||||
Text(status)
|
||||
.font(.system(size: 13))
|
||||
.foregroundStyle(HexTheme.textBase)
|
||||
.monospacedDigit()
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
.accessibilityIdentifier("multilingual-model-status")
|
||||
Text("Hex dictates with its usual model meanwhile.")
|
||||
.font(.system(size: 12))
|
||||
.foregroundStyle(HexTheme.textMuted)
|
||||
.fixedSize(horizontal: false, vertical: true)
|
||||
}
|
||||
Spacer(minLength: 0)
|
||||
}
|
||||
if let fraction = download.fraction {
|
||||
ProgressView(value: fraction)
|
||||
.tint(HexTheme.textBase)
|
||||
}
|
||||
if download.isFailed {
|
||||
HStack(spacing: 8) {
|
||||
Button("Try again", action: retry)
|
||||
.accessibilityIdentifier("multilingual-model-try-again")
|
||||
Button("Turn off", action: turnOff)
|
||||
.accessibilityIdentifier("multilingual-model-turn-off")
|
||||
}
|
||||
.font(.system(size: 13, weight: .medium))
|
||||
.foregroundStyle(HexTheme.textBase)
|
||||
.buttonStyle(MultilingualStatusButtonStyle())
|
||||
}
|
||||
}
|
||||
.padding(14)
|
||||
.background(
|
||||
download.isFailed ? HexTheme.dangerSurface : Color.white.opacity(0.055),
|
||||
in: RoundedRectangle(cornerRadius: 14, style: .continuous)
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private struct MultilingualStatusButtonStyle: ButtonStyle {
|
||||
func makeBody(configuration: Configuration) -> some View {
|
||||
configuration.label
|
||||
.padding(.horizontal, 12)
|
||||
.frame(height: 28)
|
||||
.background(Color.white.opacity(configuration.isPressed ? 0.15 : 0.09), in: Capsule())
|
||||
}
|
||||
}
|
||||
@@ -21,6 +21,17 @@ nonisolated enum ParakeetUltraModel {
|
||||
case corrupt(String)
|
||||
}
|
||||
|
||||
/// The stand-in server simulator runs download from (`HEX_ULTRA_MODEL_BASE`), in debug
|
||||
/// builds only: it serves the same files at the same paths.
|
||||
static var baseURL: String {
|
||||
#if DEBUG
|
||||
if let base = ProcessInfo.processInfo.environment["HEX_ULTRA_MODEL_BASE"], !base.isEmpty {
|
||||
return base
|
||||
}
|
||||
#endif
|
||||
return "https://huggingface.co/\(repository)/resolve/\(revision)"
|
||||
}
|
||||
|
||||
static let repository = "FluidInference/parakeet-ultra-coreml"
|
||||
static let revision = "95eaa59a39d4394f047a4dc5cce480388a60d1b6"
|
||||
|
||||
@@ -76,20 +87,32 @@ nonisolated enum ParakeetUltraModel {
|
||||
}
|
||||
|
||||
/// Every file at its full size; each was checked against its digest before it got its name.
|
||||
static func isDownloaded(in directory: URL = directory) -> Bool {
|
||||
static func isDownloaded(in directory: URL = directory, files: [File] = files) -> Bool {
|
||||
files.allSatisfy { size(of: directory.appendingPathComponent($0.path)) == $0.size }
|
||||
}
|
||||
|
||||
static func url(for file: File) -> URL {
|
||||
URL(string: "https://huggingface.co/\(repository)/resolve/\(revision)/\(file.path)")!
|
||||
/// What's on disk so far, finished files and `.partial` ones: where a download resumes.
|
||||
static func bytesOnDisk(in directory: URL = directory, files: [File] = files) -> Int64 {
|
||||
files.reduce(0) { total, file in
|
||||
let destination = directory.appendingPathComponent(file.path)
|
||||
if let size = size(of: destination), size == file.size { return total + size }
|
||||
let partial = size(of: destination.appendingPathExtension("partial")) ?? 0
|
||||
return total + min(partial, file.size)
|
||||
}
|
||||
}
|
||||
|
||||
static func url(for file: File, base: String = baseURL) -> URL {
|
||||
URL(string: "\(base)/\(file.path)")!
|
||||
}
|
||||
|
||||
/// Downloads what's missing, resuming a file from its `.partial`. `progress` gets the
|
||||
/// fraction of `totalBytes` on disk.
|
||||
/// bytes on disk, first what an earlier download left.
|
||||
static func download(
|
||||
to directory: URL = directory,
|
||||
files: [File] = files,
|
||||
base: String = baseURL,
|
||||
session: URLSession = .shared,
|
||||
progress: @escaping @Sendable (Double) -> Void
|
||||
progress: @escaping @Sendable (Int64) -> Void
|
||||
) async throws {
|
||||
try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
|
||||
var folder = directory
|
||||
@@ -97,19 +120,21 @@ nonisolated enum ParakeetUltraModel {
|
||||
values.isExcludedFromBackup = true
|
||||
try? folder.setResourceValues(values)
|
||||
|
||||
progress(bytesOnDisk(in: directory, files: files))
|
||||
var done: Int64 = 0
|
||||
for file in files {
|
||||
try Task.checkCancellation()
|
||||
let destination = directory.appendingPathComponent(file.path)
|
||||
if size(of: destination) == file.size {
|
||||
done += file.size
|
||||
progress(Double(done) / Double(totalBytes))
|
||||
continue
|
||||
}
|
||||
let before = done
|
||||
try await fetch(file, to: destination, session: session) { written in
|
||||
progress(Double(before + written) / Double(totalBytes))
|
||||
try await fetch(file, to: destination, base: base, session: session) { written in
|
||||
progress(before + written)
|
||||
}
|
||||
done += file.size
|
||||
progress(done)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -167,6 +192,7 @@ nonisolated enum ParakeetUltraModel {
|
||||
private static func fetch(
|
||||
_ file: File,
|
||||
to destination: URL,
|
||||
base: String,
|
||||
session: URLSession,
|
||||
onBytes: @escaping @Sendable (Int64) -> Void
|
||||
) async throws {
|
||||
@@ -179,7 +205,7 @@ nonisolated enum ParakeetUltraModel {
|
||||
offset = 0
|
||||
}
|
||||
if offset < file.size {
|
||||
var request = URLRequest(url: url(for: file))
|
||||
var request = URLRequest(url: url(for: file, base: base))
|
||||
if offset > 0 {
|
||||
request.setValue("bytes=\(offset)-", forHTTPHeaderField: "Range")
|
||||
}
|
||||
|
||||
@@ -10,8 +10,9 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
|
||||
enum TranscriberError: HexDescribedError {
|
||||
case notPrepared
|
||||
case emptyTranscript
|
||||
case downloadFailed
|
||||
case notEnoughMemory
|
||||
/// Not on the iPhone yet: `ParakeetUltraDownload` gets it beside the usual engine.
|
||||
case notDownloaded
|
||||
|
||||
var errorDescription: String? {
|
||||
switch self {
|
||||
@@ -19,15 +20,19 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
|
||||
String(localized: "Parakeet Ultra is not ready.")
|
||||
case .emptyTranscript:
|
||||
String(localized: "Parakeet did not detect any speech.")
|
||||
case .downloadFailed:
|
||||
String(localized: "Parakeet Ultra didn't download. Check your connection and try again.")
|
||||
case .notEnoughMemory:
|
||||
String(localized: "Parakeet Ultra needs more memory than Hex has here.")
|
||||
case .notDownloaded:
|
||||
String(localized: "Parakeet Ultra isn't downloaded yet.")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static let name = "Parakeet Ultra"
|
||||
/// After a load that ended Hex (`ParakeetLoadGuard`), beside Try again.
|
||||
nonisolated static var setAsideMessage: String {
|
||||
String(localized: "Hex closed while getting the multilingual model ready, likely short of memory.")
|
||||
}
|
||||
/// `ParakeetLoadGuard`'s and `ParakeetCompileRecord`'s keys, apart from V2's.
|
||||
static let loadGuardModel = "parakeetUltra"
|
||||
static let compileRecordKey = "parakeetUltra.compiledForSystem"
|
||||
@@ -74,47 +79,33 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
|
||||
.replacingOccurrences(of: " ", with: "\u{00A0}")
|
||||
}
|
||||
|
||||
/// What the bar says while it downloads: the size, and that Wi-Fi is better for it.
|
||||
nonisolated static var downloadDetail: String {
|
||||
String(localized: "Downloading \(downloadSize) · Wi-Fi recommended")
|
||||
/// Whether Core ML compiled it for this iOS already, so loading it takes seconds rather
|
||||
/// than the minute or more of a first load.
|
||||
nonisolated static var isCompiled: Bool {
|
||||
ParakeetCompileRecord(key: compileRecordKey).isCurrent
|
||||
}
|
||||
|
||||
/// Loads the downloaded model; the download itself is `ParakeetUltraDownload`'s, beside
|
||||
/// the engine Hex dictates with meanwhile.
|
||||
func prepare(
|
||||
progressHandler: @escaping @Sendable (ModelPreparationProgress) -> Void
|
||||
) async throws {
|
||||
guard manager == nil else { return }
|
||||
guard ParakeetUltraModel.isDownloaded() else { throw TranscriberError.notDownloaded }
|
||||
|
||||
let compiled = ParakeetCompileRecord(key: Self.compileRecordKey)
|
||||
let downloads = !ParakeetUltraModel.isDownloaded()
|
||||
let progress = ParakeetProgressFeed(
|
||||
ParakeetPreparationProgress(
|
||||
downloads: downloads,
|
||||
compiledBefore: compiled.isCurrent,
|
||||
downloadDetail: Self.downloadDetail
|
||||
),
|
||||
ParakeetPreparationProgress(downloads: false, compiledBefore: compiled.isCurrent),
|
||||
emit: progressHandler
|
||||
)
|
||||
let download: Task<Void, Error>? = downloads ? Task { try await Self.download(progress) } : nil
|
||||
let ticker = Task {
|
||||
while !Task.isCancelled {
|
||||
progress.tick()
|
||||
// Turned off mid-download: stop rather than finish a download it then deletes.
|
||||
if download != nil, !MultilingualParakeetSettings().isEnabled {
|
||||
download?.cancel()
|
||||
}
|
||||
try? await Task.sleep(for: .milliseconds(250))
|
||||
}
|
||||
}
|
||||
defer { ticker.cancel() }
|
||||
|
||||
if let download {
|
||||
try await withTaskCancellationHandler {
|
||||
try await download.value
|
||||
} onCancel: {
|
||||
download.cancel()
|
||||
}
|
||||
}
|
||||
|
||||
// As V2: only the load, where iOS can end Hex short of memory, counts for the guard,
|
||||
// and too little left to load it sets it aside for the engine Hex uses without it.
|
||||
let loads = ParakeetLoadGuard(model: Self.loadGuardModel)
|
||||
@@ -138,26 +129,6 @@ actor ParakeetUltraTranscriber: LocalTranscribing {
|
||||
progress.update { $0.finish() }
|
||||
}
|
||||
|
||||
/// Resumes each file from its `.partial`; a download that stops moving for
|
||||
/// `ParakeetTranscriber.downloadStallLimit` ends, as V2's does.
|
||||
private static func download(_ progress: ParakeetProgressFeed) async throws {
|
||||
let fraction = ProgressReading()
|
||||
do {
|
||||
try await StallWatch.run(
|
||||
limit: ParakeetTranscriber.downloadStallLimit,
|
||||
progress: { fraction.value },
|
||||
operation: {
|
||||
try await ParakeetUltraModel.download { value in
|
||||
fraction.value = value
|
||||
progress.update { $0.download(value) }
|
||||
}
|
||||
}
|
||||
)
|
||||
} catch is ParakeetUltraModel.DownloadError {
|
||||
throw TranscriberError.downloadFailed
|
||||
}
|
||||
}
|
||||
|
||||
func transcribe(_ audioURL: URL) async throws -> String {
|
||||
guard let manager else {
|
||||
throw TranscriberError.notPrepared
|
||||
|
||||
@@ -750,11 +750,21 @@ struct ExperimentsSettingsPage: View {
|
||||
SettingsToggleRow(
|
||||
title: String(localized: "Multilingual model (Parakeet)"),
|
||||
detail: multilingualParakeetFits
|
||||
? String(localized: "25 languages, even mixed in a sentence. \(Self.multilingualParakeetSize) download. Keyboard dictation opens Hex.")
|
||||
? String(localized: "25 languages, even mixed in a sentence. \(Self.multilingualParakeetSize) download, Wi-Fi recommended. Once it's ready, keyboard dictation opens Hex.")
|
||||
: String(localized: "Needs an iPhone with 6 GB of memory or more"),
|
||||
isOn: $multilingualParakeet
|
||||
)
|
||||
.disabled(!multilingualParakeetFits)
|
||||
|
||||
if multilingualParakeet {
|
||||
MultilingualModelStatus(
|
||||
download: DictationController.shared.multilingualDownload,
|
||||
retry: { DictationController.shared.retryMultilingualParakeet() },
|
||||
turnOff: { multilingualParakeet = false }
|
||||
)
|
||||
.padding(.horizontal, 12)
|
||||
.padding(.bottom, 12)
|
||||
}
|
||||
}
|
||||
}
|
||||
.animation(HexMotion.settleAnimation(reduceMotion: reduceMotion), value: adaptiveTouch)
|
||||
@@ -789,12 +799,7 @@ struct ExperimentsSettingsPage: View {
|
||||
.onChange(of: multilingualParakeet) { _, enabled in
|
||||
guard enabled != MultilingualParakeetSettings().isEnabled else { return }
|
||||
MultilingualParakeetSettings().isEnabled = enabled
|
||||
if enabled {
|
||||
// Turning it on is the way to try again after a load that ended Hex.
|
||||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
|
||||
} else {
|
||||
ParakeetUltraModel.remove()
|
||||
}
|
||||
// On: downloads beside the engine that dictates; off: stops and deletes it.
|
||||
DictationController.shared.multilingualParakeetChanged()
|
||||
}
|
||||
.onAppear {
|
||||
|
||||
@@ -1569,23 +1569,136 @@ final class DictationControllerTests: XCTestCase {
|
||||
XCTAssertFalse(ParakeetCompileRecord(key: ParakeetUltraTranscriber.compileRecordKey, defaults: defaults).isCurrent)
|
||||
}
|
||||
|
||||
func testMultilingualParakeetsDownloadSaysItsSizeAndWiFi() {
|
||||
var progress = ParakeetPreparationProgress(
|
||||
downloads: true,
|
||||
compiledBefore: false,
|
||||
downloadDetail: ParakeetUltraTranscriber.downloadDetail
|
||||
)
|
||||
progress.download(0.5)
|
||||
let detail = progress.snapshot(at: 1).detail
|
||||
XCTAssertEqual(detail, ParakeetUltraTranscriber.downloadDetail)
|
||||
XCTAssertTrue(detail.contains("632"), detail)
|
||||
XCTAssertTrue(detail.contains("Wi-Fi"), detail)
|
||||
func testMultilingualParakeetNeverTakesDictationUntilItIsReady() {
|
||||
let step = DictationController.multilingualStep
|
||||
XCTAssertEqual(step(nil, true, true), .none, "off, another language, or set aside")
|
||||
XCTAssertEqual(step("nl-NL", false, false), .download, "the usual engine dictates while it downloads")
|
||||
XCTAssertEqual(step("nl-NL", false, true), .download)
|
||||
XCTAssertEqual(step("nl-NL", true, false), .loadBeside, "a first load compiles beside the usual engine")
|
||||
XCTAssertEqual(step("nl-NL", true, true), .dictate)
|
||||
XCTAssertEqual(
|
||||
ParakeetPreparationProgress(downloads: true, compiledBefore: false).downloadDetail,
|
||||
"Downloading the speech model",
|
||||
"V2's words are unchanged"
|
||||
)
|
||||
XCTAssertTrue(DictationController.isDownloadInterruption(ParakeetUltraTranscriber.TranscriberError.downloadFailed))
|
||||
}
|
||||
|
||||
func testTheKeyboardOpensHexForMultilingualParakeetOnlyOnceItIsReady() {
|
||||
let settings = MultilingualParakeetSettings(defaults: defaults)
|
||||
XCTAssertFalse(settings.keyboardOpensHex(language: "nl-NL"), "off")
|
||||
settings.isEnabled = true
|
||||
XCTAssertFalse(settings.keyboardOpensHex(language: "nl-NL"), "on, still downloading: the keyboard dictates itself")
|
||||
settings.isModelReady = true
|
||||
XCTAssertTrue(settings.keyboardOpensHex(language: "nl-NL"))
|
||||
XCTAssertFalse(settings.keyboardOpensHex(language: nil), "a language it doesn't take")
|
||||
settings.isEnabled = false
|
||||
XCTAssertFalse(settings.keyboardOpensHex(language: "nl-NL"))
|
||||
}
|
||||
|
||||
func testOnlyASwitchTurnedOffDeletesTheMultilingualModelAtLaunch() {
|
||||
let settings = MultilingualParakeetSettings(defaults: defaults)
|
||||
XCTAssertFalse(settings.isTurnedOff, "never set: nothing to delete")
|
||||
settings.isEnabled = true
|
||||
XCTAssertFalse(settings.isTurnedOff)
|
||||
settings.isEnabled = false
|
||||
XCTAssertTrue(settings.isTurnedOff)
|
||||
XCTAssertFalse(
|
||||
MultilingualParakeetSettings(defaults: nil).isTurnedOff,
|
||||
"an App Group Hex can't read says nothing about the switch"
|
||||
)
|
||||
}
|
||||
|
||||
func testTheMultilingualDownloadResumesFromItsPartialFileAndSaysItsBytes() async throws {
|
||||
let model = try UltraStubModel()
|
||||
defer { model.remove() }
|
||||
let first = model.files[0]
|
||||
let partial = model.folder.appendingPathComponent(first.path).appendingPathExtension("partial")
|
||||
try FileManager.default.createDirectory(at: partial.deletingLastPathComponent(), withIntermediateDirectories: true)
|
||||
try model.bodies[first.path]!.prefix(1000).write(to: partial)
|
||||
XCTAssertEqual(ParakeetUltraModel.bytesOnDisk(in: model.folder, files: model.files), 1000)
|
||||
|
||||
let download = model.download()
|
||||
download.start()
|
||||
if case .downloading(let bytes) = download.state {
|
||||
XCTAssertEqual(bytes, 1000, "picks up from what's on disk")
|
||||
} else {
|
||||
XCTFail("\(download.state)")
|
||||
}
|
||||
XCTAssertEqual(download.status, "Downloading the multilingual model: 1\u{00A0}KB of 5\u{00A0}KB")
|
||||
try await waitUntil { download.state == .loading }
|
||||
|
||||
XCTAssertTrue(ParakeetUltraModel.isDownloaded(in: model.folder, files: model.files))
|
||||
XCTAssertEqual(UltraStubProtocol.requested.first ?? nil, "bytes=1000-", "resumed with a range")
|
||||
}
|
||||
|
||||
func testAMultilingualDownloadThatDropsOrStallsTriesAgainFromWhereItWas() async throws {
|
||||
let model = try UltraStubModel()
|
||||
defer { model.remove() }
|
||||
UltraStubProtocol.script([.cut(1200), .hang(800), .serve])
|
||||
let download = model.download()
|
||||
var finished = 0
|
||||
download.onDownloaded = { finished += 1 }
|
||||
download.start()
|
||||
try await waitUntil(seconds: 10) { download.state == .loading }
|
||||
|
||||
XCTAssertEqual(finished, 1)
|
||||
XCTAssertTrue(ParakeetUltraModel.isDownloaded(in: model.folder, files: model.files))
|
||||
XCTAssertEqual(
|
||||
Array(UltraStubProtocol.requested.prefix(3)),
|
||||
[nil, "bytes=1200-", "bytes=2000-"],
|
||||
"a dropped connection and a stalled one each pick up from the partial file"
|
||||
)
|
||||
}
|
||||
|
||||
func testAMultilingualDownloadThatKeepsMovingNeverRunsOutOfTries() async throws {
|
||||
let model = try UltraStubModel()
|
||||
defer { model.remove() }
|
||||
UltraStubProtocol.script(Array(repeating: .cut(400), count: ParakeetUltraDownload.automaticTries + 1))
|
||||
let download = model.download()
|
||||
download.start()
|
||||
try await waitUntil(seconds: 10) { download.state == .loading || download.isFailed }
|
||||
XCTAssertEqual(download.state, .loading, "each drop came after more bytes, so it kept going")
|
||||
XCTAssertTrue(ParakeetUltraModel.isDownloaded(in: model.folder, files: model.files))
|
||||
}
|
||||
|
||||
func testAMultilingualDownloadThatKeepsFailingSaysWhatHappened() async throws {
|
||||
let model = try UltraStubModel()
|
||||
defer { model.remove() }
|
||||
UltraStubProtocol.script(Array(repeating: .status(503), count: 10))
|
||||
let download = model.download()
|
||||
download.start()
|
||||
try await waitUntil { download.isFailed }
|
||||
XCTAssertEqual(UltraStubProtocol.requested.count, ParakeetUltraDownload.automaticTries)
|
||||
XCTAssertEqual(download.status, "Hugging Face didn't send the multilingual model (error 503). Try again later.")
|
||||
|
||||
UltraStubProtocol.script([.status(404)])
|
||||
download.restart()
|
||||
try await waitUntil { download.isFailed }
|
||||
XCTAssertEqual(UltraStubProtocol.requested.count, 1, "a refused request isn't tried again by itself")
|
||||
|
||||
XCTAssertEqual(
|
||||
ParakeetUltraDownload.message(for: URLError(.notConnectedToInternet)),
|
||||
"The multilingual model's download stopped. Check your connection. It picks up where it left off."
|
||||
)
|
||||
XCTAssertTrue(ParakeetUltraDownload.message(for: CocoaError(.fileWriteOutOfSpace)).contains("free space"))
|
||||
XCTAssertEqual(ParakeetUltraDownload.backoff(1), 2)
|
||||
XCTAssertEqual(ParakeetUltraDownload.backoff(4), 16)
|
||||
XCTAssertEqual(ParakeetUltraDownload.backoff(9), 30)
|
||||
}
|
||||
|
||||
func testTurningTheMultilingualModelOffStopsItsDownloadBeforeItIsDeleted() async throws {
|
||||
let model = try UltraStubModel()
|
||||
defer { model.remove() }
|
||||
UltraStubProtocol.script([.hang(800)])
|
||||
let download = model.download(stallLimit: .seconds(30))
|
||||
download.start()
|
||||
try await waitUntil { ParakeetUltraModel.bytesOnDisk(in: model.folder, files: model.files) == 800 }
|
||||
|
||||
await download.stop()
|
||||
XCTAssertEqual(download.state, .idle)
|
||||
ParakeetUltraModel.remove(from: model.folder)
|
||||
try await Task.sleep(for: .milliseconds(300))
|
||||
XCTAssertFalse(FileManager.default.fileExists(atPath: model.folder.path), "nothing wrote after stop")
|
||||
}
|
||||
|
||||
func testParakeetUltrasFilesArePinnedAndComplete() throws {
|
||||
@@ -1631,6 +1744,15 @@ final class DictationControllerTests: XCTestCase {
|
||||
try await settle()
|
||||
XCTAssertEqual(made, 1)
|
||||
|
||||
// Turned off (the test host's App Group has it off) while the usual engine dictates:
|
||||
// nothing to prepare again.
|
||||
controller.multilingualParakeetChanged()
|
||||
try await settle()
|
||||
XCTAssertEqual(made, 1)
|
||||
XCTAssertEqual(controller.phase, .ready)
|
||||
|
||||
// Turned off while Parakeet Ultra dictated: Hex prepares the engine it picks without it.
|
||||
controller.engineName = ParakeetUltraTranscriber.name
|
||||
controller.multilingualParakeetChanged()
|
||||
try await settle()
|
||||
XCTAssertEqual(made, 2)
|
||||
@@ -2155,6 +2277,146 @@ final class DictationControllerTests: XCTestCase {
|
||||
try await Task.sleep(for: .milliseconds(25))
|
||||
}
|
||||
}
|
||||
|
||||
private func waitUntil(seconds: Double = 5, _ condition: () -> Bool) async throws {
|
||||
let deadline = Date().addingTimeInterval(seconds)
|
||||
while !condition() {
|
||||
guard Date() < deadline else {
|
||||
XCTFail("timed out")
|
||||
return
|
||||
}
|
||||
try await Task.sleep(for: .milliseconds(20))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Two small files standing in for Parakeet Ultra's, served by `UltraStubProtocol`.
|
||||
@MainActor
|
||||
private struct UltraStubModel {
|
||||
let folder = FileManager.default.temporaryDirectory.appendingPathComponent("ultra-\(UUID().uuidString)")
|
||||
let bodies: [String: Data]
|
||||
let files: [ParakeetUltraModel.File]
|
||||
|
||||
init() throws {
|
||||
var bodies: [String: Data] = [:]
|
||||
var files: [ParakeetUltraModel.File] = []
|
||||
let scratch = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
|
||||
defer { try? FileManager.default.removeItem(at: scratch) }
|
||||
for (path, size) in [("Encoder.mlmodelc/weights/weight.bin", 3000), ("parakeet_vocab.json", 2000)] {
|
||||
let body = Data((0..<size).map { UInt8(truncatingIfNeeded: $0 &* 31 &+ path.count) })
|
||||
try body.write(to: scratch)
|
||||
bodies[path] = body
|
||||
files.append(.init(path: path, size: Int64(size), sha256: try ParakeetUltraModel.sha256(of: scratch)))
|
||||
}
|
||||
self.bodies = bodies
|
||||
self.files = files
|
||||
UltraStubProtocol.reset(bodies: bodies)
|
||||
}
|
||||
|
||||
func download(stallLimit: Duration = .seconds(1)) -> ParakeetUltraDownload {
|
||||
let configuration = URLSessionConfiguration.ephemeral
|
||||
configuration.protocolClasses = [UltraStubProtocol.self]
|
||||
return ParakeetUltraDownload(
|
||||
directory: folder,
|
||||
files: files,
|
||||
base: "https://stub.invalid/model",
|
||||
session: URLSession(configuration: configuration),
|
||||
stallLimit: stallLimit,
|
||||
backoff: { _ in 0 }
|
||||
)
|
||||
}
|
||||
|
||||
func remove() {
|
||||
try? FileManager.default.removeItem(at: folder)
|
||||
}
|
||||
}
|
||||
|
||||
/// Serves `UltraStubModel`'s files with ranges, following a script for its next requests:
|
||||
/// cut the connection after some bytes, send some and then nothing, or answer a status.
|
||||
private final class UltraStubProtocol: URLProtocol, @unchecked Sendable {
|
||||
enum Step: Sendable {
|
||||
case serve
|
||||
case cut(Int)
|
||||
case hang(Int)
|
||||
case status(Int)
|
||||
}
|
||||
|
||||
private static let lock = NSLock()
|
||||
nonisolated(unsafe) private static var bodies: [String: Data] = [:]
|
||||
nonisolated(unsafe) private static var steps: [Step] = []
|
||||
nonisolated(unsafe) private static var ranges: [String?] = []
|
||||
|
||||
static func reset(bodies: [String: Data]) {
|
||||
lock.withLock {
|
||||
self.bodies = bodies
|
||||
steps = []
|
||||
ranges = []
|
||||
}
|
||||
}
|
||||
|
||||
static func script(_ steps: [Step]) {
|
||||
lock.withLock {
|
||||
self.steps = steps
|
||||
ranges = []
|
||||
}
|
||||
}
|
||||
|
||||
/// Each request's Range header, in order.
|
||||
static var requested: [String?] {
|
||||
lock.withLock { ranges }
|
||||
}
|
||||
|
||||
override class func canInit(with request: URLRequest) -> Bool { true }
|
||||
override class func canonicalRequest(for request: URLRequest) -> URLRequest { request }
|
||||
|
||||
override func startLoading() {
|
||||
let url = request.url!
|
||||
let range = request.value(forHTTPHeaderField: "Range")
|
||||
let path = url.path.replacingOccurrences(of: "/model/", with: "")
|
||||
let (step, body): (Step, Data?) = Self.lock.withLock {
|
||||
Self.ranges.append(range)
|
||||
let step = Self.steps.isEmpty ? .serve : Self.steps.removeFirst()
|
||||
return (step, Self.bodies[path])
|
||||
}
|
||||
guard let body else {
|
||||
respond(404, url: url)
|
||||
client?.urlProtocolDidFinishLoading(self)
|
||||
return
|
||||
}
|
||||
if case .status(let status) = step {
|
||||
respond(status, url: url)
|
||||
client?.urlProtocolDidFinishLoading(self)
|
||||
return
|
||||
}
|
||||
let offset = range.flatMap { Int($0.dropFirst("bytes=".count).dropLast()) } ?? 0
|
||||
respond(offset > 0 ? 206 : 200, url: url)
|
||||
let rest = body.dropFirst(offset)
|
||||
switch step {
|
||||
case .cut(let count):
|
||||
client?.urlProtocol(self, didLoad: Data(rest.prefix(count)))
|
||||
// The connection drops a moment after the bytes arrive, as on a network.
|
||||
DispatchQueue.global().asyncAfter(deadline: .now() + 0.2) {
|
||||
self.client?.urlProtocol(self, didFailWithError: URLError(.networkConnectionLost))
|
||||
}
|
||||
case .hang(let count):
|
||||
client?.urlProtocol(self, didLoad: Data(rest.prefix(count)))
|
||||
case .serve, .status:
|
||||
client?.urlProtocol(self, didLoad: Data(rest))
|
||||
client?.urlProtocolDidFinishLoading(self)
|
||||
}
|
||||
}
|
||||
|
||||
override func stopLoading() {}
|
||||
|
||||
private func respond(_ status: Int, url: URL) {
|
||||
let response = HTTPURLResponse(
|
||||
url: url,
|
||||
statusCode: status,
|
||||
httpVersion: "HTTP/1.1",
|
||||
headerFields: ["Content-Type": "application/octet-stream"]
|
||||
)!
|
||||
client?.urlProtocol(self, didReceive: response, cacheStoragePolicy: .notAllowed)
|
||||
}
|
||||
}
|
||||
|
||||
/// The dictation language rule: an explicit choice, else the first preferred language either
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
import Foundation
|
||||
|
||||
/// Experiments › Multilingual model (Parakeet), in the shared App Group so the keyboard reads
|
||||
/// it too. Off by default. While it's on, Hex dictates its 25 languages with Parakeet Ultra
|
||||
/// (`ParakeetUltraTranscriber`), which hears them mixed within a sentence (Dutch with English
|
||||
/// words), and the Hex key hands its dictations to Hex, as before iOS 26: the model can't load
|
||||
/// within a keyboard's memory limit.
|
||||
/// it too. Off by default. While it's on and Hex has the model ready, Hex dictates its 25
|
||||
/// languages with Parakeet Ultra (`ParakeetUltraTranscriber`), which hears them mixed within a
|
||||
/// sentence (Dutch with English words), and the Hex key hands its dictations to Hex, as before
|
||||
/// iOS 26: the model can't load within a keyboard's memory limit. Until it's ready, both
|
||||
/// dictate as they would with the experiment off.
|
||||
nonisolated struct MultilingualParakeetSettings {
|
||||
static let enabledKey = "experiments.multilingualParakeet"
|
||||
/// Set by Hex once Parakeet Ultra loaded; cleared while it's missing, set aside or off.
|
||||
static let modelReadyKey = "experiments.multilingualParakeet.ready"
|
||||
static let launchArgument = "--experiment-multilingual-parakeet"
|
||||
|
||||
/// Parakeet Ultra's languages (ISO 639-1), those of NVIDIA Parakeet TDT 0.6B v3.
|
||||
@@ -29,6 +32,25 @@ nonisolated struct MultilingualParakeetSettings {
|
||||
nonmutating set { defaults?.set(newValue, forKey: Self.enabledKey) }
|
||||
}
|
||||
|
||||
/// Turned off by the person, as stored: never set reads false here, and so does an App
|
||||
/// Group iOS doesn't let Hex read yet (a launch before the iPhone's first unlock), which
|
||||
/// must not delete a model that is on.
|
||||
var isTurnedOff: Bool {
|
||||
defaults?.object(forKey: Self.enabledKey) as? Bool == false
|
||||
}
|
||||
|
||||
/// Whether Hex has Parakeet Ultra loaded at least once since it last downloaded or was let
|
||||
/// back in: until then, the keyboard dictates as it would with the experiment off.
|
||||
var isModelReady: Bool {
|
||||
get { defaults?.bool(forKey: Self.modelReadyKey) ?? false }
|
||||
nonmutating set { defaults?.set(newValue, forKey: Self.modelReadyKey) }
|
||||
}
|
||||
|
||||
/// The keyboard's Dictate and Hex keys open Hex for it: on, ready, and the language its.
|
||||
func keyboardOpensHex(language: String?) -> Bool {
|
||||
isEnabled && isModelReady && language != nil
|
||||
}
|
||||
|
||||
/// Whether this iPhone has the memory to load it at all.
|
||||
static func fits(physicalMemory: UInt64 = ProcessInfo.processInfo.physicalMemory) -> Bool {
|
||||
physicalMemory >= minimumPhysicalMemory
|
||||
|
||||
+23
-8
@@ -310,20 +310,35 @@ never at a launch by itself.
|
||||
0.6B v3) with it, which hears them mixed within a sentence (Dutch with English
|
||||
words, a TestFlight request); any other dictation language keeps the engine Hex
|
||||
picks without it. The language Ultra takes is the one set, else the one Automatic
|
||||
follows on the keyboard, else the iPhone's first. Turning it on downloads
|
||||
follows on the keyboard, else the iPhone's first. The experiment never takes
|
||||
dictation away: until Ultra is ready, Hex and the keyboard dictate as they would
|
||||
with it off, and Hex switches to Ultra once it has loaded (a dictation in flight
|
||||
isn't interrupted). Turning it on downloads
|
||||
FluidInference's Core ML build (632 MB, pinned to revision `95eaa59`, each file
|
||||
checked against its SHA-256 and resumed from its `.partial`) into Application
|
||||
Support, out of backups, through the usual preparation bar ("Downloading 632.2 MB ·
|
||||
Wi-Fi recommended", then Core ML's compile); turning it off deletes it, as does a
|
||||
launch that finds a download left with it off. FluidAudio 0.15.5 has no
|
||||
Support, out of backups, beside that engine (`ParakeetUltraDownload`): home and the
|
||||
switch say "Downloading the multilingual model: 123.4 MB of 632.2 MB" with a bar. A
|
||||
dropped connection, or bytes that stop for 20 s, tries again by itself after 2, 4, 8,
|
||||
16, then 30 s, starting over whenever it got further; in the background it waits for
|
||||
Hex to come back to the front, which picks it up at once (and starts a fresh
|
||||
connection after 25 s away). After six tries in a row that got nothing further, or a
|
||||
refused request or a full iPhone, it says what happened in plain words with Try again
|
||||
and Turn off. Once downloaded, Ultra's first load (Core ML's compile, a minute or
|
||||
more) runs beside the usual engine too, except where Parakeet V2 dictates and leaves
|
||||
no room for it (English before iOS 26), where Ultra takes V2's place for that load;
|
||||
later launches load it directly when Core ML has compiled it for this iOS. Turning it off stops the download, then deletes it;
|
||||
a launch deletes a leftover only when the switch reads off as stored, never when the
|
||||
App Group can't be read (a launch before the first unlock). FluidAudio 0.15.5 has no
|
||||
`.ultra`, so `ParakeetUltraModel` loads the four models itself and hands
|
||||
FluidAudio v3 `AsrModels` (its own loader would replace a folder it can't load
|
||||
with v3's download). It is chosen only with 1.5 GB left to load it
|
||||
(`os_proc_available_memory`), and never again after a load that ended Hex
|
||||
(`ParakeetLoadGuard`, its own keys) until the switch is turned off and on. The
|
||||
keyboard can't load it, so while it takes the dictation language, Dictate and the
|
||||
Hex key open Hex, as before iOS 26 (a warm session still dictates without
|
||||
leaving the app). Measured on a Mac (`~/dev/experiments/2026-11-21-hex-multilingual-parakeet`):
|
||||
(`ParakeetLoadGuard`, its own keys) until Try again beside the status, or the switch
|
||||
turned off and on. The keyboard can't load it, so once Hex has it ready
|
||||
(`MultilingualParakeetSettings.isModelReady`, set after a load, cleared while it's
|
||||
missing, set aside or off) and it takes the dictation language, Dictate and the Hex
|
||||
key open Hex, as before iOS 26 (a warm session still dictates without leaving the
|
||||
app); until then the keyboard dictates itself. Measured on a Mac (`~/dev/experiments/2026-11-21-hex-multilingual-parakeet`):
|
||||
mixed Dutch-English test sentences 7.5% WER against Apple Speech nl-NL's 26.9%,
|
||||
pure Dutch 0.9% (0.4%), pure English 0% (0%). Load memory, compile time and speed
|
||||
on an iPhone are unmeasured.
|
||||
|
||||
Reference in New Issue
Block a user