mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-21 17:56:51 +08:00
dev
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9a06976d0e |
iOS: Italian, French and Portuguese layouts that autocorrect in their own language; spoken @ and email addresses (#2069)
* iOS: Italian, French and Portuguese full-keyboard layouts that autocorrect in their own language A TestFlight tester on build 31 with an Italian iPhone reported that Hex kept autocorrecting their Italian into English. KeyboardLanguage only knew en, nb, es and de, so Automatic mapped an it-IT dictation language (and any other language without a layout) to English: English letters, and UITextChecker's en_US dictionary rewriting Italian words. Adds italian/french/portuguese cases. They type on the English QWERTY rows (Apple's Italian and Portuguese keyboards are QWERTY; Hex has no AZERTY yet, so French is QWERTY too), with their own long-press accent order (Italian à è ì ò ù, é right after è), their UITextChecker dictionary (it, fr, pt_BR then pt_PT), and an Italian common two-letter list. Automatic now resolves it-*, fr-* and pt-* to these; the dictation language still wins over the iPhone's keyboards, and Hebrew or Russian still get English. Automatic deliberately ignores Hex's own input mode: its primaryLanguage is the Info.plist's en-US whatever you type. Spatial autocorrect (#2059) only has en and es models; the new layouts get no model and fall back to the dictionary engine in their own language (the decoder also refuses a model of another language). The space bar already names the dictation language ("Italiano"). Snapshot scenes get Italian, French and Portuguese drafts (--dictation-language it-IT, --keyboard-layout it|fr|pt). Verified on an iOS 27.2 iPhone 17 Pro simulator: full HEXMobileTests suite (272 tests, 0 failures, 1 skipped as before). New cases in KeyboardLayoutTests, AutocorrectEngineTests and FullKeyboardEngineTests. Measured on 12 Italian sentences (19 one-slip typos, 59 correct words) with the real dictionary: English fallback fixed 4, miscorrected 4 and broke 19 correct words (Domani->Domain, stai->stay, Sono->Soon, molto->moto); Italian fixed 17, miscorrected 0 and broke none. Renders of the Italian keyboard (typing with the "ritardo" suggestion, the E and I accent menus, "Italiano" on the space bar) checked by eye. * iOS: spoken "at the rate" and "at sign" become "@", joining a spoken email address An Indian English TestFlight tester (build 32) said "at the rate", India's name for @, and got the words. Apple's on-device models don't fix it: checked on macOS 27.2 with Hex's settings and an en_IN voice, SpeechTranscriber (Hex's default) writes "My email is John at the rategmail.com." and "Send it to Priya at signyahoo.com." (gluing the domain to the word before), and DictationTranscriber writes "at the rate gmail.com", though it already writes "priya@yahoo.com" for "at sign" and "john@gmail.com" for "john at gmail dot com". SpokenCleanupRules.spokenSymbols (English only) turns "at the rate" and "at sign" into "@". Before a domain ("gmail dot com", "gmail.com", "co dot in", or glued as above), the word before (with any "dot"-joined parts) and the domain join into one lowercase address, "dot" becoming "." only there: "John at the rate Gmail dot com." becomes "john@gmail.com.". "at the rate" counts only before a domain, so "at the rate of 5%" and "at the rate we planned" stay, and "at" alone is never touched; "at sign" elsewhere is a lone "@" (but not in "at sign language"). Only the matched words change. It runs from CleanupTagger.cleanIfEnabled, which both the app's and in-keyboard dictation call: in every style but Verbatim, with Clean up what I say on or off (the experiment's guard moved below it; the tagger itself still only runs when on). TranscriptFormatter: Polished capitalized after every full stop, which would have written "john@gmail.Com."; a sentence now starts only after ".", "!" or "?" and a space, an address keeps its case (Polished and Notes), and a lone address gets no full stop. Verified on an iOS 27.2 simulator: full HEXMobileTests suite (272 tests, 0 failures, 1 skipped as before), with new cases in TranscriptCleanupTests (Apple's real outputs included) and CleanupTaggerTests. |
||
|
|
1ede7bb792 |
iOS: Spanish clean-up, paste translated or as spoken, screen stays on while recording (#2046)
* iOS: keep the screen on while Hex records or transcribes
A Spanish-speaking TestFlight tester (build 20) asked that the screen not
turn off while recording. DictationController now sets
UIApplication.isIdleTimerDisabled whenever its phase is recording or
transcribing, and clears it when the phase leaves those states (done,
cancel, failure), so Auto-Lock comes back after every dictation. Nothing
else in Hex sets the flag.
Limits: iOS honors the flag only while Hex is the foreground app (the
handoff screen and in-app recording). A warm-session or in-keyboard
dictation happens with another app in front, and a keyboard extension has
no public API to keep the screen on, so those can still dim. No private API
or value(forKey:) workaround is used.
Verified: new DictationControllerTests.testRecordingAndTranscribingKeepTheScreenOn
(recording, transcribing, done, cancel, failed) and the rest of
DictationControllerTests pass on an iPhone 18 Pro simulator (iOS 27.2).
* iOS: Spanish, German and Norwegian rules for Clean up what I say
A Spanish-speaking TestFlight tester said the corrections part works badly
in Spanish. Clean up what I say ran its cleanup tagger only for English and
returned every other language untouched, so with the experiment on a Spanish
dictation kept its fillers and spoken corrections. The tagger stays
English-only: run directly on Spanish, it changed the meaning of 8 of 18
sentences, including dropped negations ("Te digo que no puedo ir." became
"Te puedo ir.", "O sea que al final no vienes." became "O sea que vienes.").
- New SpokenCleanupRules: for es, de and nb/nn/no, a filler between commas
goes ("este", "o sea"), and a correction marker set off by commas ("digo",
"perdón", "no", "no, espera", "mejor dicho"; "nein", "ich meine"; "nei",
"nei vent", "jeg mener") drops the words it replaces. Those are counted
back to the nearest earlier word that matches the repair's first word, or
is of its kind (number, weekday, month, today or tomorrow), within the
sentence and 8 words; with neither, nothing changes. "mejor dicho", "no
mejor", "besser gesagt" and "rettere sagt" also count without commas.
Like the tagger, it only removes words; CleanupReconstruction repairs
commas and capitals.
- CleanupTagger.cleanIfEnabled runs those rules for non-English languages
instead of returning the text unchanged. The model still never loads for
them, and Verbatim still skips cleanup.
- TranscriptFormatter: the styles trim Spanish hesitations (eh, em, mmm;
never "e") and German ones (äh, ähm, öhm; never "eh"), and capitalize a
sentence that opens with ¿ or ¡.
Measured on 25 Spanish sentences I wrote, with Polished and the experiment
on (9 German and Norwegian ones too). Before, every sentence came out as
spoken. After, all 13 with fillers or corrections are cleaned ("Eh, nos
vemos mañana a las 3, digo, a las 4." -> "Nos vemos mañana a las 4.") and
all 12 that use the same words in their ordinary meaning stay unchanged
("Te digo que no puedo ir.", "Quiero este, el azul.", "Lo siento, perdón,
Ana.").
Verified: new SpokenCleanupRulesTests, updated
CleanupTaggerTests.testOtherLanguagesGetRulesNeverTheEnglishTagger (it
used to expect Norwegian "tirsdag, nei vent, torsdag" untouched), plus the
CleanupTagger, DictationController and TranscriptCleanup suites, all
passing on an iPhone 18 Pro simulator (iOS 27.2).
* iOS: Clean up what I say drops Spanish, German and Norwegian hesitations
In-keyboard dictation (iOS 26+) inserts Apple Speech's text without running
a style through TranscriptFormatter, so the Spanish and German hesitation
trim added to the styles in 71acb4f3 never reached it. The English tagger
removes "um" and "uh" in both places. SpokenCleanupRules now also drops the
language's hesitation sounds anywhere in the sentence, using the styles'
own list (TranscriptFormatter.fillerPattern): eh, em, mmm in Spanish; äh,
ähm, öhm in German; eh, ehm, øh, hm in Norwegian. A sound that asks or
exclaims ("¿eh?", "eh?", "¡Eh, tú!") stays.
Correction to 71acb4f3's message: on the 25 Spanish sentences, 14 (not 13)
came out cleaned and 11 (not 12) stayed unchanged; nothing else differs.
Verified: SpokenCleanupRulesTests (with "Mmm, creo que sí, eh, pero..." ->
"Creo que sí, pero...", "No te olvides, ¿eh?" and "Padre e hijo." kept,
Norwegian "Eh," and German "Äh," removed, German "eh egal" kept) and
CleanupTaggerTests pass on an iPhone 18 Pro simulator (iOS 27.2).
* iOS: choose per dictation whether Translate what I say pastes translated
A Spanish-speaking TestFlight tester liked Translate what I say but always
speaks Spanish and only sometimes wants the English. Until now translation
was all or nothing, set in Experiments.
- DictationTranslationSettings.pastesTranslation (App Group key
experiments.translateDictation.pastesTranslation, default true, so
existing users still get translations) is a switch that keeps the last
choice. translateIfEnabled skips translation while it says as spoken, in
the app (warm session, handoff, Action Button) and in in-keyboard
dictation alike.
- TranslationPasteChoice names both sides in English, like the rest of
Hex's copy: "Paste in English" / "Paste in Spanish" ("Paste as spoken"
when the dictation language is unknown, "Norwegian" without "Bokmål").
It exists only while the experiment is on, the keyboard has Full Access,
and you dictate in a language other than the target.
- Full keyboard: the Hex key menu gets two rows under Language, each with a
checkmark, and the space bar reads "Español → English" while dictations
paste translated.
- Dictation bar: the language tray gets a "Paste in English" chip, lit
while translating, and the language label reads "Español → English". The
label now has layout priority so it isn't cut to "Español → E…".
- Settings: a "Paste in English" toggle under Insert in mirrors the switch.
Turning the experiment on resets it to translated.
- Simulator fixtures: --translate-to <code> and --paste-as-spoken.
Verified: new TranslationPasteSwitchTests (titles and visibility, the
remembered choice, as spoken skipping translation, the model's label, and
the Hex key menu's rows, their checkmarks and their callback). Full
xcodebuild test: 230 tests, 0 failures, 1 skipped (AppleSpeechClipTests).
Rendered on an iPhone 18 Pro simulator (iOS 27.2): full-keyboard-style in
light and dark, translated and as spoken, without translation (unchanged),
full-keyboard-idle, keyboard-language, keyboard-warm, keyboard-recording and
experiments. The simulator can't actually translate (Apple's
LanguageAvailability says es -> en is unsupported there), so a translated
paste is unverified on device.
|
||
|
|
9d26cbdb87 |
iOS: fix six TestFlight tester issues (audio error, leaked error text, Live Activity, haptics, capitals, push to talk) (#2035)
* iOS: retry '!int' audio activation mixable and show audio errors as sentences
TestFlight testers saw "The operation couldn't be completed. (OSStatus error
560557684.)" in the keyboard strip and the app's setup card. 560557684 is
'!int' (AVAudioSession cannotInterruptOthers): a non-mixable session (the
Bluetooth-microphone .record category) going active from the background while
another app plays or records.
- AudioBackend: when setActive(true) fails with '!int', set .playAndRecord
with .mixWithOthers (keeping HFP when Bluetooth is allowed) and activate
once more, so the warm keepalive and captures start alongside the other
app's audio instead of failing.
- New UserFacingError (HEXShared, app and keyboard): AVAudioSession codes map
to short sentences ("Another app is using audio. Try again."); other
microphone errors get "Hex couldn't start the microphone. Try again.";
transcription and model errors keep their description only when it reads
as a sentence, never Foundation's "couldn't be completed (… error N)" or a
Swift type name.
- DictationController: every error it shows (phase .failed and the
keyboard's message) goes through UserFacingError instead of
localizedDescription.
Verified: xcodebuild test on the iPhone 18 Pro simulator, 179 tests pass,
including new tests for the code-to-sentence mapping and for a warm session
whose keepalive throws '!int' reaching the app and keyboard snapshot as
"Another app is using audio. Try again.". The background '!int' retry itself
needs a physical iPhone (music playing, Bluetooth mic allowed, warm session).
* iOS: end empty dictations quietly and keep error text human in the keyboard
Tapping Done (or ✕) in the keyboard's dictation screen without speaking
showed "Transcription failed: HEXKeyboard.(unknown context at $1…)": the
in-keyboard path printed the error's domain (a private Swift type name),
code and description for Apple Speech's "no words" result.
- New EmptyCapture rule (HEXShared): a capture under 0.3 s (or none at all)
is a tap, and an engine's no-speech error or a cancellation means nothing
was said; both end quietly. Engines say so through NoSpeechError
(Parakeet's and Apple Speech's emptyTranscript, the keyboard's
KeyboardSpeech.Failure.empty).
- KeyboardLocalDictation: Done before the microphone came up, or a capture
too short to hold a word, returns to idle without transcribing (and a late
"started" from WebKit stops the capture instead of opening a file);
no-speech ends quietly; a transcription error from a cancelled or
superseded job is dropped; every other failure shows a sentence from
UserFacingError or the keyboard's own copy, while the probe log keeps the
domain and code. KeyboardSpeech.Failure's descriptions are now sentences.
- DictationController (warm session and app): an engine's no-speech error
returns to Ready with no message instead of failing.
Verified: xcodebuild test on the iPhone 18 Pro simulator after rebasing onto
origin/dev (
|
||
|
|
ee34feb49b |
iOS: dictate in every language Apple Speech has on device
A Norwegian TestFlight tester's iPhone silently fell back to English:
Hex picked the first preferred language SpeechTranscriber supports, and
SpeechTranscriber has no Norwegian, Danish, Swedish or Hebrew. Testers
also asked for Spanish and German.
What changed:
- SpeechLanguagePicker picks the dictation language: an explicit choice
wins (SpeechTranscriber where it has the language, else
DictationTranscriber); Automatic takes the first preferred iPhone
language either module has, SpeechTranscriber first for the same
language, then the region's locale, then en-US. Nynorsk tries Bokmal.
- AppleSpeechModule wraps SpeechTranscriber and DictationTranscriber
(preset longDictation) for the same SpeechAnalyzer flow and asset
download; the app's AppleSpeechTranscriber and in-keyboard dictation
both use it and AppleSpeechLanguages, so they resolve the same language
and share downloaded assets.
- Settings > Dictation language (iOS 26): Automatic plus every locale
either module supports, by name, saved in the App Group
(settings.dictationLanguage). Picking one prepares its model now, or
after the dictation in flight. The setup card's Speech model row names
the real language ("Norwegian Bokmal · Apple").
- English-only steps follow the language: the cleanup tagger skips other
languages; Apple Intelligence cleanup runs only when the model supports
the language; filler trimming is per language (English "er" was eating
Norwegian "er" = "is"; Portuguese "um" is a word); translation's source
is the dictation language instead of a guess.
- Replaced: AppleSpeechTranscriber.preferredLocale()/isAvailable with
preferredLanguage()/shouldUse(_:). An iPhone without SpeechTranscriber
still dictates English with Parakeet, as before.
Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2),
189 tests, 0 failures, including new picker, language-change, filler and
tagger tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO
succeeds. The simulator can't transcribe (SpeechTranscriber unavailable,
DictationTranscriber "Model not available"); the same Swift files built
for macOS 26 transcribed say clips word for word: de-DE and es-ES via
SpeechTranscriber, nb-NO and he-IL via DictationTranscriber (no
punctuation from DictationTranscriber under any preset). Not tried on a
physical iPhone.
|
||
|
|
1980a8b38c | feat(ios): polish iPhone keyboard dictation, warm sessions, and Live Activity (#574) |