mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-21 17:56:51 +08:00
dev
77
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
415cbf6373 |
iOS: keep key haptics while dictating, and slide left on delete to delete words (#2080)
* iOS keyboard: key haptics are the same in every layout, and stay on while the keyboard records A TestFlight tester on build 37 felt no key haptics with the Portuguese layout added in #2069. The haptic path was traced for every layout (en, nb, es, de, it, fr, pt) and none of it depends on the language. Every key going down calls FullKeyboardView.keyFeedback through KeyboardKeyPlaneView.onKeyDown, accent and trackpad steps call selectionFeedback, and both are gated only by KeyboardViewController.allowsKeyHaptics (Full Access and Settings > Keyboard > Haptics). Layouts only change the letter rows, accent lists and the spelling dictionary. So the cause is the tester's setup or state, not the layout. Two things silence the taps with no code path to see them: - Without Full Access iOS plays no keyboard haptics. Since #2067 the keyboard types without Full Access, so a tester can type in any layout while Haptics shows On and feel nothing. - In-keyboard dictation sets this process's audio session to record (WebKit captures for it), and iOS mutes a recording session's haptics and system sounds by default. The keyboard now calls setAllowHapticsAndSystemSoundsDuringRecording(true) once in viewDidLoad. Verified on an iOS 27.2 iPhone 17 Pro simulator (Mac Studio): the full HEXMobileTests suite passes (278 tests, 0 failures, 1 skipped as before). The new testEveryLayoutAsksForTheSameKeyHaptics drives a letter, space, delete and an accent step through the real FullKeyboardView in each of the seven layouts and gets the same haptic requests in every one. A live Hex keyboard (Full Access granted) with Settings > Keyboard layout set to Portuguese showed Portuguese accents and the "olá" suggestion. A temporary log line, not committed, showed the haptic call firing with haptics allowed for each key and accent step. The Taptic output and the recording-session mute need a physical iPhone. * iOS keyboard: hold delete and slide left to delete word by word A TestFlight tester asked for the delete swipe from Apple's keyboard. - New DeleteWordSwipe (HEXShared/FullKeyboard): once a finger that went down on delete is a key width (layout.letterWidth) left of where it started, each further key width deletes one more word. Sliding back right types back the words this gesture deleted, newest first. Lifting, or a cancelled touch, keeps the result. - Words are counted as Option-Delete does: spaces and punctuation before the cursor go with the word in front of them ("one two, three. four" loses "four", "three. ", "two, "), apostrophes stay inside words ("don't"), and text that is only spaces and punctuation goes in one step. - KeyboardKeyPlaneView starts a DeleteWordSwipe with each delete press, feeds it moves, and stops hold-to-repeat once the swipe is active. A tap, a still hold, or a hold that drifts less than a key width behaves as before. - Each word deleted or restored ticks the selection haptic (gated by allowsHaptics, as the accent and trackpad ticks are). Edits go through the text proxy only (deleteBackward and insertText) and show in typing logs as delete_word / insert with src delete_swipe(_restore). Nothing new in Settings. Not done: Learn how I type's journal isn't told about words a swipe erases. Verified on an iOS 27.2 iPhone 17 Pro simulator (Mac Studio): the full HEXMobileTests suite passes (278 tests, 0 failures, 1 skipped as before), including new FullKeyboardEngineTests cases for step-by-step deletion, a tick per word, restore on slide back, commit on lift, the start of the text, and the word rule. In a live Hex keyboard in a text field ("Running 10 min late, "), driven with AXe, the swipe went to 1, 2 and 3 words deleted, sliding back restored two, and lifting kept "Running 10 ". A 1 s hold with no slide and a hold with an 18 pt drift only repeated, as before. |
||
|
|
65a2e04e3d |
iOS: let Translate what I say insert in the language you usually dictate in (#2078)
Insert in hid the current dictation language, so someone who dictates in English could never pick English as the target and translate Italian into it. The list now has every language Apple translates; dictations already in the target paste as spoken, and the row says to switch the language from the Hex key instead of showing Apple's can't-translate status. Verified: Release generic-iOS build. |
||
|
|
c5400a4686 |
iOS: Russian and Serbian (Latin) full-keyboard layouts (#2075)
A TestFlight tester (build 35) asked for Russian and Serbian (Latin) layouts in
Settings › Keyboard › Keyboard layout. KeyboardLanguage gets russian ("ru") and
serbianLatin ("sr-Latn") cases, the way #2069 added Italian, French and Portuguese.
Russian is Apple's ЙЦУКЕН: 11, 11 and 9 letters, ё on a held е and ъ on a held ь.
The long-press lists are Apple's own, read from the iOS 27.2 simulator's
TextInput_ru bundle, base letter first (Apple's combining-macron forms а̄ е̄ о̄ с̄ are
left out). Autocorrect and suggestions use UITextChecker's ru_RU dictionary.
Serbian (Latin) is Apple's QWERTZ option for it. Apple's iPhone Serbian Latin
keyboard has no keys of its own for č ć đ š ž (TextInput_sr lists QWERTY, QWERTZ and
AZERTY; only the iPad's Croatian layouts give them keys), so they are held on c, d,
s and z with Apple's lists. iOS 27.2 has no Serbian (or Croatian) UITextChecker
dictionary: suggestions and autocorrect stay off for it, never English. The
dictionary candidates are Latin-only (sr_Latn), since a Cyrillic sr dictionary
would reject every Latin word.
Automatic picks Russian for ru-* and Serbian (Latin) for sr-Latn-* dictation
languages, iPhone keyboards or preferred languages. Bare sr / sr-RS (Cyrillic)
still gets English: Hex has no Cyrillic Serbian layout. Spatial autocorrect has no
ru or sr-Latn model, so neither gets a decoder (same rule as #2069).
KeyboardLayout: Russian's nine-letter row 3 overlapped the 10-key-sized shift and
delete by about 9 pt. Where a letter row wouldn't fit, shift and delete now narrow
to fit (to about a letter's width, as on Apple's Russian keyboard); every existing
layout computes the same modW, so their frames are unchanged. Hit-testing is
untouched. DictationLanguageSwitch.nativeName names a language in its own script
("sr-Latn-RS" is "Srpski", not "Српски"); the space bar still names the dictation
language ("Русский" for ru-RU).
Snapshot scenes get Russian and Serbian drafts (--keyboard-layout ru|sr-Latn,
--accent-key е|ь|c|s).
Verified on an iOS 27.2 iPhone 17 Pro simulator: full HEXMobileTests suite, 277
tests, 0 failures, 1 skipped as before (one new Russian autocorrect case was
adjusted and re-run on its own after the full run). New cases in
KeyboardLayoutTests, AutocorrectEngineTests, FullKeyboardEngineTests and
KeyboardBridgeTests. Live in Messages with the Hex keyboard: Automatic with ru-RU
dictation shows the Russian layout and "Русский" on the space bar, "минкт"
suggests "минут", and holding е typed "ещё". Picking Srpski in Settings, holding c
and s typed "čekaj šta" with an empty strip. Compared with Apple's Russian and
Serbian Latin keyboards on the same simulator: same rows, letter-sized
shift/delete on Russian, and the same е, ь and c menus.
|
||
|
|
2e7fc060ed |
iOS: let the keyboard dictate once Hex has the mic, even if it opened before the grant (#2071)
TestFlight (build 32, iOS 27.2.1): onboarding's practice field said the microphone wasn't allowed after it had been, until Skip for now. Cause (reproduced on an iOS 27.2 simulator): a keyboard process keeps the microphone permission it read when it started. AVAudioApplication, AVAudioSession and AVCaptureDevice all still read "undetermined" in it after Hex is allowed the mic, so a Hex keyboard opened before the grant (tried in another app first, then onboarding) hits KeyboardLocalDictation's guard and shows "Allow the microphone in Hex first" on the Try the keyboard page. The app and the try page had the right state; only the keyboard's own reading was stale. What changed - KeyboardBridge.appMicrophoneAllowed (App Group key app.microphoneAllowed): the app's own reading, written by KeyboardSetupMonitor.refresh (every second on onboarding and home, on becoming active, and after the mic prompt), only when it changes. - KeyboardLocalDictation.start accepts the mic when its own reading or the app's says granted; WebKit's capture still checks, and a NotAllowedError still shows the same message. The probe log notes app=granted|no. - Nothing removed. Onboarding is unchanged: #2058's no Allow button before the prompt, and Continue after a decline, stay as they are. #2058 made this easier to hit, not the cause: build 32 needed the mic for Continue, so it took a keyboard opened earlier; now declining leads to the try page with the keyboard open, and allowing it later in Settings left that keyboard stale too. Verified (anomastudio, iPhone 18 Pro Max simulator, iOS 27.2, fresh install, mic reset, Full Access via TCC): Hex keyboard tried in Spotlight first, then onboarding with the mic allowed at the system prompt. Before: same keyboard pid, mic=undetermined, "Allow the microphone in Hex first" on the try page. After: same pid, mic=undetermined app=granted, web capture started in the practice field. xcodebuild test: 267 tests, 0 failures, 1 skipped. Not verified: a physical iPhone (the simulator's WebKit captures without a TCC check and has no speech assets), or a keyboard that started with the mic denied and was allowed in Settings. |
||
|
|
9b53afcd4c |
iOS: emoji panel no longer blurs on a real keyboard; taps and holds behave (#2070)
* iOS: no blurred emoji on a real keyboard (hide the panel's scroll edge effects) A TestFlight tester (build 32, iPhone19,2, iOS 27.2.1) found the emoji panel "pretty much unusable": the whole suggestion row was blurred smudges, and so were the grid's section names. Cause: iOS 26's automatic scroll edge effect. Inside a keyboard, where the panel sits on the system's glass, every scroll view gets a soft blur along its top, bottom, left and right edges. On the 44 pt suggestion row (and the search results row) the top and bottom effects meet and blur everything; on the grid they blur the section names along the top and the columns at either side while scrolling. The app's --snap-state scenes host the keyboard view outside a keyboard, where iOS adds no effect, so they never showed it. There is no visual effect view or layer filter in the panel. What changed - UIScrollView.hideEdgeEffects() hides all four edge effects (iOS 26+); the grid and EmojiRowView (suggestions and search results) call it. Nothing draws over them, so they need none. - A hairline (palette.separator) between the suggestion row and the grid. - FullKeyboardEngineTests: opens the panel and its search and checks every scroll view hides its four edge effects; it fails on the previous code (the 345x44 pt row). Verified on the Studio's iPhone 18 Pro simulator (iOS 27.2) in a live hosted keyboard (the real HEXKeyboard.appex in the app's hosted-keyboard conversation, driven with AXe taps): before, the row, the section names, the left column after a swipe and the search results were blurred, light and dark; after, all sharp. xcodebuild test: 267 tests, 1 skipped, 0 failures. * iOS: emoji panel clicks on a pick, not on every touch, and a hold types on release Found driving the panel in a live hosted keyboard on the simulator next to the system emoji keyboard. What changed - The grid played the key click and haptic when a cell lit up, which is also how every swipe through the grid starts (cells light on touch-down so taps feel immediate), so scrolling clicked. The click now comes with the pick (didSelect); the key pop-up still shows on touch-down and goes when the scroll starts. - Holding an emoji without skin tones typed it as soon as the hold registered (0.4 s), before the finger lifted, so sliding away couldn't take it back. It now behaves as a slow tap: its pop-up shows while held, and it's typed only if the finger lifts over it. Emoji with skin tones still open the tone picker. Verified in the live keyboard (AXe touch down, hold 1 s, lift): released over ❤️ typed it; released 200 pt away typed nothing; held 👍 opened the tones with the remembered one selected and lifting on a darker tone typed and kept it. Three taps 0.15 s apart typed the three emoji under them; the category bar jumped to Food and drink; a held delete cleared three emoji; search typed "piz" with the letter keys and its first result went into the field; return went back to the panel and ABC to the letters. A tap during a fling's deceleration only stops the scroll, as in the system emoji keyboard. xcodebuild test: 267 tests, 1 skipped, 0 failures. |
||
|
|
9a06976d0e |
iOS: Italian, French and Portuguese layouts that autocorrect in their own language; spoken @ and email addresses (#2069)
* iOS: Italian, French and Portuguese full-keyboard layouts that autocorrect in their own language A TestFlight tester on build 31 with an Italian iPhone reported that Hex kept autocorrecting their Italian into English. KeyboardLanguage only knew en, nb, es and de, so Automatic mapped an it-IT dictation language (and any other language without a layout) to English: English letters, and UITextChecker's en_US dictionary rewriting Italian words. Adds italian/french/portuguese cases. They type on the English QWERTY rows (Apple's Italian and Portuguese keyboards are QWERTY; Hex has no AZERTY yet, so French is QWERTY too), with their own long-press accent order (Italian à è ì ò ù, é right after è), their UITextChecker dictionary (it, fr, pt_BR then pt_PT), and an Italian common two-letter list. Automatic now resolves it-*, fr-* and pt-* to these; the dictation language still wins over the iPhone's keyboards, and Hebrew or Russian still get English. Automatic deliberately ignores Hex's own input mode: its primaryLanguage is the Info.plist's en-US whatever you type. Spatial autocorrect (#2059) only has en and es models; the new layouts get no model and fall back to the dictionary engine in their own language (the decoder also refuses a model of another language). The space bar already names the dictation language ("Italiano"). Snapshot scenes get Italian, French and Portuguese drafts (--dictation-language it-IT, --keyboard-layout it|fr|pt). Verified on an iOS 27.2 iPhone 17 Pro simulator: full HEXMobileTests suite (272 tests, 0 failures, 1 skipped as before). New cases in KeyboardLayoutTests, AutocorrectEngineTests and FullKeyboardEngineTests. Measured on 12 Italian sentences (19 one-slip typos, 59 correct words) with the real dictionary: English fallback fixed 4, miscorrected 4 and broke 19 correct words (Domani->Domain, stai->stay, Sono->Soon, molto->moto); Italian fixed 17, miscorrected 0 and broke none. Renders of the Italian keyboard (typing with the "ritardo" suggestion, the E and I accent menus, "Italiano" on the space bar) checked by eye. * iOS: spoken "at the rate" and "at sign" become "@", joining a spoken email address An Indian English TestFlight tester (build 32) said "at the rate", India's name for @, and got the words. Apple's on-device models don't fix it: checked on macOS 27.2 with Hex's settings and an en_IN voice, SpeechTranscriber (Hex's default) writes "My email is John at the rategmail.com." and "Send it to Priya at signyahoo.com." (gluing the domain to the word before), and DictationTranscriber writes "at the rate gmail.com", though it already writes "priya@yahoo.com" for "at sign" and "john@gmail.com" for "john at gmail dot com". SpokenCleanupRules.spokenSymbols (English only) turns "at the rate" and "at sign" into "@". Before a domain ("gmail dot com", "gmail.com", "co dot in", or glued as above), the word before (with any "dot"-joined parts) and the domain join into one lowercase address, "dot" becoming "." only there: "John at the rate Gmail dot com." becomes "john@gmail.com.". "at the rate" counts only before a domain, so "at the rate of 5%" and "at the rate we planned" stay, and "at" alone is never touched; "at sign" elsewhere is a lone "@" (but not in "at sign language"). Only the matched words change. It runs from CleanupTagger.cleanIfEnabled, which both the app's and in-keyboard dictation call: in every style but Verbatim, with Clean up what I say on or off (the experiment's guard moved below it; the tagger itself still only runs when on). TranscriptFormatter: Polished capitalized after every full stop, which would have written "john@gmail.Com."; a sentence now starts only after ".", "!" or "?" and a space, an address keeps its case (Polished and Notes), and a lone address gets no full stop. Verified on an iOS 27.2 simulator: full HEXMobileTests suite (272 tests, 0 failures, 1 skipped as before), with new cases in TranscriptCleanupTests (Apple's real outputs included) and CleanupTaggerTests. |
||
|
|
1930e8d7ac | iOS: split DictationController so dev passes the size check (#2066) | ||
|
|
b6024b2001 |
iOS: full keyboard by default, typing without Full Access, and a logging mark (#2067)
* iOS keyboard: the full keyboard by default, typing without Full Access
App Review 4.4.1 needs a keyboard extension to type, and to keep working
without Full Access and without network. Hex's default was the dictation
bar (space, delete, return, Dictate; no letter keys), and without Full
Access it showed only setup steps.
What changed
- FullKeyboardSettings.isEnabled (App Group key experiments.fullKeyboard,
what Settings › Keyboard's Full / Dictation bar row binds) now reads
unset as true: new installs and testers who never touched the old
experiment get the full keyboard; an explicit false (turned off, or
Dictation bar picked) keeps the dictation bar. --keyboard
full|dictation-bar picks it in simulator scenes.
- Without Full Access the keyboard always mounts the full keyboard,
whatever is chosen (the bar types only spaces and line breaks).
- Removed: the full keyboard's permanent "Allow Full Access to dictate
with Hex" strip notice; the strip suggests and corrects as usual.
- Without Full Access the Hex key (tap, hold, VoiceOver, or sliding for
its menu) no longer tries to dictate or open Hex: the strip shows
"Allow Full Access to dictate" and "Settings › Apps › Hex › Keyboards"
for 8 s (until a key is typed), with an Open Hex chip, the only thing
that opens Hex (hex-dictation://keyboard/setup) without Full Access.
- In-keyboard dictation (onLocalDictateTap) is offered only with Full
Access; before, it was set without it too, so the Hex key would have
started a capture that can't work there.
- Scene full-keyboard-no-access-hint; README: the default, behaviour
without Full Access, memory numbers and proof limits.
Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release, keyboard
in a real hosted field, a temporary uncommitted probe):
- Fresh install, Full Access off: full keyboard, no in-keyboard
dictation; taps through the engine typed "thw " -> "the ", "tk " ->
"to "; with Spatial autocorrect on (decoder active) also "ofthw " ->
"of the ". The hint renders light and dark.
- Full Access on: full keyboard by default; --keyboard dictation-bar
mounts the bar.
- Memory, 3 runs each, MB, before (
|
||
|
|
6d2035eca2 |
iOS: rebuild the audio engine before the warm-mic tap so a route change can't crash it (#2064)
TestFlight crash on build 32: an Objective-C exception from installTap in SystemAudioManager.startKeepalive after toggling Always use iPhone microphone and Keep mic warm. Changing the session can move the hardware sample rate, and the reused AVAudioEngine kept its old input format. startKeepalive now builds a fresh engine after configuring the session and taps with the hardware's input format, throwing (already handled by callers) if it isn't valid. Verified a Release generic-iOS build; the exception path needs a device route change. |
||
|
|
be49c35f2b |
iOS: privacy and support links in the app, Continue before the mic prompt, typing logs consent (#2063)
* iOS Privacy: Privacy policy and Support links, for App Review 5.1.1(i) App Review requires a privacy policy link inside the app. The Privacy page now has a card with two rows between Learn how I type and Advanced: - Privacy policy opens https://dev.anoma.ly/hex/privacy. - Support opens https://dev.anoma.ly/hex/iphone. Both open in Safari inside Hex (SFSafariViewController, SafariSheet). The URLs come from one base in HexLinks.swift (HexLinks.site). Moving to https://anoma.ly/hex is a one-line change once the site's production deploy serves those pages; https://anoma.ly/hex/privacy returns 404 today, while both dev.anoma.ly pages return 200. README's Privacy page entry says so. Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked at the settings-privacy render. A temporary, uncommitted hook opened the Privacy policy row's Safari sheet, which loaded the live dev.anoma.ly page. * iOS setup sheet: Continue, not Allow, before the microphone prompt App Review 5.1.1(iv) rejects a custom button that says "Allow" right before iOS's own permission prompt. The setup sheet's Microphone row had one. It now matches onboarding's Set up Hex page (#2058): the row says why ("For dictation", replacing "Required") and its button says Continue. Tapping Continue still calls requestMicrophone, so iOS asks right after. Allowed and Off are unchanged, and Off still shows Open Settings. New scene: --snap-state setup-sheet-microphone, the setup sheet before iOS has asked for the microphone. README's Setup and scene notes are updated. Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked at the setup-sheet-microphone render (For dictation, Continue) and the setup-sheet render (Allowed, no button). * iOS Privacy: Record typing logs says what it records and asks before turning on App Review 2.5.14 requires explicit consent before recording what people type. - The description now says what the feature records: "Records taps and the text you type for 14 days, on this iPhone, to export. Off by default." It replaces "Off by default. Stays on this iPhone to export" and fits on two lines. - Turning it on opens an alert before anything changes. Title: "Record typing logs?". Message: "Hex's keyboard will record each tap and the text you type or dictate with it, including the text around the cursor, for 14 days. The logs stay on this iPhone for you to export or delete. Password, code and card fields are never recorded." Turn On switches it on; Cancel leaves it off. Turning it off needs no dialog. The switch still writes only TypingLogSettings.isEnabled. - Every claim matches docs/typing-logs.md: touches on Hex's keys, the text each key produced, the text around the cursor, dictations, 14 days in the App Group, export and delete from Typing logs, and a "skipped" line only for password, one-time code and card fields. New scene: --snap-state settings-privacy-confirm opens the dialog (SettingsSnapshotFixture.confirmsTypingLogs). README and docs/typing-logs.md now describe the consent and the new path (Privacy › Advanced, no longer Experiments). Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked at the settings-privacy, settings-privacy-confirm and settings-privacy-on renders. * iOS home: Privacy policy and Support join Acknowledgements #2059 added an Acknowledgements row at the bottom of home's utility card. The privacy policy and support links read better beside it than on the Privacy page. Home's last card now has Privacy policy (hand.raised, ↗), Support (questionmark.circle, ↗) and Acknowledgements (doc.text, ›). Keyboard settings and Run setup again keep their own card above it. - Moved, not duplicated: the Privacy page's links card is removed. That page is back to "Everything stays on this iPhone", Learn how I type, and Advanced. - Links still come from HexLinks (dev.anoma.ly until production serves anoma.ly/hex/privacy) and open in Safari inside Hex. - README's Settings section now lists the links on home. Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked at the settings-home render, which still fits one screen without scrolling, and the settings-privacy render. |
||
|
|
c14df5a4f0 |
iOS: spatial autocorrect experiment with Google Books models, plus accent and two-letter fixes (#2059)
* iOS keyboard: Spatial autocorrect experiment (tap + bigram decoder), off by default
What: a noisy-channel word decoder (TypingDecoder) behind Settings ›
Experiments › Spatial autocorrect (experiments.spatialAutocorrect, off
by default; it needs the full keyboard with Autocorrect on). When on, for a layout whose
model is bundled (English, Spanish), it replaces UITextChecker's one-edit
guesses on space and in the strip:
- each tap is scored against every letter with a 2D Gaussian around the
key (sigma 0.29 x 0.26 of the key pitch), moved by Learn how I type's
average tap offsets for letters with 20+ taps;
- words come from a beam search over a memory-mapped, sorted word list
with substitutions, one insertion, deletion or transposition, free
apostrophes ("dont" -> "don't") and accented letters read from their
base key (é from e, +1 nat);
- a bigram language model on the word before (stupid backoff) weighs
them, and a missed space is tried by splitting the taps in two (or
treating a bottom-row tap as the space);
- what was typed is replaced only when the best reading beats it by 2
nats, or 7 when it is itself a word the model, UITextChecker, learned
words or the shorthand list know.
Text replacements, learned fixes, the English contraction/typo table and
words you kept still come first. FullKeyboardEngine now records each
letter's tap point for the word being typed (cleared on anything else).
scripts/build-typing-model.py writes the model format (v2: per-language
alphabet, sorted words, u8 costs, sorted bigram keys); models are kept
out of git (.gitignore). The English prototype model is built from
SymSpell's frequency lists (MIT; Google Books Ngram CC BY 3.0 and SCOWL),
the Spanish one from Tatoeba sentence counts (CC BY 2.0 FR). Typing logs
now record the setting.
Why: Record typing logs show short words and non-words left uncorrected
("tk" for "to"), missed spaces with a typo inside, and o/i taps landing
low on k/j. The UITextChecker path skips 1-2 letter words, never uses
the word before, and can't split a word that also has a substitution.
Verified (anomastudio, iPhone 18 Pro simulator, iOS 27.2): benchmark
harness (kept outside the repo) typing every tap through
FullKeyboardEngine. Synthetic held-out English, 13,870 words: output
word error rate 15.0% today vs 6.7% (6.3% with tap offsets); typos fixed
58% vs 83%; correct words wrongly changed 89 vs 97 of 9,203. Labelled
real typing logs, 172 words: 14.5% vs 4.7%. Spanish held-out, 13,301
words: 26.8% vs 8.1% (Spanish model and test from the same corpus, so
optimistic). Per letter p50 1.0 ms, p95 2.5 ms; mapping the model adds
no measurable phys_footprint. Full HEXMobileTests suite: 221 tests,
0 failures (1 skipped, as before); Release generic-iOS build with
CODE_SIGNING_ALLOWED=NO succeeds with both models bundled. No new tests.
* iOS keyboard: Spatial autocorrect models from Google Books Ngram v3
What: the English and Spanish models (HEXKeyboard/TypingModel/en.hextm,
es.hextm, 3.6 MB each) are now built from Google Books Ngram v3 (CC BY
3.0) word and word-pair counts summed over 1980-2019, keeping only
words SCOWL (sizes up to 70, lowercase, ASCII) or RLA-ES (LibreOffice's
es_ES and es_MX dictionaries, MPL 1.1) accept, and are committed.
- English: 86,133 words, 417,480 word pairs; Spanish: 110,000 words,
326,366 pairs. Pairs are the ones that tell most over the single-word
model (count x log of P(b|a) over 0.4 P(b)), as many as fit.
- Format v3: pairs grouped by first word (u32 row starts, u32 second
word, u8 cost), 5 bytes a pair instead of 9, so ~70% more pairs fit
in the same size. TypingDecoderModel reads v3 only.
- scripts/build-typing-model.py downloads, counts (each ngram file
deleted once counted, every stage cached for a resumable rerun) and
writes a model: python build-typing-model.py <en|es> <workdir> <out>.
- Tuning for the new models (dev sentences): English leans more on the
word before (lmWeight 1.5; Parameters.tuned(for:)); an accent-only
change ("esta" -> "está") needs a 2-nat margin instead of 7 (ñ for n
isn't one: "ano"/"año"); an accented letter also reads from its base
key when it has its own (ñ from n), whichever is cheaper.
- Settings › Acknowledgements (new row and sheet, AcknowledgementsView)
and ios/README.md credit Google Books Ngram (CC BY 3.0), SCOWL with its
and WordNet's notices, and RLA-ES (MPL 1.1).
- AutocorrectEngineTests: decoder ranking cases on the bundled models
("need t^o" -> "need to", a missed space with a slip -> "see how",
correct words kept, Spanish "aqui manana" -> "aquí mañana", "esta"
kept).
Why: Ryan's licence decision: CC BY with credit, no share-alike, no
non-commercial or GPL/LGPL data, and not SymSpell's word-pair list
(unverified provenance). The Spanish model was built from Tatoeba,
the same corpus as its test set.
Removed: the SymSpell- and Tatoeba-built prototype models and the v2
format (pairs as sorted u64 keys); models are no longer gitignored.
Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2; benchmark
harness outside the repo; word error rate on words that aren't names):
English held-out test, 13,870 words: today's engine 13.0%, decoder
3.6% (3.4% with tap offsets), correct words broken 4 -> 2 of 7,716;
labelled real typing logs, 172 words: 14.5% -> 4.1%; the shared
example sentence decodes exactly; Spanish held-out Tatoeba test (no
source shared with the model), 13,301 words: 25.7% -> 8.0%, broken
16 -> 20 of 7,246. Per letter p50 1.0 ms, p95 2.6 ms.
AutocorrectEngineTests 19 passed.
* iOS keyboard: Spatial autocorrect knows your own words
What: the decoder also searches TypingDecoder.UserWords, a small sorted
in-memory list of your words, beside the model: Contacts names and
other UILexicon terms, one-word text replacement outputs, words you
kept twice (LearnedWordsStore) and typo fixes Learn how I type learned.
Each scores as a moderately common word (userWordCost 11 nats) and
comes back as you spell it ("Zorbix", not "zorbix"); typed exactly,
one counts as a word, so it needs the larger margin to change.
AutocorrectEngine rebuilds the list only when a source changes
(revision bumped by loadLexiconEntries, applyLearning, suppressAndKeep)
or the model changes. The search is now generic over TypingDecoderWords
(the model or your words); the model's own narrow moved to it.
Why: a name, brand or word you use that the model lacks could only be
left alone, never reached from a typo.
Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2):
AutocorrectEngineTests 20 passed, including a new case: with "Zorbix"
in the lexicon, a slip "zorbiz" becomes "Zorbix" and "zorbix" typed
right stays. Benchmark unchanged (it has no user words).
* iOS keyboard: an accent or tilde is the cheapest autocorrect fix
What: AutocorrectEngine.singleEditKeyboardCost now scores a one-letter
substitution that only adds or changes an accent, diaeresis or tilde
(e/é, u/ü, n/ñ) at 0.15, below every other edit, instead of 1.15 (the
cost of two far-apart keys). New helper sameLetterIgnoringAccents.
Valid words are still never rewritten, so "esta" stays "esta".
Why: Spanish typed without accents was corrected into the wrong word,
because the accented guess cost more than any insertion, deletion or
neighbouring key: "aqui" -> "caqui", "podia" -> "odia", "manana" ->
"manaba". Now they become "aquí", "podía", "mañana".
Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2, the
benchmark harness kept outside the repo, today's engine with Spatial
autocorrect off): Spanish held-out Tatoeba test, 13,301 words, output
word error rate 26.8% -> 24.7%, words changed into another wrong word
1,015 -> 743, correct words broken 16 -> 16 of 7,246 (names excluded).
English test (13,870 words) unchanged at 15.0%.
* iOS keyboard: fix two-letter typos toward very common two-letter words
What: cautiousReplacement fixes a two-letter non-word (UITextChecker
doesn't know it, not ALL CAPS, no apostrophe) when one neighbouring key
or one swap turns it into a very common two-letter word of the layout's
language (KeyboardLanguage.commonTwoLetterWords, commonest first: "tk"
-> "to", "ot" -> "to", Spanish "nl" -> "no", "ee" -> "de"). English and
Spanish have lists; Norwegian and German have none yet and keep
leaving two-letter words alone. The step numbering in the comments
moves by one.
Why: autocorrect skipped every word under three letters, and the
commonest words are two letters: typing logs show "tk" for "to" left
uncorrected again and again (the o key's taps land low, on k).
Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2, benchmark
harness outside the repo, today's engine): English held-out test,
13,870 words, output word error rate 15.0% -> 13.2%; labelled real
typing logs, 172 words, 14.5% -> 10.5%; Spanish test 24.7% -> 22.9%;
correct words broken unchanged (4 of 7,716 English, 16 of 7,246
Spanish, names excluded).
|
||
|
|
9cf4d8b427 |
iOS: plain permission prompts and no pre-prompt Allow button, for App Review (#2058)
* iOS: plain purpose strings for the microphone and speech recognition
App Review 5.1.1(ii) wants purpose strings that clearly and completely say why
Hex needs access. The old speech string named an API ("SpeechTranscriber")
instead of saying what happens, and the microphone string didn't say what
becomes of the audio. Both now say what Hex does with it and that it stays on
the iPhone:
- Microphone: "Hex turns what you say into text on this iPhone. Audio never
leaves your iPhone and is deleted once it's transcribed."
- Speech recognition: "Hex turns your recordings into text with Apple's
on-device speech recognition. Nothing you say is sent to a server."
Same strings in the app and the keyboard extension (project.yml and the
generated Info.plists).
Verified: xcodegen regenerates identical Info.plists; Release build for
generic/platform=iOS succeeds and HEX.app/Info.plist carries the new strings.
* iOS onboarding: no Allow button before the microphone prompt, and no dead end when it's declined
Two App Review 5.1.1(iv) rejection patterns on the Set up Hex page:
1. A custom screen that leads into a permission prompt may not use "Allow"
(or similar) on the button that triggers it; Apple asks for "Continue" or
"Next". The Microphone row's small Allow button is gone. The row now says
why ("So Hex can hear you dictate"), and iOS asks when the person taps
Continue, or the Keyboard row's Open Settings (unchanged: it asks first so
Microphone is listed on Hex's Settings page). Denied still shows Open
Settings.
2. Continue needed the microphone granted, so anyone who declined it could
never get into Hex, though the keyboard types without it. Continue now
needs only the keyboard (added or unverifiable). Declining leads on to Try
the keyboard, and the app's setup card keeps showing the microphone until
it's allowed.
README's first-run section updated to match.
Verified: Release build for generic/platform=iOS succeeds. Not yet rendered
in a simulator.
|
||
|
|
71491f93f1 |
iOS: stamp App Store builds with their own version from an ios-beta-appstore-* tag (#2057)
Ryan wants the repo and TestFlight to stay on 0.2.0 while the App Store gets 0.2.1. ci_post_clone.sh now rewrites MARKETING_VERSION from a tag ios-beta-appstore-<version> before xcodegen; other builds are untouched. Verified sh -n and the sed against a copy of project.yml (all three targets become 0.2.1). |
||
|
|
1e353d53b0 |
iOS: settings as a short home screen with Dictation, Keyboard, Privacy and Experiments pages (#2054)
* iOS settings prototype 1: short home screen with separate pages
Ryan found the home screen cluttered: a large Keyboard setup checklist sat at
the top even when setup was nearly done, real features were filed under
Experiments, and every setting shared one long scroll. This prototype keeps
every setting and its behaviour, and changes only layout and copy.
Home now fits one screen: the Hex header, one status line ("Hex is ready", or
"Finish setup · 1 step ›"), the Dictate bar, four rows (Dictation, Keyboard,
Privacy, Experiments), then Keyboard settings ↗ and Run setup again.
- Replaced: KeyboardSetupCard with SetupStatusRow plus SetupSheet, a sheet
sized to its content. It has the same four checks, the Full Access check
field and Open Settings. Full Access that hasn't been confirmed reads "Not
confirmed yet" with a dotted circle, not as a failure. The keyboard's Open
Hex setup link opens the sheet.
- Removed: ExperimentsCard, split into pages (SettingsPages.swift):
Dictation has style, language, translate, clean up, Apple Intelligence
clean-up, custom vocabulary, keep mic warm and iPhone microphone. Keyboard
has full keyboard, then layout, autocorrect and key pop-ups only while it
is on, haptics, and the test field that was on home. Privacy has
"Everything stays on this iPhone" and Learn how I type. Experiments has
Emoji, with Clear emoji history while it's on, plus Record typing logs and
its viewer.
- Emoji used to show only while the full keyboard was on. It now shows
either way and reads "Needs the full keyboard" while it's off, because it
no longer sits under that switch.
- Haptics and Learn how I type stay visible with the full keyboard off,
because the dictation bar uses them too (KeyboardViewController).
- Copy is cut to one line per row. "Insert in" is now a Menu in the app's
muted style instead of the system picker, with the same choices and writes.
- Removed the practice field's autofocus for hosted-keyboard scenes. It was
dead code, because HEXMobileApp sends those scenes to
HostedKeyboardConversationStage.
- New --snap-state scenes: settings-home, setup-sheet, and
settings-<page>[-on][-end]. These write the page's switches (Emoji
included) to the App Group, so each render is deterministic.
Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2) on the
Mac Studio, rebased onto origin/dev with the emoji panel.
* iOS settings: typing logs under Privacy › Advanced, AI clean-up in Experiments
Ryan asked for Experiments to hold only real experiments.
- Moved: Record typing logs and its Typing logs viewer, from Experiments to a
small "Advanced" section at the bottom of Privacy (a new SettingsSectionHeader
in the old Experiments header style). It stays in App Store builds.
- Moved: Clean up with Apple Intelligence, from Dictation to Experiments. The
code still calls it an experiment (AppleIntelligenceCleanupSettings, key
experiments.appleIntelligenceCleanup). It keeps its disabled state and
reason line when Apple Intelligence can't run.
- Experiments now has Emoji (with Clear emoji history) and Clean up with
Apple Intelligence.
Nothing else changed: same settings keys, same reads and writes.
Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2), and I
looked at settings-privacy-on and settings-experiments renders.
* iOS home: status line reads "Ready" under the Hex header
"Hex is ready" repeated the header's "Hex" on the line just below it. Now it
reads "Ready" with the same green check. VoiceOver still says "Hex is ready",
and the setup sheet's title keeps "Hex is ready" because the sheet has no
Hex header.
Verified: the settings-home render on an iPhone 18 Pro simulator (iOS 27.2).
* iOS settings: Keyboard is a choice between Full and Dictation bar
The full keyboard is becoming the default and the dictation bar an option
(Ryan). On the Keyboard page, the "Full keyboard" toggle is replaced by a
"Keyboard" row with two segments: Full first, then Dictation bar, with no
Beta tag. Layout, autocorrect and key pop-ups show only while Full is chosen.
Haptics stays visible either way, because the dictation bar uses it too.
The choice still writes FullKeyboardSettings.isEnabled (Full = true). The
stored default and its name are changing on ios-full-keyboard-default, which
has no commits yet. This page binds to the setting, so it follows that change.
Verified: the build passes, and I looked at settings-keyboard (Dictation bar
chosen) and settings-keyboard-on (Full chosen) on an iPhone 18 Pro simulator
(iOS 27.2).
* iOS settings: plain, accurate descriptions for App Store review
Three descriptions now say only what the code does:
- Record typing logs (Privacy › Advanced): "Off by default. Stays on this
iPhone to export". It's off unless turned on (TypingLogSettings), the files
stay in the App Group, and they leave only through the share sheet.
- Learn how I type: "Learns from taps and the words you fix", replacing
"Taps and fixes, never what you write". The journal does keep word pairs
(a word you fixed, with at most one word of context), so "never what you
write" claimed too much.
- Clean up what I say: "Drops ums, repeats and corrected words", replacing
"...and corrections". It drops the words a spoken correction replaces, not
the correction itself.
Verified: I looked at the settings-privacy, settings-privacy-on and
settings-dictation renders on an iPhone 18 Pro simulator (iOS 27.2). Each
description fits on one line.
* iOS settings scenes: settings-*-on seeds one emoji in the history
With an empty history, Clear emoji history renders disabled. That's
accurate, but in a review screenshot it reads as broken. The -on scenes now
record one emoji (EmojiUsageStore.record), so the row shows enabled. The
plain scenes clear the history. This runs only under --snap-state settings-*.
Verified: I looked at the settings-experiments-on render on an iPhone 18 Pro
simulator (iOS 27.2).
* iOS README: describe the settings home and its pages
- New "Settings" section: home (the status line, Dictate, the four rows,
Keyboard settings, Run setup again), what each page holds, and the
--snap-state scenes that render them.
- Setup now describes the status line and setup sheet, which replaced the
setup card.
- Every "Experiments › …" path that moved now names its page: Dictation
(clean up, custom vocabulary, translate, dictation language), Keyboard
(Full, haptics, layout), Privacy (Learn how I type, and Advanced › Record
typing logs and Typing logs), Experiments (Emoji, Clean up with Apple
Intelligence).
Docs only.
|
||
|
|
6b0a07fafd |
iOS: keep the internal version at 0.2.0 (#2053)
Ryan wants the repo and TestFlight to stay on 0.2.0, which beta review approved; the App Store submission gets its own 0.2.1 build. Reverts #2052's MARKETING_VERSION 1.0 (iPhone only and the privacy manifests stay). Verified project.yml: MARKETING_VERSION 0.2.0 for all three targets, TARGETED_DEVICE_FAMILY 1 unchanged. |
||
|
|
78e4458d4c |
iOS: version 1.0, iPhone only, privacy manifests (#2052)
* iOS: add privacy manifests for the app and keyboard extension Neither HEX.app nor HEXKeyboard.appex shipped a PrivacyInfo.xcprivacy, which App Store submission needs for required-reason APIs. Both now declare no tracking, no tracking domains, no collected data, and the APIs Hex's own code uses: UserDefaults (CA92.1 app-only, 1C8F.1 App Group), file timestamps (C617.1, typing journal and typing log files in the App Group), system boot time (35F9.1, CACurrentMediaTime for elapsed time), and active keyboards (3EC4.1 custom keyboard, 54BD.1 layout choice via UITextInputMode.activeInputModes). Verified: plutil -lint passes; xcodegen picks both up as resources and a Debug simulator build places them at HEX.app/PrivacyInfo.xcprivacy and HEX.app/PlugIns/HEXKeyboard.appex/PrivacyInfo.xcprivacy. * iOS: version 1.0 for the App Store, iPhone only Ryan's App Store calls: the first release is 1.0 (the 0.2.0 TestFlight line; App Store Connect's version must equal CFBundleShortVersionString), and Hex Mobile is iPhone only for now. MARKETING_VERSION 0.2.0 -> 1.0 and TARGETED_DEVICE_FAMILY 1 for the app, keyboard and widgets; README notes both. Verified a Release generic-iOS build: Info.plists read 1.0 and UIDeviceFamily [1], with both privacy manifests bundled. |
||
|
|
cc75290551 |
iOS: emoji panel with on-device suggestions (Experiments › Emoji) (#2051)
* iOS: emoji index built from Unicode's emoji list and CLDR
What changed
- scripts/build-emoji-data.py builds HEXShared/FullKeyboard/Emoji/emoji-index-<lang>.txt
for the four keyboard layouts (en, nb, es, de) from Unicode emoji-test.txt 18.0
(pinned by SHA-256) and CLDR 49 annotations and annotationsDerived (release-49 tag),
plus Hex's own chat words in scripts/emoji-aliases-en.txt ("congrats", "bday",
"lol", "new year"; 336 words, written for Hex).
- Each file holds the catalog in panel order (8 categories, 1,923 emoji, 332 with five
uniform skin tones), each emoji's English name, its folded search words, and the
word and phrase postings already weighted (one-word name 1.0, name word 0.75,
keyword 0.6, keyword-phrase word 0.3, chat word 1.2 down to 0.9, an IDF factor for
words many emoji share), sorted by key so the keyboard can binary-search the file
in place. Non-English layouts carry English's words at 0.85.
- LICENSE-Unicode.txt (Unicode License v3) ships beside the data, as the licence
asks: both emoji-test.txt and CLDR's data are under it, which allows commercial
use with the notice.
Why
- The emoji panel (next commits) suggests emoji for what you type and searches them
on the iPhone. Precomputing the index keeps the keyboard extension's heap small:
the first version built Swift dictionaries at runtime and held 5.7 MB, which
malloc kept after release; the mapped file plus offsets measures 0.7 MB.
Verified
- The generator runs from a sources folder or downloads the pinned files; output is
deterministic. Files: en 351 KB, nb 560 KB, es 549 KB, de 610 KB.
* iOS: rank emoji for the text before the cursor, on device
What changed
- EmojiLibrary maps an emoji-index file (Data .alwaysMapped) and keeps only the emoji,
their skin tones and byte offsets: word and phrase lookups binary-search the file's
sorted keys and parse postings in place; search scans each emoji's folded name and
keyword words for prefixes (name words first, a short name the query fills before a
long one it touches, then your use and everyday emoji). Emoji newer than 15.1 are
listed only if the emoji font itself draws them as one glyph.
- EmojiSuggester reads the last eight content words (the newest counts 1, then 0.85
each word back), their forms (plurals, -ing, -ed, held-down letters: "lmaooo"),
two- to four-word phrases ("new year", "good luck") and the word still being typed;
an emoji several words point at gains a third of the rest. One emoji per family
("man running", "woman running"); your use breaks ties and fills the row, then
everyday defaults. forFinishedWord gives the strip at most three strong matches for
the word just finished ("pizza " -> 🍕).
- EmojiUsageStore keeps, in the App Group, a use count per emoji that halves every 10
days (frequency and recency in one score, at most 120 emoji) and the skin tone last
picked for each; no words. clear() empties both.
- EmojiRoutedTarget lets the keys type into the emoji search's query instead of the
field while you search.
- EmojiSemanticIndex (app target only, for the comparison test): Apple's on-device
NLEmbedding word vectors over the index's 3,257 words, standing in for a context word
with no emoji of its own. Not used by the keyboard (see Measured).
- EmojiSuggesterTests: index, folding, context, families, endings, other layouts,
strip, search, history decay and tones, the search's routing, and quality on 60
hand-written sentences (Fixtures/emoji-context-sentences.json, 30 to tune, 30 held
out) with and without the embedding.
Measured (sensible emoji in the top 5, context only; top-1 in brackets)
- Simulator, iPhone 17, iOS 27.2: keywords dev 30/30 (27), held-out 30/30 (27), 0.04 ms
a sentence; with the embedding the same counts at 0.14 ms, its index built in 120 ms.
- Mac harness, same code: CLDR keywords alone dev 26/30 (13), held-out 22/30 (14);
with Hex's chat words 30/30 (27) and 30/30 (27). A blind set of 40 messages and
sensible emoji written by a separate agent that saw no code or word lists: CLDR
alone 30/40 (13), with chat words 37/40 (28), adding the embedding 37/40 (28); 38/40
after the held-down-letter rule, which those failures suggested. Apple's sentence
embedding scored 18/40 (7) and took 7.5 s to embed the 1,923 descriptions.
- Memory on the Mac: the mapped index adds 0.72 MB; the embedding adds 5.3 MB (1 MB
NLEmbedding, 3.9 MB vectors) and none of it came back after release. No measurable
gain for that, so the keyboard uses keywords and your history only.
Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures.
* iOS: emoji panel for the full keyboard (Experiments › Emoji)
What changed
- Experiments › Full keyboard › Emoji (experiments.emoji, off by default; shown with the
full keyboard): an emoji key (face.smiling) right of 123, after the globe, in plain
text fields (not URL, email or ASCII-only ones). The bottom row takes the compact
sizes it uses beside the globe. "Clear emoji history" empties EmojiUsageStore.
- EmojiPanelView replaces the strip and the keys: a row of up to ten suggestions for
the text before the cursor and your history (one emoji per family, your skin tone),
a search button; a grid that scrolls sideways through Frequently used (your history,
then everyday emoji, 16 at least) and the eight categories in four rows of 42 pt
(three of 38 in landscape), each section's name kept at the left edge while it
scrolls; a bar with ABC, the categories and a repeating delete, drawn like the
letter keys' 123 and delete. Holding an emoji with skin tones opens the accent
picker with its five tones; the pick is typed and remembered for that emoji. Key
pop-ups and haptics follow their switches.
- Search: the letter keys type into a search field in the strip (EmojiRoutedTarget),
results above it in a row the keyboard grows 44 pt for, your suggestions while it's
empty; return, the field's close button or the emoji key go back to the panel.
Typing logs and Learn how I type see the emoji typed into the field, never the query.
- The strip, when it would be empty after a space, offers the finished word's strong
emoji ("pizza " -> 🍕, "birthday " -> 🎂 🎉 🥳), a lone one in the centre slot; never
over word suggestions, Undo dictation, a status or the clipboard chips.
- Grid cells drop their text (and bitmap) when they scroll off and prefetching is off:
scrolling every section had kept 353 cell bitmaps, 35 MB.
- FullKeyboardView+Emoji.swift holds the wiring; the engine's target is now the routed
target, which forwards everything to the field unless a search is open.
- Scenes: full-keyboard-emoji, -emoji-search (--emoji-query), -emoji-tones,
-emoji-food, -emoji-strip (each -dark and -dark-field), and -emoji-probe, which prints
the panel's footprint and open latency.
Measured in the keyboard extension (Release, iPhone 17 simulator, iOS 27.2, a field
focused with the real HEXKeyboard.appex and a temporary, uncommitted probe; the
simulator counts no GPU memory and reports no headroom), footprint:
keyboard with the emoji key, nothing loaded 19.0 MB
after "pizza " (the strip loads the index) 19.0 MB
panel open 28.7 MB
picked three suggestions, closed 22.7 MB
every section scrolled 41.6 MB (peak 48.4; 71.1 before the cell fix)
closed after that 36.5 MB
The 17.5 MB left after browsing everything is about 351 emoji glyph images the system's
text rendering caches process-wide (18.7 MB in the heap, 351 CGImages); drawing with a
font instance the panel owns or at 21 pt left it the same, and no public API frees it.
Whether iOS purges it under memory pressure is unverified on an iPhone.
Opening: 22-26 ms from the emoji key to the panel laid out the first time, 12-14 ms
after (median of 10), about 21 ms until committed.
Verified: xcodebuild test (245 tests, 1 skipped, 0 failures); every scene above in
light, dark and dark field looked at on the simulator.
* iOS: document the emoji experiment and keep its blind sentences as a check
What changed
- README: the emoji experiment under Capabilities (what the panel, search, skin tones
and strip do, what the history keeps, where the index comes from and its licence),
how to rebuild the index, and its proof limits with the quality, memory and latency
measurements and what is unverified on an iPhone (whether iOS purges the cached
emoji glyph images that browsing every section leaves, about 18 MB).
- emoji-context-sentences.json gains the 40-message blind set a separate agent wrote
without seeing code or word lists; EmojiSuggesterTests scores it beside the other
60 and fails below 35/40.
Why: the blind set is the least biased quality number; keeping it catches a ranking
change that helps the hand-written sentences at its expense.
Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures; EMOJI_QUALITY: dev 30/30 (top-1 27), held-out 30/30 (27), blind 38/40 (29),
the same with the embedding, which built in 101 ms.
|
||
|
|
81a36817b4 |
iOS keyboard: follow the keyboard's live traits for light and dark, not the proxy's appearance (#2050)
TestFlight build 29: in dark mode, switching to another app and back left the
full keyboard's keys in the light palette (white keys, black labels) over the
system's dark platter.
Cause: the palette took textDocumentProxy.keyboardAppearance first, and
updateKeyboardTraits() overrode the controller's interface style from it. The
host resolves a default field's appearance to .light or .dark when it sends
its text traits and doesn't send them again when only its style changes, so
the value goes stale. Reproduced on an iOS 27.2 simulator (iPhone 16 Pro Max,
Contacts' New Contact field): with the keyboard up, go home, the system
switches to light, return (the proxy now says .light), the system switches
back to dark. The keyboard window's traits went dark, and the platter with
them, but the palette stayed on the stale .light ("applyAppearance view=dark
win=dark appearance=light -> light"). In light mode the reverse also
happened: a stale .dark set overrideUserInterfaceStyle = .dark, which kept
both the full keyboard and the dictation bar dark on a light platter. On a
real iPhone the stale value most likely comes from the app switcher's light
and dark snapshots of the host. That device trigger is inferred; the
simulator reproduces the stale-value mechanism.
Change: FullKeyboardPalette.isDark(traits:) follows the trait collection
alone, and the controller no longer sets overrideUserInterfaceStyle.
Removed: the proxy's .light/.dark no longer decides the palette, and the
#904 override is gone. #904's behaviour stays: iOS itself gives the keyboard
window dark traits for a field that asks for a dark keyboard in a light app
(checked with --hosted-field dark-field: full keyboard and dictation bar
both dark).
Verified on the iOS 27.2 simulator with before and after screenshots: dark
mode round trip (dark keys after the fix), light mode round trip (light
keys), the dictation bar both ways, the dark field in a light app, and a
plain home and return. xcodebuild test: 247 tests, 0 failures, 1 skipped.
|
||
|
|
178c16fdc5 |
iOS: speech model fallbacks for iPhone 11 and unavailable languages; no raw errors in the keyboard (#2048)
* iOS: fall back to DictationTranscriber when a speech model won't install, and never show Apple's error text A TestFlight tester (iPhone 15, iOS 26.2, build 25, Russian) saw the full keyboard's strip read "transcription.ru asset not found after attempted download." Apple Speech lists some languages whose asset never arrives (Apple's forum shows the same for Arabic), and Hex both stopped there and showed Apple's sentence: UserFacingError (#2035) kept any description without an error code in it, and this one has none. - SpeechModelInstall (HEXShared, used by the app and in-keyboard dictation): install the chosen module's model, then the same language on DictationTranscriber when SpeechTranscriber's won't install, and go straight to DictationTranscriber for three days. When neither installs while online, Automatic sets the language aside for three days and moves to the next one ("Russian isn't available on this iPhone yet. Using English."); a language picked by hand says so and offers that one ("Try English", with Use English in the app). Offline or stalled, it says the download failed and keeps the language. Every message fits the full keyboard's strip. - StallWatch: a download whose progress hasn't moved for 60 s is cancelled and treated as failed, instead of holding the progress bar forever. - AppleSpeechAssets.reserve makes room under maximumReservedLocales by releasing other reservations first, and a reservation Hex can't get no longer fails preparation: the keyboard reserved every language it used and never released one, so the app's reserve could throw for good. - UserFacingError is now an allowlist: only Hex's own errors (HexDescribedError) show their text, a lost connection while getting the model ready says so, and anything else gets the context's sentence. This replaces the marker filter, which let Apple's readable sentences through. - DictationController re-prepares in the language Automatic moved to (at most twice), keeps a notice, and offers a language after an explicit choice fails. The dictation bar shows a failure on two smaller lines. - Snapshot scenes: full-keyboard-language-unavailable[-dark], keyboard-language-unavailable, language-unavailable, language-switched. Verified: xcodebuild build-for-testing on an iPhone 15 / iOS 27.2 simulator, and the 44 DictationController, picker, SpeechModelInstall and StallWatch tests pass (fallback, set-aside, offline, unsupported, stall, explicit offer, Automatic switch, error text). Apple Speech itself can't run in the simulator, so the install steps are fakes there; which module fails for Russian on iOS 26.2 is unverified on hardware (no iOS 26.x simulator runtime). * iOS: dictate English with Apple on iPhone 11, and stop the Parakeet relaunch loop A TestFlight tester (iPhone 11, iOS 27.3, build 16) reported the model download sticking at 50% and the app then crashing at every launch, with no crash log. iPhone 11 (A13, 4 GB) has no SpeechTranscriber (SpeechTranscriber.isAvailable is false on every A13 iPhone), so Hex used Parakeet V2 for English. FluidAudio 0.15.5 reports downloads as 0-50% and Core ML's compile as 50-100%, so 50% is MLModel(contentsOf:) compiling the encoder, which can't be cancelled; iOS ends the app there. Once the files are on disk, init saw hasCachedModel and loaded Parakeet again at once, so every launch ended the same way. iOS ending an app for memory leaves no crash report, which fits the empty crash list. - AppleSpeechTranscriber.shouldUse: English without SpeechTranscriber uses DictationTranscriber where Parakeet doesn't fit (under 5 GiB of RAM: every A13 iPhone) on iOS 26+. Parakeet stays where it fits and before iOS 26. - ParakeetLoadGuard: a load still marked as running at launch belonged to a process that ended, so the next launches don't load Parakeet by themselves until a load works; Hex says it closed while loading the model and waits for Set up. If Automatic now picks Apple Speech, it prepares that instead. - Parakeet's download half runs under StallWatch (90 s without progress ends it); the compile half isn't watched since it can't be stopped. FluidAudio's download errors (Hugging Face paths and HTML in their text) become "Parakeet V2 didn't download. Check your connection and try again." Verified: xcodebuild test on an iPhone 15 / iOS 27.2 simulator, 247 tests, 0 failures, 1 skipped (AppleSpeechClipTests, needs clips), including new tests for shouldUse on a 4 GB iPhone and the load guard across launches. Parakeet's memory use, the compile at 50% and the ended launches need a real iPhone 11; the simulator neither limits memory nor runs SpeechTranscriber. |
||
|
|
8b8b6fc0ef |
iOS onboarding: say 100% on device on the welcome page (#2047)
A tester said it isn't clear that Hex runs on device, its main feature. The welcome page now shows a '100% on device' pill under the Hex mark, and the second point reads 'Free, and nothing leaves your iPhone' (was 'Free and private, on this iPhone'). Verified HEXMobileTests on the Mac Studio and a simulator render of onboarding-welcome. |
||
|
|
1ede7bb792 |
iOS: Spanish clean-up, paste translated or as spoken, screen stays on while recording (#2046)
* iOS: keep the screen on while Hex records or transcribes
A Spanish-speaking TestFlight tester (build 20) asked that the screen not
turn off while recording. DictationController now sets
UIApplication.isIdleTimerDisabled whenever its phase is recording or
transcribing, and clears it when the phase leaves those states (done,
cancel, failure), so Auto-Lock comes back after every dictation. Nothing
else in Hex sets the flag.
Limits: iOS honors the flag only while Hex is the foreground app (the
handoff screen and in-app recording). A warm-session or in-keyboard
dictation happens with another app in front, and a keyboard extension has
no public API to keep the screen on, so those can still dim. No private API
or value(forKey:) workaround is used.
Verified: new DictationControllerTests.testRecordingAndTranscribingKeepTheScreenOn
(recording, transcribing, done, cancel, failed) and the rest of
DictationControllerTests pass on an iPhone 18 Pro simulator (iOS 27.2).
* iOS: Spanish, German and Norwegian rules for Clean up what I say
A Spanish-speaking TestFlight tester said the corrections part works badly
in Spanish. Clean up what I say ran its cleanup tagger only for English and
returned every other language untouched, so with the experiment on a Spanish
dictation kept its fillers and spoken corrections. The tagger stays
English-only: run directly on Spanish, it changed the meaning of 8 of 18
sentences, including dropped negations ("Te digo que no puedo ir." became
"Te puedo ir.", "O sea que al final no vienes." became "O sea que vienes.").
- New SpokenCleanupRules: for es, de and nb/nn/no, a filler between commas
goes ("este", "o sea"), and a correction marker set off by commas ("digo",
"perdón", "no", "no, espera", "mejor dicho"; "nein", "ich meine"; "nei",
"nei vent", "jeg mener") drops the words it replaces. Those are counted
back to the nearest earlier word that matches the repair's first word, or
is of its kind (number, weekday, month, today or tomorrow), within the
sentence and 8 words; with neither, nothing changes. "mejor dicho", "no
mejor", "besser gesagt" and "rettere sagt" also count without commas.
Like the tagger, it only removes words; CleanupReconstruction repairs
commas and capitals.
- CleanupTagger.cleanIfEnabled runs those rules for non-English languages
instead of returning the text unchanged. The model still never loads for
them, and Verbatim still skips cleanup.
- TranscriptFormatter: the styles trim Spanish hesitations (eh, em, mmm;
never "e") and German ones (äh, ähm, öhm; never "eh"), and capitalize a
sentence that opens with ¿ or ¡.
Measured on 25 Spanish sentences I wrote, with Polished and the experiment
on (9 German and Norwegian ones too). Before, every sentence came out as
spoken. After, all 13 with fillers or corrections are cleaned ("Eh, nos
vemos mañana a las 3, digo, a las 4." -> "Nos vemos mañana a las 4.") and
all 12 that use the same words in their ordinary meaning stay unchanged
("Te digo que no puedo ir.", "Quiero este, el azul.", "Lo siento, perdón,
Ana.").
Verified: new SpokenCleanupRulesTests, updated
CleanupTaggerTests.testOtherLanguagesGetRulesNeverTheEnglishTagger (it
used to expect Norwegian "tirsdag, nei vent, torsdag" untouched), plus the
CleanupTagger, DictationController and TranscriptCleanup suites, all
passing on an iPhone 18 Pro simulator (iOS 27.2).
* iOS: Clean up what I say drops Spanish, German and Norwegian hesitations
In-keyboard dictation (iOS 26+) inserts Apple Speech's text without running
a style through TranscriptFormatter, so the Spanish and German hesitation
trim added to the styles in 71acb4f3 never reached it. The English tagger
removes "um" and "uh" in both places. SpokenCleanupRules now also drops the
language's hesitation sounds anywhere in the sentence, using the styles'
own list (TranscriptFormatter.fillerPattern): eh, em, mmm in Spanish; äh,
ähm, öhm in German; eh, ehm, øh, hm in Norwegian. A sound that asks or
exclaims ("¿eh?", "eh?", "¡Eh, tú!") stays.
Correction to 71acb4f3's message: on the 25 Spanish sentences, 14 (not 13)
came out cleaned and 11 (not 12) stayed unchanged; nothing else differs.
Verified: SpokenCleanupRulesTests (with "Mmm, creo que sí, eh, pero..." ->
"Creo que sí, pero...", "No te olvides, ¿eh?" and "Padre e hijo." kept,
Norwegian "Eh," and German "Äh," removed, German "eh egal" kept) and
CleanupTaggerTests pass on an iPhone 18 Pro simulator (iOS 27.2).
* iOS: choose per dictation whether Translate what I say pastes translated
A Spanish-speaking TestFlight tester liked Translate what I say but always
speaks Spanish and only sometimes wants the English. Until now translation
was all or nothing, set in Experiments.
- DictationTranslationSettings.pastesTranslation (App Group key
experiments.translateDictation.pastesTranslation, default true, so
existing users still get translations) is a switch that keeps the last
choice. translateIfEnabled skips translation while it says as spoken, in
the app (warm session, handoff, Action Button) and in in-keyboard
dictation alike.
- TranslationPasteChoice names both sides in English, like the rest of
Hex's copy: "Paste in English" / "Paste in Spanish" ("Paste as spoken"
when the dictation language is unknown, "Norwegian" without "Bokmål").
It exists only while the experiment is on, the keyboard has Full Access,
and you dictate in a language other than the target.
- Full keyboard: the Hex key menu gets two rows under Language, each with a
checkmark, and the space bar reads "Español → English" while dictations
paste translated.
- Dictation bar: the language tray gets a "Paste in English" chip, lit
while translating, and the language label reads "Español → English". The
label now has layout priority so it isn't cut to "Español → E…".
- Settings: a "Paste in English" toggle under Insert in mirrors the switch.
Turning the experiment on resets it to translated.
- Simulator fixtures: --translate-to <code> and --paste-as-spoken.
Verified: new TranslationPasteSwitchTests (titles and visibility, the
remembered choice, as spoken skipping translation, the model's label, and
the Hex key menu's rows, their checkmarks and their callback). Full
xcodebuild test: 230 tests, 0 failures, 1 skipped (AppleSpeechClipTests).
Rendered on an iPhone 18 Pro simulator (iOS 27.2): full-keyboard-style in
light and dark, translated and as spoken, without translation (unchanged),
full-keyboard-idle, keyboard-language, keyboard-warm, keyboard-recording and
experiments. The simulator can't actually translate (Apple's
LanguageAvailability says es -> en is unsupported there), so a translated
paste is unverified on device.
|
||
|
|
6ab53fff1e |
iOS keyboard: open Hex instead of being killed when short of memory (#2045)
* iOS: open Hex instead of dictating in the keyboard when it is short of memory Testers on 4 GB iPhones (12, 12 mini, 13) saw in-keyboard dictation in other apps stop about a second after tapping Dictate and iOS switch to the system keyboard: iOS ends a keyboard extension that passes its memory limit (Apple: "The memory limits vary from model to model"; reported at ~60-70 MB by KeyboardKit, ActiveHard 77 MB by fcitx5-ios, 90 MB on a 4 GB iPad by dictus-ios #202). Starting creates the capture web view, loads its page, starts WebKit's capture and mounts the recording screen's Metal scope all at once. What changed - KeyboardLocalDictation.hasMemoryToStart: os_proc_available_memory() must leave 32 MB for the first dictation (web view, page, WebKit's first use, recording screen, Metal device and pipeline) and 16 MB once the web view is loaded (capture and recording screen only). 0 (the simulator) is enough. - KeyboardViewController.syncLocalDictationAvailability, on every bridge poll: while short, viewModel.onLocalDictateTap is nil, so Dictate, the bar's Hex mark and the full keyboard's Hex key take their existing cold start (open Hex, record there, return), as before iOS 26. A hold stays a tap there, so push to talk sends no stray stop. With room, nothing changes. The probe log notes each switch, at most every 5 s. - A web view whose WebContent process ended now also leaves the keyboard's view; before, it stayed behind (2 pt, alpha 0.02) next to the new one each later dictation made. - KeyboardProbeLog.memory() adds the process's peak footprint (ledger_phys_footprint_peak), so a tester's keyboard-capture-probe.log shows what the limit saw: [footprint, peak, headroom]. - README: the guard, and its proof limits. Tried and left out, after measuring - Releasing the capture web view after each dictation (and on memory warnings): the keyboard's footprint stayed (31.9 -> 31.8 MB; WebKit/JSC state and allocator pages stay in the process, malloc_zone_pressure_relief returned 16 KB) while the next start's spike grew (+1.9 vs +0.9 MB) and it waited for the page again (~1.5 s). The web view stays loaded, as before. - Dropping the text checker or lexicon while dictating: the full keyboard rests lower than the dictation bar in the simulator (18.3-21.0 vs 21.4-24.3 MB), so nothing showed it helps. Measured (simulator iPhone 13, iOS 27.2, Release; a temporary, uncommitted probe drove the keyboard's own Dictate and Done paths with a host field focused, logged phys_footprint at each step and every 100 ms, with footprint(1) sampling from outside; 3 runs each, MB): bar before bar after full before full after keyboard shown 21.6-24.2 21.4-24.3 18.5-21.0 18.3-20.9 web view created 25.1-27.7 24.9-27.9 22.1-24.7 22.1-24.7 web view loaded 27.5-30.0 27.5-30.5 25.5-28.7 25.8-28.6 capture started 28.5-30.8 28.1-31.3 26.7-29.6 26.8-29.4 max 3 s after tap 28.8-31.2 28.5-31.7 28.9-31.4 28.9-32.1 stop 27.8-31.8 27.7-31.0 26.0-28.6 26.1-28.6 transcriber made 27.9-31.8 27.8-31.0 26.1-28.7 26.2-28.8 lifetime peak 31.5-32.4 28.9-32.3 29.8-32.6 30.0-32.6 A second dictation adds 0.5-1.3 MB (bar) and 4.2-4.8 MB (full keyboard, which rebuilds its recording panel). With a simulated limit 24 MB above the footprint at the tap, both keyboards now hand off and stay at 21.2 / 18.3 MB instead of rising 7-13 MB; with 40 MB they dictate in the keyboard, twice. The simulator counts no GPU memory and transcription fails there ("Audio format is not supported"); a macOS stand-in put the Metal device and pipeline at ~3.5 MB and the scope's drawables at ~4.4 MB, which the thresholds allow for. Verified: xcodebuild test on the iPhone 18 Pro simulator, 212 tests pass (1 skipped as before); the Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds. Not verified on a physical iPhone: the real limit, GPU memory, and whether 32/16 MB is the right line; the probe log's peak and headroom will show it. * iOS keyboard: need 24 MB, not 16, before a second in-keyboard dictation Leaving the full keyboard's new recording strip briefly peaked 7 MB above recording (35.5 MB in a simulated iPhone 13), and a second dictation now costs up to 8 MB, so 16 MB left little margin. Compile-checked with a generic-iOS build. |
||
|
|
550eccd970 |
iOS onboarding: say 'Allow Full Access' in the Keyboard row itself (#2044)
A tester said onboarding never told them to Allow Full Access; the Settings path was small muted text below the card. The Keyboard row now carries 'Add Hex and Allow Full Access' under its title. Verified HEXMobileTests and a simulator render of onboarding-allow. |
||
|
|
42e6e8e12d |
iOS: remove the return-to-app feature; swipe back after a handoff (#2042)
Ryan decided Hex Mobile won't open other apps' URL schemes after the keyboard hands off to the Hex app: no return-to-app tricks, since they are an App Review risk and only worked for apps with a known scheme. Removed: - HostApp / HostAppCatalog (the curated scheme list) and the bridge's returnAppID / quickReturnApps settings. The stored App Group keys (settings.returnAppID, settings.quickReturnAppIDs) are just ignored; no migration. - KeyboardLaunchRequest's returnAppID and the URL's return= item. An older keyboard that still sends return= parses fine and it is ignored. - DictationController's auto-return: AutoReturnState, the URL opener, setReturnAppID, setQuickReturnApps, returnToHostApp, maybeAutoReturnAfterActivation, triggerAutoReturn, the ~1 s fallback task, noteSceneDidEnterBackground, and the attemptAutoReturn flags. - The app's Settings "Return to" row, its "Edit apps…" sheet (ReturnAppsSheet.swift), and the handoff screen's quick-jump app chips. - The dictation bar's return chip and its app tray, and the full keyboard Hex key menu's "Return to" row and page (with its return_app typing-log event). - The handoff-returning snapshot scene. Replaced by: the handoff screen's existing quiet "Swipe bar to return" cue over the home indicator, now shown whenever it records or a warm session is on (it used to hide while an auto-return was attempted). The rest of the handoff and warm-session flow is unchanged. Docs: ios/README.md (cold start, Hex key menu, proof limits) and ios/docs/typing-logs.md. Verified: xcodebuild test on an iPhone 18 Pro simulator (iOS 27.2): 220 tests, 1 skipped, 0 failures. The handoff test now checks the 10 s window alone, and the launch URL test checks that a legacy return= is ignored. Rendered --snap-state setup, warm, handoff-listening, handoff-no-home-bar, keyboard-cold and full-keyboard-style and checked them: no Return to row, chips or return trigger; the swipe cue shows on the handoff screen. |
||
|
|
deb84b6082 |
iOS: no hyphens from autocorrect, completions ranked by frequency (#2041)
* iOS keyboard: never autocorrect into a hyphen; split a missed space into two words
What: AutocorrectEngine now passes every UITextChecker guess through
usableGuess before autocorrect or the strip uses it. A guess may add no
character you didn't type except letters and a contraction's apostrophe,
so "t.b." for "tb" or "x-y" for "xy" is never applied or shown. A guess
that splits the word in two ("up-to" or "up to" for "upto") is used only
as two words with a space, and only when its letters are exactly the
typed ones (a missed space), both halves are dictionary words (two or
more letters, or English "a"/"I", since UITextChecker accepts every
single letter), and the checker offers the spaced form itself.
Why: Record typing logs showed a missed space corrected into a
hyphenated compound. The old loop skipped spaced guesses but let the
hyphenated one through as a cheap one-character insertion. Removed: that
hyphen path; replaced by the spaced form under the rules above.
Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2) for
AutocorrectEngineTests (4 new tests: missed space, the split rules,
punctuation, and the system dictionary's "nevermind" -> "never mind"),
FullKeyboardEngineTests, TypingLogTests, DictationLearningTests: 40
tests, 0 failures.
* iOS keyboard: rank completions by frequency, names last, no repeated literal
What: the suggestion strip's completions now keep UITextChecker's own
order, which is most probable first (Apple's documented order; on iOS 27
"b" lists but, be, because and "th" lists the, they, that's), instead of
re-sorting it by length and then alphabetically. UILexicon terms
(Contacts names and text replacements) now rank after the dictionary's
completions instead of ahead of everything, and a capitalised name ranks
after every other completion unless the typed word starts with a
capital. A completion equal to what was typed is dropped, one-letter
words get completions too, and when nothing is left the strip shows only
the typed word on the left instead of repeating it in the centre.
Why: Record typing logs showed contact names and job titles from the
lexicon taking the centre slot for common prefixes (a surname over "the"
for "th"), and one-letter words repeating the literal as the primary.
Removed: the lexicon-first pass (up to three terms before any spelling
guess) and the length/alphabetical sort. The frequency source is the
existing UITextChecker; no word list was added.
Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2),
AutocorrectEngineTests: 16 tests, 0 failures (4 new: frequency order
ahead of contacts, names demoted unless capitalised, no repeated
literal, and the system dictionary's order for "b" and "th").
|
||
|
|
f82adb826f |
iOS: drop the recording dot and timer so the Dot Scope fills the recording bar (#2040)
Ryan wanted only the scope and the two buttons in the full keyboard's recording bar. Removed the red dot, the clock label, the palette's recordingDot and the clock subscription that only refreshed the timer; the scope well now starts at the bar's left edge. The dictation-bar keyboard keeps its timer. Verified HEXMobileTests and simulator renders of recording and transcribing in light and dark. |
||
|
|
15965d5625 |
iOS: end the recording bar with round cancel and check buttons, the check becoming the spinner (#2039)
Ryan found the cancel button, the 'Done' text button and the separate transcribing label confusing. The bar now ends in two equal round buttons, ✕ and ✓; while transcribing the ✓ is replaced by a spinner in the same place, ✕ still cancels, and the clock stays a clock. Verified HEXMobileTests and simulator renders of recording and transcribing in light and dark. |
||
|
|
0aa6fb8fea |
iOS: grow the keyboard for a taller recording bar and show one transcribing status (#2038)
Ryan found the strip-height recording bar too thin and the Transcribing label plus 'Inserting…' on Done confusing. While dictating the full keyboard grows 16 pt through preferredHeight and a new onPreferredHeightChange the controller applies; the keys shift down by the lift the view actually has, so they stay put on screen, and the bar gets 60 pt. While transcribing the clock reads 'Transcribing…' and Done shows a spinner. Verified HEXMobileTests and simulator renders of typing, recording and transcribing in light and dark. |
||
|
|
33727933b0 |
iOS: keep dictation in the suggestion strip and leave the keys in place (#2037)
Ryan wanted dictation to use only the top bar, not cover the keyboard. The recording panel now has the strip's frame and lays out the clock, the Dot Scope well, and capsule cancel and Done buttons in one row; only the strip and the Hex mark fade out, so the keys stay visible. The style chip is dropped from the recording bar for room; the Hex key's menu still sets the style. Verified simulator renders of the recording and transcribing states in light and dark, and HEXMobileTests. |
||
|
|
221b6a45fc |
iOS: ask for the microphone before sending setup to Settings (#2036)
A tester found neither Keyboards nor Microphone on Hex's Settings page from onboarding page 2. iOS lists Microphone only after the app has asked, so page 2 puts Microphone first and the Keyboard row's Open Settings asks first when needed. A 'Don't see Keyboards?' link shows the General › Keyboard › Keyboards › Add New Keyboard path for when iOS hasn't listed Keyboards on Hex's page yet. Verified HEXMobileTests and a simulator render of page 2. |
||
|
|
9d26cbdb87 |
iOS: fix six TestFlight tester issues (audio error, leaked error text, Live Activity, haptics, capitals, push to talk) (#2035)
* iOS: retry '!int' audio activation mixable and show audio errors as sentences
TestFlight testers saw "The operation couldn't be completed. (OSStatus error
560557684.)" in the keyboard strip and the app's setup card. 560557684 is
'!int' (AVAudioSession cannotInterruptOthers): a non-mixable session (the
Bluetooth-microphone .record category) going active from the background while
another app plays or records.
- AudioBackend: when setActive(true) fails with '!int', set .playAndRecord
with .mixWithOthers (keeping HFP when Bluetooth is allowed) and activate
once more, so the warm keepalive and captures start alongside the other
app's audio instead of failing.
- New UserFacingError (HEXShared, app and keyboard): AVAudioSession codes map
to short sentences ("Another app is using audio. Try again."); other
microphone errors get "Hex couldn't start the microphone. Try again.";
transcription and model errors keep their description only when it reads
as a sentence, never Foundation's "couldn't be completed (… error N)" or a
Swift type name.
- DictationController: every error it shows (phase .failed and the
keyboard's message) goes through UserFacingError instead of
localizedDescription.
Verified: xcodebuild test on the iPhone 18 Pro simulator, 179 tests pass,
including new tests for the code-to-sentence mapping and for a warm session
whose keepalive throws '!int' reaching the app and keyboard snapshot as
"Another app is using audio. Try again.". The background '!int' retry itself
needs a physical iPhone (music playing, Bluetooth mic allowed, warm session).
* iOS: end empty dictations quietly and keep error text human in the keyboard
Tapping Done (or ✕) in the keyboard's dictation screen without speaking
showed "Transcription failed: HEXKeyboard.(unknown context at $1…)": the
in-keyboard path printed the error's domain (a private Swift type name),
code and description for Apple Speech's "no words" result.
- New EmptyCapture rule (HEXShared): a capture under 0.3 s (or none at all)
is a tap, and an engine's no-speech error or a cancellation means nothing
was said; both end quietly. Engines say so through NoSpeechError
(Parakeet's and Apple Speech's emptyTranscript, the keyboard's
KeyboardSpeech.Failure.empty).
- KeyboardLocalDictation: Done before the microphone came up, or a capture
too short to hold a word, returns to idle without transcribing (and a late
"started" from WebKit stops the capture instead of opening a file);
no-speech ends quietly; a transcription error from a cancelled or
superseded job is dropped; every other failure shows a sentence from
UserFacingError or the keyboard's own copy, while the probe log keeps the
domain and code. KeyboardSpeech.Failure's descriptions are now sentences.
- DictationController (warm session and app): an engine's no-speech error
returns to Ready with no message instead of failing.
Verified: xcodebuild test on the iPhone 18 Pro simulator after rebasing onto
origin/dev (
|
||
|
|
bb5875cea8 |
iOS: Norwegian, Spanish and German full-keyboard layouts
Testers asked for the Norwegian layout and, with it, Spanish and German. The layout follows the dictation language unless set. What changed: - KeyboardLanguage holds each layout's letter rows, long-press accents and spelling dictionary: Norsk (a-ring ending row 1, o-slash and ae row 2), Espanol (n-tilde ending row 2), Deutsch (QWERTZ, u/o/a-umlaut, sharp s on a held S), as Apple lays them out. Adding a language is a new case with those three filled in. - KeyboardLayout takes the language. Rows of 11 keys narrow only the character keys; shift, delete and the bottom row keep their English (10-key) sizes, and row 3's letters sit centred between shift and delete. letterWidth stays the 10-key width; characterWidth is new. - Experiments > Full keyboard > Keyboard layout: Automatic, English, Norsk, Espanol, Deutsch (settings.keyboardLayout). Automatic follows the dictation language (no layout yet for Hebrew and others: English), else the first iPhone keyboard or preferred language with a layout. A pick in the keyboard's language switch changes the layout at once. - Autocorrect uses the layout's UITextChecker dictionary (nb, es, de); without one it offers nothing but your text replacements instead of correcting into English. The English contractions and typo list (i -> I) apply only to English; neighbouring-key costs follow the layout. LanguageSettings.effectiveDictationLanguage now prefers what Automatic picks over the last language dictation ran in. - Typing logs record the layout's language; the snapshot stage takes --keyboard-layout, follows --dictation-language and --accent-key. - TypingLogTests pins English, since FullKeyboardView now reads the App Group's dictation language. Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2): 206 tests, 0 failures, 1 skipped (the Apple Speech clip test), with layout tests for every language at 320-440 pt and landscape (rows, 11-key widths, hit-frame tiling, balloons, accent menus) and autocorrect tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO succeeds; light and dark screenshots of each layout, its accents, shift, the settings row and the dictation bar's re-shot trigger checked by eye. The simulator has dictionaries for nb, es and de. Accent lists are from Apple's layouts as remembered, not checked on a device; the 123 and #+= pages are English's in every language. |
||
|
|
2b225b390b |
iOS: switch the dictation language from the keyboard
Testers asked to change the dictation language without opening Hex.
What changed:
- Dictation bar: a language trigger beside the style ("Norsk") opens a
tray of chips: Auto plus up to five languages.
- Full keyboard: the Hex key menu gets a Language row and page with the
same choices, and the space bar names the dictation language faintly
("English", "Norsk"), like Apple's.
- DictationLanguageSwitch picks the choices (the language set, recent
picks, then the iPhone's preferred languages, each matched to a locale
Apple Speech has; Nynorsk maps to Bokmal) and names each in its own
language, adding a region code only where two share a language.
- LanguageSettings keeps recent picks, what Automatic resolves to, and
Apple's language list (cached when Hex or the keyboard lists it); a
pick in the keyboard writes the same settings.dictationLanguage as
Hex's menu. In-keyboard dictation reads it per dictation; a warm Hex
session follows it in its 100 ms loop and prepares the language
between dictations; a keyboard handoff and the Action Button pick it
up before recording.
- Typing logs record the dictation language setting and a
dictation_language UI event.
Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2):
195 tests, 0 failures, 1 skipped (the Apple Speech clip test, which
needs HEX_SPEECH_CLIPS), including new switch, recents, follow and
space-bar tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO
succeeds; screenshots of the Hex menu's language page (light, dark),
the space bar, the dictation bar's tray and the settings row checked by
eye. The trigger's last change (chevron instead of a globe, which looked
like the next-keyboard key) was compiled, not re-shot. Not tried on a
physical iPhone.
|
||
|
|
ee34feb49b |
iOS: dictate in every language Apple Speech has on device
A Norwegian TestFlight tester's iPhone silently fell back to English:
Hex picked the first preferred language SpeechTranscriber supports, and
SpeechTranscriber has no Norwegian, Danish, Swedish or Hebrew. Testers
also asked for Spanish and German.
What changed:
- SpeechLanguagePicker picks the dictation language: an explicit choice
wins (SpeechTranscriber where it has the language, else
DictationTranscriber); Automatic takes the first preferred iPhone
language either module has, SpeechTranscriber first for the same
language, then the region's locale, then en-US. Nynorsk tries Bokmal.
- AppleSpeechModule wraps SpeechTranscriber and DictationTranscriber
(preset longDictation) for the same SpeechAnalyzer flow and asset
download; the app's AppleSpeechTranscriber and in-keyboard dictation
both use it and AppleSpeechLanguages, so they resolve the same language
and share downloaded assets.
- Settings > Dictation language (iOS 26): Automatic plus every locale
either module supports, by name, saved in the App Group
(settings.dictationLanguage). Picking one prepares its model now, or
after the dictation in flight. The setup card's Speech model row names
the real language ("Norwegian Bokmal · Apple").
- English-only steps follow the language: the cleanup tagger skips other
languages; Apple Intelligence cleanup runs only when the model supports
the language; filler trimming is per language (English "er" was eating
Norwegian "er" = "is"; Portuguese "um" is a word); translation's source
is the dictation language instead of a guess.
- Replaced: AppleSpeechTranscriber.preferredLocale()/isAvailable with
preferredLanguage()/shouldUse(_:). An iPhone without SpeechTranscriber
still dictates English with Parakeet, as before.
Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2),
189 tests, 0 failures, including new picker, language-change, filler and
tagger tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO
succeeds. The simulator can't transcribe (SpeechTranscriber unavailable,
DictationTranscriber "Model not available"); the same Swift files built
for macOS 26 transcribed say clips word for word: de-DE and es-ES via
SpeechTranscriber, nb-NO and he-IL via DictationTranscriber (no
punctuation from DictationTranscriber under any preset). Not tried on a
physical iPhone.
|
||
|
|
d21699b896 |
iOS: load the cleanup model after recording and only with memory to spare in the keyboard (#2033)
A tester's iPhone 13 Pro dropped back to the system keyboard as soon as dictation started with Clean up what I say on: the keyboard loaded the Core ML tagger on the CPU and Neural Engine at the start of recording, alongside the microphone, capture web view and speech recognizer, and iOS ended the extension. The keyboard now loads it when recording stops, runs it CPU-only in the extension (weights stay memory-mapped), and skips cleanup when os_proc_available_memory is under 80 MB. Verified HEXMobileTests on the iPhone 18 Pro simulator; not reproduced on a device. |
||
|
|
c2cae93d61 |
iOS: trim onboarding to the essentials (#2032)
Ryan found page 2 wordy and confusing (Full Access 'Not checked / Checked on the next step', a five-step list, captions and hints). Page 2 is now two rows, Keyboard and Microphone, a one-line Settings path and Continue; the Full Access and speech model rows are removed (Full Access is confirmed on page 3, the model downloads in the background). Pages 1 and 3 lose their detail lines and the waiting status. Verified HEXMobileTests and simulator renders of every onboarding scene. |
||
|
|
15186d0419 |
iOS: add a three-page first-run onboarding (#2031)
TestFlight testers turned on Allow Full Access, came back, and Hex still said Off: iOS reports Full Access only inside the keyboard, so the app can't know until the keyboard runs. The setup card alone didn't explain that. A first run now walks people through it and confirms Full Access in the keyboard itself. - Welcome: Hex mark, one-line pitch, three points, Get started. - Set up Hex: live checklist (Hex Keyboard, Allow Full Access, Microphone with Allow / Open Settings, speech model download) refreshed every second, one Open Settings button with the five steps to take there. Continue needs the keyboard added (or unverifiable) and the microphone; Full Access reads Not checked here and the model keeps downloading. - Try the keyboard: a focused field. Success once the keyboard has reported Full Access (keyboard.fullAccessSeenAt) and text is in the field, then Done. "Full Access looks off" with Open Settings appears when Full Access is still missing 4 s after a keyboard switch or new text (a switch right as the keyboard comes up is ignored). Skip for now enters the app; the existing setup card still shows until setup is complete. - Shows once: onboarding.completed / onboarding.page in the app's UserDefaults; a quit part-way resumes on the same page; a launch that finds the keyboard ready and the microphone allowed (existing users) records completion and skips it. Settings gets a Run setup again row. - Never shows under --snap-state except its own scenes: onboarding-welcome, -allow, -try, -try-done, -try-off, and onboarding-tour, which walks every page by itself for a screen recording. DictationController's fixture reuses the setup case for them (line count unchanged, it sits at its cap). - README: first-run behaviour, scenes, and what is unverified on hardware. Verified: xcodebuild test on the iPhone 18 Pro simulator (177 tests, 0 failures); Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds; simulator screenshots of every scene; a fresh install opens page 1, a saved page resumes without a false Full Access note, and an install with Full Access seen and the microphone granted skips onboarding and stores onboarding.completed = 1; a recording of the tour scene. |
||
|
|
635a6c5a2e |
iOS: add a Haptics switch to the full keyboard (#2030)
A tester asked to turn off typing haptics. Experiments › Full keyboard › Haptics (on by default) now gates the keyboard's allowsHaptics alongside Full Access, so key presses, accent picks and trackpad steps stop tapping when it's off. Typing logs record the setting. Verified HEXMobileTests on the iPhone 18 Pro simulator and a render of the toggle. |
||
|
|
5ce7cf8362 |
iOS: let setup check Full Access from the keyboard instead of showing Off (#2028)
Testers turned on Allow Full Access and the app still said Off: iOS only reports Full Access inside the keyboard, so the app learns it from keyboard.fullAccessSeenAt after the keyboard runs once. The row now says Not checked in that state and the setup card offers a field to open the Hex keyboard in, which records Full Access; the existing 1 s refresh then shows On. Verified HEXMobileTests on the iPhone 18 Pro simulator and a render of the setup scene. |
||
|
|
8c7d01672e |
iOS: add a Key pop-ups switch to the full keyboard (#2027)
A tester asked to turn off the enlarged letter that pops up over a pressed key. Experiments › Full keyboard › Key pop-ups (on by default) now controls it; off, the pressed key darkens in place as with iOS's Character Preview off. Typing logs record the setting as key_popups. Verified HEXMobileTests on the iPhone 18 Pro simulator and simulator renders of a held key with the setting on and off. |
||
|
|
dcc8479cc2 |
ios: record raw typing logs from Hex's keyboard (experiment) (#2024)
What changed - Experiments › Record typing logs (off by default, App Group key experiments.typingLogs). It is separate from Learn how I type, which is unchanged. While it is on, the keyboard writes NDJSON day files to TypingLogs/yyyy-MM-dd.ndjson in the App Group. Events: session_start/session_end (field traits, keyboard size and safe area, orientation, appearance, device, build, locale); layout (every key's frame and hit frame, the strip and its suggestion slots, and a hash, with the key list written once per file); touch (began/moved/ended/cancelled with raw x/y, majorRadius and tolerance, force, UITouch.timestamp, the key the engine resolved, and the key under the point when they differ; moves are throttled to 2 pt and 8 ms); edit (insert, delete, word delete, autocorrect, autocorrect undo, suggestion, double-space period, return and send, paste, undo dictation, trackpad cursor); candidates; context (documentContextBefore/AfterInput capped at 200 characters, after each word, return and dictation and once host changes settle, marked cleared when the host empties the field); ui (accents, trackpad, the Hex key and its menu, strip chips, globe); dictation (raw, learned fixes, spelling, tagger, Apple Intelligence, translation, formatting, inserted text, durations, engine and locale, from in-keyboard and app dictation); and dictation_outcome (from a second DictationEditTracker). The schema is in ios/docs/typing-logs.md. - Password, new password, one-time code, every credit card content type, and secure fields write one skipped line and nothing else. A new field, or new traits, starts a new session. - TypingLogger buffers events on the main thread. TypingLogStore encodes and appends them on a serial utility queue every 2 s, when the keyboard goes away, and on a memory warning. Each day is capped at 64 MB with one cap_reached line, and files are kept 14 days. When the App Group can't be reached (no Full Access), events are dropped silently. Nothing is sent. - Hooks: FullKeyboardEngine.typingLog, and the KeyboardKeyPlaneView touch path, which the UITouch overrides now reach through began/moved/ended/cancelled(id:sample:). Every call is skipped while the logger is nil, so typing is unchanged with the experiment off. KeyboardSnapshot gains optional stages, sent only while the experiment is on. KeyboardSpeech.transcribe also returns its locale. - App: the toggle and a Typing logs screen (each day with its size and event count, Share per day, Share all through the share sheet, Delete all with a confirmation). Also --snap-state experiments and typing-logs scenes, and an --experiment-typing-logs launch argument. Why Ryan wants to export a day of raw typing to his Mac, to test and train prediction models and a per-user touch model (adaptive key targets). Verified - xcodebuild test on the iPhone 18 Pro simulator: 177 tests, 0 failures. This includes 9 new TypingLogTests: line format and escaping; day rotation with layouts in full once per file; Delete all while the writer holds the file open; the daily cap (tested at 4 KB) across two writers; 14-day retention; the skip rule; a skipped session writing one line; and a real FullKeyboardView typing 25 taps through the plane's touch path (autocorrect, delete, double-space period, the 123 layer, return, a long-press é). That log's edits replay to the field's text, "Don't go yet. 7 ok\nCafé". An independent Python replay of the same file matched, and every lifted key's point was inside its logged hit frame. - Overhead, simulator Debug, 3,000 taps: about 9 µs a tap on the main thread (38 → 47 µs) and about 1 KB a tap (5.2 lines), so about 12 MB an hour at 200 characters a minute. - The Release generic-iOS build with CODE_SIGNING_ALLOWED=NO succeeds. - Simulator screenshots of the toggle and the Typing logs screen. - Not verified on a physical iPhone: real touch radii and sampling rates, how often hosts call textDidChange, whether documentIdentifier stays stable, the extension's write cost, and AirDrop from the share sheet. |
||
|
|
efb7dcafb5 |
iOS: clean up what I say with an on-device tagger (experiment) (#2023)
* ios: clean up what I say with Hex's on-device dictation tagger (experiment)
What changed
- Experiments › Clean up what I say (off by default, App Group key
experiments.cleanupTagger): Hex's trained cleanup tagger removes ums,
repeats, and the words a spoken correction replaces ("3pm, actually 4pm"),
on the iPhone, in in-keyboard dictation and in app dictation. It runs after
transcription and learned fixes (in the app also after custom vocabulary),
before Apple Intelligence cleanup, translation, and formatting, and never
for Verbatim. When both cleanups are on, Apple Intelligence now receives the
tagged text instead of the raw transcript; nothing else changes while the
experiment is off.
- Model: the BERT L-4/H-512 token classifier (ckpt-1200k-l4h512) converted to
an int8 Core ML ML Program (coremltools 9, fp16 compute, enumerated lengths
16/32/64/128 plus a length input that masks the padding; the output is
logit(DELETE) - logit(KEEP)), 28.5 MB, with its WordPiece vocabulary. Both
ship once, in HEXKeyboard.appex (HEXKeyboard/CleanupTagger/), and the app
loads them from its embedded keyboard extension. Core ML codegen is off for
the keyboard target.
- HEXShared: CleanupTokenizer (Hugging Face BertTokenizer: special tokens,
BertNormalizer, BertPreTokenizer, WordPiece, byte-exact vocabulary keys,
HF's Unicode 8.0 character classes), CleanupReconstruction (a port of
js/reconstruct.js with JavaScript's UTF-16 casing), and CleanupTagger
(chunks of whole words within 128 tokens, P(DELETE) > 0.98, one serial
queue, 150 ms budget). A timeout, an error, or a result with no words left
keeps the transcript. The model loads when a dictation starts recording with
the experiment on, stays loaded, and is freed on a memory warning. It runs
on the Neural Engine or the CPU, never the GPU.
- Call sites: one each in KeyboardLocalDictation (after DictationCorrections,
before translation) and DictationController (after VocabularySpelling,
before the Apple Intelligence pass). prepareIfEnabled runs at recording
start, in KeyboardViewController and in DictationController.startRecording.
Why
- In the simulator the first load took 0.15 s on the CPU and 0.8 s on the GPU
path, both over the 150 ms budget. So loading starts when recording starts,
and the GPU isn't used (it also added 30 MB of footprint, against 3 MB on the
CPU, which keeps the int8 weights memory-mapped).
- Threshold 0.98, not the exported 0.95: 1/45 false deletes on hand-written
clean sentences against 3/45, with the same self-correction accuracy.
Verified
- xcodebuild test on the iPhone 18 Pro simulator: 168 tests, 0 failures,
including the new CleanupTaggerTests and DictationControllerTests
testCleanupTaggerRunsAfterLearnedFixesAndSkipsVerbatim. After the fixtures
were rewritten one case per line (same content), CleanupTaggerTests and
DictationControllerTests passed again (28 tests).
- Parity on the 137 hand-written sentences at 0.98: labels match the fp32
checkpoint on all 1,128 words, and 136/137 outputs match the ONNX int8
pipeline. #120 ("The cat keeps trying to scratch that chair.") differs on
two words that ONNX keeps (P 0.9725/0.9794) and Core ML deletes (0.984).
The fp32 checkpoint also deletes them (0.9879/0.9853), so the difference
comes from ONNX's int8 activations. 128/137 outputs match the hand-written
targets. The tokenizer matches HF on 1,891 words, and reconstruction matches
reconstruct.js on 670 cases.
- Simulator, CPU path: load 140-161 ms, first sentence after load about 8 ms,
then 3.3 ms median and 7.5 ms p95 per sentence; phys_footprint +2.9 MB.
- The Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds.
HEXKeyboard.appex is 30.2 MB (model 27.8 MB, vocabulary 0.2 MB).
- Not verified on a physical iPhone: Neural Engine load time, latency, and the
keyboard extension's footprint next to Apple Speech.
Note: this adds a 28.4 MB binary (weight.bin) to git; the repo has no LFS.
* Fetch the cleanup tagger's Core ML model at build time instead of committing it
The 27 MB mlpackage would have been ~30x the largest file in the repo, kept in history forever. It is now the ios-model-cleanup-tagger-1 release asset; ios/scripts/fetch-cleanup-model.sh downloads it, checks its SHA-256, and skips when the pinned copy is already in place. Xcode Cloud runs it in ci_post_clone.sh before xcodegen; the folder is gitignored.
Verified the fetched package is byte-identical to the one the tests ran against (diff -r), and a second run is a no-op.
|
||
|
|
c351c54db0 |
ios: learn dictation and typing fixes from edits (Learn how I type) (#2022)
What changed - Learn how I type (Experiments, off by default) now follows each dictation the keyboard inserts, from the full keyboard and the dictation bar, for up to 60 s. On device it works out whether the dictation was kept, edited, rewritten (more than half changed), undone, deleted, or redictated (deleted and dictated again within 20 s). DictationEditTracker remembers each insertion by the text around it. DictationSpanLocator finds it again in documentContextBefore/AfterInput, handling nil or cut-short context. DictationWordDiff diffs it word by word, and DictationEditClassifier turns the changes into (heard, inserted, meant) pairs: substitutions, insertions, deletions, words cleanup dropped that you added back, and fillers Hex left that you deleted. These teach nothing: new writing at the edges, changes that aren't plausibly the same word, truncated edges, and translated dictations. - The journal stores only word pairs (at most 3 words a side, one word of context on each side) with counts and contradictions, plus one outcome per dictation, never the dictation itself. Older journal files still load. - A substitution seen twice and never contradicted is applied right after transcription in KeyboardLocalDictation and DictationController, before custom vocabulary and cleanup. It matches whole words only and follows case. Its meant words join the Apple Speech contextual strings; in-keyboard Apple Speech now gets them too. The app sends the raw transcript to the keyboard (KeyboardSnapshot.heard) only while Learn how I type is on and the text wasn't translated. - Typing: a typo fixed the same way twice becomes an AutocorrectEngine candidate. Space applies it unless the typo is a real word, in which case it is only suggested. An autocorrection turned down twice never auto-applies again. Turning one down means a backspace undo, or tapping the literal chip, which is the new keptTyped record. suppressAndKeep was session-only. - Typing insights lists "Dictation fixes Hex learned" with a switch per row, and the kept and edited share of dictations. Delete erases everything: the file, the switched-off list, and, through an erasedAt marker, whatever a running keyboard still holds. - There is now one journal per process (TypingJournal.sharedIfEnabled). It reloads when another process wrote the file, so the keyboard and the app's keyboard preview no longer overwrite each other. Before, each FullKeyboardView made its own journal. - The Learn how I type toggle moved out of the Full keyboard group, because it now also learns from the dictation bar. Why: so Hex predicts better from the corrections people already make, without asking and without storing what they write. Verified: xcodegen generate, then xcodebuild test on the iPhone 18 Pro simulator: 155 tests, 0 failures. That includes the new DictationEditInferenceTests, DictationEditTrackerTests, TypingJournalLearningTests, and a DictationController test. The Release generic-iOS build with CODE_SIGNING_ALLOWED=NO succeeds. Not run on a physical iPhone. |
||
|
|
232be8ce4b |
Move the Hex key into the strip and grow it into the recording bar (#2021)
The square Hex keycap right of space (with a ↗ 'opens Hex' badge) is now a bare mark at the suggestion strip's trailing edge, so the bottom row is Apple's with a full-width space bar. Starting dictation grows a capsule from the mark across the strip before the recording panel takes over; ending it shrinks the bar back into the mark. The style menu opens downward from the strip. HexKeyPlacement keeps the other placements for snapshot comparison. Verified HEXMobileTests (bottom-row expectations updated, strip-position test added) and simulator recordings of the morph in light and dark. |
||
|
|
2c80cd400d |
Add Translate what I say (experiment) (#2015)
Speak one language and insert another: after transcription and optional cleanup, the spoken language is detected with NLLanguageRecognizer and translated on the iPhone with TranslationSession(installedSource:target:) once the pack is downloaded in Experiments. Applies to in-keyboard and app dictation; same language, a missing pack, or an error keeps the original. Off by default. Verified the Release build and HEXMobileTests on the simulator; not yet on a device. |
||
|
|
011da1b38c |
Drop the fake crown from the full keyboard snapshot stage (#2014)
The stage padded 12 pt above the keyboard to imitate iOS 26's Liquid Glass crown, which iOS 27 doesn't add, so renders showed 16 pt above the suggestion pill and 4 pt below it. Without it, renders match the device: 4 pt above and below. Verified by measuring the full-keyboard-typing render at 3x. |
||
|
|
d66946319a |
Make the suggestion highlight fill its slot (#2013)
The 44 pt strip still drew a ~26 pt pill with wide insets, so the extra height read as empty space above a small selection. The strip now spans the whole strip area, the pill fills each third 4 pt in from every edge, suggestion text is 17 pt, and chips grow to 34 pt; keyboard height is unchanged. Verified HEXMobileTests and simulator renders of the typing state in light and dark. |
||
|
|
0c9fa9cddf |
Give the full keyboard's suggestion strip its full height (#2012)
The strip was 35 pt and drawn 8% above centre for iOS 26's +17 pt Liquid Glass crown, which iOS 27 doesn't add, so suggestions and the autocorrect pill looked short and clipped at the top (TestFlight build 4). Portrait strip is now 44 pt (keyboard 262 pt), landscape stays 39 pt, and strip contents are centred. Verified HEXMobileTests (layout expectations updated) and simulator renders of the typing state in light and dark. |
||
|
|
d02d8515d6 |
Replace the keyboard's blank Paste control with a Paste chip (#2011)
UIPasteControl draws as an empty blue box inside a keyboard extension (TestFlight build 4), which looked like a broken half-width strip. The strip now shows a Paste chip styled like Undo dictation that reads UIPasteboard.general on tap; iOS may ask Allow Paste the first time. Offering still checks only hasStrings/hasURLs. Removes KeyboardPasteReceiver. Verified the Release build and HEXMobileTests on the simulator; not yet on a device. |
||
|
|
b77981d63d |
Add Learn how I type, an on-device typing journal (experiment) (#2010)
Off by default under Experiments with the full keyboard. Records per-key tap offsets as averages (never a letter sequence) and single-word corrections: retyped words within a small edit distance, autocorrect fixes, undone autocorrect, and picked suggestions, capped at 2,000. Skips URL, email, credential, code, card and phone fields and fields without autocorrect. Stored in the App Group; the app shows the most common fixes and off-centre keys and can delete it all. Personalization from this data is a later step. Verified the Release build compiles and HEXMobileTests pass on the iPhone 18 Pro simulator; not yet tried on a device. |