77 Commits
Author SHA1 Message Date
Ryan Vogel 415cbf6373 iOS: keep key haptics while dictating, and slide left on delete to delete words (#2080)
* iOS keyboard: key haptics are the same in every layout, and stay on while the keyboard records

A TestFlight tester on build 37 felt no key haptics with the Portuguese layout
added in #2069. The haptic path was traced for every layout (en, nb, es, de, it,
fr, pt) and none of it depends on the language. Every key going down calls
FullKeyboardView.keyFeedback through KeyboardKeyPlaneView.onKeyDown, accent and
trackpad steps call selectionFeedback, and both are gated only by
KeyboardViewController.allowsKeyHaptics (Full Access and Settings > Keyboard >
Haptics). Layouts only change the letter rows, accent lists and the spelling
dictionary.

So the cause is the tester's setup or state, not the layout. Two things silence
the taps with no code path to see them:
- Without Full Access iOS plays no keyboard haptics. Since #2067 the keyboard
  types without Full Access, so a tester can type in any layout while Haptics
  shows On and feel nothing.
- In-keyboard dictation sets this process's audio session to record (WebKit
  captures for it), and iOS mutes a recording session's haptics and system
  sounds by default. The keyboard now calls
  setAllowHapticsAndSystemSoundsDuringRecording(true) once in viewDidLoad.

Verified on an iOS 27.2 iPhone 17 Pro simulator (Mac Studio): the full
HEXMobileTests suite passes (278 tests, 0 failures, 1 skipped as before). The
new testEveryLayoutAsksForTheSameKeyHaptics drives a letter, space, delete and
an accent step through the real FullKeyboardView in each of the seven layouts
and gets the same haptic requests in every one. A live Hex keyboard (Full Access
granted) with Settings > Keyboard layout set to Portuguese showed Portuguese
accents and the "olá" suggestion. A temporary log line, not committed, showed
the haptic call firing with haptics allowed for each key and accent step. The
Taptic output and the recording-session mute need a physical iPhone.

* iOS keyboard: hold delete and slide left to delete word by word

A TestFlight tester asked for the delete swipe from Apple's keyboard.

- New DeleteWordSwipe (HEXShared/FullKeyboard): once a finger that went down
  on delete is a key width (layout.letterWidth) left of where it started, each
  further key width deletes one more word. Sliding back right types back the
  words this gesture deleted, newest first. Lifting, or a cancelled touch,
  keeps the result.
- Words are counted as Option-Delete does: spaces and punctuation before the
  cursor go with the word in front of them ("one two, three. four" loses "four",
  "three. ", "two, "), apostrophes stay inside words ("don't"), and text that
  is only spaces and punctuation goes in one step.
- KeyboardKeyPlaneView starts a DeleteWordSwipe with each delete press, feeds
  it moves, and stops hold-to-repeat once the swipe is active. A tap, a still
  hold, or a hold that drifts less than a key width behaves as before.
- Each word deleted or restored ticks the selection haptic (gated by
  allowsHaptics, as the accent and trackpad ticks are). Edits go through the
  text proxy only (deleteBackward and insertText) and show in typing logs as
  delete_word / insert with src delete_swipe(_restore). Nothing new in
  Settings.

Not done: Learn how I type's journal isn't told about words a swipe erases.

Verified on an iOS 27.2 iPhone 17 Pro simulator (Mac Studio): the full
HEXMobileTests suite passes (278 tests, 0 failures, 1 skipped as before),
including new FullKeyboardEngineTests cases for step-by-step deletion, a tick
per word, restore on slide back, commit on lift, the start of the text, and
the word rule. In a live Hex keyboard in a text field ("Running 10 min late, "),
driven with AXe, the swipe went to 1, 2 and 3 words deleted, sliding back
restored two, and lifting kept "Running 10 ". A 1 s hold with no slide and a
hold with an 18 pt drift only repeated, as before.
2026-11-20 17:55:51 +00:00
Ryan Vogel 65a2e04e3d iOS: let Translate what I say insert in the language you usually dictate in (#2078)
Insert in hid the current dictation language, so someone who dictates in English could never pick English as the target and translate Italian into it. The list now has every language Apple translates; dictations already in the target paste as spoken, and the row says to switch the language from the Hex key instead of showing Apple's can't-translate status.

Verified: Release generic-iOS build.
2026-11-20 17:53:31 +00:00
Ryan Vogel c5400a4686 iOS: Russian and Serbian (Latin) full-keyboard layouts (#2075)
A TestFlight tester (build 35) asked for Russian and Serbian (Latin) layouts in
Settings › Keyboard › Keyboard layout. KeyboardLanguage gets russian ("ru") and
serbianLatin ("sr-Latn") cases, the way #2069 added Italian, French and Portuguese.

Russian is Apple's ЙЦУКЕН: 11, 11 and 9 letters, ё on a held е and ъ on a held ь.
The long-press lists are Apple's own, read from the iOS 27.2 simulator's
TextInput_ru bundle, base letter first (Apple's combining-macron forms а̄ е̄ о̄ с̄ are
left out). Autocorrect and suggestions use UITextChecker's ru_RU dictionary.

Serbian (Latin) is Apple's QWERTZ option for it. Apple's iPhone Serbian Latin
keyboard has no keys of its own for č ć đ š ž (TextInput_sr lists QWERTY, QWERTZ and
AZERTY; only the iPad's Croatian layouts give them keys), so they are held on c, d,
s and z with Apple's lists. iOS 27.2 has no Serbian (or Croatian) UITextChecker
dictionary: suggestions and autocorrect stay off for it, never English. The
dictionary candidates are Latin-only (sr_Latn), since a Cyrillic sr dictionary
would reject every Latin word.

Automatic picks Russian for ru-* and Serbian (Latin) for sr-Latn-* dictation
languages, iPhone keyboards or preferred languages. Bare sr / sr-RS (Cyrillic)
still gets English: Hex has no Cyrillic Serbian layout. Spatial autocorrect has no
ru or sr-Latn model, so neither gets a decoder (same rule as #2069).

KeyboardLayout: Russian's nine-letter row 3 overlapped the 10-key-sized shift and
delete by about 9 pt. Where a letter row wouldn't fit, shift and delete now narrow
to fit (to about a letter's width, as on Apple's Russian keyboard); every existing
layout computes the same modW, so their frames are unchanged. Hit-testing is
untouched. DictationLanguageSwitch.nativeName names a language in its own script
("sr-Latn-RS" is "Srpski", not "Српски"); the space bar still names the dictation
language ("Русский" for ru-RU).

Snapshot scenes get Russian and Serbian drafts (--keyboard-layout ru|sr-Latn,
--accent-key е|ь|c|s).

Verified on an iOS 27.2 iPhone 17 Pro simulator: full HEXMobileTests suite, 277
tests, 0 failures, 1 skipped as before (one new Russian autocorrect case was
adjusted and re-run on its own after the full run). New cases in
KeyboardLayoutTests, AutocorrectEngineTests, FullKeyboardEngineTests and
KeyboardBridgeTests. Live in Messages with the Hex keyboard: Automatic with ru-RU
dictation shows the Russian layout and "Русский" on the space bar, "минкт"
suggests "минут", and holding е typed "ещё". Picking Srpski in Settings, holding c
and s typed "čekaj šta" with an empty strip. Compared with Apple's Russian and
Serbian Latin keyboards on the same simulator: same rows, letter-sized
shift/delete on Russian, and the same е, ь and c menus.
2026-11-20 17:44:49 +00:00
Ryan Vogel 2e7fc060ed iOS: let the keyboard dictate once Hex has the mic, even if it opened before the grant (#2071)
TestFlight (build 32, iOS 27.2.1): onboarding's practice field said the
microphone wasn't allowed after it had been, until Skip for now.

Cause (reproduced on an iOS 27.2 simulator): a keyboard process keeps the
microphone permission it read when it started. AVAudioApplication,
AVAudioSession and AVCaptureDevice all still read "undetermined" in it after
Hex is allowed the mic, so a Hex keyboard opened before the grant (tried in
another app first, then onboarding) hits KeyboardLocalDictation's guard and
shows "Allow the microphone in Hex first" on the Try the keyboard page. The
app and the try page had the right state; only the keyboard's own reading
was stale.

What changed
- KeyboardBridge.appMicrophoneAllowed (App Group key app.microphoneAllowed):
  the app's own reading, written by KeyboardSetupMonitor.refresh (every
  second on onboarding and home, on becoming active, and after the mic
  prompt), only when it changes.
- KeyboardLocalDictation.start accepts the mic when its own reading or the
  app's says granted; WebKit's capture still checks, and a NotAllowedError
  still shows the same message. The probe log notes app=granted|no.
- Nothing removed. Onboarding is unchanged: #2058's no Allow button before
  the prompt, and Continue after a decline, stay as they are.

#2058 made this easier to hit, not the cause: build 32 needed the mic for
Continue, so it took a keyboard opened earlier; now declining leads to the
try page with the keyboard open, and allowing it later in Settings left that
keyboard stale too.

Verified (anomastudio, iPhone 18 Pro Max simulator, iOS 27.2, fresh install,
mic reset, Full Access via TCC): Hex keyboard tried in Spotlight first, then
onboarding with the mic allowed at the system prompt. Before: same keyboard
pid, mic=undetermined, "Allow the microphone in Hex first" on the try page.
After: same pid, mic=undetermined app=granted, web capture started in the
practice field. xcodebuild test: 267 tests, 0 failures, 1 skipped.
Not verified: a physical iPhone (the simulator's WebKit captures without a
TCC check and has no speech assets), or a keyboard that started with the mic
denied and was allowed in Settings.
2026-11-20 16:59:42 +00:00
Ryan Vogel 9b53afcd4c iOS: emoji panel no longer blurs on a real keyboard; taps and holds behave (#2070)
* iOS: no blurred emoji on a real keyboard (hide the panel's scroll edge effects)

A TestFlight tester (build 32, iPhone19,2, iOS 27.2.1) found the emoji panel "pretty
much unusable": the whole suggestion row was blurred smudges, and so were the grid's
section names.

Cause: iOS 26's automatic scroll edge effect. Inside a keyboard, where the panel sits on
the system's glass, every scroll view gets a soft blur along its top, bottom, left and
right edges. On the 44 pt suggestion row (and the search results row) the top and bottom
effects meet and blur everything; on the grid they blur the section names along the top
and the columns at either side while scrolling. The app's --snap-state scenes host the
keyboard view outside a keyboard, where iOS adds no effect, so they never showed it.
There is no visual effect view or layer filter in the panel.

What changed
- UIScrollView.hideEdgeEffects() hides all four edge effects (iOS 26+); the grid and
  EmojiRowView (suggestions and search results) call it. Nothing draws over them, so
  they need none.
- A hairline (palette.separator) between the suggestion row and the grid.
- FullKeyboardEngineTests: opens the panel and its search and checks every scroll view
  hides its four edge effects; it fails on the previous code (the 345x44 pt row).

Verified on the Studio's iPhone 18 Pro simulator (iOS 27.2) in a live hosted keyboard
(the real HEXKeyboard.appex in the app's hosted-keyboard conversation, driven with AXe
taps): before, the row, the section names, the left column after a swipe and the search
results were blurred, light and dark; after, all sharp. xcodebuild test: 267 tests, 1
skipped, 0 failures.

* iOS: emoji panel clicks on a pick, not on every touch, and a hold types on release

Found driving the panel in a live hosted keyboard on the simulator next to the system
emoji keyboard.

What changed
- The grid played the key click and haptic when a cell lit up, which is also how every
  swipe through the grid starts (cells light on touch-down so taps feel immediate), so
  scrolling clicked. The click now comes with the pick (didSelect); the key pop-up still
  shows on touch-down and goes when the scroll starts.
- Holding an emoji without skin tones typed it as soon as the hold registered (0.4 s),
  before the finger lifted, so sliding away couldn't take it back. It now behaves as a
  slow tap: its pop-up shows while held, and it's typed only if the finger lifts over it.
  Emoji with skin tones still open the tone picker.

Verified in the live keyboard (AXe touch down, hold 1 s, lift): released over ❤️ typed it;
released 200 pt away typed nothing; held 👍 opened the tones with the remembered one
selected and lifting on a darker tone typed and kept it. Three taps 0.15 s apart typed
the three emoji under them; the category bar jumped to Food and drink; a held delete
cleared three emoji; search typed "piz" with the letter keys and its first result went
into the field; return went back to the panel and ABC to the letters. A tap during a
fling's deceleration only stops the scroll, as in the system emoji keyboard.
xcodebuild test: 267 tests, 1 skipped, 0 failures.
2026-11-20 16:58:50 +00:00
Ryan Vogel 9a06976d0e iOS: Italian, French and Portuguese layouts that autocorrect in their own language; spoken @ and email addresses (#2069)
* iOS: Italian, French and Portuguese full-keyboard layouts that autocorrect in their own language

A TestFlight tester on build 31 with an Italian iPhone reported that Hex kept
autocorrecting their Italian into English. KeyboardLanguage only knew en, nb, es
and de, so Automatic mapped an it-IT dictation language (and any other language
without a layout) to English: English letters, and UITextChecker's en_US
dictionary rewriting Italian words.

Adds italian/french/portuguese cases. They type on the English QWERTY rows
(Apple's Italian and Portuguese keyboards are QWERTY; Hex has no AZERTY yet, so
French is QWERTY too), with their own long-press accent order (Italian à è ì ò ù,
é right after è), their UITextChecker dictionary (it, fr, pt_BR then pt_PT), and an
Italian common two-letter list. Automatic now resolves it-*, fr-* and pt-* to these;
the dictation language still wins over the iPhone's keyboards, and Hebrew or
Russian still get English. Automatic deliberately ignores Hex's own input mode:
its primaryLanguage is the Info.plist's en-US whatever you type. Spatial
autocorrect (#2059) only has en and es models; the new layouts get no model and
fall back to the dictionary engine in their own language (the decoder also
refuses a model of another language). The space bar already names the dictation
language ("Italiano").

Snapshot scenes get Italian, French and Portuguese drafts
(--dictation-language it-IT, --keyboard-layout it|fr|pt).

Verified on an iOS 27.2 iPhone 17 Pro simulator: full HEXMobileTests suite
(272 tests, 0 failures, 1 skipped as before). New cases in KeyboardLayoutTests,
AutocorrectEngineTests and FullKeyboardEngineTests. Measured on 12 Italian
sentences (19 one-slip typos, 59 correct words) with the real dictionary:
English fallback fixed 4, miscorrected 4 and broke 19 correct words
(Domani->Domain, stai->stay, Sono->Soon, molto->moto); Italian fixed 17,
miscorrected 0 and broke none. Renders of the Italian keyboard (typing with the
"ritardo" suggestion, the E and I accent menus, "Italiano" on the space bar)
checked by eye.

* iOS: spoken "at the rate" and "at sign" become "@", joining a spoken email address

An Indian English TestFlight tester (build 32) said "at the rate", India's name for
@, and got the words. Apple's on-device models don't fix it: checked on macOS 27.2
with Hex's settings and an en_IN voice, SpeechTranscriber (Hex's default) writes
"My email is John at the rategmail.com." and "Send it to Priya at signyahoo.com."
(gluing the domain to the word before), and DictationTranscriber writes "at the
rate gmail.com", though it already writes "priya@yahoo.com" for "at sign" and
"john@gmail.com" for "john at gmail dot com".

SpokenCleanupRules.spokenSymbols (English only) turns "at the rate" and "at sign"
into "@". Before a domain ("gmail dot com", "gmail.com", "co dot in", or glued as
above), the word before (with any "dot"-joined parts) and the domain join into one
lowercase address, "dot" becoming "." only there: "John at the rate Gmail dot com."
becomes "john@gmail.com.". "at the rate" counts only before a domain, so "at the
rate of 5%" and "at the rate we planned" stay, and "at" alone is never touched;
"at sign" elsewhere is a lone "@" (but not in "at sign language"). Only the
matched words change.

It runs from CleanupTagger.cleanIfEnabled, which both the app's and in-keyboard
dictation call: in every style but Verbatim, with Clean up what I say on or off
(the experiment's guard moved below it; the tagger itself still only runs when on).

TranscriptFormatter: Polished capitalized after every full stop, which would have
written "john@gmail.Com."; a sentence now starts only after ".", "!" or "?" and a
space, an address keeps its case (Polished and Notes), and a lone address gets no
full stop.

Verified on an iOS 27.2 simulator: full HEXMobileTests suite (272 tests, 0
failures, 1 skipped as before), with new cases in TranscriptCleanupTests (Apple's
real outputs included) and CleanupTaggerTests.
2026-11-20 16:43:10 +00:00
Dax 1930e8d7ac iOS: split DictationController so dev passes the size check (#2066) 2026-11-20 11:42:45 -05:00
Ryan Vogel b6024b2001 iOS: full keyboard by default, typing without Full Access, and a logging mark (#2067)
* iOS keyboard: the full keyboard by default, typing without Full Access

App Review 4.4.1 needs a keyboard extension to type, and to keep working
without Full Access and without network. Hex's default was the dictation
bar (space, delete, return, Dictate; no letter keys), and without Full
Access it showed only setup steps.

What changed
- FullKeyboardSettings.isEnabled (App Group key experiments.fullKeyboard,
  what Settings › Keyboard's Full / Dictation bar row binds) now reads
  unset as true: new installs and testers who never touched the old
  experiment get the full keyboard; an explicit false (turned off, or
  Dictation bar picked) keeps the dictation bar. --keyboard
  full|dictation-bar picks it in simulator scenes.
- Without Full Access the keyboard always mounts the full keyboard,
  whatever is chosen (the bar types only spaces and line breaks).
- Removed: the full keyboard's permanent "Allow Full Access to dictate
  with Hex" strip notice; the strip suggests and corrects as usual.
- Without Full Access the Hex key (tap, hold, VoiceOver, or sliding for
  its menu) no longer tries to dictate or open Hex: the strip shows
  "Allow Full Access to dictate" and "Settings › Apps › Hex › Keyboards"
  for 8 s (until a key is typed), with an Open Hex chip, the only thing
  that opens Hex (hex-dictation://keyboard/setup) without Full Access.
- In-keyboard dictation (onLocalDictateTap) is offered only with Full
  Access; before, it was set without it too, so the Hex key would have
  started a capture that can't work there.
- Scene full-keyboard-no-access-hint; README: the default, behaviour
  without Full Access, memory numbers and proof limits.

Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release, keyboard
in a real hosted field, a temporary uncommitted probe):
- Fresh install, Full Access off: full keyboard, no in-keyboard
  dictation; taps through the engine typed "thw " -> "the ", "tk " ->
  "to "; with Spatial autocorrect on (decoder active) also "ofthw " ->
  "of the ". The hint renders light and dark.
- Full Access on: full keyboard by default; --keyboard dictation-bar
  mounts the bar.
- Memory, 3 runs each, MB, before (78e4458d) -> after:
  full keyboard idle 18.4-18.5 -> 18.5-18.6 (18.0-18.3 without Full
  Access), first dictation max within 3 s 30.8-31.3 -> 31.2-31.8, peak
  32.1-32.5 -> 32.3-33.0; bar idle 20.7 -> 20.8, first dictation
  27.5-27.8 -> 27.3-27.9. Unchanged within noise, so #2045's 32/24 MB
  guard still covers it.
- xcodebuild test: 267 tests, 0 failures, 1 skipped (as before).
Not verified on a physical iPhone.

* iOS keyboard: show a Logging mark while typing logs record

App Review 2.5.14: a keyboard that records what is typed must say so the
whole time. Record typing logs had no indicator in the keyboard.

What changed
- KeyboardViewModel.isLoggingTyping, set by the keyboard from
  TypingLogger.isRecording (session start and end, and every bridge
  poll), so a skipped field (password, one-time code, card) shows no
  mark.
- Full keyboard: a flat "● Logging" mark (10 pt, the strip's muted ink)
  in the strip's top-leading corner, in from the keyboard's rounded
  corner; while dictating, at the start of the recording bar's well, with
  the scope moved past it, so it never disappears.
- Dictation bar: the same mark beside the Hex mark in its top row, there
  while idle and while recording.
- The dot is muted, not orange: an orange dot would read as iOS's
  microphone indicator. VoiceOver reads "Typing logs are recording".
- --typing-log-mark shows it in the full-keyboard and keyboard-* scenes;
  README.

Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release, keyboard
in a real hosted field with --experiment-typing-logs): the mark shows in
the full keyboard's strip, in its recording bar during an in-keyboard
dictation, and in the dictation bar idle and recording; the scenes
render it light and dark. Idle footprint with logging on 18.5-18.7 MB
(full) and 20.8-20.9 MB (bar), as without. Built for testing.

* iOS keyboard: cap keyboard-capture-probe.log at 512 KB

The in-keyboard dictation probe log in the App Group grew without limit:
every dictation and memory change appends to it, and nothing removed it.

What changed
- KeyboardProbeLog.write: once the file passes 512 KB, it keeps only its
  newest 256 KB, from the first whole line in them (atomic rewrite).
  Nothing user-facing reads the log; testers' copies keep their recent
  footprint lines.
- README: the cap.

Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release): a
614,400-byte log in the simulator's App Group container became 262,590
bytes after the keyboard's next in-keyboard dictation wrote to it,
starting on a whole filler line and ending with that dictation's lines.
2026-11-20 16:41:37 +00:00
Ryan Vogel 6d2035eca2 iOS: rebuild the audio engine before the warm-mic tap so a route change can't crash it (#2064)
TestFlight crash on build 32: an Objective-C exception from installTap in SystemAudioManager.startKeepalive after toggling Always use iPhone microphone and Keep mic warm. Changing the session can move the hardware sample rate, and the reused AVAudioEngine kept its old input format. startKeepalive now builds a fresh engine after configuring the session and taps with the hardware's input format, throwing (already handled by callers) if it isn't valid.

Verified a Release generic-iOS build; the exception path needs a device route change.
2026-11-20 16:25:47 +00:00
Ryan Vogel be49c35f2b iOS: privacy and support links in the app, Continue before the mic prompt, typing logs consent (#2063)
* iOS Privacy: Privacy policy and Support links, for App Review 5.1.1(i)

App Review requires a privacy policy link inside the app. The Privacy page
now has a card with two rows between Learn how I type and Advanced:
- Privacy policy opens https://dev.anoma.ly/hex/privacy.
- Support opens https://dev.anoma.ly/hex/iphone.

Both open in Safari inside Hex (SFSafariViewController, SafariSheet).

The URLs come from one base in HexLinks.swift (HexLinks.site). Moving to
https://anoma.ly/hex is a one-line change once the site's production deploy
serves those pages; https://anoma.ly/hex/privacy returns 404 today, while
both dev.anoma.ly pages return 200. README's Privacy page entry says so.

Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked
at the settings-privacy render. A temporary, uncommitted hook opened the
Privacy policy row's Safari sheet, which loaded the live dev.anoma.ly page.

* iOS setup sheet: Continue, not Allow, before the microphone prompt

App Review 5.1.1(iv) rejects a custom button that says "Allow" right before
iOS's own permission prompt. The setup sheet's Microphone row had one. It now
matches onboarding's Set up Hex page (#2058): the row says why ("For
dictation", replacing "Required") and its button says Continue. Tapping
Continue still calls requestMicrophone, so iOS asks right after. Allowed and
Off are unchanged, and Off still shows Open Settings.

New scene: --snap-state setup-sheet-microphone, the setup sheet before iOS
has asked for the microphone. README's Setup and scene notes are updated.

Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked
at the setup-sheet-microphone render (For dictation, Continue) and the
setup-sheet render (Allowed, no button).

* iOS Privacy: Record typing logs says what it records and asks before turning on

App Review 2.5.14 requires explicit consent before recording what people
type.

- The description now says what the feature records: "Records taps and the
  text you type for 14 days, on this iPhone, to export. Off by default." It
  replaces "Off by default. Stays on this iPhone to export" and fits on two
  lines.
- Turning it on opens an alert before anything changes. Title: "Record
  typing logs?". Message: "Hex's keyboard will record each tap and the text
  you type or dictate with it, including the text around the cursor, for 14
  days. The logs stay on this iPhone for you to export or delete. Password,
  code and card fields are never recorded." Turn On switches it on; Cancel
  leaves it off. Turning it off needs no dialog. The switch still writes only
  TypingLogSettings.isEnabled.
- Every claim matches docs/typing-logs.md: touches on Hex's keys, the text
  each key produced, the text around the cursor, dictations, 14 days in the
  App Group, export and delete from Typing logs, and a "skipped" line only for
  password, one-time code and card fields.

New scene: --snap-state settings-privacy-confirm opens the dialog
(SettingsSnapshotFixture.confirmsTypingLogs). README and docs/typing-logs.md
now describe the consent and the new path (Privacy › Advanced, no longer
Experiments).

Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked
at the settings-privacy, settings-privacy-confirm and settings-privacy-on
renders.

* iOS home: Privacy policy and Support join Acknowledgements

#2059 added an Acknowledgements row at the bottom of home's utility card. The
privacy policy and support links read better beside it than on the Privacy
page. Home's last card now has Privacy policy (hand.raised, ↗), Support
(questionmark.circle, ↗) and Acknowledgements (doc.text, ›). Keyboard settings
and Run setup again keep their own card above it.

- Moved, not duplicated: the Privacy page's links card is removed. That page
  is back to "Everything stays on this iPhone", Learn how I type, and
  Advanced.
- Links still come from HexLinks (dev.anoma.ly until production serves
  anoma.ly/hex/privacy) and open in Safari inside Hex.
- README's Settings section now lists the links on home.

Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2). I looked
at the settings-home render, which still fits one screen without scrolling,
and the settings-privacy render.
2026-11-20 16:23:37 +00:00
Ryan Vogel c14df5a4f0 iOS: spatial autocorrect experiment with Google Books models, plus accent and two-letter fixes (#2059)
* iOS keyboard: Spatial autocorrect experiment (tap + bigram decoder), off by default

What: a noisy-channel word decoder (TypingDecoder) behind Settings ›
Experiments › Spatial autocorrect (experiments.spatialAutocorrect, off
by default; it needs the full keyboard with Autocorrect on). When on, for a layout whose
model is bundled (English, Spanish), it replaces UITextChecker's one-edit
guesses on space and in the strip:
- each tap is scored against every letter with a 2D Gaussian around the
  key (sigma 0.29 x 0.26 of the key pitch), moved by Learn how I type's
  average tap offsets for letters with 20+ taps;
- words come from a beam search over a memory-mapped, sorted word list
  with substitutions, one insertion, deletion or transposition, free
  apostrophes ("dont" -> "don't") and accented letters read from their
  base key (é from e, +1 nat);
- a bigram language model on the word before (stupid backoff) weighs
  them, and a missed space is tried by splitting the taps in two (or
  treating a bottom-row tap as the space);
- what was typed is replaced only when the best reading beats it by 2
  nats, or 7 when it is itself a word the model, UITextChecker, learned
  words or the shorthand list know.
Text replacements, learned fixes, the English contraction/typo table and
words you kept still come first. FullKeyboardEngine now records each
letter's tap point for the word being typed (cleared on anything else).
scripts/build-typing-model.py writes the model format (v2: per-language
alphabet, sorted words, u8 costs, sorted bigram keys); models are kept
out of git (.gitignore). The English prototype model is built from
SymSpell's frequency lists (MIT; Google Books Ngram CC BY 3.0 and SCOWL),
the Spanish one from Tatoeba sentence counts (CC BY 2.0 FR). Typing logs
now record the setting.

Why: Record typing logs show short words and non-words left uncorrected
("tk" for "to"), missed spaces with a typo inside, and o/i taps landing
low on k/j. The UITextChecker path skips 1-2 letter words, never uses
the word before, and can't split a word that also has a substitution.

Verified (anomastudio, iPhone 18 Pro simulator, iOS 27.2): benchmark
harness (kept outside the repo) typing every tap through
FullKeyboardEngine. Synthetic held-out English, 13,870 words: output
word error rate 15.0% today vs 6.7% (6.3% with tap offsets); typos fixed
58% vs 83%; correct words wrongly changed 89 vs 97 of 9,203. Labelled
real typing logs, 172 words: 14.5% vs 4.7%. Spanish held-out, 13,301
words: 26.8% vs 8.1% (Spanish model and test from the same corpus, so
optimistic). Per letter p50 1.0 ms, p95 2.5 ms; mapping the model adds
no measurable phys_footprint. Full HEXMobileTests suite: 221 tests,
0 failures (1 skipped, as before); Release generic-iOS build with
CODE_SIGNING_ALLOWED=NO succeeds with both models bundled. No new tests.

* iOS keyboard: Spatial autocorrect models from Google Books Ngram v3

What: the English and Spanish models (HEXKeyboard/TypingModel/en.hextm,
es.hextm, 3.6 MB each) are now built from Google Books Ngram v3 (CC BY
3.0) word and word-pair counts summed over 1980-2019, keeping only
words SCOWL (sizes up to 70, lowercase, ASCII) or RLA-ES (LibreOffice's
es_ES and es_MX dictionaries, MPL 1.1) accept, and are committed.
- English: 86,133 words, 417,480 word pairs; Spanish: 110,000 words,
  326,366 pairs. Pairs are the ones that tell most over the single-word
  model (count x log of P(b|a) over 0.4 P(b)), as many as fit.
- Format v3: pairs grouped by first word (u32 row starts, u32 second
  word, u8 cost), 5 bytes a pair instead of 9, so ~70% more pairs fit
  in the same size. TypingDecoderModel reads v3 only.
- scripts/build-typing-model.py downloads, counts (each ngram file
  deleted once counted, every stage cached for a resumable rerun) and
  writes a model: python build-typing-model.py <en|es> <workdir> <out>.
- Tuning for the new models (dev sentences): English leans more on the
  word before (lmWeight 1.5; Parameters.tuned(for:)); an accent-only
  change ("esta" -> "está") needs a 2-nat margin instead of 7 (ñ for n
  isn't one: "ano"/"año"); an accented letter also reads from its base
  key when it has its own (ñ from n), whichever is cheaper.
- Settings › Acknowledgements (new row and sheet, AcknowledgementsView)
  and ios/README.md credit Google Books Ngram (CC BY 3.0), SCOWL with its
  and WordNet's notices, and RLA-ES (MPL 1.1).
- AutocorrectEngineTests: decoder ranking cases on the bundled models
  ("need t^o" -> "need to", a missed space with a slip -> "see how",
  correct words kept, Spanish "aqui manana" -> "aquí mañana", "esta"
  kept).

Why: Ryan's licence decision: CC BY with credit, no share-alike, no
non-commercial or GPL/LGPL data, and not SymSpell's word-pair list
(unverified provenance). The Spanish model was built from Tatoeba,
the same corpus as its test set.

Removed: the SymSpell- and Tatoeba-built prototype models and the v2
format (pairs as sorted u64 keys); models are no longer gitignored.

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2; benchmark
harness outside the repo; word error rate on words that aren't names):
English held-out test, 13,870 words: today's engine 13.0%, decoder
3.6% (3.4% with tap offsets), correct words broken 4 -> 2 of 7,716;
labelled real typing logs, 172 words: 14.5% -> 4.1%; the shared
example sentence decodes exactly; Spanish held-out Tatoeba test (no
source shared with the model), 13,301 words: 25.7% -> 8.0%, broken
16 -> 20 of 7,246. Per letter p50 1.0 ms, p95 2.6 ms.
AutocorrectEngineTests 19 passed.

* iOS keyboard: Spatial autocorrect knows your own words

What: the decoder also searches TypingDecoder.UserWords, a small sorted
in-memory list of your words, beside the model: Contacts names and
other UILexicon terms, one-word text replacement outputs, words you
kept twice (LearnedWordsStore) and typo fixes Learn how I type learned.
Each scores as a moderately common word (userWordCost 11 nats) and
comes back as you spell it ("Zorbix", not "zorbix"); typed exactly,
one counts as a word, so it needs the larger margin to change.
AutocorrectEngine rebuilds the list only when a source changes
(revision bumped by loadLexiconEntries, applyLearning, suppressAndKeep)
or the model changes. The search is now generic over TypingDecoderWords
(the model or your words); the model's own narrow moved to it.

Why: a name, brand or word you use that the model lacks could only be
left alone, never reached from a typo.

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2):
AutocorrectEngineTests 20 passed, including a new case: with "Zorbix"
in the lexicon, a slip "zorbiz" becomes "Zorbix" and "zorbix" typed
right stays. Benchmark unchanged (it has no user words).

* iOS keyboard: an accent or tilde is the cheapest autocorrect fix

What: AutocorrectEngine.singleEditKeyboardCost now scores a one-letter
substitution that only adds or changes an accent, diaeresis or tilde
(e/é, u/ü, n/ñ) at 0.15, below every other edit, instead of 1.15 (the
cost of two far-apart keys). New helper sameLetterIgnoringAccents.
Valid words are still never rewritten, so "esta" stays "esta".

Why: Spanish typed without accents was corrected into the wrong word,
because the accented guess cost more than any insertion, deletion or
neighbouring key: "aqui" -> "caqui", "podia" -> "odia", "manana" ->
"manaba". Now they become "aquí", "podía", "mañana".

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2, the
benchmark harness kept outside the repo, today's engine with Spatial
autocorrect off): Spanish held-out Tatoeba test, 13,301 words, output
word error rate 26.8% -> 24.7%, words changed into another wrong word
1,015 -> 743, correct words broken 16 -> 16 of 7,246 (names excluded).
English test (13,870 words) unchanged at 15.0%.

* iOS keyboard: fix two-letter typos toward very common two-letter words

What: cautiousReplacement fixes a two-letter non-word (UITextChecker
doesn't know it, not ALL CAPS, no apostrophe) when one neighbouring key
or one swap turns it into a very common two-letter word of the layout's
language (KeyboardLanguage.commonTwoLetterWords, commonest first: "tk"
-> "to", "ot" -> "to", Spanish "nl" -> "no", "ee" -> "de"). English and
Spanish have lists; Norwegian and German have none yet and keep
leaving two-letter words alone. The step numbering in the comments
moves by one.

Why: autocorrect skipped every word under three letters, and the
commonest words are two letters: typing logs show "tk" for "to" left
uncorrected again and again (the o key's taps land low, on k).

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2, benchmark
harness outside the repo, today's engine): English held-out test,
13,870 words, output word error rate 15.0% -> 13.2%; labelled real
typing logs, 172 words, 14.5% -> 10.5%; Spanish test 24.7% -> 22.9%;
correct words broken unchanged (4 of 7,716 English, 16 of 7,246
Spanish, names excluded).
2026-11-20 16:18:07 +00:00
Ryan Vogel 9cf4d8b427 iOS: plain permission prompts and no pre-prompt Allow button, for App Review (#2058)
* iOS: plain purpose strings for the microphone and speech recognition

App Review 5.1.1(ii) wants purpose strings that clearly and completely say why
Hex needs access. The old speech string named an API ("SpeechTranscriber")
instead of saying what happens, and the microphone string didn't say what
becomes of the audio. Both now say what Hex does with it and that it stays on
the iPhone:

- Microphone: "Hex turns what you say into text on this iPhone. Audio never
  leaves your iPhone and is deleted once it's transcribed."
- Speech recognition: "Hex turns your recordings into text with Apple's
  on-device speech recognition. Nothing you say is sent to a server."

Same strings in the app and the keyboard extension (project.yml and the
generated Info.plists).

Verified: xcodegen regenerates identical Info.plists; Release build for
generic/platform=iOS succeeds and HEX.app/Info.plist carries the new strings.

* iOS onboarding: no Allow button before the microphone prompt, and no dead end when it's declined

Two App Review 5.1.1(iv) rejection patterns on the Set up Hex page:

1. A custom screen that leads into a permission prompt may not use "Allow"
   (or similar) on the button that triggers it; Apple asks for "Continue" or
   "Next". The Microphone row's small Allow button is gone. The row now says
   why ("So Hex can hear you dictate"), and iOS asks when the person taps
   Continue, or the Keyboard row's Open Settings (unchanged: it asks first so
   Microphone is listed on Hex's Settings page). Denied still shows Open
   Settings.
2. Continue needed the microphone granted, so anyone who declined it could
   never get into Hex, though the keyboard types without it. Continue now
   needs only the keyboard (added or unverifiable). Declining leads on to Try
   the keyboard, and the app's setup card keeps showing the microphone until
   it's allowed.

README's first-run section updated to match.

Verified: Release build for generic/platform=iOS succeeds. Not yet rendered
in a simulator.
2026-11-20 16:13:36 +00:00
Ryan Vogel 71491f93f1 iOS: stamp App Store builds with their own version from an ios-beta-appstore-* tag (#2057)
Ryan wants the repo and TestFlight to stay on 0.2.0 while the App Store gets 0.2.1. ci_post_clone.sh now rewrites MARKETING_VERSION from a tag ios-beta-appstore-<version> before xcodegen; other builds are untouched.

Verified sh -n and the sed against a copy of project.yml (all three targets become 0.2.1).
2026-11-20 16:11:59 +00:00
Ryan Vogel 1e353d53b0 iOS: settings as a short home screen with Dictation, Keyboard, Privacy and Experiments pages (#2054)
* iOS settings prototype 1: short home screen with separate pages

Ryan found the home screen cluttered: a large Keyboard setup checklist sat at
the top even when setup was nearly done, real features were filed under
Experiments, and every setting shared one long scroll. This prototype keeps
every setting and its behaviour, and changes only layout and copy.

Home now fits one screen: the Hex header, one status line ("Hex is ready", or
"Finish setup · 1 step ›"), the Dictate bar, four rows (Dictation, Keyboard,
Privacy, Experiments), then Keyboard settings ↗ and Run setup again.

- Replaced: KeyboardSetupCard with SetupStatusRow plus SetupSheet, a sheet
  sized to its content. It has the same four checks, the Full Access check
  field and Open Settings. Full Access that hasn't been confirmed reads "Not
  confirmed yet" with a dotted circle, not as a failure. The keyboard's Open
  Hex setup link opens the sheet.
- Removed: ExperimentsCard, split into pages (SettingsPages.swift):
  Dictation has style, language, translate, clean up, Apple Intelligence
  clean-up, custom vocabulary, keep mic warm and iPhone microphone. Keyboard
  has full keyboard, then layout, autocorrect and key pop-ups only while it
  is on, haptics, and the test field that was on home. Privacy has
  "Everything stays on this iPhone" and Learn how I type. Experiments has
  Emoji, with Clear emoji history while it's on, plus Record typing logs and
  its viewer.
- Emoji used to show only while the full keyboard was on. It now shows
  either way and reads "Needs the full keyboard" while it's off, because it
  no longer sits under that switch.
- Haptics and Learn how I type stay visible with the full keyboard off,
  because the dictation bar uses them too (KeyboardViewController).
- Copy is cut to one line per row. "Insert in" is now a Menu in the app's
  muted style instead of the system picker, with the same choices and writes.
- Removed the practice field's autofocus for hosted-keyboard scenes. It was
  dead code, because HEXMobileApp sends those scenes to
  HostedKeyboardConversationStage.
- New --snap-state scenes: settings-home, setup-sheet, and
  settings-<page>[-on][-end]. These write the page's switches (Emoji
  included) to the App Group, so each render is deterministic.

Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2) on the
Mac Studio, rebased onto origin/dev with the emoji panel.

* iOS settings: typing logs under Privacy › Advanced, AI clean-up in Experiments

Ryan asked for Experiments to hold only real experiments.

- Moved: Record typing logs and its Typing logs viewer, from Experiments to a
  small "Advanced" section at the bottom of Privacy (a new SettingsSectionHeader
  in the old Experiments header style). It stays in App Store builds.
- Moved: Clean up with Apple Intelligence, from Dictation to Experiments. The
  code still calls it an experiment (AppleIntelligenceCleanupSettings, key
  experiments.appleIntelligenceCleanup). It keeps its disabled state and
  reason line when Apple Intelligence can't run.
- Experiments now has Emoji (with Clear emoji history) and Clean up with
  Apple Intelligence.

Nothing else changed: same settings keys, same reads and writes.

Verified: the build passes on an iPhone 18 Pro simulator (iOS 27.2), and I
looked at settings-privacy-on and settings-experiments renders.

* iOS home: status line reads "Ready" under the Hex header

"Hex is ready" repeated the header's "Hex" on the line just below it. Now it
reads "Ready" with the same green check. VoiceOver still says "Hex is ready",
and the setup sheet's title keeps "Hex is ready" because the sheet has no
Hex header.

Verified: the settings-home render on an iPhone 18 Pro simulator (iOS 27.2).

* iOS settings: Keyboard is a choice between Full and Dictation bar

The full keyboard is becoming the default and the dictation bar an option
(Ryan). On the Keyboard page, the "Full keyboard" toggle is replaced by a
"Keyboard" row with two segments: Full first, then Dictation bar, with no
Beta tag. Layout, autocorrect and key pop-ups show only while Full is chosen.
Haptics stays visible either way, because the dictation bar uses it too.

The choice still writes FullKeyboardSettings.isEnabled (Full = true). The
stored default and its name are changing on ios-full-keyboard-default, which
has no commits yet. This page binds to the setting, so it follows that change.

Verified: the build passes, and I looked at settings-keyboard (Dictation bar
chosen) and settings-keyboard-on (Full chosen) on an iPhone 18 Pro simulator
(iOS 27.2).

* iOS settings: plain, accurate descriptions for App Store review

Three descriptions now say only what the code does:
- Record typing logs (Privacy › Advanced): "Off by default. Stays on this
  iPhone to export". It's off unless turned on (TypingLogSettings), the files
  stay in the App Group, and they leave only through the share sheet.
- Learn how I type: "Learns from taps and the words you fix", replacing
  "Taps and fixes, never what you write". The journal does keep word pairs
  (a word you fixed, with at most one word of context), so "never what you
  write" claimed too much.
- Clean up what I say: "Drops ums, repeats and corrected words", replacing
  "...and corrections". It drops the words a spoken correction replaces, not
  the correction itself.

Verified: I looked at the settings-privacy, settings-privacy-on and
settings-dictation renders on an iPhone 18 Pro simulator (iOS 27.2). Each
description fits on one line.

* iOS settings scenes: settings-*-on seeds one emoji in the history

With an empty history, Clear emoji history renders disabled. That's
accurate, but in a review screenshot it reads as broken. The -on scenes now
record one emoji (EmojiUsageStore.record), so the row shows enabled. The
plain scenes clear the history. This runs only under --snap-state settings-*.

Verified: I looked at the settings-experiments-on render on an iPhone 18 Pro
simulator (iOS 27.2).

* iOS README: describe the settings home and its pages

- New "Settings" section: home (the status line, Dictate, the four rows,
  Keyboard settings, Run setup again), what each page holds, and the
  --snap-state scenes that render them.
- Setup now describes the status line and setup sheet, which replaced the
  setup card.
- Every "Experiments › …" path that moved now names its page: Dictation
  (clean up, custom vocabulary, translate, dictation language), Keyboard
  (Full, haptics, layout), Privacy (Learn how I type, and Advanced › Record
  typing logs and Typing logs), Experiments (Emoji, Clean up with Apple
  Intelligence).

Docs only.
2026-11-20 16:04:10 +00:00
Ryan Vogel 6b0a07fafd iOS: keep the internal version at 0.2.0 (#2053)
Ryan wants the repo and TestFlight to stay on 0.2.0, which beta review approved; the App Store submission gets its own 0.2.1 build. Reverts #2052's MARKETING_VERSION 1.0 (iPhone only and the privacy manifests stay).

Verified project.yml: MARKETING_VERSION 0.2.0 for all three targets, TARGETED_DEVICE_FAMILY 1 unchanged.
2026-11-20 16:02:24 +00:00
Ryan Vogel 78e4458d4c iOS: version 1.0, iPhone only, privacy manifests (#2052)
* iOS: add privacy manifests for the app and keyboard extension

Neither HEX.app nor HEXKeyboard.appex shipped a PrivacyInfo.xcprivacy, which
App Store submission needs for required-reason APIs. Both now declare no
tracking, no tracking domains, no collected data, and the APIs Hex's own
code uses: UserDefaults (CA92.1 app-only, 1C8F.1 App Group), file
timestamps (C617.1, typing journal and typing log files in the App Group),
system boot time (35F9.1, CACurrentMediaTime for elapsed time), and active
keyboards (3EC4.1 custom keyboard, 54BD.1 layout choice via
UITextInputMode.activeInputModes).

Verified: plutil -lint passes; xcodegen picks both up as resources and a
Debug simulator build places them at HEX.app/PrivacyInfo.xcprivacy and
HEX.app/PlugIns/HEXKeyboard.appex/PrivacyInfo.xcprivacy.

* iOS: version 1.0 for the App Store, iPhone only

Ryan's App Store calls: the first release is 1.0 (the 0.2.0 TestFlight line; App Store Connect's version must equal CFBundleShortVersionString), and Hex Mobile is iPhone only for now. MARKETING_VERSION 0.2.0 -> 1.0 and TARGETED_DEVICE_FAMILY 1 for the app, keyboard and widgets; README notes both.

Verified a Release generic-iOS build: Info.plists read 1.0 and UIDeviceFamily [1], with both privacy manifests bundled.
2026-11-20 15:54:01 +00:00
Ryan Vogel cc75290551 iOS: emoji panel with on-device suggestions (Experiments › Emoji) (#2051)
* iOS: emoji index built from Unicode's emoji list and CLDR

What changed
- scripts/build-emoji-data.py builds HEXShared/FullKeyboard/Emoji/emoji-index-<lang>.txt
  for the four keyboard layouts (en, nb, es, de) from Unicode emoji-test.txt 18.0
  (pinned by SHA-256) and CLDR 49 annotations and annotationsDerived (release-49 tag),
  plus Hex's own chat words in scripts/emoji-aliases-en.txt ("congrats", "bday",
  "lol", "new year"; 336 words, written for Hex).
- Each file holds the catalog in panel order (8 categories, 1,923 emoji, 332 with five
  uniform skin tones), each emoji's English name, its folded search words, and the
  word and phrase postings already weighted (one-word name 1.0, name word 0.75,
  keyword 0.6, keyword-phrase word 0.3, chat word 1.2 down to 0.9, an IDF factor for
  words many emoji share), sorted by key so the keyboard can binary-search the file
  in place. Non-English layouts carry English's words at 0.85.
- LICENSE-Unicode.txt (Unicode License v3) ships beside the data, as the licence
  asks: both emoji-test.txt and CLDR's data are under it, which allows commercial
  use with the notice.

Why
- The emoji panel (next commits) suggests emoji for what you type and searches them
  on the iPhone. Precomputing the index keeps the keyboard extension's heap small:
  the first version built Swift dictionaries at runtime and held 5.7 MB, which
  malloc kept after release; the mapped file plus offsets measures 0.7 MB.

Verified
- The generator runs from a sources folder or downloads the pinned files; output is
  deterministic. Files: en 351 KB, nb 560 KB, es 549 KB, de 610 KB.

* iOS: rank emoji for the text before the cursor, on device

What changed
- EmojiLibrary maps an emoji-index file (Data .alwaysMapped) and keeps only the emoji,
  their skin tones and byte offsets: word and phrase lookups binary-search the file's
  sorted keys and parse postings in place; search scans each emoji's folded name and
  keyword words for prefixes (name words first, a short name the query fills before a
  long one it touches, then your use and everyday emoji). Emoji newer than 15.1 are
  listed only if the emoji font itself draws them as one glyph.
- EmojiSuggester reads the last eight content words (the newest counts 1, then 0.85
  each word back), their forms (plurals, -ing, -ed, held-down letters: "lmaooo"),
  two- to four-word phrases ("new year", "good luck") and the word still being typed;
  an emoji several words point at gains a third of the rest. One emoji per family
  ("man running", "woman running"); your use breaks ties and fills the row, then
  everyday defaults. forFinishedWord gives the strip at most three strong matches for
  the word just finished ("pizza " -> 🍕).
- EmojiUsageStore keeps, in the App Group, a use count per emoji that halves every 10
  days (frequency and recency in one score, at most 120 emoji) and the skin tone last
  picked for each; no words. clear() empties both.
- EmojiRoutedTarget lets the keys type into the emoji search's query instead of the
  field while you search.
- EmojiSemanticIndex (app target only, for the comparison test): Apple's on-device
  NLEmbedding word vectors over the index's 3,257 words, standing in for a context word
  with no emoji of its own. Not used by the keyboard (see Measured).
- EmojiSuggesterTests: index, folding, context, families, endings, other layouts,
  strip, search, history decay and tones, the search's routing, and quality on 60
  hand-written sentences (Fixtures/emoji-context-sentences.json, 30 to tune, 30 held
  out) with and without the embedding.

Measured (sensible emoji in the top 5, context only; top-1 in brackets)
- Simulator, iPhone 17, iOS 27.2: keywords dev 30/30 (27), held-out 30/30 (27), 0.04 ms
  a sentence; with the embedding the same counts at 0.14 ms, its index built in 120 ms.
- Mac harness, same code: CLDR keywords alone dev 26/30 (13), held-out 22/30 (14);
  with Hex's chat words 30/30 (27) and 30/30 (27). A blind set of 40 messages and
  sensible emoji written by a separate agent that saw no code or word lists: CLDR
  alone 30/40 (13), with chat words 37/40 (28), adding the embedding 37/40 (28); 38/40
  after the held-down-letter rule, which those failures suggested. Apple's sentence
  embedding scored 18/40 (7) and took 7.5 s to embed the 1,923 descriptions.
- Memory on the Mac: the mapped index adds 0.72 MB; the embedding adds 5.3 MB (1 MB
  NLEmbedding, 3.9 MB vectors) and none of it came back after release. No measurable
  gain for that, so the keyboard uses keywords and your history only.

Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures.

* iOS: emoji panel for the full keyboard (Experiments › Emoji)

What changed
- Experiments › Full keyboard › Emoji (experiments.emoji, off by default; shown with the
  full keyboard): an emoji key (face.smiling) right of 123, after the globe, in plain
  text fields (not URL, email or ASCII-only ones). The bottom row takes the compact
  sizes it uses beside the globe. "Clear emoji history" empties EmojiUsageStore.
- EmojiPanelView replaces the strip and the keys: a row of up to ten suggestions for
  the text before the cursor and your history (one emoji per family, your skin tone),
  a search button; a grid that scrolls sideways through Frequently used (your history,
  then everyday emoji, 16 at least) and the eight categories in four rows of 42 pt
  (three of 38 in landscape), each section's name kept at the left edge while it
  scrolls; a bar with ABC, the categories and a repeating delete, drawn like the
  letter keys' 123 and delete. Holding an emoji with skin tones opens the accent
  picker with its five tones; the pick is typed and remembered for that emoji. Key
  pop-ups and haptics follow their switches.
- Search: the letter keys type into a search field in the strip (EmojiRoutedTarget),
  results above it in a row the keyboard grows 44 pt for, your suggestions while it's
  empty; return, the field's close button or the emoji key go back to the panel.
  Typing logs and Learn how I type see the emoji typed into the field, never the query.
- The strip, when it would be empty after a space, offers the finished word's strong
  emoji ("pizza " -> 🍕, "birthday " -> 🎂 🎉 🥳), a lone one in the centre slot; never
  over word suggestions, Undo dictation, a status or the clipboard chips.
- Grid cells drop their text (and bitmap) when they scroll off and prefetching is off:
  scrolling every section had kept 353 cell bitmaps, 35 MB.
- FullKeyboardView+Emoji.swift holds the wiring; the engine's target is now the routed
  target, which forwards everything to the field unless a search is open.
- Scenes: full-keyboard-emoji, -emoji-search (--emoji-query), -emoji-tones,
  -emoji-food, -emoji-strip (each -dark and -dark-field), and -emoji-probe, which prints
  the panel's footprint and open latency.

Measured in the keyboard extension (Release, iPhone 17 simulator, iOS 27.2, a field
focused with the real HEXKeyboard.appex and a temporary, uncommitted probe; the
simulator counts no GPU memory and reports no headroom), footprint:
  keyboard with the emoji key, nothing loaded   19.0 MB
  after "pizza " (the strip loads the index)    19.0 MB
  panel open                                     28.7 MB
  picked three suggestions, closed               22.7 MB
  every section scrolled                         41.6 MB (peak 48.4; 71.1 before the cell fix)
  closed after that                              36.5 MB
The 17.5 MB left after browsing everything is about 351 emoji glyph images the system's
text rendering caches process-wide (18.7 MB in the heap, 351 CGImages); drawing with a
font instance the panel owns or at 21 pt left it the same, and no public API frees it.
Whether iOS purges it under memory pressure is unverified on an iPhone.
Opening: 22-26 ms from the emoji key to the panel laid out the first time, 12-14 ms
after (median of 10), about 21 ms until committed.

Verified: xcodebuild test (245 tests, 1 skipped, 0 failures); every scene above in
light, dark and dark field looked at on the simulator.

* iOS: document the emoji experiment and keep its blind sentences as a check

What changed
- README: the emoji experiment under Capabilities (what the panel, search, skin tones
  and strip do, what the history keeps, where the index comes from and its licence),
  how to rebuild the index, and its proof limits with the quality, memory and latency
  measurements and what is unverified on an iPhone (whether iOS purges the cached
  emoji glyph images that browsing every section leaves, about 18 MB).
- emoji-context-sentences.json gains the 40-message blind set a separate agent wrote
  without seeing code or word lists; EmojiSuggesterTests scores it beside the other
  60 and fails below 35/40.

Why: the blind set is the least biased quality number; keeping it catches a ranking
change that helps the hand-written sentences at its expense.

Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures; EMOJI_QUALITY: dev 30/30 (top-1 27), held-out 30/30 (27), blind 38/40 (29),
the same with the embedding, which built in 101 ms.
2026-11-20 15:53:24 +00:00
Ryan Vogel 81a36817b4 iOS keyboard: follow the keyboard's live traits for light and dark, not the proxy's appearance (#2050)
TestFlight build 29: in dark mode, switching to another app and back left the
full keyboard's keys in the light palette (white keys, black labels) over the
system's dark platter.

Cause: the palette took textDocumentProxy.keyboardAppearance first, and
updateKeyboardTraits() overrode the controller's interface style from it. The
host resolves a default field's appearance to .light or .dark when it sends
its text traits and doesn't send them again when only its style changes, so
the value goes stale. Reproduced on an iOS 27.2 simulator (iPhone 16 Pro Max,
Contacts' New Contact field): with the keyboard up, go home, the system
switches to light, return (the proxy now says .light), the system switches
back to dark. The keyboard window's traits went dark, and the platter with
them, but the palette stayed on the stale .light ("applyAppearance view=dark
win=dark appearance=light -> light"). In light mode the reverse also
happened: a stale .dark set overrideUserInterfaceStyle = .dark, which kept
both the full keyboard and the dictation bar dark on a light platter. On a
real iPhone the stale value most likely comes from the app switcher's light
and dark snapshots of the host. That device trigger is inferred; the
simulator reproduces the stale-value mechanism.

Change: FullKeyboardPalette.isDark(traits:) follows the trait collection
alone, and the controller no longer sets overrideUserInterfaceStyle.
Removed: the proxy's .light/.dark no longer decides the palette, and the
#904 override is gone. #904's behaviour stays: iOS itself gives the keyboard
window dark traits for a field that asks for a dark keyboard in a light app
(checked with --hosted-field dark-field: full keyboard and dictation bar
both dark).

Verified on the iOS 27.2 simulator with before and after screenshots: dark
mode round trip (dark keys after the fix), light mode round trip (light
keys), the dictation bar both ways, the dark field in a light app, and a
plain home and return. xcodebuild test: 247 tests, 0 failures, 1 skipped.
2026-11-20 15:53:21 +00:00
Ryan Vogel 178c16fdc5 iOS: speech model fallbacks for iPhone 11 and unavailable languages; no raw errors in the keyboard (#2048)
* iOS: fall back to DictationTranscriber when a speech model won't install, and never show Apple's error text

A TestFlight tester (iPhone 15, iOS 26.2, build 25, Russian) saw the full
keyboard's strip read "transcription.ru asset not found after attempted
download." Apple Speech lists some languages whose asset never arrives
(Apple's forum shows the same for Arabic), and Hex both stopped there and
showed Apple's sentence: UserFacingError (#2035) kept any description without
an error code in it, and this one has none.

- SpeechModelInstall (HEXShared, used by the app and in-keyboard dictation):
  install the chosen module's model, then the same language on
  DictationTranscriber when SpeechTranscriber's won't install, and go straight
  to DictationTranscriber for three days. When neither installs while online,
  Automatic sets the language aside for three days and moves to the next one
  ("Russian isn't available on this iPhone yet. Using English."); a language
  picked by hand says so and offers that one ("Try English", with Use English
  in the app). Offline or stalled, it says the download failed and keeps the
  language. Every message fits the full keyboard's strip.
- StallWatch: a download whose progress hasn't moved for 60 s is cancelled
  and treated as failed, instead of holding the progress bar forever.
- AppleSpeechAssets.reserve makes room under maximumReservedLocales by
  releasing other reservations first, and a reservation Hex can't get no
  longer fails preparation: the keyboard reserved every language it used and
  never released one, so the app's reserve could throw for good.
- UserFacingError is now an allowlist: only Hex's own errors
  (HexDescribedError) show their text, a lost connection while getting the
  model ready says so, and anything else gets the context's sentence. This
  replaces the marker filter, which let Apple's readable sentences through.
- DictationController re-prepares in the language Automatic moved to
  (at most twice), keeps a notice, and offers a language after an explicit
  choice fails. The dictation bar shows a failure on two smaller lines.
- Snapshot scenes: full-keyboard-language-unavailable[-dark],
  keyboard-language-unavailable, language-unavailable, language-switched.

Verified: xcodebuild build-for-testing on an iPhone 15 / iOS 27.2 simulator,
and the 44 DictationController, picker, SpeechModelInstall and StallWatch
tests pass (fallback, set-aside, offline, unsupported, stall, explicit offer,
Automatic switch, error text). Apple Speech itself can't run in the
simulator, so the install steps are fakes there; which module fails for
Russian on iOS 26.2 is unverified on hardware (no iOS 26.x simulator runtime).

* iOS: dictate English with Apple on iPhone 11, and stop the Parakeet relaunch loop

A TestFlight tester (iPhone 11, iOS 27.3, build 16) reported the model
download sticking at 50% and the app then crashing at every launch, with no
crash log. iPhone 11 (A13, 4 GB) has no SpeechTranscriber
(SpeechTranscriber.isAvailable is false on every A13 iPhone), so Hex used
Parakeet V2 for English. FluidAudio 0.15.5 reports downloads as 0-50% and
Core ML's compile as 50-100%, so 50% is MLModel(contentsOf:) compiling the
encoder, which can't be cancelled; iOS ends the app there. Once the files are
on disk, init saw hasCachedModel and loaded Parakeet again at once, so every
launch ended the same way. iOS ending an app for memory leaves no crash
report, which fits the empty crash list.

- AppleSpeechTranscriber.shouldUse: English without SpeechTranscriber uses
  DictationTranscriber where Parakeet doesn't fit (under 5 GiB of RAM: every
  A13 iPhone) on iOS 26+. Parakeet stays where it fits and before iOS 26.
- ParakeetLoadGuard: a load still marked as running at launch belonged to a
  process that ended, so the next launches don't load Parakeet by themselves
  until a load works; Hex says it closed while loading the model and waits
  for Set up. If Automatic now picks Apple Speech, it prepares that instead.
- Parakeet's download half runs under StallWatch (90 s without progress ends
  it); the compile half isn't watched since it can't be stopped. FluidAudio's
  download errors (Hugging Face paths and HTML in their text) become "Parakeet
  V2 didn't download. Check your connection and try again."

Verified: xcodebuild test on an iPhone 15 / iOS 27.2 simulator, 247 tests,
0 failures, 1 skipped (AppleSpeechClipTests, needs clips), including new
tests for shouldUse on a 4 GB iPhone and the load guard across launches.
Parakeet's memory use, the compile at 50% and the ended launches need a real
iPhone 11; the simulator neither limits memory nor runs SpeechTranscriber.
2026-11-20 15:04:43 +00:00
Ryan Vogel 8b8b6fc0ef iOS onboarding: say 100% on device on the welcome page (#2047)
A tester said it isn't clear that Hex runs on device, its main feature. The welcome page now shows a '100% on device' pill under the Hex mark, and the second point reads 'Free, and nothing leaves your iPhone' (was 'Free and private, on this iPhone').

Verified HEXMobileTests on the Mac Studio and a simulator render of onboarding-welcome.
2026-11-20 14:45:20 +00:00
Ryan Vogel 1ede7bb792 iOS: Spanish clean-up, paste translated or as spoken, screen stays on while recording (#2046)
* iOS: keep the screen on while Hex records or transcribes

A Spanish-speaking TestFlight tester (build 20) asked that the screen not
turn off while recording. DictationController now sets
UIApplication.isIdleTimerDisabled whenever its phase is recording or
transcribing, and clears it when the phase leaves those states (done,
cancel, failure), so Auto-Lock comes back after every dictation. Nothing
else in Hex sets the flag.

Limits: iOS honors the flag only while Hex is the foreground app (the
handoff screen and in-app recording). A warm-session or in-keyboard
dictation happens with another app in front, and a keyboard extension has
no public API to keep the screen on, so those can still dim. No private API
or value(forKey:) workaround is used.

Verified: new DictationControllerTests.testRecordingAndTranscribingKeepTheScreenOn
(recording, transcribing, done, cancel, failed) and the rest of
DictationControllerTests pass on an iPhone 18 Pro simulator (iOS 27.2).

* iOS: Spanish, German and Norwegian rules for Clean up what I say

A Spanish-speaking TestFlight tester said the corrections part works badly
in Spanish. Clean up what I say ran its cleanup tagger only for English and
returned every other language untouched, so with the experiment on a Spanish
dictation kept its fillers and spoken corrections. The tagger stays
English-only: run directly on Spanish, it changed the meaning of 8 of 18
sentences, including dropped negations ("Te digo que no puedo ir." became
"Te puedo ir.", "O sea que al final no vienes." became "O sea que vienes.").

- New SpokenCleanupRules: for es, de and nb/nn/no, a filler between commas
  goes ("este", "o sea"), and a correction marker set off by commas ("digo",
  "perdón", "no", "no, espera", "mejor dicho"; "nein", "ich meine"; "nei",
  "nei vent", "jeg mener") drops the words it replaces. Those are counted
  back to the nearest earlier word that matches the repair's first word, or
  is of its kind (number, weekday, month, today or tomorrow), within the
  sentence and 8 words; with neither, nothing changes. "mejor dicho", "no
  mejor", "besser gesagt" and "rettere sagt" also count without commas.
  Like the tagger, it only removes words; CleanupReconstruction repairs
  commas and capitals.
- CleanupTagger.cleanIfEnabled runs those rules for non-English languages
  instead of returning the text unchanged. The model still never loads for
  them, and Verbatim still skips cleanup.
- TranscriptFormatter: the styles trim Spanish hesitations (eh, em, mmm;
  never "e") and German ones (äh, ähm, öhm; never "eh"), and capitalize a
  sentence that opens with ¿ or ¡.

Measured on 25 Spanish sentences I wrote, with Polished and the experiment
on (9 German and Norwegian ones too). Before, every sentence came out as
spoken. After, all 13 with fillers or corrections are cleaned ("Eh, nos
vemos mañana a las 3, digo, a las 4." -> "Nos vemos mañana a las 4.") and
all 12 that use the same words in their ordinary meaning stay unchanged
("Te digo que no puedo ir.", "Quiero este, el azul.", "Lo siento, perdón,
Ana.").

Verified: new SpokenCleanupRulesTests, updated
CleanupTaggerTests.testOtherLanguagesGetRulesNeverTheEnglishTagger (it
used to expect Norwegian "tirsdag, nei vent, torsdag" untouched), plus the
CleanupTagger, DictationController and TranscriptCleanup suites, all
passing on an iPhone 18 Pro simulator (iOS 27.2).

* iOS: Clean up what I say drops Spanish, German and Norwegian hesitations

In-keyboard dictation (iOS 26+) inserts Apple Speech's text without running
a style through TranscriptFormatter, so the Spanish and German hesitation
trim added to the styles in 71acb4f3 never reached it. The English tagger
removes "um" and "uh" in both places. SpokenCleanupRules now also drops the
language's hesitation sounds anywhere in the sentence, using the styles'
own list (TranscriptFormatter.fillerPattern): eh, em, mmm in Spanish; äh,
ähm, öhm in German; eh, ehm, øh, hm in Norwegian. A sound that asks or
exclaims ("¿eh?", "eh?", "¡Eh, tú!") stays.

Correction to 71acb4f3's message: on the 25 Spanish sentences, 14 (not 13)
came out cleaned and 11 (not 12) stayed unchanged; nothing else differs.

Verified: SpokenCleanupRulesTests (with "Mmm, creo que sí, eh, pero..." ->
"Creo que sí, pero...", "No te olvides, ¿eh?" and "Padre e hijo." kept,
Norwegian "Eh," and German "Äh," removed, German "eh egal" kept) and
CleanupTaggerTests pass on an iPhone 18 Pro simulator (iOS 27.2).

* iOS: choose per dictation whether Translate what I say pastes translated

A Spanish-speaking TestFlight tester liked Translate what I say but always
speaks Spanish and only sometimes wants the English. Until now translation
was all or nothing, set in Experiments.

- DictationTranslationSettings.pastesTranslation (App Group key
  experiments.translateDictation.pastesTranslation, default true, so
  existing users still get translations) is a switch that keeps the last
  choice. translateIfEnabled skips translation while it says as spoken, in
  the app (warm session, handoff, Action Button) and in in-keyboard
  dictation alike.
- TranslationPasteChoice names both sides in English, like the rest of
  Hex's copy: "Paste in English" / "Paste in Spanish" ("Paste as spoken"
  when the dictation language is unknown, "Norwegian" without "Bokmål").
  It exists only while the experiment is on, the keyboard has Full Access,
  and you dictate in a language other than the target.
- Full keyboard: the Hex key menu gets two rows under Language, each with a
  checkmark, and the space bar reads "Español → English" while dictations
  paste translated.
- Dictation bar: the language tray gets a "Paste in English" chip, lit
  while translating, and the language label reads "Español → English". The
  label now has layout priority so it isn't cut to "Español → E…".
- Settings: a "Paste in English" toggle under Insert in mirrors the switch.
  Turning the experiment on resets it to translated.
- Simulator fixtures: --translate-to <code> and --paste-as-spoken.

Verified: new TranslationPasteSwitchTests (titles and visibility, the
remembered choice, as spoken skipping translation, the model's label, and
the Hex key menu's rows, their checkmarks and their callback). Full
xcodebuild test: 230 tests, 0 failures, 1 skipped (AppleSpeechClipTests).
Rendered on an iPhone 18 Pro simulator (iOS 27.2): full-keyboard-style in
light and dark, translated and as spoken, without translation (unchanged),
full-keyboard-idle, keyboard-language, keyboard-warm, keyboard-recording and
experiments. The simulator can't actually translate (Apple's
LanguageAvailability says es -> en is unsupported there), so a translated
paste is unverified on device.
2026-11-20 14:39:28 +00:00
Ryan Vogel 6ab53fff1e iOS keyboard: open Hex instead of being killed when short of memory (#2045)
* iOS: open Hex instead of dictating in the keyboard when it is short of memory

Testers on 4 GB iPhones (12, 12 mini, 13) saw in-keyboard dictation in other
apps stop about a second after tapping Dictate and iOS switch to the system
keyboard: iOS ends a keyboard extension that passes its memory limit (Apple:
"The memory limits vary from model to model"; reported at ~60-70 MB by
KeyboardKit, ActiveHard 77 MB by fcitx5-ios, 90 MB on a 4 GB iPad by
dictus-ios #202). Starting creates the capture web view, loads its page, starts
WebKit's capture and mounts the recording screen's Metal scope all at once.

What changed
- KeyboardLocalDictation.hasMemoryToStart: os_proc_available_memory() must
  leave 32 MB for the first dictation (web view, page, WebKit's first use,
  recording screen, Metal device and pipeline) and 16 MB once the web view is
  loaded (capture and recording screen only). 0 (the simulator) is enough.
- KeyboardViewController.syncLocalDictationAvailability, on every bridge poll:
  while short, viewModel.onLocalDictateTap is nil, so Dictate, the bar's Hex
  mark and the full keyboard's Hex key take their existing cold start (open
  Hex, record there, return), as before iOS 26. A hold stays a tap there, so
  push to talk sends no stray stop. With room, nothing changes. The probe log
  notes each switch, at most every 5 s.
- A web view whose WebContent process ended now also leaves the keyboard's
  view; before, it stayed behind (2 pt, alpha 0.02) next to the new one each
  later dictation made.
- KeyboardProbeLog.memory() adds the process's peak footprint
  (ledger_phys_footprint_peak), so a tester's keyboard-capture-probe.log shows
  what the limit saw: [footprint, peak, headroom].
- README: the guard, and its proof limits.

Tried and left out, after measuring
- Releasing the capture web view after each dictation (and on memory
  warnings): the keyboard's footprint stayed (31.9 -> 31.8 MB; WebKit/JSC state
  and allocator pages stay in the process, malloc_zone_pressure_relief returned
  16 KB) while the next start's spike grew (+1.9 vs +0.9 MB) and it waited for
  the page again (~1.5 s). The web view stays loaded, as before.
- Dropping the text checker or lexicon while dictating: the full keyboard rests
  lower than the dictation bar in the simulator (18.3-21.0 vs 21.4-24.3 MB), so
  nothing showed it helps.

Measured (simulator iPhone 13, iOS 27.2, Release; a temporary, uncommitted
probe drove the keyboard's own Dictate and Done paths with a host field
focused, logged phys_footprint at each step and every 100 ms, with footprint(1)
sampling from outside; 3 runs each, MB):
                      bar before  bar after   full before full after
  keyboard shown      21.6-24.2   21.4-24.3   18.5-21.0   18.3-20.9
  web view created    25.1-27.7   24.9-27.9   22.1-24.7   22.1-24.7
  web view loaded     27.5-30.0   27.5-30.5   25.5-28.7   25.8-28.6
  capture started     28.5-30.8   28.1-31.3   26.7-29.6   26.8-29.4
  max 3 s after tap   28.8-31.2   28.5-31.7   28.9-31.4   28.9-32.1
  stop                27.8-31.8   27.7-31.0   26.0-28.6   26.1-28.6
  transcriber made    27.9-31.8   27.8-31.0   26.1-28.7   26.2-28.8
  lifetime peak       31.5-32.4   28.9-32.3   29.8-32.6   30.0-32.6
A second dictation adds 0.5-1.3 MB (bar) and 4.2-4.8 MB (full keyboard, which
rebuilds its recording panel). With a simulated limit 24 MB above the footprint
at the tap, both keyboards now hand off and stay at 21.2 / 18.3 MB instead of
rising 7-13 MB; with 40 MB they dictate in the keyboard, twice. The simulator
counts no GPU memory and transcription fails there ("Audio format is not
supported"); a macOS stand-in put the Metal device and pipeline at ~3.5 MB and
the scope's drawables at ~4.4 MB, which the thresholds allow for.

Verified: xcodebuild test on the iPhone 18 Pro simulator, 212 tests pass (1 skipped as
before);
the Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds. Not
verified on a physical iPhone: the real limit, GPU memory, and whether 32/16 MB
is the right line; the probe log's peak and headroom will show it.

* iOS keyboard: need 24 MB, not 16, before a second in-keyboard dictation

Leaving the full keyboard's new recording strip briefly peaked 7 MB above recording (35.5 MB in a simulated iPhone 13), and a second dictation now costs up to 8 MB, so 16 MB left little margin.

Compile-checked with a generic-iOS build.
2026-11-20 14:33:16 +00:00
Ryan Vogel 550eccd970 iOS onboarding: say 'Allow Full Access' in the Keyboard row itself (#2044)
A tester said onboarding never told them to Allow Full Access; the Settings path was small muted text below the card. The Keyboard row now carries 'Add Hex and Allow Full Access' under its title.

Verified HEXMobileTests and a simulator render of onboarding-allow.
2026-11-20 14:12:19 +00:00
Ryan Vogel 42e6e8e12d iOS: remove the return-to-app feature; swipe back after a handoff (#2042)
Ryan decided Hex Mobile won't open other apps' URL schemes after the
keyboard hands off to the Hex app: no return-to-app tricks, since they
are an App Review risk and only worked for apps with a known scheme.

Removed:
- HostApp / HostAppCatalog (the curated scheme list) and the bridge's
  returnAppID / quickReturnApps settings. The stored App Group keys
  (settings.returnAppID, settings.quickReturnAppIDs) are just ignored;
  no migration.
- KeyboardLaunchRequest's returnAppID and the URL's return= item. An
  older keyboard that still sends return= parses fine and it is ignored.
- DictationController's auto-return: AutoReturnState, the URL opener,
  setReturnAppID, setQuickReturnApps, returnToHostApp,
  maybeAutoReturnAfterActivation, triggerAutoReturn, the ~1 s fallback
  task, noteSceneDidEnterBackground, and the attemptAutoReturn flags.
- The app's Settings "Return to" row, its "Edit apps…" sheet
  (ReturnAppsSheet.swift), and the handoff screen's quick-jump app chips.
- The dictation bar's return chip and its app tray, and the full
  keyboard Hex key menu's "Return to" row and page (with its
  return_app typing-log event).
- The handoff-returning snapshot scene.

Replaced by: the handoff screen's existing quiet "Swipe bar to return"
cue over the home indicator, now shown whenever it records or a warm
session is on (it used to hide while an auto-return was attempted).
The rest of the handoff and warm-session flow is unchanged.

Docs: ios/README.md (cold start, Hex key menu, proof limits) and
ios/docs/typing-logs.md.

Verified: xcodebuild test on an iPhone 18 Pro simulator (iOS 27.2):
220 tests, 1 skipped, 0 failures. The handoff test now checks the 10 s
window alone, and the launch URL test checks that a legacy return= is
ignored. Rendered --snap-state setup, warm, handoff-listening,
handoff-no-home-bar, keyboard-cold and full-keyboard-style and checked
them: no Return to row, chips or return trigger; the swipe cue shows on
the handoff screen.
2026-11-20 13:58:31 +00:00
Ryan Vogel deb84b6082 iOS: no hyphens from autocorrect, completions ranked by frequency (#2041)
* iOS keyboard: never autocorrect into a hyphen; split a missed space into two words

What: AutocorrectEngine now passes every UITextChecker guess through
usableGuess before autocorrect or the strip uses it. A guess may add no
character you didn't type except letters and a contraction's apostrophe,
so "t.b." for "tb" or "x-y" for "xy" is never applied or shown. A guess
that splits the word in two ("up-to" or "up to" for "upto") is used only
as two words with a space, and only when its letters are exactly the
typed ones (a missed space), both halves are dictionary words (two or
more letters, or English "a"/"I", since UITextChecker accepts every
single letter), and the checker offers the spaced form itself.

Why: Record typing logs showed a missed space corrected into a
hyphenated compound. The old loop skipped spaced guesses but let the
hyphenated one through as a cheap one-character insertion. Removed: that
hyphen path; replaced by the spaced form under the rules above.

Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2) for
AutocorrectEngineTests (4 new tests: missed space, the split rules,
punctuation, and the system dictionary's "nevermind" -> "never mind"),
FullKeyboardEngineTests, TypingLogTests, DictationLearningTests: 40
tests, 0 failures.

* iOS keyboard: rank completions by frequency, names last, no repeated literal

What: the suggestion strip's completions now keep UITextChecker's own
order, which is most probable first (Apple's documented order; on iOS 27
"b" lists but, be, because and "th" lists the, they, that's), instead of
re-sorting it by length and then alphabetically. UILexicon terms
(Contacts names and text replacements) now rank after the dictionary's
completions instead of ahead of everything, and a capitalised name ranks
after every other completion unless the typed word starts with a
capital. A completion equal to what was typed is dropped, one-letter
words get completions too, and when nothing is left the strip shows only
the typed word on the left instead of repeating it in the centre.

Why: Record typing logs showed contact names and job titles from the
lexicon taking the centre slot for common prefixes (a surname over "the"
for "th"), and one-letter words repeating the literal as the primary.
Removed: the lexicon-first pass (up to three terms before any spelling
guess) and the length/alphabetical sort. The frequency source is the
existing UITextChecker; no word list was added.

Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2),
AutocorrectEngineTests: 16 tests, 0 failures (4 new: frequency order
ahead of contacts, names demoted unless capitalised, no repeated
literal, and the system dictionary's order for "b" and "th").
2026-11-20 13:49:49 +00:00
Ryan Vogel f82adb826f iOS: drop the recording dot and timer so the Dot Scope fills the recording bar (#2040)
Ryan wanted only the scope and the two buttons in the full keyboard's recording bar. Removed the red dot, the clock label, the palette's recordingDot and the clock subscription that only refreshed the timer; the scope well now starts at the bar's left edge. The dictation-bar keyboard keeps its timer.

Verified HEXMobileTests and simulator renders of recording and transcribing in light and dark.
2026-11-20 13:48:30 +00:00
Ryan Vogel 15965d5625 iOS: end the recording bar with round cancel and check buttons, the check becoming the spinner (#2039)
Ryan found the cancel button, the 'Done' text button and the separate transcribing label confusing. The bar now ends in two equal round buttons, ✕ and ✓; while transcribing the ✓ is replaced by a spinner in the same place, ✕ still cancels, and the clock stays a clock.

Verified HEXMobileTests and simulator renders of recording and transcribing in light and dark.
2026-11-20 13:45:09 +00:00
Ryan Vogel 0aa6fb8fea iOS: grow the keyboard for a taller recording bar and show one transcribing status (#2038)
Ryan found the strip-height recording bar too thin and the Transcribing label plus 'Inserting…' on Done confusing. While dictating the full keyboard grows 16 pt through preferredHeight and a new onPreferredHeightChange the controller applies; the keys shift down by the lift the view actually has, so they stay put on screen, and the bar gets 60 pt. While transcribing the clock reads 'Transcribing…' and Done shows a spinner.

Verified HEXMobileTests and simulator renders of typing, recording and transcribing in light and dark.
2026-11-20 13:41:32 +00:00
Ryan Vogel 33727933b0 iOS: keep dictation in the suggestion strip and leave the keys in place (#2037)
Ryan wanted dictation to use only the top bar, not cover the keyboard. The recording panel now has the strip's frame and lays out the clock, the Dot Scope well, and capsule cancel and Done buttons in one row; only the strip and the Hex mark fade out, so the keys stay visible. The style chip is dropped from the recording bar for room; the Hex key's menu still sets the style.

Verified simulator renders of the recording and transcribing states in light and dark, and HEXMobileTests.
2026-11-20 13:35:48 +00:00
Ryan Vogel 221b6a45fc iOS: ask for the microphone before sending setup to Settings (#2036)
A tester found neither Keyboards nor Microphone on Hex's Settings page from onboarding page 2. iOS lists Microphone only after the app has asked, so page 2 puts Microphone first and the Keyboard row's Open Settings asks first when needed. A 'Don't see Keyboards?' link shows the General › Keyboard › Keyboards › Add New Keyboard path for when iOS hasn't listed Keyboards on Hex's page yet.

Verified HEXMobileTests and a simulator render of page 2.
2026-11-20 13:27:33 +00:00
Ryan Vogel 9d26cbdb87 iOS: fix six TestFlight tester issues (audio error, leaked error text, Live Activity, haptics, capitals, push to talk) (#2035)
* iOS: retry '!int' audio activation mixable and show audio errors as sentences

TestFlight testers saw "The operation couldn't be completed. (OSStatus error
560557684.)" in the keyboard strip and the app's setup card. 560557684 is
'!int' (AVAudioSession cannotInterruptOthers): a non-mixable session (the
Bluetooth-microphone .record category) going active from the background while
another app plays or records.

- AudioBackend: when setActive(true) fails with '!int', set .playAndRecord
  with .mixWithOthers (keeping HFP when Bluetooth is allowed) and activate
  once more, so the warm keepalive and captures start alongside the other
  app's audio instead of failing.
- New UserFacingError (HEXShared, app and keyboard): AVAudioSession codes map
  to short sentences ("Another app is using audio. Try again."); other
  microphone errors get "Hex couldn't start the microphone. Try again.";
  transcription and model errors keep their description only when it reads
  as a sentence, never Foundation's "couldn't be completed (… error N)" or a
  Swift type name.
- DictationController: every error it shows (phase .failed and the
  keyboard's message) goes through UserFacingError instead of
  localizedDescription.

Verified: xcodebuild test on the iPhone 18 Pro simulator, 179 tests pass,
including new tests for the code-to-sentence mapping and for a warm session
whose keepalive throws '!int' reaching the app and keyboard snapshot as
"Another app is using audio. Try again.". The background '!int' retry itself
needs a physical iPhone (music playing, Bluetooth mic allowed, warm session).

* iOS: end empty dictations quietly and keep error text human in the keyboard

Tapping Done (or ✕) in the keyboard's dictation screen without speaking
showed "Transcription failed: HEXKeyboard.(unknown context at $1…)": the
in-keyboard path printed the error's domain (a private Swift type name),
code and description for Apple Speech's "no words" result.

- New EmptyCapture rule (HEXShared): a capture under 0.3 s (or none at all)
  is a tap, and an engine's no-speech error or a cancellation means nothing
  was said; both end quietly. Engines say so through NoSpeechError
  (Parakeet's and Apple Speech's emptyTranscript, the keyboard's
  KeyboardSpeech.Failure.empty).
- KeyboardLocalDictation: Done before the microphone came up, or a capture
  too short to hold a word, returns to idle without transcribing (and a late
  "started" from WebKit stops the capture instead of opening a file);
  no-speech ends quietly; a transcription error from a cancelled or
  superseded job is dropped; every other failure shows a sentence from
  UserFacingError or the keyboard's own copy, while the probe log keeps the
  domain and code. KeyboardSpeech.Failure's descriptions are now sentences.
- DictationController (warm session and app): an engine's no-speech error
  returns to Ready with no message instead of failing.

Verified: xcodebuild test on the iPhone 18 Pro simulator after rebasing onto
origin/dev (bba07c64), 210 tests pass (1 skipped as before), including new
tests for the empty-capture rule and for a warm keyboard dictation whose
transcriber hears nothing (Ready, no message) next to one that fails (a
sentence). The in-keyboard WebKit capture path needs a physical iPhone.

* iOS: start Live Activities only for system dictations and end them reliably

Testers saw Hex's activity (dots + ✓, expanded "Copied") stay in the
Dynamic Island, even with Keep mic warm Off. The ✓/Copied look is the
widget's stale rendering of an activity nobody ended: every recording
(keyboard warm dictations and in-app ones included) requested an activity,
and the end ran in a detached task that a suspending app (microphone just
released) could skip, so it went stale and stayed for hours.

- DictationController requests an activity only for dictations that copy
  (Action Button, Control Center, Siri/Shortcuts); the keyboard and the app
  show their own recording UI. Removed: the activity for keyboard and in-app
  dictations.
- LiveActivityCoordinator: ending (cancel, failure, short capture, warm
  session cancel) ends every Hex activity there is at that moment with
  .immediate; Copied still lingers 2 s via .after. Ends run under a UIKit
  background task, and a dictation that is transcribing keeps one until its
  activity ends. A current activity the system already ended is replaced
  rather than updated.
- Stale activities end at launch (DictationController.shared init), when Hex
  becomes active, and when the warm session ends or Keep mic warm is set to
  Off, unless a dictation is recording or transcribing.
- README: when Live Activities show and end.

Verified: xcodebuild test on the iPhone 18 Pro simulator, 210 tests pass
(1 skipped as before). A temporary, uncommitted test on the simulator showed
no activity for a warm keyboard dictation, one for a copy dictation that
ended after Copied, one ended at once on cancel, and an orphan activity
ended by Keep mic warm › Off. The Dynamic Island itself, and background
suspension timing, need a physical iPhone.

* iOS: give the dictation bar's keys the full keyboard's light haptic

The full keyboard taps under the finger (Experiments › Haptics, with Full
Access); the dictation bar's keys had no haptic at all.

- KeyboardViewModel gains onKeyFeedback; every control on the dictation bar
  calls it: Hex mark, Dictate / Set up, ✕ and Done, Swipe and its return-app
  chips, the style and language triggers and chips, the tray's close button,
  space, return, and delete (each repeat, as the full keyboard's delete).
- KeyboardViewController plays a light impact at the full keyboard's 0.65
  intensity, under the same rule the full keyboard now shares
  (allowsKeyHaptics: Full Access and FullKeyboardSettings().haptics).
- Experiments: the Haptics switch moves out of the Full keyboard group so it
  shows whether or not the full keyboard is on; same key
  (settings.keyboardHaptics), subtitle now says it covers Hex's keys.

Verified: xcodebuild test on the iPhone 18 Pro simulator, 210 tests pass (1
skipped as before); Experiments rendered with --snap-state experiments shows
Haptics after Keyboard layout, outside the Full keyboard rows. The haptic
itself (and its Full Access gate) needs a physical iPhone; the simulator has
no Taptic Engine.

* iOS: lowercase a dictation that continues a sentence

A tester typed a few words, dictated, and the dictation started with a
capital as if it began a sentence: speech engines capitalize every result.

- TranscriptFormatter.startsSentence(after:): the text before the cursor ends
  a sentence when it is empty or only spaces, or its last mark is . ! ? … 。
  or a line break (closing quotes and brackets after them count too).
- TranscriptFormatter.continuingSentence: mid-sentence, the first letter goes
  lowercase unless the first word is "I" or an I' contraction, all capitals
  (2+ letters), has capitals inside (McDonald's), or its lowercase form isn't
  in the dictation language's dictionary (likely a name).
- insertionText takes that dictionary (isWord; nil keeps the case), never
  adds a second space after one already there (spaces at the start of the
  dictation are dropped), and adds none after an opening ( [ { " “ ‘ « ¿ ¡.
- KeyboardViewController (the one place both keyboards insert a dictation)
  passes a UITextChecker for the dictation language (Apple Speech's last
  resolved language, else Settings' or Automatic's, else the first
  preferred language), mapped by KeyboardLanguage.spellCheckLanguage(
  forDictation:available:). German and Luxembourgish keep the case: on the
  simulator de_DE accepts "auto" and "haus", so the name guard can't protect
  their nouns. A language with no dictionary keeps the case too.
- README: the rule.

Verified: xcodebuild test on the iPhone 18 Pro simulator, 213 tests pass (1
skipped as before), with new tests for the sentence-start rule, the
exceptions, spacing, and the language-to-dictionary mapping. A temporary,
uncommitted test on the simulator's real UITextChecker found "sarah" and
"london" unknown and "the"/"that's" known in en_US, es and de_DE. How each
host app reports documentContextBeforeInput is unverified on a device.

* iOS: push to talk on the Hex key and the dictation bar's Dictate

A tester asked to hold to talk instead of tapping Dictate and then Done.

Rule: holding still for 0.35 s (HexKeyGesture.holdDuration, the Hex key's
existing hold) starts dictation at once, and lifting the finger finishes it
as Done does. A quick tap keeps tap to start, Done to finish. On the Hex key,
sliding more than 10 pt before the hold fires still opens the style menu;
once dictation has started, moving changes nothing. Push to talk applies
only when dictation runs in the keyboard (warm session or in-keyboard
dictation); a hold that opens Hex stays a tap.

- HexKeyGesture: a fired hold's touch-up is now .release (was ignored).
  Changed: before, a hold started dictation and the release did nothing.
- HexKeyControl calls onHoldRelease; FullKeyboardView finishes the dictation
  the hold started (model.onStopTap, logged as dictation_stop from
  hex_key_release). The recording fade keeps a still-tracking Hex key in
  the window at alpha 0 instead of hiding it, so its touch keeps arriving.
- KeyboardSurfaceView: the well tracks the press with a simultaneous
  DragGesture on the same state machine (the Dictate button stops taking
  touches once recording shows), starts with the light haptic
  (onKeyFeedback: Haptics setting and Full Access), and finishes on release;
  the button's own touch-up action skips a press the hold took.
- Accessibility hints mention holding; README and docs/typing-logs.md
  describe it.

Verified: xcodebuild test on the iPhone 18 Pro simulator, 213 tests pass (1
skipped as before), including the updated HexKeyGesture tests (hold then
release finishes; moving after the hold is ignored). The dictation bar
renders as before (--snap-state keyboard-warm). Real touches (hold timing,
the bar's buttons with the well's gesture, the Hex key's touch surviving the
recording panel) were not driven on the simulator and need a physical
iPhone.
2026-11-20 13:21:16 +00:00
Ryan Vogel bb5875cea8 iOS: Norwegian, Spanish and German full-keyboard layouts
Testers asked for the Norwegian layout and, with it, Spanish and
German. The layout follows the dictation language unless set.

What changed:
- KeyboardLanguage holds each layout's letter rows, long-press accents
  and spelling dictionary: Norsk (a-ring ending row 1, o-slash and ae
  row 2), Espanol (n-tilde ending row 2), Deutsch (QWERTZ, u/o/a-umlaut,
  sharp s on a held S), as Apple lays them out. Adding a language is a
  new case with those three filled in.
- KeyboardLayout takes the language. Rows of 11 keys narrow only the
  character keys; shift, delete and the bottom row keep their English
  (10-key) sizes, and row 3's letters sit centred between shift and
  delete. letterWidth stays the 10-key width; characterWidth is new.
- Experiments > Full keyboard > Keyboard layout: Automatic, English,
  Norsk, Espanol, Deutsch (settings.keyboardLayout). Automatic follows
  the dictation language (no layout yet for Hebrew and others: English),
  else the first iPhone keyboard or preferred language with a layout. A
  pick in the keyboard's language switch changes the layout at once.
- Autocorrect uses the layout's UITextChecker dictionary (nb, es, de);
  without one it offers nothing but your text replacements instead of
  correcting into English. The English contractions and typo list
  (i -> I) apply only to English; neighbouring-key costs follow the
  layout. LanguageSettings.effectiveDictationLanguage now prefers what
  Automatic picks over the last language dictation ran in.
- Typing logs record the layout's language; the snapshot stage takes
  --keyboard-layout, follows --dictation-language and --accent-key.
- TypingLogTests pins English, since FullKeyboardView now reads the
  App Group's dictation language.

Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2):
206 tests, 0 failures, 1 skipped (the Apple Speech clip test), with
layout tests for every language at 320-440 pt and landscape (rows,
11-key widths, hit-frame tiling, balloons, accent menus) and autocorrect
tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO succeeds;
light and dark screenshots of each layout, its accents, shift, the
settings row and the dictation bar's re-shot trigger checked by eye. The
simulator has dictionaries for nb, es and de. Accent lists are from
Apple's layouts as remembered, not checked on a device; the 123 and #+=
pages are English's in every language.
2026-11-20 07:52:48 -05:00
Ryan Vogel 2b225b390b iOS: switch the dictation language from the keyboard
Testers asked to change the dictation language without opening Hex.

What changed:
- Dictation bar: a language trigger beside the style ("Norsk") opens a
  tray of chips: Auto plus up to five languages.
- Full keyboard: the Hex key menu gets a Language row and page with the
  same choices, and the space bar names the dictation language faintly
  ("English", "Norsk"), like Apple's.
- DictationLanguageSwitch picks the choices (the language set, recent
  picks, then the iPhone's preferred languages, each matched to a locale
  Apple Speech has; Nynorsk maps to Bokmal) and names each in its own
  language, adding a region code only where two share a language.
- LanguageSettings keeps recent picks, what Automatic resolves to, and
  Apple's language list (cached when Hex or the keyboard lists it); a
  pick in the keyboard writes the same settings.dictationLanguage as
  Hex's menu. In-keyboard dictation reads it per dictation; a warm Hex
  session follows it in its 100 ms loop and prepares the language
  between dictations; a keyboard handoff and the Action Button pick it
  up before recording.
- Typing logs record the dictation language setting and a
  dictation_language UI event.

Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2):
195 tests, 0 failures, 1 skipped (the Apple Speech clip test, which
needs HEX_SPEECH_CLIPS), including new switch, recents, follow and
space-bar tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO
succeeds; screenshots of the Hex menu's language page (light, dark),
the space bar, the dictation bar's tray and the settings row checked by
eye. The trigger's last change (chevron instead of a globe, which looked
like the next-keyboard key) was compiled, not re-shot. Not tried on a
physical iPhone.
2026-11-20 07:52:38 -05:00
Ryan Vogel ee34feb49b iOS: dictate in every language Apple Speech has on device
A Norwegian TestFlight tester's iPhone silently fell back to English:
Hex picked the first preferred language SpeechTranscriber supports, and
SpeechTranscriber has no Norwegian, Danish, Swedish or Hebrew. Testers
also asked for Spanish and German.

What changed:
- SpeechLanguagePicker picks the dictation language: an explicit choice
  wins (SpeechTranscriber where it has the language, else
  DictationTranscriber); Automatic takes the first preferred iPhone
  language either module has, SpeechTranscriber first for the same
  language, then the region's locale, then en-US. Nynorsk tries Bokmal.
- AppleSpeechModule wraps SpeechTranscriber and DictationTranscriber
  (preset longDictation) for the same SpeechAnalyzer flow and asset
  download; the app's AppleSpeechTranscriber and in-keyboard dictation
  both use it and AppleSpeechLanguages, so they resolve the same language
  and share downloaded assets.
- Settings > Dictation language (iOS 26): Automatic plus every locale
  either module supports, by name, saved in the App Group
  (settings.dictationLanguage). Picking one prepares its model now, or
  after the dictation in flight. The setup card's Speech model row names
  the real language ("Norwegian Bokmal · Apple").
- English-only steps follow the language: the cleanup tagger skips other
  languages; Apple Intelligence cleanup runs only when the model supports
  the language; filler trimming is per language (English "er" was eating
  Norwegian "er" = "is"; Portuguese "um" is a word); translation's source
  is the dictation language instead of a guess.
- Replaced: AppleSpeechTranscriber.preferredLocale()/isAvailable with
  preferredLanguage()/shouldUse(_:). An iPhone without SpeechTranscriber
  still dictates English with Parakeet, as before.

Verified: xcodebuild test on the iPhone 18 Pro simulator (iOS 27.2),
189 tests, 0 failures, including new picker, language-change, filler and
tagger tests; Release generic-iOS build with CODE_SIGNING_ALLOWED=NO
succeeds. The simulator can't transcribe (SpeechTranscriber unavailable,
DictationTranscriber "Model not available"); the same Swift files built
for macOS 26 transcribed say clips word for word: de-DE and es-ES via
SpeechTranscriber, nb-NO and he-IL via DictationTranscriber (no
punctuation from DictationTranscriber under any preset). Not tried on a
physical iPhone.
2026-11-20 07:52:32 -05:00
Ryan Vogel d21699b896 iOS: load the cleanup model after recording and only with memory to spare in the keyboard (#2033)
A tester's iPhone 13 Pro dropped back to the system keyboard as soon as dictation started with Clean up what I say on: the keyboard loaded the Core ML tagger on the CPU and Neural Engine at the start of recording, alongside the microphone, capture web view and speech recognizer, and iOS ended the extension. The keyboard now loads it when recording stops, runs it CPU-only in the extension (weights stay memory-mapped), and skips cleanup when os_proc_available_memory is under 80 MB.

Verified HEXMobileTests on the iPhone 18 Pro simulator; not reproduced on a device.
2026-11-20 12:47:32 +00:00
Ryan Vogel c2cae93d61 iOS: trim onboarding to the essentials (#2032)
Ryan found page 2 wordy and confusing (Full Access 'Not checked / Checked on the next step', a five-step list, captions and hints). Page 2 is now two rows, Keyboard and Microphone, a one-line Settings path and Continue; the Full Access and speech model rows are removed (Full Access is confirmed on page 3, the model downloads in the background). Pages 1 and 3 lose their detail lines and the waiting status.

Verified HEXMobileTests and simulator renders of every onboarding scene.
2026-11-20 12:44:24 +00:00
Ryan Vogel 15186d0419 iOS: add a three-page first-run onboarding (#2031)
TestFlight testers turned on Allow Full Access, came back, and Hex still said
Off: iOS reports Full Access only inside the keyboard, so the app can't know
until the keyboard runs. The setup card alone didn't explain that. A first
run now walks people through it and confirms Full Access in the keyboard
itself.

- Welcome: Hex mark, one-line pitch, three points, Get started.
- Set up Hex: live checklist (Hex Keyboard, Allow Full Access, Microphone
  with Allow / Open Settings, speech model download) refreshed every second,
  one Open Settings button with the five steps to take there. Continue needs
  the keyboard added (or unverifiable) and the microphone; Full Access reads
  Not checked here and the model keeps downloading.
- Try the keyboard: a focused field. Success once the keyboard has reported
  Full Access (keyboard.fullAccessSeenAt) and text is in the field, then Done.
  "Full Access looks off" with Open Settings appears when Full Access is
  still missing 4 s after a keyboard switch or new text (a switch right as
  the keyboard comes up is ignored). Skip for now enters the app; the
  existing setup card still shows until setup is complete.
- Shows once: onboarding.completed / onboarding.page in the app's
  UserDefaults; a quit part-way resumes on the same page; a launch that finds
  the keyboard ready and the microphone allowed (existing users) records
  completion and skips it. Settings gets a Run setup again row.
- Never shows under --snap-state except its own scenes: onboarding-welcome,
  -allow, -try, -try-done, -try-off, and onboarding-tour, which walks every
  page by itself for a screen recording. DictationController's fixture
  reuses the setup case for them (line count unchanged, it sits at its cap).
- README: first-run behaviour, scenes, and what is unverified on hardware.

Verified: xcodebuild test on the iPhone 18 Pro simulator (177 tests, 0
failures); Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds;
simulator screenshots of every scene; a fresh install opens page 1, a saved
page resumes without a false Full Access note, and an install with Full
Access seen and the microphone granted skips onboarding and stores
onboarding.completed = 1; a recording of the tour scene.
2026-11-20 12:36:45 +00:00
Ryan Vogel 635a6c5a2e iOS: add a Haptics switch to the full keyboard (#2030)
A tester asked to turn off typing haptics. Experiments › Full keyboard › Haptics (on by default) now gates the keyboard's allowsHaptics alongside Full Access, so key presses, accent picks and trackpad steps stop tapping when it's off. Typing logs record the setting.

Verified HEXMobileTests on the iPhone 18 Pro simulator and a render of the toggle.
2026-11-20 11:59:34 +00:00
Ryan Vogel 5ce7cf8362 iOS: let setup check Full Access from the keyboard instead of showing Off (#2028)
Testers turned on Allow Full Access and the app still said Off: iOS only reports Full Access inside the keyboard, so the app learns it from keyboard.fullAccessSeenAt after the keyboard runs once. The row now says Not checked in that state and the setup card offers a field to open the Hex keyboard in, which records Full Access; the existing 1 s refresh then shows On.

Verified HEXMobileTests on the iPhone 18 Pro simulator and a render of the setup scene.
2026-11-20 11:53:51 +00:00
Ryan Vogel 8c7d01672e iOS: add a Key pop-ups switch to the full keyboard (#2027)
A tester asked to turn off the enlarged letter that pops up over a pressed key. Experiments › Full keyboard › Key pop-ups (on by default) now controls it; off, the pressed key darkens in place as with iOS's Character Preview off. Typing logs record the setting as key_popups.

Verified HEXMobileTests on the iPhone 18 Pro simulator and simulator renders of a held key with the setting on and off.
2026-11-20 11:47:18 +00:00
Ryan Vogel dcc8479cc2 ios: record raw typing logs from Hex's keyboard (experiment) (#2024)
What changed
- Experiments › Record typing logs (off by default, App Group key
  experiments.typingLogs). It is separate from Learn how I type, which is
  unchanged. While it is on, the keyboard writes NDJSON day files to
  TypingLogs/yyyy-MM-dd.ndjson in the App Group. Events:
  session_start/session_end (field traits, keyboard size and safe area,
  orientation, appearance, device, build, locale); layout (every key's frame
  and hit frame, the strip and its suggestion slots, and a hash, with the key
  list written once per file); touch (began/moved/ended/cancelled with raw
  x/y, majorRadius and tolerance, force, UITouch.timestamp, the key the engine
  resolved, and the key under the point when they differ; moves are throttled
  to 2 pt and 8 ms); edit (insert, delete, word delete, autocorrect,
  autocorrect undo, suggestion, double-space period, return and send, paste,
  undo dictation, trackpad cursor); candidates; context
  (documentContextBefore/AfterInput capped at 200 characters, after each word,
  return and dictation and once host changes settle, marked cleared when the
  host empties the field); ui (accents, trackpad, the Hex key and its menu,
  strip chips, globe); dictation (raw, learned fixes, spelling, tagger, Apple
  Intelligence, translation, formatting, inserted text, durations, engine and
  locale, from in-keyboard and app dictation); and dictation_outcome (from a
  second DictationEditTracker). The schema is in ios/docs/typing-logs.md.
- Password, new password, one-time code, every credit card content type, and
  secure fields write one skipped line and nothing else. A new field, or new
  traits, starts a new session.
- TypingLogger buffers events on the main thread. TypingLogStore encodes and
  appends them on a serial utility queue every 2 s, when the keyboard goes
  away, and on a memory warning. Each day is capped at 64 MB with one
  cap_reached line, and files are kept 14 days. When the App Group can't be
  reached (no Full Access), events are dropped silently. Nothing is sent.
- Hooks: FullKeyboardEngine.typingLog, and the KeyboardKeyPlaneView touch
  path, which the UITouch overrides now reach through
  began/moved/ended/cancelled(id:sample:). Every call is skipped while the
  logger is nil, so typing is unchanged with the experiment off.
  KeyboardSnapshot gains optional stages, sent only while the experiment is
  on. KeyboardSpeech.transcribe also returns its locale.
- App: the toggle and a Typing logs screen (each day with its size and event
  count, Share per day, Share all through the share sheet, Delete all with a
  confirmation). Also --snap-state experiments and typing-logs scenes, and an
  --experiment-typing-logs launch argument.

Why
Ryan wants to export a day of raw typing to his Mac, to test and train
prediction models and a per-user touch model (adaptive key targets).

Verified
- xcodebuild test on the iPhone 18 Pro simulator: 177 tests, 0 failures.
  This includes 9 new TypingLogTests: line format and escaping; day rotation
  with layouts in full once per file; Delete all while the writer holds the
  file open; the daily cap (tested at 4 KB) across two writers; 14-day
  retention; the skip rule; a skipped session writing one line; and a real
  FullKeyboardView typing 25 taps through the plane's touch path
  (autocorrect, delete, double-space period, the 123 layer, return, a
  long-press é). That log's edits replay to the field's text, "Don't go yet.
  7 ok\nCafé". An independent Python replay of the same file matched, and
  every lifted key's point was inside its logged hit frame.
- Overhead, simulator Debug, 3,000 taps: about 9 µs a tap on the main thread
  (38 → 47 µs) and about 1 KB a tap (5.2 lines), so about 12 MB an hour at
  200 characters a minute.
- The Release generic-iOS build with CODE_SIGNING_ALLOWED=NO succeeds.
- Simulator screenshots of the toggle and the Typing logs screen.
- Not verified on a physical iPhone: real touch radii and sampling rates, how
  often hosts call textDidChange, whether documentIdentifier stays stable,
  the extension's write cost, and AirDrop from the share sheet.
2026-11-20 02:13:16 +00:00
Ryan Vogel efb7dcafb5 iOS: clean up what I say with an on-device tagger (experiment) (#2023)
* ios: clean up what I say with Hex's on-device dictation tagger (experiment)

What changed
- Experiments › Clean up what I say (off by default, App Group key
  experiments.cleanupTagger): Hex's trained cleanup tagger removes ums,
  repeats, and the words a spoken correction replaces ("3pm, actually 4pm"),
  on the iPhone, in in-keyboard dictation and in app dictation. It runs after
  transcription and learned fixes (in the app also after custom vocabulary),
  before Apple Intelligence cleanup, translation, and formatting, and never
  for Verbatim. When both cleanups are on, Apple Intelligence now receives the
  tagged text instead of the raw transcript; nothing else changes while the
  experiment is off.
- Model: the BERT L-4/H-512 token classifier (ckpt-1200k-l4h512) converted to
  an int8 Core ML ML Program (coremltools 9, fp16 compute, enumerated lengths
  16/32/64/128 plus a length input that masks the padding; the output is
  logit(DELETE) - logit(KEEP)), 28.5 MB, with its WordPiece vocabulary. Both
  ship once, in HEXKeyboard.appex (HEXKeyboard/CleanupTagger/), and the app
  loads them from its embedded keyboard extension. Core ML codegen is off for
  the keyboard target.
- HEXShared: CleanupTokenizer (Hugging Face BertTokenizer: special tokens,
  BertNormalizer, BertPreTokenizer, WordPiece, byte-exact vocabulary keys,
  HF's Unicode 8.0 character classes), CleanupReconstruction (a port of
  js/reconstruct.js with JavaScript's UTF-16 casing), and CleanupTagger
  (chunks of whole words within 128 tokens, P(DELETE) > 0.98, one serial
  queue, 150 ms budget). A timeout, an error, or a result with no words left
  keeps the transcript. The model loads when a dictation starts recording with
  the experiment on, stays loaded, and is freed on a memory warning. It runs
  on the Neural Engine or the CPU, never the GPU.
- Call sites: one each in KeyboardLocalDictation (after DictationCorrections,
  before translation) and DictationController (after VocabularySpelling,
  before the Apple Intelligence pass). prepareIfEnabled runs at recording
  start, in KeyboardViewController and in DictationController.startRecording.

Why
- In the simulator the first load took 0.15 s on the CPU and 0.8 s on the GPU
  path, both over the 150 ms budget. So loading starts when recording starts,
  and the GPU isn't used (it also added 30 MB of footprint, against 3 MB on the
  CPU, which keeps the int8 weights memory-mapped).
- Threshold 0.98, not the exported 0.95: 1/45 false deletes on hand-written
  clean sentences against 3/45, with the same self-correction accuracy.

Verified
- xcodebuild test on the iPhone 18 Pro simulator: 168 tests, 0 failures,
  including the new CleanupTaggerTests and DictationControllerTests
  testCleanupTaggerRunsAfterLearnedFixesAndSkipsVerbatim. After the fixtures
  were rewritten one case per line (same content), CleanupTaggerTests and
  DictationControllerTests passed again (28 tests).
- Parity on the 137 hand-written sentences at 0.98: labels match the fp32
  checkpoint on all 1,128 words, and 136/137 outputs match the ONNX int8
  pipeline. #120 ("The cat keeps trying to scratch that chair.") differs on
  two words that ONNX keeps (P 0.9725/0.9794) and Core ML deletes (0.984).
  The fp32 checkpoint also deletes them (0.9879/0.9853), so the difference
  comes from ONNX's int8 activations. 128/137 outputs match the hand-written
  targets. The tokenizer matches HF on 1,891 words, and reconstruction matches
  reconstruct.js on 670 cases.
- Simulator, CPU path: load 140-161 ms, first sentence after load about 8 ms,
  then 3.3 ms median and 7.5 ms p95 per sentence; phys_footprint +2.9 MB.
- The Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds.
  HEXKeyboard.appex is 30.2 MB (model 27.8 MB, vocabulary 0.2 MB).
- Not verified on a physical iPhone: Neural Engine load time, latency, and the
  keyboard extension's footprint next to Apple Speech.

Note: this adds a 28.4 MB binary (weight.bin) to git; the repo has no LFS.

* Fetch the cleanup tagger's Core ML model at build time instead of committing it

The 27 MB mlpackage would have been ~30x the largest file in the repo, kept in history forever. It is now the ios-model-cleanup-tagger-1 release asset; ios/scripts/fetch-cleanup-model.sh downloads it, checks its SHA-256, and skips when the pinned copy is already in place. Xcode Cloud runs it in ci_post_clone.sh before xcodegen; the folder is gitignored.

Verified the fetched package is byte-identical to the one the tests ran against (diff -r), and a second run is a no-op.
2026-11-20 01:29:41 +00:00
Ryan Vogel c351c54db0 ios: learn dictation and typing fixes from edits (Learn how I type) (#2022)
What changed
- Learn how I type (Experiments, off by default) now follows each dictation
  the keyboard inserts, from the full keyboard and the dictation bar, for up
  to 60 s. On device it works out whether the dictation was kept, edited,
  rewritten (more than half changed), undone, deleted, or redictated (deleted
  and dictated again within 20 s). DictationEditTracker remembers each
  insertion by the text around it. DictationSpanLocator finds it again in
  documentContextBefore/AfterInput, handling nil or cut-short context.
  DictationWordDiff diffs it word by word, and DictationEditClassifier turns
  the changes into (heard, inserted, meant) pairs: substitutions, insertions,
  deletions, words cleanup dropped that you added back, and fillers Hex left
  that you deleted. These teach nothing: new writing at the edges, changes
  that aren't plausibly the same word, truncated edges, and translated
  dictations.
- The journal stores only word pairs (at most 3 words a side, one word of
  context on each side) with counts and contradictions, plus one outcome per
  dictation, never the dictation itself. Older journal files still load.
- A substitution seen twice and never contradicted is applied right after
  transcription in KeyboardLocalDictation and DictationController, before
  custom vocabulary and cleanup. It matches whole words only and follows
  case. Its meant words join the Apple Speech contextual strings; in-keyboard
  Apple Speech now gets them too. The app sends the raw transcript to the
  keyboard (KeyboardSnapshot.heard) only while Learn how I type is on and the
  text wasn't translated.
- Typing: a typo fixed the same way twice becomes an AutocorrectEngine
  candidate. Space applies it unless the typo is a real word, in which case
  it is only suggested. An autocorrection turned down twice never auto-applies
  again. Turning one down means a backspace undo, or tapping the literal chip,
  which is the new keptTyped record. suppressAndKeep was session-only.
- Typing insights lists "Dictation fixes Hex learned" with a switch per row,
  and the kept and edited share of dictations. Delete erases everything: the
  file, the switched-off list, and, through an erasedAt marker, whatever a
  running keyboard still holds.
- There is now one journal per process (TypingJournal.sharedIfEnabled). It
  reloads when another process wrote the file, so the keyboard and the app's
  keyboard preview no longer overwrite each other. Before, each
  FullKeyboardView made its own journal.
- The Learn how I type toggle moved out of the Full keyboard group, because
  it now also learns from the dictation bar.

Why: so Hex predicts better from the corrections people already make,
without asking and without storing what they write.

Verified: xcodegen generate, then xcodebuild test on the iPhone 18 Pro
simulator: 155 tests, 0 failures. That includes the new
DictationEditInferenceTests, DictationEditTrackerTests,
TypingJournalLearningTests, and a DictationController test. The Release
generic-iOS build with CODE_SIGNING_ALLOWED=NO succeeds. Not run on a
physical iPhone.
2026-11-20 01:04:58 +00:00
Ryan Vogel 232be8ce4b Move the Hex key into the strip and grow it into the recording bar (#2021)
The square Hex keycap right of space (with a ↗ 'opens Hex' badge) is now a bare mark at the suggestion strip's trailing edge, so the bottom row is Apple's with a full-width space bar. Starting dictation grows a capsule from the mark across the strip before the recording panel takes over; ending it shrinks the bar back into the mark. The style menu opens downward from the strip. HexKeyPlacement keeps the other placements for snapshot comparison.

Verified HEXMobileTests (bottom-row expectations updated, strip-position test added) and simulator recordings of the morph in light and dark.
2026-11-19 23:57:59 +00:00
Ryan Vogel 2c80cd400d Add Translate what I say (experiment) (#2015)
Speak one language and insert another: after transcription and optional cleanup, the spoken language is detected with NLLanguageRecognizer and translated on the iPhone with TranslationSession(installedSource:target:) once the pack is downloaded in Experiments. Applies to in-keyboard and app dictation; same language, a missing pack, or an error keeps the original. Off by default.

Verified the Release build and HEXMobileTests on the simulator; not yet on a device.
2026-11-19 21:20:07 +00:00
Ryan Vogel 011da1b38c Drop the fake crown from the full keyboard snapshot stage (#2014)
The stage padded 12 pt above the keyboard to imitate iOS 26's Liquid Glass crown, which iOS 27 doesn't add, so renders showed 16 pt above the suggestion pill and 4 pt below it. Without it, renders match the device: 4 pt above and below.

Verified by measuring the full-keyboard-typing render at 3x.
2026-11-19 21:16:39 +00:00
Ryan Vogel d66946319a Make the suggestion highlight fill its slot (#2013)
The 44 pt strip still drew a ~26 pt pill with wide insets, so the extra height read as empty space above a small selection. The strip now spans the whole strip area, the pill fills each third 4 pt in from every edge, suggestion text is 17 pt, and chips grow to 34 pt; keyboard height is unchanged.

Verified HEXMobileTests and simulator renders of the typing state in light and dark.
2026-11-19 21:15:08 +00:00
Ryan Vogel 0c9fa9cddf Give the full keyboard's suggestion strip its full height (#2012)
The strip was 35 pt and drawn 8% above centre for iOS 26's +17 pt Liquid Glass crown, which iOS 27 doesn't add, so suggestions and the autocorrect pill looked short and clipped at the top (TestFlight build 4). Portrait strip is now 44 pt (keyboard 262 pt), landscape stays 39 pt, and strip contents are centred.

Verified HEXMobileTests (layout expectations updated) and simulator renders of the typing state in light and dark.
2026-11-19 21:11:35 +00:00
Ryan Vogel d02d8515d6 Replace the keyboard's blank Paste control with a Paste chip (#2011)
UIPasteControl draws as an empty blue box inside a keyboard extension (TestFlight build 4), which looked like a broken half-width strip. The strip now shows a Paste chip styled like Undo dictation that reads UIPasteboard.general on tap; iOS may ask Allow Paste the first time. Offering still checks only hasStrings/hasURLs. Removes KeyboardPasteReceiver.

Verified the Release build and HEXMobileTests on the simulator; not yet on a device.
2026-11-19 21:06:32 +00:00
Ryan Vogel b77981d63d Add Learn how I type, an on-device typing journal (experiment) (#2010)
Off by default under Experiments with the full keyboard. Records per-key tap offsets as averages (never a letter sequence) and single-word corrections: retyped words within a small edit distance, autocorrect fixes, undone autocorrect, and picked suggestions, capped at 2,000. Skips URL, email, credential, code, card and phone fields and fields without autocorrect. Stored in the App Group; the app shows the most common fixes and off-centre keys and can delete it all. Personalization from this data is a later step.

Verified the Release build compiles and HEXMobileTests pass on the iPhone 18 Pro simulator; not yet tried on a device.
2026-11-19 21:04:08 +00:00