Commit Graph
3 Commits
Author SHA1 Message Date
Ryan Vogel c14df5a4f0 iOS: spatial autocorrect experiment with Google Books models, plus accent and two-letter fixes (#2059)
* iOS keyboard: Spatial autocorrect experiment (tap + bigram decoder), off by default

What: a noisy-channel word decoder (TypingDecoder) behind Settings ›
Experiments › Spatial autocorrect (experiments.spatialAutocorrect, off
by default; it needs the full keyboard with Autocorrect on). When on, for a layout whose
model is bundled (English, Spanish), it replaces UITextChecker's one-edit
guesses on space and in the strip:
- each tap is scored against every letter with a 2D Gaussian around the
  key (sigma 0.29 x 0.26 of the key pitch), moved by Learn how I type's
  average tap offsets for letters with 20+ taps;
- words come from a beam search over a memory-mapped, sorted word list
  with substitutions, one insertion, deletion or transposition, free
  apostrophes ("dont" -> "don't") and accented letters read from their
  base key (é from e, +1 nat);
- a bigram language model on the word before (stupid backoff) weighs
  them, and a missed space is tried by splitting the taps in two (or
  treating a bottom-row tap as the space);
- what was typed is replaced only when the best reading beats it by 2
  nats, or 7 when it is itself a word the model, UITextChecker, learned
  words or the shorthand list know.
Text replacements, learned fixes, the English contraction/typo table and
words you kept still come first. FullKeyboardEngine now records each
letter's tap point for the word being typed (cleared on anything else).
scripts/build-typing-model.py writes the model format (v2: per-language
alphabet, sorted words, u8 costs, sorted bigram keys); models are kept
out of git (.gitignore). The English prototype model is built from
SymSpell's frequency lists (MIT; Google Books Ngram CC BY 3.0 and SCOWL),
the Spanish one from Tatoeba sentence counts (CC BY 2.0 FR). Typing logs
now record the setting.

Why: Record typing logs show short words and non-words left uncorrected
("tk" for "to"), missed spaces with a typo inside, and o/i taps landing
low on k/j. The UITextChecker path skips 1-2 letter words, never uses
the word before, and can't split a word that also has a substitution.

Verified (anomastudio, iPhone 18 Pro simulator, iOS 27.2): benchmark
harness (kept outside the repo) typing every tap through
FullKeyboardEngine. Synthetic held-out English, 13,870 words: output
word error rate 15.0% today vs 6.7% (6.3% with tap offsets); typos fixed
58% vs 83%; correct words wrongly changed 89 vs 97 of 9,203. Labelled
real typing logs, 172 words: 14.5% vs 4.7%. Spanish held-out, 13,301
words: 26.8% vs 8.1% (Spanish model and test from the same corpus, so
optimistic). Per letter p50 1.0 ms, p95 2.5 ms; mapping the model adds
no measurable phys_footprint. Full HEXMobileTests suite: 221 tests,
0 failures (1 skipped, as before); Release generic-iOS build with
CODE_SIGNING_ALLOWED=NO succeeds with both models bundled. No new tests.

* iOS keyboard: Spatial autocorrect models from Google Books Ngram v3

What: the English and Spanish models (HEXKeyboard/TypingModel/en.hextm,
es.hextm, 3.6 MB each) are now built from Google Books Ngram v3 (CC BY
3.0) word and word-pair counts summed over 1980-2019, keeping only
words SCOWL (sizes up to 70, lowercase, ASCII) or RLA-ES (LibreOffice's
es_ES and es_MX dictionaries, MPL 1.1) accept, and are committed.
- English: 86,133 words, 417,480 word pairs; Spanish: 110,000 words,
  326,366 pairs. Pairs are the ones that tell most over the single-word
  model (count x log of P(b|a) over 0.4 P(b)), as many as fit.
- Format v3: pairs grouped by first word (u32 row starts, u32 second
  word, u8 cost), 5 bytes a pair instead of 9, so ~70% more pairs fit
  in the same size. TypingDecoderModel reads v3 only.
- scripts/build-typing-model.py downloads, counts (each ngram file
  deleted once counted, every stage cached for a resumable rerun) and
  writes a model: python build-typing-model.py <en|es> <workdir> <out>.
- Tuning for the new models (dev sentences): English leans more on the
  word before (lmWeight 1.5; Parameters.tuned(for:)); an accent-only
  change ("esta" -> "está") needs a 2-nat margin instead of 7 (ñ for n
  isn't one: "ano"/"año"); an accented letter also reads from its base
  key when it has its own (ñ from n), whichever is cheaper.
- Settings › Acknowledgements (new row and sheet, AcknowledgementsView)
  and ios/README.md credit Google Books Ngram (CC BY 3.0), SCOWL with its
  and WordNet's notices, and RLA-ES (MPL 1.1).
- AutocorrectEngineTests: decoder ranking cases on the bundled models
  ("need t^o" -> "need to", a missed space with a slip -> "see how",
  correct words kept, Spanish "aqui manana" -> "aquí mañana", "esta"
  kept).

Why: Ryan's licence decision: CC BY with credit, no share-alike, no
non-commercial or GPL/LGPL data, and not SymSpell's word-pair list
(unverified provenance). The Spanish model was built from Tatoeba,
the same corpus as its test set.

Removed: the SymSpell- and Tatoeba-built prototype models and the v2
format (pairs as sorted u64 keys); models are no longer gitignored.

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2; benchmark
harness outside the repo; word error rate on words that aren't names):
English held-out test, 13,870 words: today's engine 13.0%, decoder
3.6% (3.4% with tap offsets), correct words broken 4 -> 2 of 7,716;
labelled real typing logs, 172 words: 14.5% -> 4.1%; the shared
example sentence decodes exactly; Spanish held-out Tatoeba test (no
source shared with the model), 13,301 words: 25.7% -> 8.0%, broken
16 -> 20 of 7,246. Per letter p50 1.0 ms, p95 2.6 ms.
AutocorrectEngineTests 19 passed.

* iOS keyboard: Spatial autocorrect knows your own words

What: the decoder also searches TypingDecoder.UserWords, a small sorted
in-memory list of your words, beside the model: Contacts names and
other UILexicon terms, one-word text replacement outputs, words you
kept twice (LearnedWordsStore) and typo fixes Learn how I type learned.
Each scores as a moderately common word (userWordCost 11 nats) and
comes back as you spell it ("Zorbix", not "zorbix"); typed exactly,
one counts as a word, so it needs the larger margin to change.
AutocorrectEngine rebuilds the list only when a source changes
(revision bumped by loadLexiconEntries, applyLearning, suppressAndKeep)
or the model changes. The search is now generic over TypingDecoderWords
(the model or your words); the model's own narrow moved to it.

Why: a name, brand or word you use that the model lacks could only be
left alone, never reached from a typo.

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2):
AutocorrectEngineTests 20 passed, including a new case: with "Zorbix"
in the lexicon, a slip "zorbiz" becomes "Zorbix" and "zorbix" typed
right stays. Benchmark unchanged (it has no user words).

* iOS keyboard: an accent or tilde is the cheapest autocorrect fix

What: AutocorrectEngine.singleEditKeyboardCost now scores a one-letter
substitution that only adds or changes an accent, diaeresis or tilde
(e/é, u/ü, n/ñ) at 0.15, below every other edit, instead of 1.15 (the
cost of two far-apart keys). New helper sameLetterIgnoringAccents.
Valid words are still never rewritten, so "esta" stays "esta".

Why: Spanish typed without accents was corrected into the wrong word,
because the accented guess cost more than any insertion, deletion or
neighbouring key: "aqui" -> "caqui", "podia" -> "odia", "manana" ->
"manaba". Now they become "aquí", "podía", "mañana".

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2, the
benchmark harness kept outside the repo, today's engine with Spatial
autocorrect off): Spanish held-out Tatoeba test, 13,301 words, output
word error rate 26.8% -> 24.7%, words changed into another wrong word
1,015 -> 743, correct words broken 16 -> 16 of 7,246 (names excluded).
English test (13,870 words) unchanged at 15.0%.

* iOS keyboard: fix two-letter typos toward very common two-letter words

What: cautiousReplacement fixes a two-letter non-word (UITextChecker
doesn't know it, not ALL CAPS, no apostrophe) when one neighbouring key
or one swap turns it into a very common two-letter word of the layout's
language (KeyboardLanguage.commonTwoLetterWords, commonest first: "tk"
-> "to", "ot" -> "to", Spanish "nl" -> "no", "ee" -> "de"). English and
Spanish have lists; Norwegian and German have none yet and keep
leaving two-letter words alone. The step numbering in the comments
moves by one.

Why: autocorrect skipped every word under three letters, and the
commonest words are two letters: typing logs show "tk" for "to" left
uncorrected again and again (the o key's taps land low, on k).

Verified (Mac Studio, iPhone 18 Pro simulator, iOS 27.2, benchmark
harness outside the repo, today's engine): English held-out test,
13,870 words, output word error rate 15.0% -> 13.2%; labelled real
typing logs, 172 words, 14.5% -> 10.5%; Spanish test 24.7% -> 22.9%;
correct words broken unchanged (4 of 7,716 English, 16 of 7,246
Spanish, names excluded).
2026-11-20 16:18:07 +00:00
Ryan Vogel cc75290551 iOS: emoji panel with on-device suggestions (Experiments › Emoji) (#2051)
* iOS: emoji index built from Unicode's emoji list and CLDR

What changed
- scripts/build-emoji-data.py builds HEXShared/FullKeyboard/Emoji/emoji-index-<lang>.txt
  for the four keyboard layouts (en, nb, es, de) from Unicode emoji-test.txt 18.0
  (pinned by SHA-256) and CLDR 49 annotations and annotationsDerived (release-49 tag),
  plus Hex's own chat words in scripts/emoji-aliases-en.txt ("congrats", "bday",
  "lol", "new year"; 336 words, written for Hex).
- Each file holds the catalog in panel order (8 categories, 1,923 emoji, 332 with five
  uniform skin tones), each emoji's English name, its folded search words, and the
  word and phrase postings already weighted (one-word name 1.0, name word 0.75,
  keyword 0.6, keyword-phrase word 0.3, chat word 1.2 down to 0.9, an IDF factor for
  words many emoji share), sorted by key so the keyboard can binary-search the file
  in place. Non-English layouts carry English's words at 0.85.
- LICENSE-Unicode.txt (Unicode License v3) ships beside the data, as the licence
  asks: both emoji-test.txt and CLDR's data are under it, which allows commercial
  use with the notice.

Why
- The emoji panel (next commits) suggests emoji for what you type and searches them
  on the iPhone. Precomputing the index keeps the keyboard extension's heap small:
  the first version built Swift dictionaries at runtime and held 5.7 MB, which
  malloc kept after release; the mapped file plus offsets measures 0.7 MB.

Verified
- The generator runs from a sources folder or downloads the pinned files; output is
  deterministic. Files: en 351 KB, nb 560 KB, es 549 KB, de 610 KB.

* iOS: rank emoji for the text before the cursor, on device

What changed
- EmojiLibrary maps an emoji-index file (Data .alwaysMapped) and keeps only the emoji,
  their skin tones and byte offsets: word and phrase lookups binary-search the file's
  sorted keys and parse postings in place; search scans each emoji's folded name and
  keyword words for prefixes (name words first, a short name the query fills before a
  long one it touches, then your use and everyday emoji). Emoji newer than 15.1 are
  listed only if the emoji font itself draws them as one glyph.
- EmojiSuggester reads the last eight content words (the newest counts 1, then 0.85
  each word back), their forms (plurals, -ing, -ed, held-down letters: "lmaooo"),
  two- to four-word phrases ("new year", "good luck") and the word still being typed;
  an emoji several words point at gains a third of the rest. One emoji per family
  ("man running", "woman running"); your use breaks ties and fills the row, then
  everyday defaults. forFinishedWord gives the strip at most three strong matches for
  the word just finished ("pizza " -> 🍕).
- EmojiUsageStore keeps, in the App Group, a use count per emoji that halves every 10
  days (frequency and recency in one score, at most 120 emoji) and the skin tone last
  picked for each; no words. clear() empties both.
- EmojiRoutedTarget lets the keys type into the emoji search's query instead of the
  field while you search.
- EmojiSemanticIndex (app target only, for the comparison test): Apple's on-device
  NLEmbedding word vectors over the index's 3,257 words, standing in for a context word
  with no emoji of its own. Not used by the keyboard (see Measured).
- EmojiSuggesterTests: index, folding, context, families, endings, other layouts,
  strip, search, history decay and tones, the search's routing, and quality on 60
  hand-written sentences (Fixtures/emoji-context-sentences.json, 30 to tune, 30 held
  out) with and without the embedding.

Measured (sensible emoji in the top 5, context only; top-1 in brackets)
- Simulator, iPhone 17, iOS 27.2: keywords dev 30/30 (27), held-out 30/30 (27), 0.04 ms
  a sentence; with the embedding the same counts at 0.14 ms, its index built in 120 ms.
- Mac harness, same code: CLDR keywords alone dev 26/30 (13), held-out 22/30 (14);
  with Hex's chat words 30/30 (27) and 30/30 (27). A blind set of 40 messages and
  sensible emoji written by a separate agent that saw no code or word lists: CLDR
  alone 30/40 (13), with chat words 37/40 (28), adding the embedding 37/40 (28); 38/40
  after the held-down-letter rule, which those failures suggested. Apple's sentence
  embedding scored 18/40 (7) and took 7.5 s to embed the 1,923 descriptions.
- Memory on the Mac: the mapped index adds 0.72 MB; the embedding adds 5.3 MB (1 MB
  NLEmbedding, 3.9 MB vectors) and none of it came back after release. No measurable
  gain for that, so the keyboard uses keywords and your history only.

Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures.

* iOS: emoji panel for the full keyboard (Experiments › Emoji)

What changed
- Experiments › Full keyboard › Emoji (experiments.emoji, off by default; shown with the
  full keyboard): an emoji key (face.smiling) right of 123, after the globe, in plain
  text fields (not URL, email or ASCII-only ones). The bottom row takes the compact
  sizes it uses beside the globe. "Clear emoji history" empties EmojiUsageStore.
- EmojiPanelView replaces the strip and the keys: a row of up to ten suggestions for
  the text before the cursor and your history (one emoji per family, your skin tone),
  a search button; a grid that scrolls sideways through Frequently used (your history,
  then everyday emoji, 16 at least) and the eight categories in four rows of 42 pt
  (three of 38 in landscape), each section's name kept at the left edge while it
  scrolls; a bar with ABC, the categories and a repeating delete, drawn like the
  letter keys' 123 and delete. Holding an emoji with skin tones opens the accent
  picker with its five tones; the pick is typed and remembered for that emoji. Key
  pop-ups and haptics follow their switches.
- Search: the letter keys type into a search field in the strip (EmojiRoutedTarget),
  results above it in a row the keyboard grows 44 pt for, your suggestions while it's
  empty; return, the field's close button or the emoji key go back to the panel.
  Typing logs and Learn how I type see the emoji typed into the field, never the query.
- The strip, when it would be empty after a space, offers the finished word's strong
  emoji ("pizza " -> 🍕, "birthday " -> 🎂 🎉 🥳), a lone one in the centre slot; never
  over word suggestions, Undo dictation, a status or the clipboard chips.
- Grid cells drop their text (and bitmap) when they scroll off and prefetching is off:
  scrolling every section had kept 353 cell bitmaps, 35 MB.
- FullKeyboardView+Emoji.swift holds the wiring; the engine's target is now the routed
  target, which forwards everything to the field unless a search is open.
- Scenes: full-keyboard-emoji, -emoji-search (--emoji-query), -emoji-tones,
  -emoji-food, -emoji-strip (each -dark and -dark-field), and -emoji-probe, which prints
  the panel's footprint and open latency.

Measured in the keyboard extension (Release, iPhone 17 simulator, iOS 27.2, a field
focused with the real HEXKeyboard.appex and a temporary, uncommitted probe; the
simulator counts no GPU memory and reports no headroom), footprint:
  keyboard with the emoji key, nothing loaded   19.0 MB
  after "pizza " (the strip loads the index)    19.0 MB
  panel open                                     28.7 MB
  picked three suggestions, closed               22.7 MB
  every section scrolled                         41.6 MB (peak 48.4; 71.1 before the cell fix)
  closed after that                              36.5 MB
The 17.5 MB left after browsing everything is about 351 emoji glyph images the system's
text rendering caches process-wide (18.7 MB in the heap, 351 CGImages); drawing with a
font instance the panel owns or at 21 pt left it the same, and no public API frees it.
Whether iOS purges it under memory pressure is unverified on an iPhone.
Opening: 22-26 ms from the emoji key to the panel laid out the first time, 12-14 ms
after (median of 10), about 21 ms until committed.

Verified: xcodebuild test (245 tests, 1 skipped, 0 failures); every scene above in
light, dark and dark field looked at on the simulator.

* iOS: document the emoji experiment and keep its blind sentences as a check

What changed
- README: the emoji experiment under Capabilities (what the panel, search, skin tones
  and strip do, what the history keeps, where the index comes from and its licence),
  how to rebuild the index, and its proof limits with the quality, memory and latency
  measurements and what is unverified on an iPhone (whether iOS purges the cached
  emoji glyph images that browsing every section leaves, about 18 MB).
- emoji-context-sentences.json gains the 40-message blind set a separate agent wrote
  without seeing code or word lists; EmojiSuggesterTests scores it beside the other
  60 and fails below 35/40.

Why: the blind set is the least biased quality number; keeping it catches a ranking
change that helps the hand-written sentences at its expense.

Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures; EMOJI_QUALITY: dev 30/30 (top-1 27), held-out 30/30 (27), blind 38/40 (29),
the same with the embedding, which built in 101 ms.
2026-11-20 15:53:24 +00:00
Ryan Vogel efb7dcafb5 iOS: clean up what I say with an on-device tagger (experiment) (#2023)
* ios: clean up what I say with Hex's on-device dictation tagger (experiment)

What changed
- Experiments › Clean up what I say (off by default, App Group key
  experiments.cleanupTagger): Hex's trained cleanup tagger removes ums,
  repeats, and the words a spoken correction replaces ("3pm, actually 4pm"),
  on the iPhone, in in-keyboard dictation and in app dictation. It runs after
  transcription and learned fixes (in the app also after custom vocabulary),
  before Apple Intelligence cleanup, translation, and formatting, and never
  for Verbatim. When both cleanups are on, Apple Intelligence now receives the
  tagged text instead of the raw transcript; nothing else changes while the
  experiment is off.
- Model: the BERT L-4/H-512 token classifier (ckpt-1200k-l4h512) converted to
  an int8 Core ML ML Program (coremltools 9, fp16 compute, enumerated lengths
  16/32/64/128 plus a length input that masks the padding; the output is
  logit(DELETE) - logit(KEEP)), 28.5 MB, with its WordPiece vocabulary. Both
  ship once, in HEXKeyboard.appex (HEXKeyboard/CleanupTagger/), and the app
  loads them from its embedded keyboard extension. Core ML codegen is off for
  the keyboard target.
- HEXShared: CleanupTokenizer (Hugging Face BertTokenizer: special tokens,
  BertNormalizer, BertPreTokenizer, WordPiece, byte-exact vocabulary keys,
  HF's Unicode 8.0 character classes), CleanupReconstruction (a port of
  js/reconstruct.js with JavaScript's UTF-16 casing), and CleanupTagger
  (chunks of whole words within 128 tokens, P(DELETE) > 0.98, one serial
  queue, 150 ms budget). A timeout, an error, or a result with no words left
  keeps the transcript. The model loads when a dictation starts recording with
  the experiment on, stays loaded, and is freed on a memory warning. It runs
  on the Neural Engine or the CPU, never the GPU.
- Call sites: one each in KeyboardLocalDictation (after DictationCorrections,
  before translation) and DictationController (after VocabularySpelling,
  before the Apple Intelligence pass). prepareIfEnabled runs at recording
  start, in KeyboardViewController and in DictationController.startRecording.

Why
- In the simulator the first load took 0.15 s on the CPU and 0.8 s on the GPU
  path, both over the 150 ms budget. So loading starts when recording starts,
  and the GPU isn't used (it also added 30 MB of footprint, against 3 MB on the
  CPU, which keeps the int8 weights memory-mapped).
- Threshold 0.98, not the exported 0.95: 1/45 false deletes on hand-written
  clean sentences against 3/45, with the same self-correction accuracy.

Verified
- xcodebuild test on the iPhone 18 Pro simulator: 168 tests, 0 failures,
  including the new CleanupTaggerTests and DictationControllerTests
  testCleanupTaggerRunsAfterLearnedFixesAndSkipsVerbatim. After the fixtures
  were rewritten one case per line (same content), CleanupTaggerTests and
  DictationControllerTests passed again (28 tests).
- Parity on the 137 hand-written sentences at 0.98: labels match the fp32
  checkpoint on all 1,128 words, and 136/137 outputs match the ONNX int8
  pipeline. #120 ("The cat keeps trying to scratch that chair.") differs on
  two words that ONNX keeps (P 0.9725/0.9794) and Core ML deletes (0.984).
  The fp32 checkpoint also deletes them (0.9879/0.9853), so the difference
  comes from ONNX's int8 activations. 128/137 outputs match the hand-written
  targets. The tokenizer matches HF on 1,891 words, and reconstruction matches
  reconstruct.js on 670 cases.
- Simulator, CPU path: load 140-161 ms, first sentence after load about 8 ms,
  then 3.3 ms median and 7.5 ms p95 per sentence; phys_footprint +2.9 MB.
- The Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds.
  HEXKeyboard.appex is 30.2 MB (model 27.8 MB, vocabulary 0.2 MB).
- Not verified on a physical iPhone: Neural Engine load time, latency, and the
  keyboard extension's footprint next to Apple Speech.

Note: this adds a 28.4 MB binary (weight.bin) to git; the repo has no LFS.

* Fetch the cleanup tagger's Core ML model at build time instead of committing it

The 27 MB mlpackage would have been ~30x the largest file in the repo, kept in history forever. It is now the ios-model-cleanup-tagger-1 release asset; ios/scripts/fetch-cleanup-model.sh downloads it, checks its SHA-256, and skips when the pinned copy is already in place. Xcode Cloud runs it in ci_post_clone.sh before xcodegen; the folder is gitignored.

Verified the fetched package is byte-identical to the one the tests ran against (diff -r), and a second run is a no-op.
2026-11-20 01:29:41 +00:00