Files
Ryan Vogel a8f9b7073d Hex for Android: on-device dictation keyboard (#2074)
* Build the native Android dictation keyboard

Add a three-page Compose setup and an InputMethodService with QWERTY editing, system IME switching, tap dictation and hold-to-talk. Capture inside the keyboard and decode a checksum-pinned, MIT-licensed Whisper tiny model so a fresh install works offline without an app switch, playback focus request or audio files.

Verified on the Studio API 35 ARM64 emulator: clean assembleDebug, three debug/release unit cases, system permission and keyboard setup, actual key editing, two WAVs through the guest microphone, insertion into a separate editor, cancellation and continued media output. The merged debug manifest has no Internet permission.

* Document Android setup and the recognizer decision

Record the supported toolchain, model and runtime licences, offline-first trade-off, and phase 1 limits. Document the headless emulator microphone path used for verification so the next phase can repeat it.

Checked the Studio build/run commands and both recognizers on the API 35 image with airplane mode enabled and media playing.

* Measure on-device Android dictation across four languages

Replace the phase-one single-phrase evidence with a fixed 40-clip multilingual, partly music-mixed corpus. Record per-clip errors, load, warm inference, memory, two-core sensitivity and delivered platform-service results separately. Keep an independent foreground editor and media track for mic-to-insertion verification.

Verified on an API 35 ARM64 Studio emulator: Whisper tiny/base/small, Parakeet TDT v3 int8 and strictly on-device Android recognition. Small in its short window made 15/285 word errors; the platform service returned 33/40. Both native runners also decoded the notebook clip after tightening the PCM input checks.

* Use verified Whisper small for styled, offline Android dictation

Replace tiny-only English recognition with a pinned small model selected by the multilingual benchmark. Keep bundled tiny for first-run offline use and download failures. Snapshot the style, cleanup, language and translation task per recording so a paste override affects only the intended editor and dictation.

Replace the recording status text and old meter with the iOS-style dot waveform, round cancel/finish controls and in-place spinner. Explain the sole network permission on welcome and bundle verified Inter v4.1 and its OFL notice for Compose and the keyboard.

Verified assembleDebug and tests on the Studio. The Android formatter matched 41 applicable iOS examples. Exercised tiny before download, observed progress and verified small and Inter hashes. The real IME inserted all four styles and both Spanish paste choices in airplane mode, with music continuing in an external editor; checked recording and transcribing in both themes.

* Add offline spatial typing assistance to the Android keyboard

Replace literal-only English and Spanish key handling with a native Kotlin Gaussian tap decoder, context-weighted word pairs, missed-space readings, and confidence margins. Map the pinned, uncompressed iOS-compatible Google Books v3 assets; credit the CC BY counts, SCOWL and MPL word filters. Use hashed local rejection counts so a second undo stops that correction without saving typed text.

Give the idle strip three frequency-ranked slots and add independent default-on key previews and system-aware haptics, plus slide-to-select accents. Commit space on the same key-up as letters: the previous posted click could reorder rapid input. Make a confident known spelling harder to replace than an unknown one.

Verified on Studio API 35: assembleDebug and tests, byte-identical English and Spanish models, the exact centre-tapped typo sentence in both themes, one-backspace undo and suppression after two undos across an IME restart, ranked suggestions, accent insertion, independent switches and the system vibration gate.

* Benchmark Android typing against iOS on synthetic key taps

Replay the 2,000 held-out English sentences through the installed Kotlin decoder and suggestion path, keeping only public synthetic rows from the mixed iOS experiment. Preserve aggregate scores, the unmodified Google Books model builder, and the two model digests; no labelled personal typing, offsets, or raw logs are committed.

Verified 1,461 errors over 13,870 words (10.53% Android; 8.39% iOS without offsets using the byte-identical model; 33.81% raw). On the Studio emulator the real editor received letters in 1.41 ms median and spaces in 3.16 ms across five exact sentence replays. Document the confidence margins, timing boundaries, feedback controls, licences and reproducible aggregate scorer.

* Align Android typing with iOS and add German and Bokmål layouts and models

Apply R1's English typo and contraction rules before spatial search and
match its language weight, shortlist and valid-spelling margins. A mapped
SCOWL spelling list supplies the validity check missing on Android. Scale
vertical spread and the missing-space cost to real Android row geometry so
"seehkw" still splits into "see how". Keep the existing en/es model bytes.

Build the new German and Bokmål HXTM files with the shared iOS writer.
German uses all Google Books v3 shards; Bokmål uses NB book counts filtered
by the complete 2022 Ordbank snapshot. Pin the new assets, credit their
sources and licences, and ship their real layouts, accents and suggestions.

Replace the post-setup list with the iOS-style Home, local Dictate bar and
four settings pages. Follow the system language by default; make the spoken
language switchable on space, and optionally pin an independent layout.

Verified on the Studio API 35 ARM64 emulator: 1,027 English and 1,374 Spanish
word errors vs R1's 1,163 and 1,387 on the same 2,000 synthetic sentences
per language. Ran all five unit tests in debug and release, inspected the
packaged assets, and checked external-editor suggestions, corrections, undo
and offline microphone insertion, plus Home and settings in both themes.
The Swift reader round-tripped both new models; the identical iOS files
are in the separate local commit 230a437e.

* Build a signed, shrunk Android 16 upload bundle

Replace the API 35 release defaults with API 36, an external upload key, R8 and resource shrinking. Omit the debug probes, benchmark receiver and unused permission, silence native speech diagnostics, and guard password-field context at the input connection. Give the download a generic user agent and ship the Android privacy policy offline.

Verified on the Studio: jarsigner, bundletool and byte-identical installed splits on API 31 and API 36; all ELF load segments and APK native offsets meet 16 KB alignment. Fresh model downloads and offline insertion passed on both. Binder traces showed zero reads in four password variants versus ten in an ordinary editor. Five tests passed in each build variant.

* Prepare the Android Play listing and device handoff

Add the en-US listing, Data safety and IARC drafts, four release screenshots, launcher-derived store art and a physical-phone checklist. Replace the obsolete API 35 build instructions with the signed API 36 workflow. Flag that the public privacy URL still needs the Android text and that suggestions can include strong language.

Verified title and description limits, native screenshot and artwork formats, model downloads and offline microphone insertion in an external editor on 2.5 GiB API 31 and 16 KB API 36 emulators. Recorded bundletool download size and sampled memory from the same signed release.
2026-11-20 17:27:17 +00:00
..

Hex for Android

Hold the Hex key, speak and release to insert text in the current app. A tap starts hands-free dictation; ✓ finishes and ✕ discards. The live dot waveform stays above the keys. While Hex transcribes, a spinner takes the checkmark’s place and ✕ still cancels.

Speech model and privacy

Whisper small Q5_1 is the default; multilingual tiny Q5_1 is bundled and works from first launch in airplane mode. The Android benchmark compares tiny, base, small, Parakeet and Android’s strictly on-device provider on the same 40 clips. Hex’s small configuration reduced WER from 17.54% to 5.26% and used 344 MiB peak native RSS on short clips. Norwegian remains the weakest language.

Welcome explains that the internet is used only to fetch the 190 MB speech model. Get started begins a fixed HTTPS download; setup shows progress and lets you continue with tiny or retry. ModelStore verifies the exact byte count and SHA-256 before atomically installing it in private, non-backed-up storage. The keyboard never makes a network request. Missing or unusable small weights fall back to tiny. Voice, words and translations stay on the device; Hex has no telemetry, transcript history or network speech service.

Keyboard and styles

Automatic uses the first supported Android system language, or English if none is supported. Hold the language-labelled space bar to switch immediately between Automatic, English, Español, Deutsch and Norsk (Bokmål). Swipe up on the Hex key or tap Hex menu to change the spoken language or style:

  • Polished (default): trims standalone hesitation sounds, fixes spacing/capitals and closes sentences.
  • Casual: lowercase texting, without the period at the end of a single statement.
  • Notes: “bullet point,” “next bullet,” “new line” and “new paragraph” become list items and breaks.
  • Verbatim: keeps the recognizer’s words and punctuation; no cleanup is applied.

Clean up what I say is off by default. The deterministic rules remove hesitations and resolve explicit self-corrections against nearby words of the same kind. Spanish, German and Norwegian use the iOS word lists and pause boundaries; English uses explicit markers, not iOS’s learned tagger or an LLM. Ambiguous repairs stay as spoken. Ordinary negation and language-specific words such as German “eh” and Norwegian “er” stay intact.

Translate what I say is also off by default. With Español and translation enabled, Next dictation offers Paste in English or Paste as spoken. The choice is captured when recording begins; after a one-off spoken-language insertion, the next dictation returns to English. Translation is multilingual Whisper’s native speech-to-English task, not a second text model or online SDK. The tested Spanish WAV inserted both “El café todavía está caliente.” and “The coffee is still hot.” offline.

Typing assistance

All four layouts have a bundled, memory-mapped HXTM v3 word and word-pair model. English and Spanish keep the original iOS Google Books files. German uses Google Books; Bokmål uses the National Library of Norway’s book counts filtered by Norsk ordbank. Start typing to see ranked suggestions; tap one to insert it. A space brings up next-word predictions. Passwords, addresses and editors that disable suggestions use literal typing.

Keyboard › Keyboard layout follows dictation by default; an explicit layout can differ from the space bar’s spoken language. Spanish has a dedicated ñ and ¿/¡ on the second symbols page. German is QWERTZ with ü, ö and ä; hold s for ß. Bokmål has å, ø and æ.

On space, the Kotlin decoder compares each tap with the measured key centres and combines those distances with the previous word’s probability. It can recover a missing space. A known spelling requires a higher confidence margin than a nonword. Immediately press Delete to restore exactly what you typed, without the space. Undo the same replacement twice and Hex keeps that spelling in later sessions. Only rejection counts keyed by hashes are saved locally; there is no typing journal.

Keyboard settings and Hex menu → Typing feedback control key pop-ups and key haptics independently. Both default on; Hex follows system keyboard vibration. Hold a letter, slide to an accent, and release. Turning off pop-ups leaves the deliberate accent picker available.

English/Spanish typing benchmark, model sources and licences

Settings

After setup, Home shows Ready or Finish setup, a Dictate bar and four destinations, following the iOS navigation in #2054. Tap Dictate, speak, then Finish dictation: Hex displays the local result on Home. Dictation controls style, language, translation, cleanup and the offline speech model. Keyboard has layout, autocorrect, key pop-ups, haptics and a practice field. Privacy explains storage and links the model credits. Experiments › Spatial autocorrect controls tap-based correction; it starts on to preserve existing Android behavior. Turning it off keeps dictionary completions and the fixed English typo/contraction rules; turning off Keyboard › Autocorrect disables all automatic replacements.

Privacy › Privacy policy opens the Android policy text even in airplane mode.

Phase 4 verification

On identical held-out synthetic taps and unchanged English and Spanish iOS models, Android’s English word error rate fell from 10.53% to 7.40%, against 8.39% for iOS R1 without offsets. Spanish was 10.33% against 10.43% for iOS (phase 3 Android: 10.06%). The step-by-step benchmark explains the original 298 extra English errors; the replay table gives raw counts, exact sentences and timing.

The Studio verified Spanish, German and Bokmål suggestions and space-triggered corrections in a separate Android app, including ¿¡, ß and æøå. The German and Bokmål assets were also copied, byte-for-byte, into the iOS typing-model folder in local commit 230a437e. The production Swift reader checked all 95,000 words per language and every stored pair score; no iOS source or existing English/Spanish model changed.

With airplane mode on and Wi-Fi off, fixed audio went through the emulator microphone and Whisper small into that editor. Spanish “Nos vemos mañana en la estación” and German “Wir treffen uns morgen am Bahnhof” matched. Bokmål “Tusen takk for hjelpen” also matched; on corpus clip 40 Hex inserted “Biblioteket stenger tydelig på lørdag” instead of “Biblioteket stenger tidlig på lørdag.” The same Spanish clip also produced a transcript from Home’s Dictate bar.

Five unit tests passed in each of the debug and release variants. Home, the four settings pages and all four detail pages were captured in light and dark. Deselecting Hex as the current keyboard changed Ready to Finish setup · 1 step; selecting it again through setup restored Ready. The local APK, full-size screenshots, replay results and Studio log are under build/phase4-proof/.

How it works

  • App: Kotlin and Jetpack Compose. Welcome → microphone permission and Android’s keyboard settings/picker → a real practice field. Setup reads live system state; it remembers the page, not the words.
  • Keyboard: HexKeyboard is an InputMethodService with native key views. Letters, shift/caps lock, symbols, delete, editor actions and next keyboard use Android input APIs. It commits through the active InputConnection; changing editors or hiding the keyboard discards pending results. Password fields allow typing only.
  • Audio: MicCapture records up to 60 s of mono 16 kHz PCM in RAM and releases the microphone before CPU inference. The recognizer interface operates on that capture and its language/task. Hex requests no audio focus and does not change playback routing.
  • Type: unmodified InterVariable.ttf from Rasmus Andersson and the Inter Project, v4.1, licensed under SIL OFL 1.1. One Compose font family and native TextView helper cover Hex’s text, including keys, menus, strip labels and tabular digits (tnum). Android supplies missing glyphs; permission dialogs and the system keyboard picker use the OS font. Gradle verifies the bundled TTF’s SHA-256.
  • Licences: OpenAI Whisper weights and whisper.cpp/ggml are MIT. Kotlin/AndroidX are Apache 2.0; libc++ is Apache 2.0 with LLVM exceptions. Bundled credits and licence texts. No ML Kit, Parakeet or third-party language packs are shipped.

Build and run

Use the Studio’s JDK 21, SDK platform 36, NDK 27.2.12479018 and CMake 3.22.1. The Play packaging checks use Build Tools 36.0.0 and bundletool 1.18.3. From the synced android/ directory, with an ARM64 emulator or phone running:

export JAVA_HOME="/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home"
export ANDROID_HOME="$HOME/Library/Android/sdk"
export PATH="$ANDROID_HOME/platform-tools:$JAVA_HOME/bin:$PATH"
./gradlew assembleDebug
./gradlew test
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n ly.anoma.hex/.MainActivity

On the headless Studio, start an ARM64 AVD with -audio none -grpc 8566 -grpc-use-token. inject.py uses the matching emulator discovery record and streams paced mono PCM16 into the guest microphone. Generate the bindings from the SDK emulator’s lib/emulator_controller.proto with grpcio-tools, then put them on PYTHONPATH. No recording or transcript is injected directly into the application.

For the upload bundle, run ./gradlew :app:bundleRelease. Release packaging requires a private properties file at ~/.config/hex-android/keystore.properties or the path in HEX_ANDROID_KEYSTORE_PROPERTIES. It must contain storeFile (relative to that file, or absolute), keyAlias, storePassword and keyPassword. Release uses R8 and resource shrinking and never falls back to the debug key. Gradle writes app/build/outputs/bundle/release/app-release.aab.

Scope: Android 12+ (min API 31), target 36, ARM64, four typing models and four dictation languages. See the Play release evidence, listing drafts, phone screenshots and real-device checklist. The Studio also exercised offline fallback, all four styles, per-dictation translation, both appearances and insertion into a separate app with music playing in earlier phases.