Files
Ryan Vogel a8f9b7073d Hex for Android: on-device dictation keyboard (#2074)
* Build the native Android dictation keyboard

Add a three-page Compose setup and an InputMethodService with QWERTY editing, system IME switching, tap dictation and hold-to-talk. Capture inside the keyboard and decode a checksum-pinned, MIT-licensed Whisper tiny model so a fresh install works offline without an app switch, playback focus request or audio files.

Verified on the Studio API 35 ARM64 emulator: clean assembleDebug, three debug/release unit cases, system permission and keyboard setup, actual key editing, two WAVs through the guest microphone, insertion into a separate editor, cancellation and continued media output. The merged debug manifest has no Internet permission.

* Document Android setup and the recognizer decision

Record the supported toolchain, model and runtime licences, offline-first trade-off, and phase 1 limits. Document the headless emulator microphone path used for verification so the next phase can repeat it.

Checked the Studio build/run commands and both recognizers on the API 35 image with airplane mode enabled and media playing.

* Measure on-device Android dictation across four languages

Replace the phase-one single-phrase evidence with a fixed 40-clip multilingual, partly music-mixed corpus. Record per-clip errors, load, warm inference, memory, two-core sensitivity and delivered platform-service results separately. Keep an independent foreground editor and media track for mic-to-insertion verification.

Verified on an API 35 ARM64 Studio emulator: Whisper tiny/base/small, Parakeet TDT v3 int8 and strictly on-device Android recognition. Small in its short window made 15/285 word errors; the platform service returned 33/40. Both native runners also decoded the notebook clip after tightening the PCM input checks.

* Use verified Whisper small for styled, offline Android dictation

Replace tiny-only English recognition with a pinned small model selected by the multilingual benchmark. Keep bundled tiny for first-run offline use and download failures. Snapshot the style, cleanup, language and translation task per recording so a paste override affects only the intended editor and dictation.

Replace the recording status text and old meter with the iOS-style dot waveform, round cancel/finish controls and in-place spinner. Explain the sole network permission on welcome and bundle verified Inter v4.1 and its OFL notice for Compose and the keyboard.

Verified assembleDebug and tests on the Studio. The Android formatter matched 41 applicable iOS examples. Exercised tiny before download, observed progress and verified small and Inter hashes. The real IME inserted all four styles and both Spanish paste choices in airplane mode, with music continuing in an external editor; checked recording and transcribing in both themes.

* Add offline spatial typing assistance to the Android keyboard

Replace literal-only English and Spanish key handling with a native Kotlin Gaussian tap decoder, context-weighted word pairs, missed-space readings, and confidence margins. Map the pinned, uncompressed iOS-compatible Google Books v3 assets; credit the CC BY counts, SCOWL and MPL word filters. Use hashed local rejection counts so a second undo stops that correction without saving typed text.

Give the idle strip three frequency-ranked slots and add independent default-on key previews and system-aware haptics, plus slide-to-select accents. Commit space on the same key-up as letters: the previous posted click could reorder rapid input. Make a confident known spelling harder to replace than an unknown one.

Verified on Studio API 35: assembleDebug and tests, byte-identical English and Spanish models, the exact centre-tapped typo sentence in both themes, one-backspace undo and suppression after two undos across an IME restart, ranked suggestions, accent insertion, independent switches and the system vibration gate.

* Benchmark Android typing against iOS on synthetic key taps

Replay the 2,000 held-out English sentences through the installed Kotlin decoder and suggestion path, keeping only public synthetic rows from the mixed iOS experiment. Preserve aggregate scores, the unmodified Google Books model builder, and the two model digests; no labelled personal typing, offsets, or raw logs are committed.

Verified 1,461 errors over 13,870 words (10.53% Android; 8.39% iOS without offsets using the byte-identical model; 33.81% raw). On the Studio emulator the real editor received letters in 1.41 ms median and spaces in 3.16 ms across five exact sentence replays. Document the confidence margins, timing boundaries, feedback controls, licences and reproducible aggregate scorer.

* Align Android typing with iOS and add German and Bokmål layouts and models

Apply R1's English typo and contraction rules before spatial search and
match its language weight, shortlist and valid-spelling margins. A mapped
SCOWL spelling list supplies the validity check missing on Android. Scale
vertical spread and the missing-space cost to real Android row geometry so
"seehkw" still splits into "see how". Keep the existing en/es model bytes.

Build the new German and Bokmål HXTM files with the shared iOS writer.
German uses all Google Books v3 shards; Bokmål uses NB book counts filtered
by the complete 2022 Ordbank snapshot. Pin the new assets, credit their
sources and licences, and ship their real layouts, accents and suggestions.

Replace the post-setup list with the iOS-style Home, local Dictate bar and
four settings pages. Follow the system language by default; make the spoken
language switchable on space, and optionally pin an independent layout.

Verified on the Studio API 35 ARM64 emulator: 1,027 English and 1,374 Spanish
word errors vs R1's 1,163 and 1,387 on the same 2,000 synthetic sentences
per language. Ran all five unit tests in debug and release, inspected the
packaged assets, and checked external-editor suggestions, corrections, undo
and offline microphone insertion, plus Home and settings in both themes.
The Swift reader round-tripped both new models; the identical iOS files
are in the separate local commit 230a437e.

* Build a signed, shrunk Android 16 upload bundle

Replace the API 35 release defaults with API 36, an external upload key, R8 and resource shrinking. Omit the debug probes, benchmark receiver and unused permission, silence native speech diagnostics, and guard password-field context at the input connection. Give the download a generic user agent and ship the Android privacy policy offline.

Verified on the Studio: jarsigner, bundletool and byte-identical installed splits on API 31 and API 36; all ELF load segments and APK native offsets meet 16 KB alignment. Fresh model downloads and offline insertion passed on both. Binder traces showed zero reads in four password variants versus ten in an ordinary editor. Five tests passed in each build variant.

* Prepare the Android Play listing and device handoff

Add the en-US listing, Data safety and IARC drafts, four release screenshots, launcher-derived store art and a physical-phone checklist. Replace the obsolete API 35 build instructions with the signed API 36 workflow. Flag that the public privacy URL still needs the Android text and that suggestions can include strong language.

Verified title and description limits, native screenshot and artwork formats, model downloads and offline microphone insertion in an external editor on 2.5 GiB API 31 and 16 KB API 36 emulators. Recorded bundletool download size and sampled memory from the same signed release.
2026-11-20 17:27:17 +00:00

148 lines
6.4 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env python3
"""Build German and Bokmål HXTM v3 files with the shared iOS count reader / model writer.
python3 build-layout-model.py {de|nb} SOURCE_CACHE OUTPUT.hextm
German: every untagged German Google Books v3 shard, 1980–2019. Bokmål: NB N-gram
2026 digibok, language nob, same years, filtered by Norsk ordbank full forms.
"""
import collections
import concurrent.futures
import csv
import importlib
import hashlib
import http.client
import io
import json
import re
import subprocess
import sys
import unicodedata
import urllib.request
import shutil
import time
import tarfile
from pathlib import Path
builder = importlib.import_module("build-typing-model")
builder.FILES["de"] = ("ger", 8, 181)
builder.LETTERS["de"] = "abcdefghijklmnopqrstuvwxyzäöüß"
builder.MAX_WORDS["de"] = 95_000
# German uses only the corpus, without importing another spelling dictionary.
# The iOS reader lowercases before adding counts; keep the same alphabet rule
# on Unicode scalars before the shared writer assigns their compact codes.
def german_words(lang, work):
return re.compile(r"[a-zäöüß]{1,32}\Z").fullmatch
builder.word_list = german_words
NB = "https://www.nb.no/sbfil/ngram/ngram_2026/digibok-"
ORDBANK = "https://www.nb.no/sbfil/leksikalske_databaser/ordbank/20220201_norsk_ordbank_nob_2005.tar.gz"
WORD = re.compile(r"[a-zæøåéèêóôüäö]{1,32}\Z")
def normalize(s):
return unicodedata.normalize("NFC", s).lower()
def nb_counts(path, is_pair, accept):
counts = collections.Counter()
with subprocess.Popen(["zstd", "-dc", path], stdout=subprocess.PIPE) as process:
for line in process.stdout:
if b'"nob"' not in line:
continue
row = json.loads(line)
if row["lang"] != "nob":
continue
first = normalize(row["first"])
if not accept(first):
continue
if is_pair:
second = normalize(row["second"])
if not accept(second):
continue
key = (first + " " + second).encode("utf-8")
else:
key = first
n = sum(v for y, v in row["counts"].items() if "1980" <= y <= "2019")
if n:
counts[key] += n
if process.wait():
raise ValueError("Incomplete zstd stream: " + str(path))
return counts
def download_nb(url, path, size):
# The NB server may close long transfers. Resume independently verified byte ranges.
if path.is_file() and path.stat().st_size == size:
return
parts = path.with_suffix(".ranges")
parts.mkdir(exist_ok=True)
block = 4 << 20
def read(i):
start, end = i * block, min((i+1)*block, size)
part = parts / str(i)
for attempt in range(9):
pos = start + (part.stat().st_size if part.exists() else 0)
if pos == end:
return part
if pos > end:
raise ValueError("Oversized NB range")
req = urllib.request.Request(url, headers={"Range": f"bytes={pos}-{end-1}", "Accept-Encoding": "identity"})
try:
with urllib.request.urlopen(req, timeout=60) as res:
if res.status != 206 or res.headers.get("Content-Range") != f"bytes {pos}-{end-1}/{size}":
raise ValueError("NB returned an unexpected range")
with part.open("ab") as f:
shutil.copyfileobj(res, f, length=262144)
if part.stat().st_size == end-start:
return part
except (OSError, http.client.IncompleteRead, EOFError) as e:
if attempt == 8:
raise RuntimeError(f"Could not fetch {url}, range {i}") from e
time.sleep(min(2**attempt, 10))
raise IOError("Incomplete NB range " + str(i))
with concurrent.futures.ThreadPoolExecutor(max_workers=16) as pool:
complete = list(pool.map(read, range((size + block - 1)//block)))
assembled = path.with_suffix(".assembling")
with assembled.open("wb") as out:
for part in complete:
with part.open("rb") as f:
shutil.copyfileobj(f, out)
if assembled.stat().st_size != size:
raise IOError("Incomplete NB archive")
assembled.replace(path)
shutil.rmtree(parts)
def norwegian(work, out):
work.mkdir(parents=True, exist_ok=True)
bank = work / "ordbank-2022.tar.gz"
builder.fetch(ORDBANK, str(bank))
if hashlib.sha256(bank.read_bytes()).hexdigest() != "57d91cb3f17b85befa50a3b56fcaed9800ca665b39486fe97b668add914d5f60":
raise ValueError("Unexpected Norsk ordbank 2022 release")
# The 2026 inflection-only TSV omits uninflected words such as på and og.
with tarfile.open(bank, "r:gz") as archive, archive.extractfile("fullformsliste.txt") as entry:
rows = csv.DictReader(io.TextIOWrapper(entry, encoding="latin-1"), delimiter="\t", quoting=csv.QUOTE_NONE)
forms = {normalize(r["OPPSLAG"]) for r in rows if r["NORMERING"] == "normert" and r["TILDATO"] == "4000"}
paths = [work / ("digibok-" + n + ".jsonl.zst") for n in ("unigram", "bigram")]
archives = [(1_269_473_951, "b430ef4e19caf1a12bcf47e1ee4f6866f27943ae133e141c57d1a3db4dd8f0c9"),
(3_832_769_124, "b6d2f3ca5492288d5f1c66fc0a3976a7239502cb690f77d493a1cdd97c5ebe1e")]
for path, (size, expected) in zip(paths, archives):
download_nb(NB + path.name.removeprefix("digibok-"), path, size)
digest = hashlib.sha256()
with path.open("rb") as src:
for block in iter(lambda: src.read(1 << 20), b""):
digest.update(block)
if digest.hexdigest() != expected:
raise ValueError("Unexpected NB archive: " + str(path))
uni = nb_counts(paths[0], False, WORD.fullmatch)
total = sum(uni.values())
vocab = dict(sorted(((w, n) for w, n in uni.items() if w in forms),
key=lambda item: (-item[1], item[0]))[:95_000])
pairs = nb_counts(paths[1], True, vocab.__contains__)
print(f"Bokmål: {total:,} word occurrences; {len(forms):,} Ordbank forms; {len(pairs):,} filtered pairs", flush=True)
builder.write_model("nb", vocab, total, pairs, str(out))
if __name__ == "__main__":
if len(sys.argv) != 4 or sys.argv[1] not in ("de", "nb"):
sys.exit(__doc__)
lang, work, out = sys.argv[1:]
if lang == "de":
builder.main(lang, work, out)
else:
norwegian(Path(work), Path(out))