Files
hex/ios/HEXKeyboard/KeyboardViewController.swift
Ryan Vogel b6024b2001 iOS: full keyboard by default, typing without Full Access, and a logging mark (#2067)
* iOS keyboard: the full keyboard by default, typing without Full Access

App Review 4.4.1 needs a keyboard extension to type, and to keep working
without Full Access and without network. Hex's default was the dictation
bar (space, delete, return, Dictate; no letter keys), and without Full
Access it showed only setup steps.

What changed
- FullKeyboardSettings.isEnabled (App Group key experiments.fullKeyboard,
  what Settings › Keyboard's Full / Dictation bar row binds) now reads
  unset as true: new installs and testers who never touched the old
  experiment get the full keyboard; an explicit false (turned off, or
  Dictation bar picked) keeps the dictation bar. --keyboard
  full|dictation-bar picks it in simulator scenes.
- Without Full Access the keyboard always mounts the full keyboard,
  whatever is chosen (the bar types only spaces and line breaks).
- Removed: the full keyboard's permanent "Allow Full Access to dictate
  with Hex" strip notice; the strip suggests and corrects as usual.
- Without Full Access the Hex key (tap, hold, VoiceOver, or sliding for
  its menu) no longer tries to dictate or open Hex: the strip shows
  "Allow Full Access to dictate" and "Settings › Apps › Hex › Keyboards"
  for 8 s (until a key is typed), with an Open Hex chip, the only thing
  that opens Hex (hex-dictation://keyboard/setup) without Full Access.
- In-keyboard dictation (onLocalDictateTap) is offered only with Full
  Access; before, it was set without it too, so the Hex key would have
  started a capture that can't work there.
- Scene full-keyboard-no-access-hint; README: the default, behaviour
  without Full Access, memory numbers and proof limits.

Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release, keyboard
in a real hosted field, a temporary uncommitted probe):
- Fresh install, Full Access off: full keyboard, no in-keyboard
  dictation; taps through the engine typed "thw " -> "the ", "tk " ->
  "to "; with Spatial autocorrect on (decoder active) also "ofthw " ->
  "of the ". The hint renders light and dark.
- Full Access on: full keyboard by default; --keyboard dictation-bar
  mounts the bar.
- Memory, 3 runs each, MB, before (78e4458d) -> after:
  full keyboard idle 18.4-18.5 -> 18.5-18.6 (18.0-18.3 without Full
  Access), first dictation max within 3 s 30.8-31.3 -> 31.2-31.8, peak
  32.1-32.5 -> 32.3-33.0; bar idle 20.7 -> 20.8, first dictation
  27.5-27.8 -> 27.3-27.9. Unchanged within noise, so #2045's 32/24 MB
  guard still covers it.
- xcodebuild test: 267 tests, 0 failures, 1 skipped (as before).
Not verified on a physical iPhone.

* iOS keyboard: show a Logging mark while typing logs record

App Review 2.5.14: a keyboard that records what is typed must say so the
whole time. Record typing logs had no indicator in the keyboard.

What changed
- KeyboardViewModel.isLoggingTyping, set by the keyboard from
  TypingLogger.isRecording (session start and end, and every bridge
  poll), so a skipped field (password, one-time code, card) shows no
  mark.
- Full keyboard: a flat "● Logging" mark (10 pt, the strip's muted ink)
  in the strip's top-leading corner, in from the keyboard's rounded
  corner; while dictating, at the start of the recording bar's well, with
  the scope moved past it, so it never disappears.
- Dictation bar: the same mark beside the Hex mark in its top row, there
  while idle and while recording.
- The dot is muted, not orange: an orange dot would read as iOS's
  microphone indicator. VoiceOver reads "Typing logs are recording".
- --typing-log-mark shows it in the full-keyboard and keyboard-* scenes;
  README.

Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release, keyboard
in a real hosted field with --experiment-typing-logs): the mark shows in
the full keyboard's strip, in its recording bar during an in-keyboard
dictation, and in the dictation bar idle and recording; the scenes
render it light and dark. Idle footprint with logging on 18.5-18.7 MB
(full) and 20.8-20.9 MB (bar), as without. Built for testing.

* iOS keyboard: cap keyboard-capture-probe.log at 512 KB

The in-keyboard dictation probe log in the App Group grew without limit:
every dictation and memory change appends to it, and nothing removed it.

What changed
- KeyboardProbeLog.write: once the file passes 512 KB, it keeps only its
  newest 256 KB, from the first whole line in them (atomic rewrite).
  Nothing user-facing reads the log; testers' copies keep their recent
  footprint lines.
- README: the cap.

Verified (anomastudio, iPhone 13 simulator, iOS 27.2, Release): a
614,400-byte log in the simulator's App Group container became 262,590
bytes after the keyboard's next in-keyboard dictation wrote to it,
starting on a whole filler line and ending with that dictation's lines.
2026-11-20 16:41:37 +00:00

769 lines
33 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import SwiftUI
import UIKit
@MainActor
final class KeyboardViewController: UIInputViewController {
private let bridge = KeyboardBridge()
private let viewModel = KeyboardViewModel()
private let localDictation = KeyboardLocalDictation()
private var pollTimer: Timer?
private var coldStartJobID = UUID().uuidString
private var lastURLRefreshBucket: Int = 0
private var lastFullAccessNote: TimeInterval = 0
private var surface: UIView?
private var fullKeyboard: FullKeyboardView?
private var heightConstraint: NSLayoutConstraint?
private var pollInterval: TimeInterval = 0
private var didHandOffCurrentField = false
private var didWarmLexicon = false
/// Learn how I type: what becomes of each dictation Hex inserts.
private let dictationEdits = DictationEditTracker()
private lazy var documentTarget = DocumentProxyTextTarget { [weak self] in self?.textDocumentProxy }
/// Record typing logs: the process's logger while the experiment is on.
private var typingLog: TypingLogger?
/// The field the current typing log session is for.
private var typingLogTraits: TypingLogTraits?
/// Follows dictations for typing logs' outcomes, apart from Learn how I type's tracker.
private let logDictationEdits = DictationEditTracker()
private var pendingHostContext: DispatchWorkItem?
private var isShown = false
/// The dictation bar's key haptic; the full keyboard has its own generator.
private lazy var barHaptics = UIImpactFeedbackGenerator(style: .light)
/// Haptics need Full Access, and Settings › Keyboard › Haptics covers both keyboards.
private var allowsKeyHaptics: Bool {
hasFullAccess && FullKeyboardSettings().haptics
}
override func viewDidLoad() {
super.viewDidLoad()
configureView()
syncFromBridge(animated: false)
}
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
coldStartJobID = UUID().uuidString
didHandOffCurrentField = false
// The keyboard may come back for another field: settle on where the dictation was last seen.
dictationEdits.finish(nil)
mountSurface()
fullKeyboard?.prepareForField()
isShown = true
beginTypingLogSession(trigger: "appear")
handOffNumericPadIfNeeded()
syncFromBridge(animated: false)
schedulePoll(force: true)
}
override func viewDidAppear(_ animated: Bool) {
super.viewDidAppear(animated)
warmFullKeyboardLexiconIfNeeded()
listDictationLanguagesIfNeeded()
}
override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
if dictationEdits.isTracking { dictationEdits.finish(editContext) }
TypingJournal.sharedIfEnabled()?.save()
isShown = false
endTypingLogSession(reason: "disappear", settleDictation: editContext)
pollTimer?.invalidate()
pollTimer = nil
}
override func didReceiveMemoryWarning() {
super.didReceiveMemoryWarning()
fullKeyboard?.handleMemoryWarning()
typingLog?.flush()
}
override func viewWillLayoutSubviews() {
super.viewWillLayoutSubviews()
if let fullKeyboard, heightConstraint?.constant != fullKeyboard.preferredHeight {
heightConstraint?.constant = fullKeyboard.preferredHeight
preferredContentSize.height = fullKeyboard.preferredHeight
}
}
override func textWillChange(_ textInput: UITextInput?) {
super.textWillChange(textInput)
if KeyboardFieldRouting.route(for: textDocumentProxy.keyboardType) != .handOff {
didHandOffCurrentField = false
}
}
override func textDidChange(_ textInput: UITextInput?) {
super.textDidChange(textInput)
updateKeyboardTraits()
followTypingLogField()
fullKeyboard?.textDidChange()
observeDictationEdits(fromKeys: false)
noteHostTextChange()
handOffNumericPadIfNeeded()
}
// MARK: - Record typing logs
/// Starts a session for the field the keyboard is in, while the experiment is on. A
/// field that must not be logged gets a `skipped` line and nothing else.
private func beginTypingLogSession(trigger: String) {
guard let log = TypingLogger.sharedIfEnabled() else {
typingLog = nil
return
}
typingLog = log
log.contextSource = { [weak self] in
self?.typingLogContext ?? TypingLogContext(before: nil, after: nil, hasText: false)
}
let traits = TypingLogTraits(textDocumentProxy, document: textDocumentProxy.documentIdentifier)
typingLogTraits = traits
log.beginSession(typingLogSessionInfo(trigger: trigger, traits: traits))
attachTypingLog()
syncLoggingMark()
}
/// `settleDictation` is the text the last dictation's outcome is judged on (nil: where
/// it was last seen, for a field the keyboard already left).
private func endTypingLogSession(reason: String, settleDictation context: DictationEditTracker.Context?) {
pendingHostContext?.cancel()
pendingHostContext = nil
guard let log = typingLog else { return }
if logDictationEdits.isTracking { logDictationEdits.finish(context) }
fullKeyboard?.typingLog = nil
log.endSession(reason: reason)
typingLogTraits = nil
syncLoggingMark()
}
/// Both keyboards show a small mark the whole time typing logs record this field.
private func syncLoggingMark() {
let logging = typingLog?.isRecording == true
if viewModel.isLoggingTyping != logging {
viewModel.isLoggingTyping = logging
}
}
private func attachTypingLog() {
guard let fullKeyboard else { return }
let log = typingLog?.isRecording == true ? typingLog : nil
guard fullKeyboard.typingLog !== log || log != nil else { return }
fullKeyboard.typingLog = log
fullKeyboard.typingLogSessionBegan()
}
/// Turned on or off while the keyboard is shown.
private func syncTypingLogEnabled() {
let enabled = TypingLogSettings().isEnabled
if enabled, typingLog == nil, isShown {
beginTypingLogSession(trigger: "enabled")
} else if !enabled, typingLog != nil {
endTypingLogSession(reason: "disabled", settleDictation: nil)
typingLog = nil
_ = TypingLogger.sharedIfEnabled()
}
}
/// Another field, or the same field with new traits: a new session (which may be skipped).
private func followTypingLogField() {
guard typingLog != nil, isShown else { return }
let traits = TypingLogTraits(textDocumentProxy, document: textDocumentProxy.documentIdentifier)
guard traits.signature != typingLogTraits?.signature else { return }
endTypingLogSession(reason: "field_changed", settleDictation: nil)
beginTypingLogSession(trigger: "field")
}
/// The host changed the text or moved the cursor. Hosts call this in bursts (and some
/// after Hex's own edits), so the context is written once things settle, at once when
/// the field was emptied.
private func noteHostTextChange() {
guard let log = typingLog, log.isRecording else { return }
pendingHostContext?.cancel()
guard textDocumentProxy.hasText else {
pendingHostContext = nil
log.context("text_did_change")
return
}
let work = DispatchWorkItem { [weak self] in
MainActor.assumeIsolated {
self?.pendingHostContext = nil
self?.typingLog?.context("text_did_change")
}
}
pendingHostContext = work
DispatchQueue.main.asyncAfter(deadline: .now() + 0.3, execute: work)
}
private var typingLogContext: TypingLogContext {
TypingLogContext(
before: textDocumentProxy.documentContextBeforeInput,
after: textDocumentProxy.documentContextAfterInput,
hasText: textDocumentProxy.hasText
)
}
private func typingLogSessionInfo(trigger: String, traits: TypingLogTraits) -> TypingLogSessionInfo {
let bundle = Bundle.main.infoDictionary
let orientation = fullKeyboard?.orientation ?? FullKeyboardView.orientation(for: traitCollection)
let dark = textDocumentProxy.keyboardAppearance == .dark || traitCollection.userInterfaceStyle == .dark
return TypingLogSessionInfo(
trigger: trigger,
surface: fullKeyboard == nil ? "bar" : "full",
traits: traits,
bounds: view.bounds.size,
safeArea: view.safeAreaInsets,
orientation: orientation == .landscape ? "landscape" : "portrait",
appearance: dark ? "dark" : "light",
device: TypingLogSessionInfo.deviceModel,
system: "\(UIDevice.current.systemName) \(UIDevice.current.systemVersion)",
version: bundle?["CFBundleShortVersionString"] as? String ?? "",
build: bundle?["CFBundleVersion"] as? String ?? "",
locale: Locale.current.identifier,
languages: Array(Locale.preferredLanguages.prefix(4)),
keyboardLanguage: primaryLanguage,
inputMode: textDocumentProxy.documentInputMode?.primaryLanguage,
fullAccess: hasFullAccess,
settings: [
TypingLogField("autocorrect", .bool(FullKeyboardSettings().autocorrect)),
TypingLogField("spatialAutocorrect", .bool(FullKeyboardSettings().spatialAutocorrect)),
TypingLogField("key_popups", .bool(FullKeyboardSettings().keyPopups)),
TypingLogField("haptics", .bool(FullKeyboardSettings().haptics)),
TypingLogField("learnHowIType", .bool(TypingJournalSettings().isEnabled)),
TypingLogField("cleanupTagger", .bool(CleanupTaggerSettings().isEnabled)),
TypingLogField("hexKeyPlacement", .string(HexKeyPlacement.current.rawValue)),
TypingLogField("keyboardLayout", .string(fullKeyboard?.engine.language.rawValue ?? "")),
TypingLogField("dictationLanguage", .string(LanguageSettings().dictationLanguage ?? "automatic")),
TypingLogField("style", .string(bridge.style.rawValue)),
]
)
}
// MARK: - Learn how I type
/// Dictations are followed only while Learn how I type is on, in plain text fields
/// that allow autocorrect.
private var learnsFromDictation: Bool {
guard TypingJournalSettings().isEnabled,
case .keys(let kind, _) = KeyboardFieldRouting.route(for: textDocumentProxy.keyboardType)
else { return false }
return TypingJournal.allows(field: kind, target: documentTarget)
}
private var editContext: DictationEditTracker.Context {
DictationEditTracker.Context(
before: textDocumentProxy.documentContextBeforeInput,
after: textDocumentProxy.documentContextAfterInput,
hasText: textDocumentProxy.hasText
)
}
private func observeDictationEdits(fromKeys: Bool) {
guard dictationEdits.isTracking || logDictationEdits.isTracking else { return }
let context = editContext
if logDictationEdits.isTracking { logDictationEdits.observe(context, fromKeys: fromKeys) }
if dictationEdits.isTracking { dictationEdits.observe(context, fromKeys: fromKeys) }
}
/// A return key labelled Send ends the dictation's edit window: what is there is sent.
private func returnKeyWillInsert() {
guard textDocumentProxy.returnKeyType == .send else { return }
if logDictationEdits.isTracking { logDictationEdits.finish(editContext) }
guard dictationEdits.isTracking else { return }
dictationEdits.finish(editContext)
}
private func handOffNumericPadIfNeeded() {
guard fullKeyboard != nil else { return }
if KeyboardFieldRouting.route(for: textDocumentProxy.keyboardType) == .handOff {
guard !didHandOffCurrentField else { return }
didHandOffCurrentField = true
if needsInputModeSwitchKey {
advanceToNextInputMode()
}
} else {
didHandOffCurrentField = false
}
}
private func warmFullKeyboardLexiconIfNeeded() {
guard let fullKeyboard else { return }
fullKeyboard.warmUpTextCheckerAsync()
guard !didWarmLexicon else { return }
didWarmLexicon = true
// UIKit calls back on its lexicon-request queue. A main-actor closure here trips
// Swift's runtime isolation check (EXC_BREAKPOINT, TestFlight build 3), so the
// callback is nonisolated and hops to the main actor itself.
requestSupplementaryLexicon { @Sendable [weak self] lexicon in
Task { @MainActor in
self?.fullKeyboard?.loadLexicon(lexicon)
}
}
}
private func configureView() {
view.backgroundColor = .clear
viewModel.inputViewController = self
dictationEdits.onResult = { result in
TypingJournal.sharedIfEnabled()?.recordDictation(result)
}
logDictationEdits.onResult = { [weak self] result in
self?.typingLog?.dictationOutcome(result)
}
localDictation.corrections = {
TypingJournal.sharedIfEnabled()?.dictationCorrections() ?? []
}
viewModel.onDictateTap = { [weak self] in self?.startWarmDictation() }
viewModel.onKeyFeedback = { [weak self] in
guard let self, self.allowsKeyHaptics else { return }
self.barHaptics.impactOccurred(intensity: 0.65)
self.barHaptics.prepare()
}
viewModel.onStopTap = { [weak self] in
guard let self else { return }
if self.localDictation.isActive {
self.localDictation.stop()
// Loading the cleanup model while recording starts ran older iPhones out of
// keyboard memory, so it loads after the microphone closes.
CleanupTagger.prepareIfEnabled(
style: self.bridge.style,
language: LanguageSettings().effectiveDictationLanguage
)
} else {
self.stopWarmDictation()
}
}
viewModel.onCancelTap = { [weak self] in
guard let self else { return }
if self.localDictation.isActive { self.localDictation.cancel() } else { self.cancelWarmDictation() }
}
syncLocalDictationAvailability(access: dictationAccess(now: Date().timeIntervalSince1970))
viewModel.onStyleSelected = { [weak self] style in
guard let self else { return }
self.bridge.style = style
if self.viewModel.style != style {
self.viewModel.style = style
}
self.refreshColdStartURL(now: Date().timeIntervalSince1970, force: true)
}
viewModel.onLanguageSelected = { [weak self] language in
LanguageSettings().chooseDictationLanguage(language)
self?.languageSignature = nil
self?.syncLanguageState()
}
viewModel.onPasteTranslationSelected = { [weak self] translated in
DictationTranslationSettings().pastesTranslation = translated
self?.languageSignature = nil
self?.syncLanguageState()
}
viewModel.onSpace = { [weak self] in
self?.typingLog?.edit("insert", [
TypingLogField("inserted", .string(" ")),
TypingLogField("src", .string("bar")),
])
self?.textDocumentProxy.insertText(" ")
self?.observeDictationEdits(fromKeys: true)
self?.typingLog?.context("space")
}
viewModel.onDelete = { [weak self] in
if let self, let log = self.typingLog {
let deleted = self.textDocumentProxy.documentContextBeforeInput?.last.map { String($0) }
log.edit("delete", [
deleted.map { TypingLogField("deleted", .string($0)) } ?? TypingLogField("deletedCount", .int(1)),
TypingLogField("src", .string("bar")),
])
}
self?.textDocumentProxy.deleteBackward()
self?.observeDictationEdits(fromKeys: true)
}
viewModel.onReturn = { [weak self] in
if let self {
self.typingLog?.edit("return", [
TypingLogField("inserted", .string("\n")),
TypingLogField("returnKeyType", .string(TypingLogTraits.name(self.textDocumentProxy.returnKeyType ?? .default))),
TypingLogField("send", .bool(self.textDocumentProxy.returnKeyType == .send)),
TypingLogField("src", .string("bar")),
])
}
self?.returnKeyWillInsert()
self?.textDocumentProxy.insertText("\n")
self?.observeDictationEdits(fromKeys: true)
self?.typingLog?.context("return")
}
viewModel.onOpenURL = { [weak self] url in
guard let self else { return }
ContainingAppOpener.open(url, from: self)
}
let heightConstraint = view.heightAnchor.constraint(equalToConstant: 140)
heightConstraint.priority = UILayoutPriority(999)
heightConstraint.isActive = true
self.heightConstraint = heightConstraint
mountSurface()
}
/// The full keyboard (iPhone only), or the dictation bar once it was chosen. Without Full
/// Access always the full keyboard: the bar types nothing but spaces and line breaks,
/// and its Dictate needs Full Access. Only switches while idle.
private func mountSurface() {
let wantsFull = (FullKeyboardSettings().isEnabled || !hasFullAccess)
&& UIDevice.current.userInterfaceIdiom == .phone
guard surface == nil || wantsFull != (fullKeyboard != nil) else { return }
guard surface == nil || viewModel.stagePhase == .idle || viewModel.stagePhase == .failed else { return }
children.forEach { child in
child.willMove(toParent: nil)
child.removeFromParent()
}
surface?.removeFromSuperview()
fullKeyboard = nil
didWarmLexicon = false
let mounted: UIView
if wantsFull {
let keyboard = FullKeyboardView(
model: viewModel,
target: DocumentProxyTextTarget { [weak self] in self?.textDocumentProxy }
)
keyboard.hostController = self
keyboard.onPreferredHeightChange = { [weak self] in
self?.view.setNeedsLayout()
self?.view.layoutIfNeeded()
}
keyboard.allowsHaptics = { [weak self] in
self?.allowsKeyHaptics ?? false
}
keyboard.onTextEdit = { [weak self] in self?.observeDictationEdits(fromKeys: true) }
keyboard.onUndoDictation = { [weak self] in
self?.dictationEdits.didUndo()
self?.logDictationEdits.didUndo()
}
keyboard.onReturnKey = { [weak self] in self?.returnKeyWillInsert() }
fullKeyboard = keyboard
mounted = keyboard
heightConstraint?.constant = keyboard.preferredHeight
} else {
let hosting = UIHostingController(
rootView: KeyboardSurfaceView(model: viewModel, defersDotScopeUntilActive: true)
)
hosting.view.backgroundColor = .clear
addChild(hosting)
hosting.didMove(toParent: self)
mounted = hosting.view
heightConstraint?.constant = 140
}
preferredContentSize.height = heightConstraint?.constant ?? 140
mounted.translatesAutoresizingMaskIntoConstraints = false
view.addSubview(mounted)
NSLayoutConstraint.activate([
mounted.leadingAnchor.constraint(equalTo: view.leadingAnchor),
mounted.trailingAnchor.constraint(equalTo: view.trailingAnchor),
mounted.topAnchor.constraint(equalTo: view.topAnchor),
mounted.bottomAnchor.constraint(equalTo: view.bottomAnchor),
])
surface = mounted
attachTypingLog()
schedulePoll(force: false)
}
/// The dictation bar polls at 25 Hz; the full keyboard at 4 Hz, and 30 Hz while recording.
private func schedulePoll(force: Bool) {
let busy = viewModel.stagePhase == .recording || viewModel.stagePhase == .transcribing
|| viewModel.pendingCommand != nil
let interval: TimeInterval = fullKeyboard == nil ? 0.04 : (busy ? 1.0 / 30 : 0.25)
guard force || (pollTimer != nil && interval != pollInterval) else { return }
pollInterval = interval
pollTimer?.invalidate()
pollTimer = Timer.scheduledTimer(withTimeInterval: interval, repeats: true) { [weak self] _ in
MainActor.assumeIsolated {
self?.syncFromBridge(animated: true)
}
}
}
/// Full Access, which dictation needs (Hex's App Group, the microphone). In the simulator
/// a live Hex session stands in for it.
private func dictationAccess(now: TimeInterval) -> Bool {
#if targetEnvironment(simulator)
return hasFullAccess || bridge.snapshot().isAvailable(at: now)
#else
return hasFullAccess
#endif
}
private func syncFromBridge(animated: Bool) {
let now = Date().timeIntervalSince1970
updateKeyboardTraits()
fullKeyboard?.syncSettings()
syncTypingLogEnabled()
syncLoggingMark()
let effectiveFullAccess = dictationAccess(now: now)
syncLocalDictationAvailability(access: effectiveFullAccess)
guard effectiveFullAccess else {
viewModel.applyBridgeState(
hasFullAccess: false,
snapshot: .offline,
isModelInstalled: false,
style: viewModel.style,
now: now,
clearPending: true,
animated: animated
)
return
}
if now - lastFullAccessNote >= 5 {
lastFullAccessNote = now
bridge.fullAccessSeenAt = now
}
if dictationEdits.isTracking {
dictationEdits.tick { editContext }
}
if logDictationEdits.isTracking {
logDictationEdits.tick { editContext }
}
let localSnapshot = localDictation.snapshot(now: now)
let snap = localSnapshot ?? bridge.snapshot()
var insertedResult = false
if snap.hasDeliverableResult(at: now),
let resultID = snap.resultID,
let rawTranscript = snap.transcript,
bridge.lastInsertedResultID != resultID {
let context = editContext
let insertion = TranscriptFormatter.insertionText(
for: rawTranscript,
beforeContext: context.before,
afterContext: context.after,
isWord: dictationWordCheck()
)
if learnsFromDictation {
dictationEdits.willInsert(heard: snap.heard, inserted: insertion, context: context)
} else {
dictationEdits.finish(context)
}
if let log = typingLog, log.isRecording {
// A translation can't be compared with what was heard.
let heard = snap.stages?.translated == nil ? (snap.heard ?? snap.stages?.raw) : nil
logDictationEdits.willInsert(heard: heard, inserted: insertion, context: context)
} else if logDictationEdits.isTracking {
logDictationEdits.finish(context)
}
textDocumentProxy.insertText(insertion)
fullKeyboard?.didInsertDictation(insertion)
typingLog?.dictation(
source: localSnapshot == nil ? "app" : "keyboard",
resultID: resultID,
style: bridge.style.rawValue,
stages: snap.stages,
delivered: rawTranscript,
inserted: insertion
)
observeDictationEdits(fromKeys: true)
bridge.lastInsertedResultID = resultID
localDictation.markDelivered(resultID: resultID)
insertedResult = true
coldStartJobID = UUID().uuidString
}
syncLanguageState()
viewModel.applyBridgeState(
hasFullAccess: true,
snapshot: snap,
isModelInstalled: bridge.isModelInstalled,
style: bridge.style,
now: now,
clearPending: insertedResult,
animated: animated
)
refreshColdStartURL(now: now, force: false)
schedulePoll(force: false)
}
// MARK: - In-keyboard dictation
/// Whether the Dictate and Hex keys dictate in the keyboard; nil until first decided.
private var localDictationAvailable: Bool?
private var lastAvailabilityNote: TimeInterval = 0
/// In-keyboard dictation (iOS 26) needs memory for its capture web view and recording
/// screen, and iOS ends a keyboard that passes its limit (older iPhones, many apps open).
/// While the keyboard is short of it, `onLocalDictateTap` is nil, so Dictate and the Hex
/// key take the cold start that opens Hex, as before iOS 26 (a hold stays a tap there).
/// Without Full Access it is nil too, and the Hex key says dictation needs Full Access.
private func syncLocalDictationAvailability(access: Bool) {
guard KeyboardLocalDictation.isSupported else { return }
guard access || localDictation.isActive else {
if viewModel.onLocalDictateTap != nil { viewModel.onLocalDictateTap = nil }
localDictationAvailable = nil
return
}
let available = localDictation.isActive || localDictation.hasMemoryToStart
guard available != localDictationAvailable else { return }
// Noted at most every 5 s, as the footprint can hover at the line.
let now = Date().timeIntervalSince1970
if localDictationAvailable != nil || !available, now - lastAvailabilityNote >= 5 {
lastAvailabilityNote = now
let change = available ? "memory back, dictating in the keyboard" : "short of memory, Dictate opens Hex"
KeyboardProbeLog.shared.write("\(change) \(KeyboardProbeLog.memory())")
}
localDictationAvailable = available
viewModel.onLocalDictateTap = available ? { [weak self] in
guard let self else { return }
self.localDictation.start(in: self.view, allowBluetoothMic: self.bridge.allowBluetoothMic)
} : nil
}
// MARK: - Dictation language
/// The dictation language's dictionary, for lowercasing a dictation that continues a
/// sentence; nil (case kept) when the iPhone has no dictionary for it.
private func dictationWordCheck() -> ((String) -> Bool)? {
let settings = LanguageSettings()
let language = settings.resolvedDictationLanguage
?? settings.effectiveDictationLanguage
?? Locale.preferredLanguages.first
?? "en-US"
guard TranscriptFormatter.lowercasesContinuations(in: language),
let code = KeyboardLanguage.spellCheckLanguage(
forDictation: language,
available: UITextChecker.availableLanguages
) else { return nil }
let checker = SystemTextChecker(language: code)
return { checker.isWordValid($0) }
}
/// What `syncLanguageState` last read, so the 25 Hz poll only works out the choices on a change.
private var languageSignature: String?
/// Apple Speech's languages are listed once per keyboard process.
private static var didListDictationLanguages = false
/// Settings › Dictation language, what Automatic picks, and the switch's choices, plus
/// Translate what I say's target and paste switch, from the App Group (iOS 26, where
/// dictation is Apple Speech).
private func syncLanguageState() {
guard KeyboardLocalDictation.isSupported else { return }
let settings = LanguageSettings()
let explicit = settings.dictationLanguage
let automatic = settings.automaticDictationLanguage
let recent = settings.recentDictationLanguages
let supported = settings.supportedDictationLanguages
let translation = DictationTranslationSettings()
let target = translation.isEnabled ? translation.targetLanguage : nil
let pastesTranslation = translation.pastesTranslation
let signature = [
explicit ?? "", automatic ?? "", recent.joined(separator: ","), "\(supported.count)",
target ?? "", "\(pastesTranslation)",
].joined(separator: "|")
guard signature != languageSignature else { return }
languageSignature = signature
let choices = DictationLanguageSwitch.choices(
current: explicit,
recent: recent,
preferred: Locale.preferredLanguages,
supported: supported
)
if viewModel.dictationLanguage != explicit { viewModel.dictationLanguage = explicit }
if viewModel.automaticLanguage != automatic { viewModel.automaticLanguage = automatic }
if viewModel.languageChoices != choices { viewModel.languageChoices = choices }
if viewModel.translationTarget != target { viewModel.translationTarget = target }
if viewModel.pastesTranslation != pastesTranslation { viewModel.pastesTranslation = pastesTranslation }
}
/// Asks Apple Speech which languages it has and what Automatic picks, so the switch can
/// offer them before Hex has listed them.
private func listDictationLanguagesIfNeeded() {
guard #available(iOS 26.0, *), hasFullAccess, !Self.didListDictationLanguages else { return }
Self.didListDictationLanguages = true
Task { [weak self] in
_ = await AppleSpeechLanguages.options()
_ = await AppleSpeechLanguages.automatic()
self?.languageSignature = nil
self?.syncLanguageState()
}
}
private func refreshColdStartURL(now: TimeInterval, force: Bool) {
let bucket = Int(now / 2.0)
guard force || bucket != lastURLRefreshBucket else { return }
lastURLRefreshBucket = bucket
mountSurface()
let request: KeyboardLaunchRequest
if bridge.isModelInstalled {
request = .startRecording(
jobID: coldStartJobID,
requestedAt: now,
style: bridge.style
)
} else {
request = .startSession(requestedAt: now)
}
viewModel.coldStartURL = request.url
}
private func updateKeyboardTraits() {
let globe = needsInputModeSwitchKey
if viewModel.needsGlobeKey != globe {
viewModel.needsGlobeKey = globe
}
// No interface style override: iOS already gives the keyboard the field's appearance
// (a field that asks for a dark keyboard gets dark traits in light mode, like the system
// keyboard), while the document proxy's `keyboardAppearance` can go stale.
let nextReturn: String = switch textDocumentProxy.returnKeyType {
case .send: "Send"
case .google, .yahoo, .search: "Search"
case .go: "Go"
case .done: "Done"
case .join: "Join"
case .next: "Next"
case .route: "Route"
case .continue: "Continue"
case .emergencyCall: "SOS"
default: "Return"
}
if viewModel.returnKeyLabel != nextReturn {
viewModel.returnKeyLabel = nextReturn
}
}
private func startWarmDictation() {
let now = Date().timeIntervalSince1970
guard viewModel.hasFullAccess, viewModel.currentSnapshot.isAvailable(at: now) else { return }
let jobID = UUID().uuidString
viewModel.setPendingCommand(
KeyboardViewModel.PendingCommand(kind: .start, jobID: jobID, sentAt: now),
now: now,
animated: true
)
bridge.send(.start, jobID: jobID, style: viewModel.style, sentAt: now)
}
private func stopWarmDictation() {
let now = Date().timeIntervalSince1970
guard let jobID = viewModel.currentSnapshot.jobID ?? viewModel.pendingCommand?.jobID else { return }
viewModel.setPendingCommand(
KeyboardViewModel.PendingCommand(kind: .stop, jobID: jobID, sentAt: now),
now: now,
animated: true
)
bridge.send(.stop, jobID: jobID, style: viewModel.style, sentAt: now)
}
private func cancelWarmDictation() {
let now = Date().timeIntervalSince1970
guard let jobID = viewModel.currentSnapshot.jobID ?? viewModel.pendingCommand?.jobID else { return }
viewModel.setPendingCommand(
KeyboardViewModel.PendingCommand(kind: .cancel, jobID: jobID, sentAt: now),
now: now,
animated: true
)
bridge.send(.cancel, jobID: jobID, style: viewModel.style, sentAt: now)
}
}