Files
hex/ios/scripts/build-emoji-data.py
Ryan Vogel cc75290551 iOS: emoji panel with on-device suggestions (Experiments › Emoji) (#2051)
* iOS: emoji index built from Unicode's emoji list and CLDR

What changed
- scripts/build-emoji-data.py builds HEXShared/FullKeyboard/Emoji/emoji-index-<lang>.txt
  for the four keyboard layouts (en, nb, es, de) from Unicode emoji-test.txt 18.0
  (pinned by SHA-256) and CLDR 49 annotations and annotationsDerived (release-49 tag),
  plus Hex's own chat words in scripts/emoji-aliases-en.txt ("congrats", "bday",
  "lol", "new year"; 336 words, written for Hex).
- Each file holds the catalog in panel order (8 categories, 1,923 emoji, 332 with five
  uniform skin tones), each emoji's English name, its folded search words, and the
  word and phrase postings already weighted (one-word name 1.0, name word 0.75,
  keyword 0.6, keyword-phrase word 0.3, chat word 1.2 down to 0.9, an IDF factor for
  words many emoji share), sorted by key so the keyboard can binary-search the file
  in place. Non-English layouts carry English's words at 0.85.
- LICENSE-Unicode.txt (Unicode License v3) ships beside the data, as the licence
  asks: both emoji-test.txt and CLDR's data are under it, which allows commercial
  use with the notice.

Why
- The emoji panel (next commits) suggests emoji for what you type and searches them
  on the iPhone. Precomputing the index keeps the keyboard extension's heap small:
  the first version built Swift dictionaries at runtime and held 5.7 MB, which
  malloc kept after release; the mapped file plus offsets measures 0.7 MB.

Verified
- The generator runs from a sources folder or downloads the pinned files; output is
  deterministic. Files: en 351 KB, nb 560 KB, es 549 KB, de 610 KB.

* iOS: rank emoji for the text before the cursor, on device

What changed
- EmojiLibrary maps an emoji-index file (Data .alwaysMapped) and keeps only the emoji,
  their skin tones and byte offsets: word and phrase lookups binary-search the file's
  sorted keys and parse postings in place; search scans each emoji's folded name and
  keyword words for prefixes (name words first, a short name the query fills before a
  long one it touches, then your use and everyday emoji). Emoji newer than 15.1 are
  listed only if the emoji font itself draws them as one glyph.
- EmojiSuggester reads the last eight content words (the newest counts 1, then 0.85
  each word back), their forms (plurals, -ing, -ed, held-down letters: "lmaooo"),
  two- to four-word phrases ("new year", "good luck") and the word still being typed;
  an emoji several words point at gains a third of the rest. One emoji per family
  ("man running", "woman running"); your use breaks ties and fills the row, then
  everyday defaults. forFinishedWord gives the strip at most three strong matches for
  the word just finished ("pizza " -> 🍕).
- EmojiUsageStore keeps, in the App Group, a use count per emoji that halves every 10
  days (frequency and recency in one score, at most 120 emoji) and the skin tone last
  picked for each; no words. clear() empties both.
- EmojiRoutedTarget lets the keys type into the emoji search's query instead of the
  field while you search.
- EmojiSemanticIndex (app target only, for the comparison test): Apple's on-device
  NLEmbedding word vectors over the index's 3,257 words, standing in for a context word
  with no emoji of its own. Not used by the keyboard (see Measured).
- EmojiSuggesterTests: index, folding, context, families, endings, other layouts,
  strip, search, history decay and tones, the search's routing, and quality on 60
  hand-written sentences (Fixtures/emoji-context-sentences.json, 30 to tune, 30 held
  out) with and without the embedding.

Measured (sensible emoji in the top 5, context only; top-1 in brackets)
- Simulator, iPhone 17, iOS 27.2: keywords dev 30/30 (27), held-out 30/30 (27), 0.04 ms
  a sentence; with the embedding the same counts at 0.14 ms, its index built in 120 ms.
- Mac harness, same code: CLDR keywords alone dev 26/30 (13), held-out 22/30 (14);
  with Hex's chat words 30/30 (27) and 30/30 (27). A blind set of 40 messages and
  sensible emoji written by a separate agent that saw no code or word lists: CLDR
  alone 30/40 (13), with chat words 37/40 (28), adding the embedding 37/40 (28); 38/40
  after the held-down-letter rule, which those failures suggested. Apple's sentence
  embedding scored 18/40 (7) and took 7.5 s to embed the 1,923 descriptions.
- Memory on the Mac: the mapped index adds 0.72 MB; the embedding adds 5.3 MB (1 MB
  NLEmbedding, 3.9 MB vectors) and none of it came back after release. No measurable
  gain for that, so the keyboard uses keywords and your history only.

Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures.

* iOS: emoji panel for the full keyboard (Experiments › Emoji)

What changed
- Experiments › Full keyboard › Emoji (experiments.emoji, off by default; shown with the
  full keyboard): an emoji key (face.smiling) right of 123, after the globe, in plain
  text fields (not URL, email or ASCII-only ones). The bottom row takes the compact
  sizes it uses beside the globe. "Clear emoji history" empties EmojiUsageStore.
- EmojiPanelView replaces the strip and the keys: a row of up to ten suggestions for
  the text before the cursor and your history (one emoji per family, your skin tone),
  a search button; a grid that scrolls sideways through Frequently used (your history,
  then everyday emoji, 16 at least) and the eight categories in four rows of 42 pt
  (three of 38 in landscape), each section's name kept at the left edge while it
  scrolls; a bar with ABC, the categories and a repeating delete, drawn like the
  letter keys' 123 and delete. Holding an emoji with skin tones opens the accent
  picker with its five tones; the pick is typed and remembered for that emoji. Key
  pop-ups and haptics follow their switches.
- Search: the letter keys type into a search field in the strip (EmojiRoutedTarget),
  results above it in a row the keyboard grows 44 pt for, your suggestions while it's
  empty; return, the field's close button or the emoji key go back to the panel.
  Typing logs and Learn how I type see the emoji typed into the field, never the query.
- The strip, when it would be empty after a space, offers the finished word's strong
  emoji ("pizza " -> 🍕, "birthday " -> 🎂 🎉 🥳), a lone one in the centre slot; never
  over word suggestions, Undo dictation, a status or the clipboard chips.
- Grid cells drop their text (and bitmap) when they scroll off and prefetching is off:
  scrolling every section had kept 353 cell bitmaps, 35 MB.
- FullKeyboardView+Emoji.swift holds the wiring; the engine's target is now the routed
  target, which forwards everything to the field unless a search is open.
- Scenes: full-keyboard-emoji, -emoji-search (--emoji-query), -emoji-tones,
  -emoji-food, -emoji-strip (each -dark and -dark-field), and -emoji-probe, which prints
  the panel's footprint and open latency.

Measured in the keyboard extension (Release, iPhone 17 simulator, iOS 27.2, a field
focused with the real HEXKeyboard.appex and a temporary, uncommitted probe; the
simulator counts no GPU memory and reports no headroom), footprint:
  keyboard with the emoji key, nothing loaded   19.0 MB
  after "pizza " (the strip loads the index)    19.0 MB
  panel open                                     28.7 MB
  picked three suggestions, closed               22.7 MB
  every section scrolled                         41.6 MB (peak 48.4; 71.1 before the cell fix)
  closed after that                              36.5 MB
The 17.5 MB left after browsing everything is about 351 emoji glyph images the system's
text rendering caches process-wide (18.7 MB in the heap, 351 CGImages); drawing with a
font instance the panel owns or at 21 pt left it the same, and no public API frees it.
Whether iOS purges it under memory pressure is unverified on an iPhone.
Opening: 22-26 ms from the emoji key to the panel laid out the first time, 12-14 ms
after (median of 10), about 21 ms until committed.

Verified: xcodebuild test (245 tests, 1 skipped, 0 failures); every scene above in
light, dark and dark field looked at on the simulator.

* iOS: document the emoji experiment and keep its blind sentences as a check

What changed
- README: the emoji experiment under Capabilities (what the panel, search, skin tones
  and strip do, what the history keeps, where the index comes from and its licence),
  how to rebuild the index, and its proof limits with the quality, memory and latency
  measurements and what is unverified on an iPhone (whether iOS purges the cached
  emoji glyph images that browsing every section leaves, about 18 MB).
- emoji-context-sentences.json gains the 40-message blind set a separate agent wrote
  without seeing code or word lists; EmojiSuggesterTests scores it beside the other
  60 and fails below 35/40.

Why: the blind set is the least biased quality number; keeping it catches a ranking
change that helps the hand-written sentences at its expense.

Verified: xcodebuild test on an iPhone 17 simulator (iOS 27.2): 245 tests, 1 skipped,
0 failures; EMOJI_QUALITY: dev 30/30 (top-1 27), held-out 30/30 (27), blind 38/40 (29),
the same with the embedding, which built in 101 ms.
2026-11-20 15:53:24 +00:00

307 lines
13 KiB
Python
Executable File
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env python3
"""Builds the emoji panel's index files from Unicode's data and Hex's chat words.
Sources:
- emoji-test.txt 18.0 (Unicode License v3): the emoji, their order, groups, versions
and skin tones
- CLDR 49 annotations and annotationsDerived (Unicode License v3): names and keywords
in English, Norwegian, Spanish and German
- scripts/emoji-aliases-en.txt (Hex's own): chat words such as "congrats" and "bday"
scripts/build-emoji-data.py [--sources DIR] [--no-aliases] [--out DIR]
With --sources, reads emoji-test.txt, ann-<cldr>.xml and der-<cldr>.xml from DIR
instead of downloading them. Writes HEXShared/FullKeyboard/Emoji/emoji-index-<lang>.txt:
one file per keyboard layout, its language's words with English's beside them,
already folded, weighted and sorted, so the keyboard only reads it into flat buffers.
Folding (must match EmojiText.fold in Swift): NFD, drop combining marks, lowercase,
curly apostrophes made straight.
"""
import argparse
import hashlib
import math
import pathlib
import re
import sys
import unicodedata
import urllib.request
import xml.etree.ElementTree as ET
EMOJI_TEST_URL = "https://www.unicode.org/Public/18.0.0/emoji/emoji-test.txt"
EMOJI_TEST_SHA256 = "8f3735cda1f92a779d78af67cf86066bb1f07143dc22f2ac29394d9bc57ab21a"
CLDR_TAG = "release-49"
CLDR_BASE = f"https://raw.githubusercontent.com/unicode-org/cldr/{CLDR_TAG}/common"
# Hex's keyboard layouts; CLDR files Norwegian Bokmål under "no".
LANGUAGES = {"en": "en", "nb": "no", "es": "es", "de": "de"}
HERE = pathlib.Path(__file__).resolve().parent
# The panel's categories, in the order its bar shows them.
CATEGORIES = {
"Smileys & Emotion": 0,
"People & Body": 0,
"Animals & Nature": 1,
"Food & Drink": 2,
"Activities": 3,
"Travel & Places": 4,
"Objects": 5,
"Symbols": 6,
"Flags": 7,
}
TONES = [0x1F3FB, 0x1F3FC, 0x1F3FD, 0x1F3FE, 0x1F3FF]
LINE = re.compile(r"^([0-9A-F ]+?)\s*;\s*([\w-]+)\s*#\s*\S+\s+E(\d+\.\d+)\s+(.+)$")
# Words too common to say anything about an emoji, in the four layouts' languages.
# The keyboard reads this list from the index, so it lives only here.
STOPWORDS = """
a an the i im me my mine you your youre we our us it its is are was were be been being am
to of in on at for with and or but so just this that these those have has had do does did
will would can could should shall not all some any about from by up out if then than too
very really get got he she they them his her their what when where how who why there here
now again into onto over as also still let lets gonna wanna dont cant ll ve re much more
most one ones other off such only own same both each few s t d m
og jeg du det er en et ei som pa til med har av ikke vi de den seg for om fra var men sa kan skal
el la los las un una y o que de del en es por con para lo se su al mi tu yo muy ya
der die das und ist ich nicht ein eine zu mit sich auf fur dem den im es wir ihr sie aber auch so
""".split()
STOP = set(STOPWORDS)
def fold(text):
text = unicodedata.normalize("NFD", text.replace("’", "'"))
return "".join(c for c in text if not unicodedata.combining(c)).lower()
def words(text):
"""Folded words: letters and digits, an apostrophe's tail dropped ("that's" -> "that")."""
out = []
for part in re.split(r"[^\w']+|_", fold(text)):
head = part.split("'", 1)[0]
if head:
out.append(head)
return out
def fetch(url):
with urllib.request.urlopen(url, timeout=60) as response:
return response.read()
def read_sources(folder):
if folder:
test = (folder / "emoji-test.txt").read_bytes()
ann = {lang: (folder / f"ann-{cldr}.xml").read_bytes() for lang, cldr in LANGUAGES.items()}
der = {lang: (folder / f"der-{cldr}.xml").read_bytes() for lang, cldr in LANGUAGES.items()}
else:
test = fetch(EMOJI_TEST_URL)
ann = {lang: fetch(f"{CLDR_BASE}/annotations/{cldr}.xml") for lang, cldr in LANGUAGES.items()}
der = {lang: fetch(f"{CLDR_BASE}/annotationsDerived/{cldr}.xml") for lang, cldr in LANGUAGES.items()}
digest = hashlib.sha256(test).hexdigest()
if digest != EMOJI_TEST_SHA256:
sys.exit(f"emoji-test.txt changed: {digest}")
return test.decode("utf-8"), ann, der
def parse_emoji(text):
"""Fully-qualified emoji in file order, skin tone variants folded into their base."""
group = ""
bases = [] # [codepoints, group, version, name, tones]
by_key = {}
variants = []
for raw in text.splitlines():
if raw.startswith("# group:"):
group = raw.split(":", 1)[1].strip()
continue
match = LINE.match(raw)
if not match or match.group(2) != "fully-qualified" or group == "Component":
continue
points = tuple(int(p, 16) for p in match.group(1).split())
tones = [p for p in points if p in TONES]
if tones:
variants.append((points, tones))
continue
key = tuple(p for p in points if p != 0xFE0F)
by_key[key] = len(bases)
bases.append([points, group, match.group(3), match.group(4), [None] * 5])
for points, tones in variants:
# One tone for everyone in it: the picker offers five.
if len(set(tones)) != 1:
continue
key = tuple(p for p in points if p not in TONES and p != 0xFE0F)
index = by_key.get(key)
if index is None:
continue
slot = TONES.index(tones[0])
if bases[index][4][slot] is None:
bases[index][4][slot] = points
return bases
def to_string(points):
return "".join(chr(p) for p in points)
def annotations(*sources):
"""CLDR names and keywords by emoji (FE0F removed); the first source wins."""
names = {}
keywords = {}
for source in sources:
root = ET.fromstring(source)
for node in root.iter("annotation"):
emoji = node.get("cp", "").replace("️", "")
text = (node.text or "").strip()
if not emoji or not text:
continue
if node.get("type") == "tts":
names.setdefault(emoji, text)
else:
keywords.setdefault(emoji, [k.strip() for k in text.split("|") if k.strip()])
return {e: (names.get(e, ""), keywords.get(e, [])) for e in set(names) | set(keywords)}
def family(name):
"""'man running', 'woman running' and 'person running facing right' are one family."""
name = name.split(":", 1)[0]
if name.endswith(" facing right"):
name = name[: -len(" facing right")]
for prefix in ("man ", "woman ", "person ", "men ", "women ", "people "):
if name.startswith(prefix):
return name[len(prefix):]
return name
class Index:
def __init__(self, count):
self.words = {} # key -> {entry: weight}
self.phrases = {}
self.alias_keys = set()
self.search_names = [[] for _ in range(count)]
self.search_keywords = [[] for _ in range(count)]
def _add(self, table, key, entry, weight):
hits = table.setdefault(key, {})
hits[entry] = max(hits.get(entry, 0), weight)
def name(self, text, entry, scale):
parts = words(text)
for part in parts:
if part not in self.search_names[entry]:
self.search_names[entry].append(part)
content = [p for p in parts if p not in STOP and len(p) >= 2]
# A one-word name ("pizza") is the strongest match there is.
weight = 1.0 if len(content) <= 1 else 0.75
for part in content:
self._add(self.words, part, entry, weight * scale)
if 1 < len(parts) <= 4:
self._add(self.phrases, " ".join(parts), entry, 0.95 * scale)
def keyword(self, text, entry, scale):
parts = words(text)
for part in parts:
if part not in self.search_keywords[entry] and part not in self.search_names[entry]:
self.search_keywords[entry].append(part)
content = [p for p in parts if p not in STOP and len(p) >= 2]
if len(parts) > 1:
if len(parts) <= 4:
self._add(self.phrases, " ".join(parts), entry, 0.85 * scale)
for part in content:
self._add(self.words, part, entry, 0.3 * scale)
else:
for part in content:
self._add(self.words, part, entry, 0.6 * scale)
def alias(self, text, entries):
"""Hex's chat words: the first emoji listed is the best answer."""
parts = words(text)
if not parts:
return
key = " ".join(parts)
self.alias_keys.add(key)
table = self.words if len(parts) == 1 else self.phrases
for rank, entry in enumerate(entries):
self._add(table, key, entry, max(0.9, 1.2 - 0.1 * rank))
def finish(self, table):
"""A word that names many emoji ("face", "hand") says little about any one of them."""
out = []
for key in sorted(table):
hits = table[key]
factor = 1.0
if key not in self.alias_keys:
factor = 1 / (1 + 0.5 * math.log2(max(1.0, len(hits) / 2)))
ranked = sorted(hits.items(), key=lambda item: (-item[1] * factor, item[0]))
out.append((key, [(entry, weight * factor) for entry, weight in ranked]))
return out
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--sources", type=pathlib.Path)
parser.add_argument("--no-aliases", action="store_true", help="CLDR words only, for measuring")
parser.add_argument("--out", type=pathlib.Path, default=HERE.parent / "HEXShared" / "FullKeyboard" / "Emoji")
args = parser.parse_args()
args.out.mkdir(parents=True, exist_ok=True)
test, ann, der = read_sources(args.sources)
bases = parse_emoji(test)
ordered = [b for _, b in sorted(enumerate(bases), key=lambda item: (CATEGORIES[item[1][1]], item[0]))]
position = {to_string(b[0]): i for i, b in enumerate(ordered)}
tables = {lang: annotations(ann[lang], der[lang]) for lang in LANGUAGES}
aliases = []
if not args.no_aliases:
for raw in (HERE / "emoji-aliases-en.txt").read_text(encoding="utf-8").splitlines():
if raw and not raw.startswith("#"):
word, targets = raw.split("\t")
aliases.append((word, [position[e] for e in targets.split(" ") if e in position]))
families = {}
for lang in LANGUAGES:
langs = [lang] if lang == "en" else [lang, "en"]
index = Index(len(ordered))
english_names = []
for i, (points, _, _, test_name, _) in enumerate(ordered):
emoji = to_string(points)
for rank, source in enumerate(langs):
# English beside another layout counts a little less than the layout's own words.
scale = 1.0 if rank == 0 else 0.85
name, keywords = tables[source].get(emoji.replace("️", ""), ("", []))
if source == "en":
english_names.append(name or test_name)
name = name or test_name
# A flag's name ("flag: Japan") also gives its place as a keyword.
if ":" in name and "\U0001F1E6" <= emoji[0] <= "\U0001F1FF":
keywords = keywords + [name.split(":", 1)[1].strip()]
index.name(name, i, scale)
for keyword in keywords:
index.keyword(keyword, i, scale)
if langs[-1] == "en":
for word, entries in aliases:
index.alias(word, entries)
lines = [
"# Generated by scripts/build-emoji-data.py; do not edit.\n",
"# Unicode emoji-test.txt 18.0 and CLDR 49 (Unicode License v3, LICENSE-Unicode.txt),\n",
"# and Hex's own chat words (scripts/emoji-aliases-en.txt).\n",
"# S stopwords; E emoji, category, version, skin tones, family, name, search words\n",
"# (name's | keywords'); W word, entry:weight...; P phrase, entry:weight...\n",
"S\t" + " ".join(STOPWORDS) + "\n",
]
for i, (points, group, version, _, tones) in enumerate(ordered):
name = english_names[i]
family_id = families.setdefault(family(name), len(families))
toned = " ".join(to_string(t) for t in tones) if all(tones) else ""
search = " ".join(index.search_names[i]) + "|" + " ".join(index.search_keywords[i])
lines.append(f"E\t{to_string(points)}\t{CATEGORIES[group]}\t{version}\t{toned}\t{family_id}\t{name}\t{search}\n")
for tag, table in (("W", index.words), ("P", index.phrases)):
for key, hits in index.finish(table):
postings = ",".join(f"{entry}:{weight:.3f}" for entry, weight in hits)
lines.append(f"{tag}\t{key}\t{postings}\n")
path = args.out / f"emoji-index-{lang}.txt"
path.write_text("".join(lines), encoding="utf-8")
print(f"{path.name}: {len(ordered)} emoji, {len(index.words)} words, {len(index.phrases)} phrases, {path.stat().st_size // 1024} KB")
if __name__ == "__main__":
main()