mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 04:50:36 +08:00
The listening section quoted a strobe reading in ms; it now quotes a 0.31 check line and names --unit ms.
219 lines
12 KiB
Markdown
219 lines
12 KiB
Markdown
# Performance benchmarks
|
|
|
|
Hex is benchmarked with [strobe](https://github.com/anomalyco/strobe) as a native
|
|
app. strobe builds two variants (for example `origin/dev` and your checkout),
|
|
launches each in a fresh private home, and compares them in alternating runs.
|
|
|
|
```sh
|
|
strobe bench perf/idle.bench.ts --base origin/dev --head .
|
|
strobe bench perf/launch.bench.ts --base origin/dev --head .
|
|
strobe bench perf/idle.bench.ts --check # one short, untimed pass
|
|
strobe bench perf/idle.bench.ts --filter "window-closed*" --base origin/dev --head .
|
|
strobe report experiment <exp_…> --format pr --update-pr <n> # the results block in PR n
|
|
```
|
|
|
|
`--filter` and `--target` take names, comma lists, and globs.
|
|
|
|
## In a pull request
|
|
|
|
Write the PR's own text yourself: what strobe showed that led to the change
|
|
(the command and recording id), and what the change does. Then let strobe
|
|
place its evidence:
|
|
|
|
```sh
|
|
strobe report experiment <exp_…> --format pr --update-pr <n>
|
|
```
|
|
|
|
It adds one generated block to PR `n`'s body, or replaces that experiment's
|
|
block after a rerun, and leaves the rest of the body alone. Don't edit inside
|
|
the block (strobe refuses to update a hand-edited one); put your reading of
|
|
the numbers above or below it. Without `--update-pr`, the block goes to
|
|
stdout for pasting.
|
|
|
|
Window cases name the screen they measure with `expect` (the window title
|
|
and the pane's heading). A block on another screen, such as a first-run
|
|
dialog over the pane, is refused `WrongScreen` instead of timed, and the
|
|
block's `On screen:` line says what showed.
|
|
|
|
Run timed benches on a quiet machine. Close other apps and keep the session
|
|
unlocked.
|
|
|
|
## What runs
|
|
|
|
- `strobe.json` builds `hex` with `cargo build --release` into
|
|
`{cache}/cargo-target`, a target folder strobe shares between variants.
|
|
The build output is `hex.exe` on Windows and `hex` on macOS and Linux.
|
|
- The build's env keeps release's code and adds line tables in `hex.pdb`, so
|
|
CPU profiles name Hex's functions: `CARGO_PROFILE_RELEASE_DEBUG=line-tables-only`,
|
|
`CARGO_PROFILE_RELEASE_STRIP=false`, and `CARGO_PROFILE_RELEASE_INCREMENTAL=false`.
|
|
It needs no profile or feature in `Cargo.toml`, so every commit builds the
|
|
same way: a `--base` from before the benches existed, and `--bisect` across
|
|
history. Such a build is never shipped.
|
|
- On Windows, `build.windows` sets `RUSTUP_TOOLCHAIN=stable-x86_64-pc-windows-msvc`,
|
|
so a machine whose default Rust toolchain is GNU still builds with MSVC
|
|
(GNU fails with "dlltool not found"). Windows builds also need the Vulkan
|
|
SDK (`docs/windows.md`).
|
|
- Each variant keeps the GPU driver's compiled shaders across its own
|
|
launches (strobe's default, `gpuCache: "warm"`), as a machine that ran Hex
|
|
before has them. Base and head never share them, and discarded warm-up
|
|
launches fill them before the timed pairs. Without them, every launch
|
|
compiled Hex's shaders again: a dictation spent 4.6 to 5.3 s of each 6 s
|
|
run in the driver's compiler. So `cold-launch` is a returning user's launch
|
|
(Hex not running, shaders cached). A first launch after an install or a
|
|
driver update would need a target with `gpuCache: "cold"`.
|
|
|
|
## Isolation
|
|
|
|
The `hex` app profile in `strobe.json` keeps a launched copy away from your
|
|
own Hex:
|
|
|
|
- `HEX_APPLICATION_SUPPORT_DIR={home}/hex-data`: settings, history, logs,
|
|
models, locks, and `local-api.json` live in the private home.
|
|
- A data root other than the default claims its own Windows single-instance
|
|
event, so the copy never signals or yields to your running Hex, and on
|
|
Linux never hands the launch to your systemd unit. Inside a redirected
|
|
home the default (`AppData/Roaming/hex`, `.local/share/hex`) follows the
|
|
redirection, so name another folder, as `hex-data` does.
|
|
- `HEX_DISABLE_UPDATES=1`: no update checks and no network.
|
|
- `USERPROFILE` (Windows) and `HOME` locate `~/.config/hex`, so your
|
|
`hex.config.ts` stays out of the copy.
|
|
- Every folder Hex and Annotate derive from yours follows the copy's
|
|
environment on Windows too (`USERPROFILE`, `APPDATA`, `LOCALAPPDATA`;
|
|
`app_paths` and `scrawl_router::user_dirs`), where `dirs` would ask the
|
|
shell for your own: Annotate's captures folder and its retention, Scrawl's
|
|
router folders and the Scrawl settings import, the meetings folder, and the
|
|
Hugging Face cache. Annotate stays on in the benches, as it is for users,
|
|
and works only inside the private home.
|
|
- `HF_HUB_CACHE={home}/.cache/huggingface/hub`: the copy never finds models
|
|
in your Hugging Face cache (on Windows that scan otherwise reads your real
|
|
profile).
|
|
- The seeded `settings.json` keeps the listener off (`enable_on_launch: false`):
|
|
no keyboard hook, no microphone, and no model load. History is kept
|
|
forever, so a seeded `history.json` never expires mid-run. The seeded
|
|
`onboarding-complete` marker skips first-run setup. The two `scrawl-*`
|
|
migrations skip importing Scrawl's settings from your home.
|
|
- `guard` stops a run if your own `settings.json` or `local-api.json`
|
|
changes, or your Hex exits. Each guard names its OS.
|
|
- macOS: a variant runs as a bare `hex` executable, not `Hex.app`, so it has
|
|
no bundle id and shares no Preferences, saved state, or permissions with
|
|
your installed Hex. `moved` tells strobe that the copy keeps its data in
|
|
the private home (`HEX_APPLICATION_SUPPORT_DIR`), not in your
|
|
`~/Library/Application Support/hex`, so your own Hex there does not stop
|
|
the bench. strobe brings the copy's window to the front only after a
|
|
minute without keyboard or mouse input. Every profile records the
|
|
`anomaly-hex-identity` migration: without it, a copy whose setup isn't
|
|
ready (the copy has no Microphone grant) reopens the first-run tour over
|
|
the window, and every click lands on the tour.
|
|
|
|
The copy loads no model, except in the listening and dictation cases below.
|
|
|
|
## Cases
|
|
|
|
| File | Case | Primary | Measures | Locked session |
|
|
| --- | --- | --- | --- | --- |
|
|
| `perf/idle.bench.ts` | `window-open` | `cpu.work` | 8 s at rest with the window on General | waits for an unlock |
|
|
| `perf/idle.bench.ts` | `window-closed` | `cpu.work` | 8 s at rest, tray only (`app --hidden`) | runs |
|
|
| `perf/idle.bench.ts` | `window-closed-memory` | `memory.app.private` | The same tray's private memory and dedicated GPU memory (`gpu.vram`) | runs |
|
|
| `perf/launch.bench.ts` | `cold-launch` | `launch.ready` | Process start to the first painted window | waits for an unlock |
|
|
| `perf/launch.bench.ts` | `hidden-launch` | `launch.ready` | Process start to `local API started`, tray only | runs |
|
|
| `perf/listening.bench.ts` | `listening-at-rest` | `cpu.work` | 8 s listening at rest, microphone released and open | runs |
|
|
| `perf/dictation.bench.ts` | `recording` | `cpu.wakes` | 6 s of one dictation recording a fixture, microphone open and released | runs |
|
|
| `perf/interaction.bench.ts` | `switch-panes` | `ui.cpu` | General → History → Output → General, each click until the next pane's content is in the accessibility tree | waits for an unlock |
|
|
| `perf/interaction.bench.ts` | `history-scroll` | `ui.cpu` | History's list scrolled down 6,000 points and back at 3,000 points a second | waits for an unlock |
|
|
| `perf/interaction.bench.ts` | `history-search` | `ui.cpu` | "cache key" typed into History's search, a key every 120 ms; each key searches 10,000 entries | waits for an unlock |
|
|
| `perf/interaction.bench.ts` | `model-picker` | `ui.cpu` | The speech model picker opened, its list of every model, back to the comparison, and closed | waits for an unlock |
|
|
|
|
Secondary metrics are thread wakeups (`cpu.wakes`), GPU use, and private
|
|
memory. `strobe query <rec> threads` names the threads; Hex names every
|
|
long-lived thread, and the window's thread is `hex-ui` on Windows.
|
|
|
|
Tray cases use `window: false`: strobe neither waits for nor checks a window,
|
|
so they also run while the session is locked. Cases with a window, or with
|
|
presented frames, wait for an unlock.
|
|
|
|
To see where one case spends its time, record it once:
|
|
|
|
```sh
|
|
strobe record --bench perf/idle.bench.ts#window-closed --head .
|
|
strobe query <rec> threads
|
|
```
|
|
|
|
## Listening cases
|
|
|
|
`perf/listening.bench.ts` runs Hex with its listener on: the
|
|
`hex-listening` profile (microphone released while idle) and
|
|
`hex-listening-warm` (microphone open, the default). The copy installs its
|
|
keyboard hook and loads the default English model, which
|
|
`perf/seeds/listening.ts` copies from this machine's own Hex install into
|
|
`perf/seeds/.generated`. strobe shares a seed file this large (64 MB and up)
|
|
read only between launches instead of copying it into each private home. The warm profile opens the machine's real microphone; neither records.
|
|
|
|
Setup waits until the copy's event log shows `session_started`, which
|
|
Windows and Linux write once the model has loaded and warmed, then 2 s
|
|
more, and at least 10 s in all. A new build's first launch compiles the GPU
|
|
driver's shaders during that load: about 5 s of CPU on an idle machine, and
|
|
over 10 s on a busy one. A fixed 10 s wait let that compile run into the
|
|
measured runs, which then read 1 to 2.5 s of `cpu.work` instead of tens of
|
|
milliseconds. strobe prints times in microseconds (`--unit ms` for
|
|
milliseconds): a check of the released profile on a busy Windows machine
|
|
reads `cpu.work 27,219 µs · cpu.wakes 134/s`.
|
|
|
|
As in the dictation cases, the profiles move the copy's shortcuts to F19
|
|
and F18 and unbind Paste Last, so a key pressed while a bench runs on a
|
|
machine in use starts nothing. The copy's keyboard and mouse hooks still
|
|
see every key and mouse move, so input on the bench machine adds
|
|
`cpu.work` and `cpu.wakes`: compare runs from an idle machine.
|
|
|
|
## Dictation cases
|
|
|
|
`perf/dictation.bench.ts` runs whole dictations with no microphone and no
|
|
paste. The `hex-dictation` profiles set `HEX_AUDIO_FIXTURE` to `speech.wav`,
|
|
which `perf/seeds/dictation.ts` makes with this machine's own
|
|
text-to-speech voice (one sentence at 48 kHz, then 2 s of silence) beside
|
|
the model the listening seed links. Every microphone open in the copy plays
|
|
that file instead, from its start, in 10 ms chunks at real-time pace, and
|
|
no device is opened, so nothing is recorded.
|
|
|
|
The bench drives the copy through its local API as `hex dictation` does.
|
|
Each dictation starts as the sentence begins (with the microphone open,
|
|
when the looping file comes round) and ends with `dictation stop --print`,
|
|
which returns the processed text instead of pasting it. The profiles move
|
|
the copy's shortcuts to F19 and F18 and unbind Paste Last, so a key pressed
|
|
while a bench runs on a machine in use reaches nothing.
|
|
|
|
Only bench builds that have `HEX_AUDIO_FIXTURE`. An older build ignores it
|
|
and opens the real microphone; setup then refuses the block, but only
|
|
after the microphone was open.
|
|
|
|
## Interaction cases
|
|
|
|
`perf/interaction.bench.ts` uses the Settings window the way a person does.
|
|
strobe drives it from outside, through what Windows gives assistive
|
|
technology: it finds each control in Hex's accessibility tree (UI
|
|
Automation, which Rune builds with AccessKit) and posts mouse and key input to
|
|
the window. Hex needs no build feature, flag, or bridge for this; a release
|
|
build answers the same way for a screen reader.
|
|
|
|
- Selectors are element keys, which the tree exposes as UI Automation
|
|
`AutomationId`s: `nav-history`, `history-search`, `transcription-control`.
|
|
A key path (`model0/choose-model`) names a key inside a keyed ancestor.
|
|
`role=<role>` and `text=<name>` also work. Keep the keys these cases use
|
|
stable; `docs/features/accessibility.md` lists them.
|
|
- Each click waits (`until`) for a key that only the next view has, so a run
|
|
ends when the change is in the tree, not after a fixed wait.
|
|
- `interaction.*` counts an input only when the accessibility tree changed
|
|
within a second of it, so the change shows to strobe as well as on screen.
|
|
- The home holds a full History: 10,000 dictations, Hex's cap, from
|
|
`perf/seeds/history.ts` (written once into `perf/seeds/.generated/history`).
|
|
The list shows the newest 200, as search results do.
|
|
- Choosing a model is not a case: it downloads or loads a model, which costs
|
|
the network and the model, not the window. `model-picker` measures the
|
|
picker's own views.
|
|
- They need an unlocked session: Windows hit-tests every point to the lock
|
|
screen while it is locked, so strobe refuses pointer input then.
|
|
|
|
```sh
|
|
strobe bench perf/interaction.bench.ts --base origin/dev --head .
|
|
strobe bench perf/interaction.bench.ts --filter "history-*" --check # untimed smoke run
|
|
```
|