* Graduate six meeting labs, move four into Meetings settings, and throw capsules into corners (t1532) * Regenerate goldens and count call rows from the provider list (t1532) * Trace a recording pill hover and reversal offscreen (t1571) * Keep the recording meter anchored and show each audio track (t1571) * Keep light and dark recording snapshots from Linux (t1571) * Draw dictation’s listening field inside the meeting pill (t1571) * Refresh meeting pill goldens with the listening field (t1571) * Paint just the listening dots on the meeting surface (t1571) * Keep dots-only recording goldens in light and dark (t1571) * Keep recording controls and the dictation HUD dark on light desktops (t1571) * Refresh the dark overlays on light-host goldens (t1571) * Keep the finished meeting in its corner and let Close dismiss it (t1571) * Refresh finished summary goldens with the close control (t1571) * Open shipped features from What’s new (t1574) * Keep release-specific links distinct from feature navigation (t1574) * Document and translate the feature feed (t1574) * Verify dark and light navigation to both controls (t1574) * Show feature icons and calendar-day headings in What’s new (t1574) * Verify local-day boundaries and refresh both feed goldens (t1574) * Separate What's new navigation from build details (t1574) * Check What’s new selection through accessibility (t1574) * Refresh only the lower sidebar in the Linux goldens (t1574) * Drop the graduated labs from the Labs card and catalog (t1532) * Keep the merged meeting files inside the source cap (t1532) * Regenerate the Labs and Meetings goldens for the cleanup (t1532) * Drop the graduated labs from the website catalog (t1532) * Keep the throw's frame hook on every capsule (t1532)
14 KiB
Live transcription streams (/streams)
/streams lets another app on this computer transcribe its own audio live on Hex's
loaded dictation model. The app records the microphone (or anything else) and
pushes 16 kHz mono samples; Hex sends partial text while it listens, one final
segment per window, and the final text when the app finishes.
This is the API of the planned shared engine service (ROADMAP.md, "Shared
engine service"). Today the running Hex app serves it; later a launchd service
(anomaly.hex.engine) will serve the same routes and the Hex UI will become
one more client. Clients written against this document should not need to
change.
The code is src/local_api/streams.rs (routes) and src/transcription/stream.rs
(windows, priority, lease).
Availability
- The running Hex app on macOS, Linux, and Windows, with a GGUF model loaded by dictation (on Linux and Windows, once the listener is on).
GET /capabilitiesreports"streams": truewhen the routes exist, and"outputProcessing": truewhenprocess=globalwould run (Labs › Process API transcripts on, and the running app installed its processor).- Streams never load, download, or switch a model. They use whatever model
and language dictation holds; if there is none, creating a stream answers
503 model-not-loaded.
Discovery and auth
Read the owner-only discovery file local-api.json in Hex's data folder
(~/Library/Application Support/hex on macOS, $XDG_DATA_HOME/hex or
~/.local/share/hex on Linux, %APPDATA%\hex on Windows) (port, token,
apiVersion), then call http://127.0.0.1:<port> with:
| Header | On | Value |
|---|---|---|
Authorization |
every request | Bearer <token> from discovery |
X-Hex-Stream-Token |
every /streams/{id}/… request |
the stream's ownerToken |
Content-Type |
create / audio | application/json / application/octet-stream |
Content-Length |
every request with a body | required; chunked bodies are refused |
A wrong or missing stream token answers 404 stream-not-found, the same as an
unknown id, so ids can't be probed.
Audio format
16 000 Hz, mono, little-endian, declared when the stream is created:
f32le: 32-bit floats in [-1, 1] (values outside are clamped; NaN or infinity is refused).s16le: signed 16-bit integers.
Each audio request carries whole samples only. Push small chunks as you record them; 100 ms to 1 s per request is typical.
Endpoints
POST /streams
{ "sampleFormat": "f32le", "sampleRate": 16000, "model": "parakeet_v3", "language": "en" }
model and language are optional; when given they must match what
dictation holds, or the answer is 409 model-mismatch with the held ones:
{ "code": "model-mismatch", "model": "parakeet_v3", "language": "auto" }
201:
{
"id": 3,
"ownerToken": "hex_9f…",
"model": "parakeet_v3",
"language": "auto",
"sampleRate": 16000,
"sampleFormat": "f32le",
"partials": true,
"leaseMs": 10000
}
partials is true only when the loaded model can stream on its device; a
model that can't sends segments and the final only.
POST /streams/{id}/audio
Body: raw samples in the declared format, at most 256 KiB. 204 when queued.
Renews the lease.
GET /streams/{id}/events
Server-sent events, one JSON object per data: line, with : keep-alive
comments every 15 s. Each subscriber gets every segment from the start, then
live events, then the end, and the response closes. A newer subscriber
replaces an older one. Events stay readable after the stream ends, until the
next stream is created.
data: {"type":"ready","id":3,"model":"parakeet_v3","language":"auto","sampleFormat":"f32le","partials":true}
data: {"type":"partial","segment":0,"stable":"so the plan for","tentative":" today"}
data: {"type":"segment","segment":0,"startMs":0,"endMs":7420,"text":"So the plan for today is the release."}
data: {"type":"final","text":"So the plan for today is the release. …","segments":[…],"durationMs":31250}
type |
Meaning |
|---|---|
ready |
Sent first: the stream's model and format. |
partial |
The window being transcribed. stable is committed text the model won't revise; tentative may still change. Replaces the previous partial for the same segment. Only when partials is true. |
segment |
A finished window's final text (empty windows send none). |
final |
Every segment joined; the stream has ended. |
error |
{"code","message"}; the stream has ended (see codes below). |
cancelled |
The stream was cancelled. |
Word timings are not sent.
POST /streams/{id}/finish
Transcribes what's left and answers with the final, as the final event:
{ "text": "…", "segments": [{ "segment": 0, "startMs": 0, "endMs": 7420, "text": "…" }], "durationMs": 31250 }
?process=global (Labs › Process API transcripts) adds the text after Hex's
Global-mode output processing, exactly as a paste with no matching app or
website mode gets it: global replacements, Global mode corrections, the
optional OpenCode rewrite, and Global's selected transformations. true and
1 are aliases for global; any other value, or a repeated parameter,
answers 400 invalid-request. Without the parameter the response is exactly
as above.
{ "text": "…", "segments": [ … ], "durationMs": 31250, "processed": "…" }
processed is the paste-ready text. Processing is best-effort and follows
the paste path: a failed stage keeps its input, so the call still succeeds.
It never pastes, saves History, updates Paste Last, or touches the clipboard.
Without an installed processor (for example hex service) the answer is 409 processing-unavailable, even while the lab is off; with a processor installed
and the lab off it is 403 processing-disabled. Processing runs one at a time
across the local API; while another holds the slot the answer is 429 queue-full. All of these are checked before the stream finishes, and a
refused finish leaves the stream running so the caller can finish again.
No more audio is accepted once finishing. It waits up to 120 s (for example
behind a long dictation) and then answers 504 stream-finish-timeout; the
stream keeps going, so call finish again or read the events. A stream that
already failed answers 409 with its {"code","message"}; a cancelled one
409 stream-cancelled.
POST /streams/{id}/heartbeat
204; renews the lease. 409 stream-ended once the stream has ended.
POST /streams/{id}/cancel
204. Discards everything; the stream ends with cancelled.
Lease
A stream ends with error lease-expired when it receives neither audio nor
a heartbeat for 10 s (leaseMs). A paused recorder should send a heartbeat
every few seconds. The lease doesn't run while finishing.
Limits
| Limit | Value |
|---|---|
| Live streams | 1 (a second create answers 409 stream-busy; an ended stream doesn't count) |
| Create body | 1 KiB |
| Audio request | 256 KiB, read within 5 s |
| Backlog (received, not yet transcribed) | 120 s of audio, then 429 stream-backlog-full |
| Window | cut at pauses, at most 25 s |
| Lease | 10 s |
| Finish wait | 120 s |
Scheduling: dictation always comes first
Hex's dictation shares the engine, and the hotkey must never wait for a stream. So:
- Audio is cut into windows at pauses (at most 25 s). Each window is fed to the engine as a yielding live stream, at most 0.4 s ahead of what the engine has decoded, so a dictation request queues behind very little.
- Any dictation request (the shortcut's capture stream, a whole-clip transcription) closes the yielding stream first, and a yielding stream is refused while a dictation stream is open.
- From the shortcut press until dictation's text is done, a stream feeds the engine nothing; its audio waits in the backlog and catches up afterwards.
- A window whose live stream dictation closed restarts from the window's start once dictation is idle. A window that completes while that can't happen, or any window on a model that can't stream, is transcribed offline as one foreground request on the loaded model: after any dictation already sent, before meeting chunks (the same class as capture-word requests).
Dictation's capture never waits on the engine: its audio is recorded independently and losslessly. Only its transcription can wait, and only for work already running, because a decode can't be aborted on Metal: the feed in flight (at most 0.4 s of audio, a few tens of milliseconds, inside the 300 ms hold threshold) or, rarely, one offline window (up to 25 s of audio) that started while dictation was idle. Streams go ahead of meeting chunks because a stream's app is waiting for text now, while meetings catch up.
Error codes
Errors are JSON {"code": "…"} (some add message), except 401, which
has no body.
| Status | Code | When |
|---|---|---|
| 400 | invalid-request |
Create body isn't the JSON above, or process is not global, true, or 1 (or is repeated). |
| 400 | unsupported-sample-rate |
sampleRate isn't 16000. |
| 400 | unsupported-sample-format |
Not f32le or s16le. |
| 400 | unknown-model |
model isn't a Hex model id. |
| 400 | invalid-audio |
Partial sample or non-finite float. |
| 401 | (no body) | Missing or wrong bearer token. |
| 403 | processing-disabled |
process was asked for with a processor installed but Labs › Process API transcripts off. |
| 404 | stream-not-found |
Unknown id, replaced stream, or wrong stream token. |
| 409 | model-mismatch |
Requested model/language isn't the held one. |
| 409 | processing-unavailable |
process was asked for and no processor is installed (for example hex service). |
| 409 | stream-busy |
Another stream is live. |
| 409 | stream-ended |
Audio or heartbeat after the end. |
| 409 | stream-finishing |
Audio after finish. |
| 409 | stream-cancelled |
finish on a cancelled stream. |
| 409 | lease-expired, model-changed, transcription-failed, hex-quitting |
finish on a stream that failed that way (same codes as the error event). |
| 411 | length-required |
No Content-Length. |
| 413 | request-too-large, audio-too-large |
Over the limits. |
| 415 | unsupported-content-type |
Wrong Content-Type. |
| 429 | queue-full |
A processing run is already under way; retry after it ends. |
| 429 | stream-backlog-full |
120 s not yet transcribed; slow down or wait. |
| 503 | engine-unavailable |
Hex's dictation engine isn't running. |
| 503 | model-not-loaded |
Dictation holds no GGUF model (for example Apple Speech). |
| 503 | shutting-down |
Hex is quitting. |
| 504 | stream-finish-timeout |
finish waited 120 s; the stream continues. |
model-changed means the user switched dictation's model during the stream;
start a new one. hex-quitting means Hex quit.
Meeting finished events (GET /meetings/events)
Not part of /streams, but on the same local API with the same discovery
and bearer token. Code: src/local_api/meeting_events.rs
(route) and src/meeting/finished_events.rs (the hub the recorder publishes
to).
Server-sent events with : keep-alive comments every 15 s. Once per meeting,
after its final transcript and Markdown are written and its manifest reads
complete (a Stop, or the final pass of a meeting Hex quit while recording):
data: {"type":"meeting.finished","id":"1730700000000-4242"}
Only the id is sent: no title, path, transcript, or audio. Read the rest with
hex meetings show <id> (or hex mcp). One subscriber at a time; a
second answers 409 subscriber-busy. Nothing is queued while nobody listens,
so a client that reconnects should list meetings to catch up. A meeting whose
final transcript fails sends nothing. The stream ends when Hex quits or the
toggle turns off. hex service serves the route but records no meetings, so
it never sends one.
Catching up after being offline
The event is a best-effort signal, not a queue. To miss nothing, treat the meetings folder as the record and the event as a hint to look now:
- Keep a cursor: the end time (
timestamp_ms + duration_ms, Unix ms) of the newest meeting you've handled, and the ids you've handled near it. - On start or reconnect, subscribe to
GET /meetings/eventsfirst, then runhex meetings list --since <cursor> --limit 200 --json(or MCP'slist_meetingswithsince). Subscribing first means a meeting that finishes while you list still arrives as an event. - A meeting is finished when its
statusiscompleteand itstranscript_sourceisfinal; read it withhex meetings show <id> --json. Leave the cursor before any meeting that is stillrecording,transcribing, orinterrupted, so the next list sees it again. - Delivery is at least once: the same id can come from the list and the
event, or from overlapping lists. Deduplicate by meeting
id(stable for the meeting's life) and keep your own receipts of what you processed.
Limits:
- One list returns at most 200 meetings, newest first, with
truncatedset when there were more; page back with--until <oldest timestamp_ms>. --since/--untilinclude every meeting whose recording overlaps the range, so the meeting at the cursor comes back; the dedupe handles it.- A meeting retention already removed is gone and can't be caught up.
- A meeting Hex quit while recording reads
interrupteduntil its final pass on the next launch with Meetings on; then it readscompleteand the event is sent. - There is no separate transcript revision id; the meeting id is the key. The final transcript is published atomically, and the event is sent once per meeting.
Final-transcript readiness is separate from notes. The event and complete
mean the transcript and Markdown are written; notes (Apple Intelligence, or
OpenCode with Summarize with OpenCode) are written after that, may fail, and
can be regenerated later. Read them from notes in hex meetings show <id> --json (null until written; generated_at_ms changes when they're
rewritten). Subscribing, listing, and showing are read-only: they run no
command, upload nothing, and write nothing.