Files
Kit Langton 5f01839ea8 feat(meetings): graduate meeting labs into normal behaviour and Meetings settings (#1821)
* Graduate six meeting labs, move four into Meetings settings, and throw capsules into corners (t1532)

* Regenerate goldens and count call rows from the provider list (t1532)

* Trace a recording pill hover and reversal offscreen (t1571)

* Keep the recording meter anchored and show each audio track (t1571)

* Keep light and dark recording snapshots from Linux (t1571)

* Draw dictation’s listening field inside the meeting pill (t1571)

* Refresh meeting pill goldens with the listening field (t1571)

* Paint just the listening dots on the meeting surface (t1571)

* Keep dots-only recording goldens in light and dark (t1571)

* Keep recording controls and the dictation HUD dark on light desktops (t1571)

* Refresh the dark overlays on light-host goldens (t1571)

* Keep the finished meeting in its corner and let Close dismiss it (t1571)

* Refresh finished summary goldens with the close control (t1571)

* Open shipped features from What’s new (t1574)

* Keep release-specific links distinct from feature navigation (t1574)

* Document and translate the feature feed (t1574)

* Verify dark and light navigation to both controls (t1574)

* Show feature icons and calendar-day headings in What’s new (t1574)

* Verify local-day boundaries and refresh both feed goldens (t1574)

* Separate What's new navigation from build details (t1574)

* Check What’s new selection through accessibility (t1574)

* Refresh only the lower sidebar in the Linux goldens (t1574)

* Drop the graduated labs from the Labs card and catalog (t1532)

* Keep the merged meeting files inside the source cap (t1532)

* Regenerate the Labs and Meetings goldens for the cleanup (t1532)

* Drop the graduated labs from the website catalog (t1532)

* Keep the throw's frame hook on every capsule (t1532)
2026-11-16 00:51:48 -08:00

310 lines
14 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Live transcription streams (`/streams`)
`/streams` lets another app on this computer transcribe its own audio live on Hex's
loaded dictation model. The app records the microphone (or anything else) and
pushes 16 kHz mono samples; Hex sends partial text while it listens, one final
segment per window, and the final text when the app finishes.
This is the API of the planned shared engine service (`ROADMAP.md`, "Shared
engine service"). Today the running Hex app serves it; later a launchd service
(`anomaly.hex.engine`) will serve the same routes and the Hex UI will become
one more client. Clients written against this document should not need to
change.
The code is `src/local_api/streams.rs` (routes) and `src/transcription/stream.rs`
(windows, priority, lease).
## Availability
- The running Hex app on macOS, Linux, and Windows, with a GGUF model loaded
by dictation (on Linux and Windows, once the listener is on).
- `GET /capabilities` reports `"streams": true` when the routes exist, and
`"outputProcessing": true` when `process=global` would run (Labs › Process
API transcripts on, and the running app installed its processor).
- Streams never load, download, or switch a model. They use whatever model
and language dictation holds; if there is none, creating a stream answers
`503 model-not-loaded`.
## Discovery and auth
Read the owner-only discovery file `local-api.json` in Hex's data folder
(`~/Library/Application Support/hex` on macOS, `$XDG_DATA_HOME/hex` or
`~/.local/share/hex` on Linux, `%APPDATA%\hex` on Windows) (`port`, `token`,
`apiVersion`), then call `http://127.0.0.1:<port>` with:
| Header | On | Value |
| --- | --- | --- |
| `Authorization` | every request | `Bearer <token>` from discovery |
| `X-Hex-Stream-Token` | every `/streams/{id}/…` request | the stream's `ownerToken` |
| `Content-Type` | create / audio | `application/json` / `application/octet-stream` |
| `Content-Length` | every request with a body | required; chunked bodies are refused |
A wrong or missing stream token answers `404 stream-not-found`, the same as an
unknown id, so ids can't be probed.
## Audio format
16 000 Hz, mono, little-endian, declared when the stream is created:
- `f32le`: 32-bit floats in [-1, 1] (values outside are clamped; NaN or
infinity is refused).
- `s16le`: signed 16-bit integers.
Each audio request carries whole samples only. Push small chunks as you record
them; 100 ms to 1 s per request is typical.
## Endpoints
### `POST /streams`
```json
{ "sampleFormat": "f32le", "sampleRate": 16000, "model": "parakeet_v3", "language": "en" }
```
`model` and `language` are optional; when given they must match what
dictation holds, or the answer is `409 model-mismatch` with the held ones:
```json
{ "code": "model-mismatch", "model": "parakeet_v3", "language": "auto" }
```
`201`:
```json
{
"id": 3,
"ownerToken": "hex_9f…",
"model": "parakeet_v3",
"language": "auto",
"sampleRate": 16000,
"sampleFormat": "f32le",
"partials": true,
"leaseMs": 10000
}
```
`partials` is `true` only when the loaded model can stream on its device; a
model that can't sends segments and the final only.
### `POST /streams/{id}/audio`
Body: raw samples in the declared format, at most 256 KiB. `204` when queued.
Renews the lease.
### `GET /streams/{id}/events`
Server-sent events, one JSON object per `data:` line, with `: keep-alive`
comments every 15 s. Each subscriber gets every segment from the start, then
live events, then the end, and the response closes. A newer subscriber
replaces an older one. Events stay readable after the stream ends, until the
next stream is created.
```text
data: {"type":"ready","id":3,"model":"parakeet_v3","language":"auto","sampleFormat":"f32le","partials":true}
data: {"type":"partial","segment":0,"stable":"so the plan for","tentative":" today"}
data: {"type":"segment","segment":0,"startMs":0,"endMs":7420,"text":"So the plan for today is the release."}
data: {"type":"final","text":"So the plan for today is the release. …","segments":[…],"durationMs":31250}
```
| `type` | Meaning |
| --- | --- |
| `ready` | Sent first: the stream's model and format. |
| `partial` | The window being transcribed. `stable` is committed text the model won't revise; `tentative` may still change. Replaces the previous partial for the same `segment`. Only when `partials` is true. |
| `segment` | A finished window's final text (empty windows send none). |
| `final` | Every segment joined; the stream has ended. |
| `error` | `{"code","message"}`; the stream has ended (see codes below). |
| `cancelled` | The stream was cancelled. |
Word timings are not sent.
### `POST /streams/{id}/finish`
Transcribes what's left and answers with the final, as the `final` event:
```json
{ "text": "…", "segments": [{ "segment": 0, "startMs": 0, "endMs": 7420, "text": "…" }], "durationMs": 31250 }
```
`?process=global` (Labs › Process API transcripts) adds the text after Hex's
Global-mode output processing, exactly as a paste with no matching app or
website mode gets it: global replacements, Global mode corrections, the
optional OpenCode rewrite, and Global's selected transformations. `true` and
`1` are aliases for `global`; any other value, or a repeated parameter,
answers `400 invalid-request`. Without the parameter the response is exactly
as above.
```json
{ "text": "…", "segments": [ … ], "durationMs": 31250, "processed": "…" }
```
`processed` is the paste-ready text. Processing is best-effort and follows
the paste path: a failed stage keeps its input, so the call still succeeds.
It never pastes, saves History, updates Paste Last, or touches the clipboard.
Without an installed processor (for example `hex service`) the answer is `409
processing-unavailable`, even while the lab is off; with a processor installed
and the lab off it is `403 processing-disabled`. Processing runs one at a time
across the local API; while another holds the slot the answer is `429
queue-full`. All of these are checked before the stream finishes, and a
refused `finish` leaves the stream running so the caller can finish again.
No more audio is accepted once finishing. It waits up to 120 s (for example
behind a long dictation) and then answers `504 stream-finish-timeout`; the
stream keeps going, so call `finish` again or read the events. A stream that
already failed answers `409` with its `{"code","message"}`; a cancelled one
`409 stream-cancelled`.
### `POST /streams/{id}/heartbeat`
`204`; renews the lease. `409 stream-ended` once the stream has ended.
### `POST /streams/{id}/cancel`
`204`. Discards everything; the stream ends with `cancelled`.
## Lease
A stream ends with `error` `lease-expired` when it receives neither audio nor
a heartbeat for 10 s (`leaseMs`). A paused recorder should send a heartbeat
every few seconds. The lease doesn't run while finishing.
## Limits
| Limit | Value |
| --- | --- |
| Live streams | 1 (a second create answers `409 stream-busy`; an ended stream doesn't count) |
| Create body | 1 KiB |
| Audio request | 256 KiB, read within 5 s |
| Backlog (received, not yet transcribed) | 120 s of audio, then `429 stream-backlog-full` |
| Window | cut at pauses, at most 25 s |
| Lease | 10 s |
| Finish wait | 120 s |
## Scheduling: dictation always comes first
Hex's dictation shares the engine, and the hotkey must never wait for a
stream. So:
- Audio is cut into windows at pauses (at most 25 s). Each window is fed to
the engine as a *yielding* live stream, at most 0.4 s ahead of what the
engine has decoded, so a dictation request queues behind very little.
- Any dictation request (the shortcut's capture stream, a whole-clip
transcription) closes the yielding stream first, and a yielding stream is
refused while a dictation stream is open.
- From the shortcut press until dictation's text is done, a stream feeds the
engine nothing; its audio waits in the backlog and catches up afterwards.
- A window whose live stream dictation closed restarts from the window's start
once dictation is idle. A window that completes while that can't happen, or
any window on a model that can't stream, is transcribed offline as one
foreground request on the loaded model: after any dictation already sent,
before meeting chunks (the same class as capture-word requests).
Dictation's capture never waits on the engine: its audio is recorded
independently and losslessly. Only its transcription can wait, and only for
work already running, because a decode can't be aborted on Metal: the feed in
flight (at most 0.4 s of audio, a few tens of milliseconds, inside the 300 ms
hold threshold) or, rarely, one offline window (up to 25 s of audio) that
started while dictation was idle. Streams go ahead of meeting chunks because
a stream's app is waiting for text now, while meetings catch up.
## Error codes
Errors are JSON `{"code": "…"}` (some add `message`), except `401`, which
has no body.
| Status | Code | When |
| --- | --- | --- |
| 400 | `invalid-request` | Create body isn't the JSON above, or `process` is not `global`, `true`, or `1` (or is repeated). |
| 400 | `unsupported-sample-rate` | `sampleRate` isn't 16000. |
| 400 | `unsupported-sample-format` | Not `f32le` or `s16le`. |
| 400 | `unknown-model` | `model` isn't a Hex model id. |
| 400 | `invalid-audio` | Partial sample or non-finite float. |
| 401 | (no body) | Missing or wrong bearer token. |
| 403 | `processing-disabled` | `process` was asked for with a processor installed but Labs › Process API transcripts off. |
| 404 | `stream-not-found` | Unknown id, replaced stream, or wrong stream token. |
| 409 | `model-mismatch` | Requested model/language isn't the held one. |
| 409 | `processing-unavailable` | `process` was asked for and no processor is installed (for example `hex service`). |
| 409 | `stream-busy` | Another stream is live. |
| 409 | `stream-ended` | Audio or heartbeat after the end. |
| 409 | `stream-finishing` | Audio after `finish`. |
| 409 | `stream-cancelled` | `finish` on a cancelled stream. |
| 409 | `lease-expired`, `model-changed`, `transcription-failed`, `hex-quitting` | `finish` on a stream that failed that way (same codes as the `error` event). |
| 411 | `length-required` | No `Content-Length`. |
| 413 | `request-too-large`, `audio-too-large` | Over the limits. |
| 415 | `unsupported-content-type` | Wrong `Content-Type`. |
| 429 | `queue-full` | A processing run is already under way; retry after it ends. |
| 429 | `stream-backlog-full` | 120 s not yet transcribed; slow down or wait. |
| 503 | `engine-unavailable` | Hex's dictation engine isn't running. |
| 503 | `model-not-loaded` | Dictation holds no GGUF model (for example Apple Speech). |
| 503 | `shutting-down` | Hex is quitting. |
| 504 | `stream-finish-timeout` | `finish` waited 120 s; the stream continues. |
`model-changed` means the user switched dictation's model during the stream;
start a new one. `hex-quitting` means Hex quit.
## Meeting finished events (`GET /meetings/events`)
Not part of `/streams`, but on the same local API with the same discovery
and bearer token. Code: `src/local_api/meeting_events.rs`
(route) and `src/meeting/finished_events.rs` (the hub the recorder publishes
to).
Server-sent events with `: keep-alive` comments every 15 s. Once per meeting,
after its final transcript and Markdown are written and its manifest reads
`complete` (a Stop, or the final pass of a meeting Hex quit while recording):
```text
data: {"type":"meeting.finished","id":"1730700000000-4242"}
```
Only the id is sent: no title, path, transcript, or audio. Read the rest with
`hex meetings show <id>` (or `hex mcp`). One subscriber at a time; a
second answers `409 subscriber-busy`. Nothing is queued while nobody listens,
so a client that reconnects should list meetings to catch up. A meeting whose
final transcript fails sends nothing. The stream ends when Hex quits or the
toggle turns off. `hex service` serves the route but records no meetings, so
it never sends one.
### Catching up after being offline
The event is a best-effort signal, not a queue. To miss nothing, treat the
meetings folder as the record and the event as a hint to look now:
1. Keep a cursor: the end time (`timestamp_ms + duration_ms`, Unix ms) of the
newest meeting you've handled, and the ids you've handled near it.
2. On start or reconnect, subscribe to `GET /meetings/events` first, then run
`hex meetings list --since <cursor> --limit 200 --json` (or MCP's
`list_meetings` with `since`). Subscribing first means a meeting that
finishes while you list still arrives as an event.
3. A meeting is finished when its `status` is `complete` and its
`transcript_source` is `final`; read it with `hex meetings show <id> --json`.
Leave the cursor before any meeting that is still `recording`,
`transcribing`, or `interrupted`, so the next list sees it again.
4. Delivery is at least once: the same id can come from the list and the
event, or from overlapping lists. Deduplicate by meeting `id` (stable for
the meeting's life) and keep your own receipts of what you processed.
Limits:
- One list returns at most 200 meetings, newest first, with `truncated` set
when there were more; page back with `--until <oldest timestamp_ms>`.
- `--since`/`--until` include every meeting whose recording overlaps the
range, so the meeting at the cursor comes back; the dedupe handles it.
- A meeting retention already removed is gone and can't be caught up.
- A meeting Hex quit while recording reads `interrupted` until its final
pass on the next launch with Meetings on; then it reads `complete` and the
event is sent.
- There is no separate transcript revision id; the meeting id is the key.
The final transcript is published atomically, and the event is sent once
per meeting.
Final-transcript readiness is separate from notes. The event and `complete`
mean the transcript and Markdown are written; notes (Apple Intelligence, or
OpenCode with Summarize with OpenCode) are written after that, may fail, and
can be regenerated later. Read them from `notes` in `hex meetings show <id>
--json` (`null` until written; `generated_at_ms` changes when they're
rewritten). Subscribing, listing, and showing are read-only: they run no
command, upload nothing, and write nothing.