mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 03:08:04 +08:00
* Graduate six meeting labs, move four into Meetings settings, and throw capsules into corners (t1532) * Regenerate goldens and count call rows from the provider list (t1532) * Trace a recording pill hover and reversal offscreen (t1571) * Keep the recording meter anchored and show each audio track (t1571) * Keep light and dark recording snapshots from Linux (t1571) * Draw dictation’s listening field inside the meeting pill (t1571) * Refresh meeting pill goldens with the listening field (t1571) * Paint just the listening dots on the meeting surface (t1571) * Keep dots-only recording goldens in light and dark (t1571) * Keep recording controls and the dictation HUD dark on light desktops (t1571) * Refresh the dark overlays on light-host goldens (t1571) * Keep the finished meeting in its corner and let Close dismiss it (t1571) * Refresh finished summary goldens with the close control (t1571) * Open shipped features from What’s new (t1574) * Keep release-specific links distinct from feature navigation (t1574) * Document and translate the feature feed (t1574) * Verify dark and light navigation to both controls (t1574) * Show feature icons and calendar-day headings in What’s new (t1574) * Verify local-day boundaries and refresh both feed goldens (t1574) * Separate What's new navigation from build details (t1574) * Check What’s new selection through accessibility (t1574) * Refresh only the lower sidebar in the Linux goldens (t1574) * Drop the graduated labs from the Labs card and catalog (t1532) * Keep the merged meeting files inside the source cap (t1532) * Regenerate the Labs and Meetings goldens for the cleanup (t1532) * Drop the graduated labs from the website catalog (t1532) * Keep the throw's frame hook on every capsule (t1532)
310 lines
14 KiB
Markdown
310 lines
14 KiB
Markdown
# Live transcription streams (`/streams`)
|
||
|
||
`/streams` lets another app on this computer transcribe its own audio live on Hex's
|
||
loaded dictation model. The app records the microphone (or anything else) and
|
||
pushes 16 kHz mono samples; Hex sends partial text while it listens, one final
|
||
segment per window, and the final text when the app finishes.
|
||
|
||
This is the API of the planned shared engine service (`ROADMAP.md`, "Shared
|
||
engine service"). Today the running Hex app serves it; later a launchd service
|
||
(`anomaly.hex.engine`) will serve the same routes and the Hex UI will become
|
||
one more client. Clients written against this document should not need to
|
||
change.
|
||
|
||
The code is `src/local_api/streams.rs` (routes) and `src/transcription/stream.rs`
|
||
(windows, priority, lease).
|
||
|
||
## Availability
|
||
|
||
- The running Hex app on macOS, Linux, and Windows, with a GGUF model loaded
|
||
by dictation (on Linux and Windows, once the listener is on).
|
||
- `GET /capabilities` reports `"streams": true` when the routes exist, and
|
||
`"outputProcessing": true` when `process=global` would run (Labs › Process
|
||
API transcripts on, and the running app installed its processor).
|
||
- Streams never load, download, or switch a model. They use whatever model
|
||
and language dictation holds; if there is none, creating a stream answers
|
||
`503 model-not-loaded`.
|
||
|
||
## Discovery and auth
|
||
|
||
Read the owner-only discovery file `local-api.json` in Hex's data folder
|
||
(`~/Library/Application Support/hex` on macOS, `$XDG_DATA_HOME/hex` or
|
||
`~/.local/share/hex` on Linux, `%APPDATA%\hex` on Windows) (`port`, `token`,
|
||
`apiVersion`), then call `http://127.0.0.1:<port>` with:
|
||
|
||
| Header | On | Value |
|
||
| --- | --- | --- |
|
||
| `Authorization` | every request | `Bearer <token>` from discovery |
|
||
| `X-Hex-Stream-Token` | every `/streams/{id}/…` request | the stream's `ownerToken` |
|
||
| `Content-Type` | create / audio | `application/json` / `application/octet-stream` |
|
||
| `Content-Length` | every request with a body | required; chunked bodies are refused |
|
||
|
||
A wrong or missing stream token answers `404 stream-not-found`, the same as an
|
||
unknown id, so ids can't be probed.
|
||
|
||
## Audio format
|
||
|
||
16 000 Hz, mono, little-endian, declared when the stream is created:
|
||
|
||
- `f32le`: 32-bit floats in [-1, 1] (values outside are clamped; NaN or
|
||
infinity is refused).
|
||
- `s16le`: signed 16-bit integers.
|
||
|
||
Each audio request carries whole samples only. Push small chunks as you record
|
||
them; 100 ms to 1 s per request is typical.
|
||
|
||
## Endpoints
|
||
|
||
### `POST /streams`
|
||
|
||
```json
|
||
{ "sampleFormat": "f32le", "sampleRate": 16000, "model": "parakeet_v3", "language": "en" }
|
||
```
|
||
|
||
`model` and `language` are optional; when given they must match what
|
||
dictation holds, or the answer is `409 model-mismatch` with the held ones:
|
||
|
||
```json
|
||
{ "code": "model-mismatch", "model": "parakeet_v3", "language": "auto" }
|
||
```
|
||
|
||
`201`:
|
||
|
||
```json
|
||
{
|
||
"id": 3,
|
||
"ownerToken": "hex_9f…",
|
||
"model": "parakeet_v3",
|
||
"language": "auto",
|
||
"sampleRate": 16000,
|
||
"sampleFormat": "f32le",
|
||
"partials": true,
|
||
"leaseMs": 10000
|
||
}
|
||
```
|
||
|
||
`partials` is `true` only when the loaded model can stream on its device; a
|
||
model that can't sends segments and the final only.
|
||
|
||
### `POST /streams/{id}/audio`
|
||
|
||
Body: raw samples in the declared format, at most 256 KiB. `204` when queued.
|
||
Renews the lease.
|
||
|
||
### `GET /streams/{id}/events`
|
||
|
||
Server-sent events, one JSON object per `data:` line, with `: keep-alive`
|
||
comments every 15 s. Each subscriber gets every segment from the start, then
|
||
live events, then the end, and the response closes. A newer subscriber
|
||
replaces an older one. Events stay readable after the stream ends, until the
|
||
next stream is created.
|
||
|
||
```text
|
||
data: {"type":"ready","id":3,"model":"parakeet_v3","language":"auto","sampleFormat":"f32le","partials":true}
|
||
|
||
data: {"type":"partial","segment":0,"stable":"so the plan for","tentative":" today"}
|
||
|
||
data: {"type":"segment","segment":0,"startMs":0,"endMs":7420,"text":"So the plan for today is the release."}
|
||
|
||
data: {"type":"final","text":"So the plan for today is the release. …","segments":[…],"durationMs":31250}
|
||
```
|
||
|
||
| `type` | Meaning |
|
||
| --- | --- |
|
||
| `ready` | Sent first: the stream's model and format. |
|
||
| `partial` | The window being transcribed. `stable` is committed text the model won't revise; `tentative` may still change. Replaces the previous partial for the same `segment`. Only when `partials` is true. |
|
||
| `segment` | A finished window's final text (empty windows send none). |
|
||
| `final` | Every segment joined; the stream has ended. |
|
||
| `error` | `{"code","message"}`; the stream has ended (see codes below). |
|
||
| `cancelled` | The stream was cancelled. |
|
||
|
||
Word timings are not sent.
|
||
|
||
### `POST /streams/{id}/finish`
|
||
|
||
Transcribes what's left and answers with the final, as the `final` event:
|
||
|
||
```json
|
||
{ "text": "…", "segments": [{ "segment": 0, "startMs": 0, "endMs": 7420, "text": "…" }], "durationMs": 31250 }
|
||
```
|
||
|
||
`?process=global` (Labs › Process API transcripts) adds the text after Hex's
|
||
Global-mode output processing, exactly as a paste with no matching app or
|
||
website mode gets it: global replacements, Global mode corrections, the
|
||
optional OpenCode rewrite, and Global's selected transformations. `true` and
|
||
`1` are aliases for `global`; any other value, or a repeated parameter,
|
||
answers `400 invalid-request`. Without the parameter the response is exactly
|
||
as above.
|
||
|
||
```json
|
||
{ "text": "…", "segments": [ … ], "durationMs": 31250, "processed": "…" }
|
||
```
|
||
|
||
`processed` is the paste-ready text. Processing is best-effort and follows
|
||
the paste path: a failed stage keeps its input, so the call still succeeds.
|
||
It never pastes, saves History, updates Paste Last, or touches the clipboard.
|
||
Without an installed processor (for example `hex service`) the answer is `409
|
||
processing-unavailable`, even while the lab is off; with a processor installed
|
||
and the lab off it is `403 processing-disabled`. Processing runs one at a time
|
||
across the local API; while another holds the slot the answer is `429
|
||
queue-full`. All of these are checked before the stream finishes, and a
|
||
refused `finish` leaves the stream running so the caller can finish again.
|
||
|
||
No more audio is accepted once finishing. It waits up to 120 s (for example
|
||
behind a long dictation) and then answers `504 stream-finish-timeout`; the
|
||
stream keeps going, so call `finish` again or read the events. A stream that
|
||
already failed answers `409` with its `{"code","message"}`; a cancelled one
|
||
`409 stream-cancelled`.
|
||
|
||
### `POST /streams/{id}/heartbeat`
|
||
|
||
`204`; renews the lease. `409 stream-ended` once the stream has ended.
|
||
|
||
### `POST /streams/{id}/cancel`
|
||
|
||
`204`. Discards everything; the stream ends with `cancelled`.
|
||
|
||
## Lease
|
||
|
||
A stream ends with `error` `lease-expired` when it receives neither audio nor
|
||
a heartbeat for 10 s (`leaseMs`). A paused recorder should send a heartbeat
|
||
every few seconds. The lease doesn't run while finishing.
|
||
|
||
## Limits
|
||
|
||
| Limit | Value |
|
||
| --- | --- |
|
||
| Live streams | 1 (a second create answers `409 stream-busy`; an ended stream doesn't count) |
|
||
| Create body | 1 KiB |
|
||
| Audio request | 256 KiB, read within 5 s |
|
||
| Backlog (received, not yet transcribed) | 120 s of audio, then `429 stream-backlog-full` |
|
||
| Window | cut at pauses, at most 25 s |
|
||
| Lease | 10 s |
|
||
| Finish wait | 120 s |
|
||
|
||
## Scheduling: dictation always comes first
|
||
|
||
Hex's dictation shares the engine, and the hotkey must never wait for a
|
||
stream. So:
|
||
|
||
- Audio is cut into windows at pauses (at most 25 s). Each window is fed to
|
||
the engine as a *yielding* live stream, at most 0.4 s ahead of what the
|
||
engine has decoded, so a dictation request queues behind very little.
|
||
- Any dictation request (the shortcut's capture stream, a whole-clip
|
||
transcription) closes the yielding stream first, and a yielding stream is
|
||
refused while a dictation stream is open.
|
||
- From the shortcut press until dictation's text is done, a stream feeds the
|
||
engine nothing; its audio waits in the backlog and catches up afterwards.
|
||
- A window whose live stream dictation closed restarts from the window's start
|
||
once dictation is idle. A window that completes while that can't happen, or
|
||
any window on a model that can't stream, is transcribed offline as one
|
||
foreground request on the loaded model: after any dictation already sent,
|
||
before meeting chunks (the same class as capture-word requests).
|
||
|
||
Dictation's capture never waits on the engine: its audio is recorded
|
||
independently and losslessly. Only its transcription can wait, and only for
|
||
work already running, because a decode can't be aborted on Metal: the feed in
|
||
flight (at most 0.4 s of audio, a few tens of milliseconds, inside the 300 ms
|
||
hold threshold) or, rarely, one offline window (up to 25 s of audio) that
|
||
started while dictation was idle. Streams go ahead of meeting chunks because
|
||
a stream's app is waiting for text now, while meetings catch up.
|
||
|
||
## Error codes
|
||
|
||
Errors are JSON `{"code": "…"}` (some add `message`), except `401`, which
|
||
has no body.
|
||
|
||
| Status | Code | When |
|
||
| --- | --- | --- |
|
||
| 400 | `invalid-request` | Create body isn't the JSON above, or `process` is not `global`, `true`, or `1` (or is repeated). |
|
||
| 400 | `unsupported-sample-rate` | `sampleRate` isn't 16000. |
|
||
| 400 | `unsupported-sample-format` | Not `f32le` or `s16le`. |
|
||
| 400 | `unknown-model` | `model` isn't a Hex model id. |
|
||
| 400 | `invalid-audio` | Partial sample or non-finite float. |
|
||
| 401 | (no body) | Missing or wrong bearer token. |
|
||
| 403 | `processing-disabled` | `process` was asked for with a processor installed but Labs › Process API transcripts off. |
|
||
| 404 | `stream-not-found` | Unknown id, replaced stream, or wrong stream token. |
|
||
| 409 | `model-mismatch` | Requested model/language isn't the held one. |
|
||
| 409 | `processing-unavailable` | `process` was asked for and no processor is installed (for example `hex service`). |
|
||
| 409 | `stream-busy` | Another stream is live. |
|
||
| 409 | `stream-ended` | Audio or heartbeat after the end. |
|
||
| 409 | `stream-finishing` | Audio after `finish`. |
|
||
| 409 | `stream-cancelled` | `finish` on a cancelled stream. |
|
||
| 409 | `lease-expired`, `model-changed`, `transcription-failed`, `hex-quitting` | `finish` on a stream that failed that way (same codes as the `error` event). |
|
||
| 411 | `length-required` | No `Content-Length`. |
|
||
| 413 | `request-too-large`, `audio-too-large` | Over the limits. |
|
||
| 415 | `unsupported-content-type` | Wrong `Content-Type`. |
|
||
| 429 | `queue-full` | A processing run is already under way; retry after it ends. |
|
||
| 429 | `stream-backlog-full` | 120 s not yet transcribed; slow down or wait. |
|
||
| 503 | `engine-unavailable` | Hex's dictation engine isn't running. |
|
||
| 503 | `model-not-loaded` | Dictation holds no GGUF model (for example Apple Speech). |
|
||
| 503 | `shutting-down` | Hex is quitting. |
|
||
| 504 | `stream-finish-timeout` | `finish` waited 120 s; the stream continues. |
|
||
|
||
`model-changed` means the user switched dictation's model during the stream;
|
||
start a new one. `hex-quitting` means Hex quit.
|
||
|
||
## Meeting finished events (`GET /meetings/events`)
|
||
|
||
Not part of `/streams`, but on the same local API with the same discovery
|
||
and bearer token. Code: `src/local_api/meeting_events.rs`
|
||
(route) and `src/meeting/finished_events.rs` (the hub the recorder publishes
|
||
to).
|
||
|
||
Server-sent events with `: keep-alive` comments every 15 s. Once per meeting,
|
||
after its final transcript and Markdown are written and its manifest reads
|
||
`complete` (a Stop, or the final pass of a meeting Hex quit while recording):
|
||
|
||
```text
|
||
data: {"type":"meeting.finished","id":"1730700000000-4242"}
|
||
```
|
||
|
||
Only the id is sent: no title, path, transcript, or audio. Read the rest with
|
||
`hex meetings show <id>` (or `hex mcp`). One subscriber at a time; a
|
||
second answers `409 subscriber-busy`. Nothing is queued while nobody listens,
|
||
so a client that reconnects should list meetings to catch up. A meeting whose
|
||
final transcript fails sends nothing. The stream ends when Hex quits or the
|
||
toggle turns off. `hex service` serves the route but records no meetings, so
|
||
it never sends one.
|
||
|
||
### Catching up after being offline
|
||
|
||
The event is a best-effort signal, not a queue. To miss nothing, treat the
|
||
meetings folder as the record and the event as a hint to look now:
|
||
|
||
1. Keep a cursor: the end time (`timestamp_ms + duration_ms`, Unix ms) of the
|
||
newest meeting you've handled, and the ids you've handled near it.
|
||
2. On start or reconnect, subscribe to `GET /meetings/events` first, then run
|
||
`hex meetings list --since <cursor> --limit 200 --json` (or MCP's
|
||
`list_meetings` with `since`). Subscribing first means a meeting that
|
||
finishes while you list still arrives as an event.
|
||
3. A meeting is finished when its `status` is `complete` and its
|
||
`transcript_source` is `final`; read it with `hex meetings show <id> --json`.
|
||
Leave the cursor before any meeting that is still `recording`,
|
||
`transcribing`, or `interrupted`, so the next list sees it again.
|
||
4. Delivery is at least once: the same id can come from the list and the
|
||
event, or from overlapping lists. Deduplicate by meeting `id` (stable for
|
||
the meeting's life) and keep your own receipts of what you processed.
|
||
|
||
Limits:
|
||
|
||
- One list returns at most 200 meetings, newest first, with `truncated` set
|
||
when there were more; page back with `--until <oldest timestamp_ms>`.
|
||
- `--since`/`--until` include every meeting whose recording overlaps the
|
||
range, so the meeting at the cursor comes back; the dedupe handles it.
|
||
- A meeting retention already removed is gone and can't be caught up.
|
||
- A meeting Hex quit while recording reads `interrupted` until its final
|
||
pass on the next launch with Meetings on; then it reads `complete` and the
|
||
event is sent.
|
||
- There is no separate transcript revision id; the meeting id is the key.
|
||
The final transcript is published atomically, and the event is sent once
|
||
per meeting.
|
||
|
||
Final-transcript readiness is separate from notes. The event and `complete`
|
||
mean the transcript and Markdown are written; notes (Apple Intelligence, or
|
||
OpenCode with Summarize with OpenCode) are written after that, may fail, and
|
||
can be regenerated later. Read them from `notes` in `hex meetings show <id>
|
||
--json` (`null` until written; `generated_at_ms` changes when they're
|
||
rewritten). Subscribing, listing, and showing are read-only: they run no
|
||
command, upload nothing, and write nothing.
|