Files
Dax cdb96f585e capture: keep all audio in long captures (#1795) (#1894)
* capture: stream audio to disk so long captures keep all of it

A capture's timeline now writes 16 kHz mono PCM16 to its bundle's
audio.wav as it records, with the leading offset and gaps as silence, so
memory stays flat and the 20-minute drop is gone. An 8-hour guard bounds
disk use only; past it the rest is one recorded gap.

A finished capture is transcribed from the WAV in 30 s pieces, the
reader hex transcribe already used (lifted to transcription/pieces.rs),
and each piece's words move onto the capture clock. The Parakeet, Apple
(path-based), and GGUF paths keep every word without holding the
session. The pipeline never rewrites a streamed audio.wav from empty
samples, an iPhone narration is read back where it must be re-based,
and a cancelled capture closes and removes its file.

* capture: say where a capture's audio stopped

A streamed capture keeps all of its audio, but the 8-hour disk guard (or
a failed audio write) still stops it early, and only the process log
said so. The timeline now carries the stop point onto RecordedAudio,
the bundle stores it as audioStoppedAtMs (serde default, so old bundles
load unchanged), and three places say it plainly: summary.md, `hex
capture show`, and `hex capture status`.

The guard records its point at a 1 s test limit, and the summary line is
tested at 12:34 and 8:00:00 so both clock shapes are covered.

* capture: fill the replay's stop point from its bundle

The 15 s replay builds Written too, on macOS only, so the new field
broke the macOS clippy job while Linux never compiled that function.

* capture: keep a recording out of recovery until it stops

Review fixes for the streamed timeline.

A capture now streams to `audio.wav.partial` and renames it to `audio.wav`
in `Timeline::finish`, so a bundle with a movie and an in-progress audio
file is never mistaken for a capture whose build never ran (recover's
orphan rule, Scrawl.app's `scrawl recover`) and retention cleans a crashed
one like any other bundle. Cancel and discard remove the partial; a failed
publish reads it back into memory rather than pointing at a missing file.

Also: a write failure's stop point is the last sample that reached the
file, not the end of the failed chunk; a trim failure counts every written
sample; the capture's words keep one running frame total instead of
adding truncated per-piece millis; an iPhone narration's `since` drains the
decoded samples instead of copying them and treats a missing file as no
audio; the phone and replay paths carry the audio stop point into their
bundles; and empty audio answers `source: "none"`, as `transcribe_with`
does.

Tests: a movie plus partial audio is never owed, retention trashes a
crashed bundle's media, a write failure records its stop point, a missing
streamed file shifts to no samples, and the replay shift moves the stop
point with the clock.

* capture: publish audio.wav only once its build journals the capture

Follow-ups from review.

A capture now streams to audio.wav.partial and keeps that name after it
stops: the partial is renamed to audio.wav right after the build writes
unfinished.json, so audio.wav never exists without a journal, and a
stopped capture waiting in the finish queue no longer looks like an
orphan to Scrawl.app's recover. A partial that cannot take the name is
read back and written by the build, and its partial name removed; the
replay removes the unshifted partial once the narration is read for
shifting.

Also: a periodic header flush error is now a write failure (the rest of
the capture is a gap) rather than a log line; the midpoint of a failed
write records where the file stopped; and the words test cuts pieces at
odd sample counts and computes offsets from the running frame total, so
per-piece millisecond drift would fail it.

Tests: a streamed audio takes its final name only when the build settles
it (and falls back when it cannot), a capture that has not been
journaled is never owed a rebuild, a mid-capture write failure stops
where the file stopped, and the word offsets are exact at odd piece
lengths.

* capture: consume a streamed recording's partial when it is read

Follow-ups from review.

`RecordedAudio::since` and `into_memory` now remove the partial once its
samples are read: an iPhone narration (capture.rs's `host_narration`
path) used to leave the whole streamed file in the finished bundle. The
replay relies on the same path instead of removing it by hand, and the
light zip leaves `audio.wav.partial` out defensively. A read that fails
is an error to `into_memory` (so a build keeps the only copy) while
`since` still treats a missing file as no audio for an iPhone.

A failed publish now reads the audio back only when it produced the
samples the recording had; a failed read keeps the partial and removes
any half-copied `audio.wav`. `publish_audio` takes `&mut self` and points
the recording at the published file.

A journaled bundle whose `audio.wav` is missing adopts `audio.wav.partial`
before rebuilding. Docs say the guarantee is a streamed capture's, and
that a crash while recording or queued is retention's job, not
recovery's.

Tests: reading a streamed recording consumes the partial; a finished
iPhone bundle keeps none; a streamed audio is published only when its
build settles it, including the read-back and unreadable-partial cases; a
journaled capture adopts its partial.

* capture: keep a failed audio publish recoverable

Follow-ups from review.

The publish fallback read the partial with `into_memory`, which deletes
it, and then ignored whether its own write landed: a failed write left
the audio nowhere. It now reads through `read_streamed` (no deletion),
writes `audio.wav` and checks the result, and removes the partial only
once that write succeeded. A failed read or write removes any half
`audio.wav` and fails the build, which is the pipeline's existing failed
path: `recover::finished` runs only for an `Ok` build, so `unfinished.json`
stays and the next launch's recovery adopts the partial instead of the
capture quietly finishing without its audio.

`publish_audio` also treats a landed copy as success even when deleting
the partial fails, logging it rather than reporting failure and letting
the fallback overwrite a good `audio.wav`.

Tests: a blocked target keeps the partial and fails the build; a
fallback write that lands consumes the partial and writes `audio.wav`;
an unreadable partial is kept.
2026-11-16 10:28:54 -05:00

401 lines
13 KiB
Rust

use serde_json::Value;
use super::{
CONTROL_VERSION, ControlReply, ControlResponse, EXIT_NO_SPEECH, EXIT_NOT_RUNNING, EXIT_REFUSED,
Output, fail,
};
use crate::control::{ModelInfo, StatusReport, settings::SettingValue};
use crate::dictation::vocabulary::VocabularyEntry;
use scrawl_capture::model::format_length;
/// Prints `reply` and returns its exit code.
pub(crate) fn print(reply: ControlReply, output: Output) -> u8 {
let code = exit_code(&reply);
if output.json {
let response = ControlResponse {
version: CONTROL_VERSION,
reply,
};
match serde_json::to_string_pretty(&response) {
Ok(json) => println!("{json}"),
Err(error) => return fail(&error.to_string()),
}
return code;
}
match reply {
ControlReply::Done { message, .. } => println!("{message}"),
ControlReply::Status(report) => print!("{}", human_status(&report)),
ControlReply::Settings { settings } => print!("{}", human_settings(&settings)),
ControlReply::Models { models } => print!("{}", human_models(&models)),
ControlReply::Vocabulary { entries } => print!("{}", human_vocabulary(&entries)),
ControlReply::ScrawlCapture(report) => print!("{}", human_scrawl_capture(&report)),
ControlReply::ScrawlCaptures { captures } => print!("{}", human_captures(&captures)),
ControlReply::ScrawlCaptureDetail(detail) => print!("{}", human_capture_detail(&detail)),
ControlReply::ScrawlCaptureRoute { explanation } => print!("{explanation}"),
ControlReply::Transcript { text } => println!("{text}"),
ControlReply::Error { message, .. } => eprintln!("hex: {message}"),
}
code
}
pub(super) fn human_scrawl_capture(report: &crate::control::ScrawlCaptureReport) -> String {
use crate::control::ScrawlCaptureState;
let mut out = String::new();
let bundle = |bundle: &Option<std::path::PathBuf>| {
bundle
.as_ref()
.map_or_else(String::new, |bundle| bundle.display().to_string())
};
match (report.state, report.changed) {
(ScrawlCaptureState::Recording, true) => {
out.push_str(&format!("Capturing into {}\n", bundle(&report.bundle)));
}
(ScrawlCaptureState::Recording, false) => out.push_str(&format!(
"Capturing into {} ({} ms)\n",
bundle(&report.bundle),
report.elapsed_ms.unwrap_or_default()
)),
(ScrawlCaptureState::Idle, true) => {
out.push_str(&format!("Ended the capture {}\n", bundle(&report.bundle)));
}
(ScrawlCaptureState::Idle, false) => out.push_str("No capture is recording\n"),
}
for finishing in &report.finishing {
out.push_str(&format!("Writing {}\n", finishing.display()));
}
if let Some(last) = &report.last {
let words = last
.words
.map_or_else(String::new, |words| format!(", {words} words"));
let gaps = last
.gaps
.filter(|gaps| *gaps > 0)
.map_or_else(String::new, |gaps| format!(", {gaps} audio gaps"));
let missing = last
.missing_transcript
.as_ref()
.map_or_else(String::new, |why| format!(", no transcript ({why})"));
let stopped = last.audio_stopped_at_ms.map_or_else(String::new, |at| {
format!(", audio stopped at {}", format_length(at))
});
out.push_str(&format!(
"Last: {} {}{words}{gaps}{stopped}{missing}\n",
last.stage,
last.bundle.display()
));
}
out
}
pub(super) fn human_captures(captures: &[crate::control::ScrawlCaptureSummary]) -> String {
if captures.is_empty() {
return "No captures\n".into();
}
captures
.iter()
.map(|capture| format!("{}\n", capture_line(capture)))
.collect()
}
/// `1792933491953.scrawl 2026-10-26 14:02 0:04 window · Brave 12 words → Hex`
pub(super) fn capture_line(capture: &crate::control::ScrawlCaptureSummary) -> String {
let seconds = capture.duration_ms / 1_000;
let mut line = format!(
"{} {} {}:{:02} {}",
capture.id,
local_minute(capture.created_at_ms),
seconds / 60,
seconds % 60,
capture.mode
);
if let Some(app) = &capture.app {
line.push_str(&format!(" · {app}"));
}
line.push_str(&format!(" {} words", capture.words));
if let Some(sent) = &capture.sent_to {
line.push_str(&format!(" → {}", sent.title));
}
if capture.queued {
line.push_str(" (queued)");
}
line
}
pub(super) fn human_capture_detail(detail: &crate::control::ScrawlCaptureDetail) -> String {
let mut out = format!(
"{}\n{}\n",
capture_line(&detail.summary),
detail.summary.bundle.display()
);
if let Some(title) = &detail.window_title {
out.push_str(&format!("Window: {title}\n"));
}
if let Some(sent) = &detail.summary.sent_to {
out.push_str(&format!("Sent to: {} ({})\n", sent.title, sent.session));
}
out.push_str(&format!(
"{} marks, {} keyframes\n",
detail.marks, detail.keyframes
));
for (label, path) in [
("Video", &detail.video),
("Audio", &detail.audio),
("Contact sheet", &detail.contact_sheet),
] {
if let Some(path) = path {
out.push_str(&format!("{label}: {}\n", path.display()));
}
}
if let Some(note) = &detail.note {
out.push_str(&format!("\nNote: {note}\n"));
}
if let Some(at) = detail.audio_stopped_at_ms {
out.push_str(&format!(
"\nAudio stopped at {}; the rest of this capture has no audio.\n",
format_length(at)
));
}
match (&detail.missing_transcript, detail.transcript.trim()) {
(Some(why), _) => out.push_str(&format!("\nNo transcript: {why}\n")),
(None, "") => out.push_str("\nNo words\n"),
(None, text) => out.push_str(&format!("\n{text}\n")),
}
out
}
/// `2026-10-26 14:02` in local time where Hex knows it.
pub(super) fn local_minute(ms: u64) -> String {
let iso = crate::meeting::query::iso_local(ms);
iso.get(..16)
.map_or(iso.clone(), |minute| minute.replace('T', " "))
}
pub(super) fn exit_code(reply: &ControlReply) -> u8 {
match reply {
ControlReply::Error { code, .. } if code == "not-running" => EXIT_NOT_RUNNING,
ControlReply::Error { code, .. } if code == "no-speech" => EXIT_NO_SPEECH,
ControlReply::Error { .. } => EXIT_REFUSED,
_ => 0,
}
}
pub(super) fn human_value(value: &Value) -> String {
match value {
Value::String(text) => text.clone(),
Value::Null => "none".into(),
other => other.to_string(),
}
}
pub(super) fn human_settings(settings: &[SettingValue]) -> String {
if let [setting] = settings {
return format!("{}\n", human_value(&setting.value));
}
let width = settings
.iter()
.map(|setting| setting.key.len())
.max()
.unwrap_or(0);
settings
.iter()
.map(|setting| {
let mut value = human_value(&setting.value);
if value.len() > 60 {
value = format!("{}…", value.chars().take(59).collect::<String>());
}
format!(
"{:width$} {value}{}\n",
setting.key,
if !setting.available {
" (not on this OS)"
} else if setting.writable {
""
} else {
" (read-only)"
}
)
})
.collect()
}
pub(super) fn human_models(models: &[ModelInfo]) -> String {
let width = models.iter().map(|model| model.id.len()).max().unwrap_or(0);
let name_width = models
.iter()
.map(|model| model.name.len())
.max()
.unwrap_or(0);
models
.iter()
.map(|model| {
let languages = if model.languages.len() > 6 {
format!("{} languages", model.languages.len())
} else {
model.languages.join(", ")
};
format!(
"{} {:width$} {:name_width$} {:>8} {:13} {languages}{}\n",
if model.selected { "*" } else { " " },
model.id,
model.name,
model.size,
if model.installed {
"installed"
} else {
"not installed"
},
if model.source.is_empty() {
String::new()
} else {
format!(" ({})", model.source)
},
)
})
.collect()
}
pub(super) fn human_vocabulary(entries: &[VocabularyEntry]) -> String {
if entries.is_empty() {
return "The vocabulary is empty; `hex vocabulary add <term>` adds one\n".into();
}
entries
.iter()
.map(|entry| match entry.sounds_like.as_slice() {
[] => format!("{}\n", entry.term),
spellings => format!("{} (sounds like {})\n", entry.term, spellings.join(", ")),
})
.collect()
}
pub(super) fn duration(seconds: u64) -> String {
if seconds >= 3600 {
format!(
"{}:{:02}:{:02}",
seconds / 3600,
seconds / 60 % 60,
seconds % 60
)
} else {
format!("{}:{:02}", seconds / 60, seconds % 60)
}
}
pub(super) fn on(enabled: bool) -> &'static str {
if enabled { "on" } else { "off" }
}
pub(super) fn human_status(report: &StatusReport) -> String {
let memory = report
.memory_footprint_bytes
.map(|bytes| format!(", {} MB", bytes / 1_000_000))
.unwrap_or_default();
let capture = match report.capture.recording {
Some(kind) => format!(
"recording {}",
serde_json::to_value(kind)
.ok()
.and_then(|value| value.as_str().map(str::to_owned))
.unwrap_or_default()
),
None => serde_json::to_value(report.capture.state)
.ok()
.and_then(|value| value.as_str().map(str::to_owned))
.unwrap_or_default(),
};
let pending = match report.capture.pending_jobs {
0 => String::new(),
jobs => format!(" ({jobs} pending)"),
};
let meeting = match (&report.meeting.title, report.meeting.elapsed_seconds) {
(Some(title), Some(elapsed)) => {
format!(
"{} “{title}” for {}",
report.meeting.state,
duration(elapsed)
)
}
_ if report.features.meetings => report.meeting.state.clone(),
_ => "off".into(),
};
let model = &report.model;
let mut model_line = format!(
"{} ({}){}",
model.name,
model.language,
if model.installed {
""
} else {
", not installed"
}
);
if let Some(fallback) = &model.fallback {
use crate::transcription::custom_models::FallbackReason;
let reason = match fallback.reason {
FallbackReason::Missing => "it is missing",
FallbackReason::Changed => "it changed since it was added",
FallbackReason::LabsOff => "its Labs toggle is off",
};
model_line.push_str(&format!(
"; using {} ({}) because {reason}",
fallback.name, fallback.language
));
}
if let Some(preparing) = &model.preparing {
model_line.push_str(&format!(
"; preparing {preparing} ({} MB)",
model.downloaded_bytes / 1_000_000
));
}
if let Some(error) = &model.error {
model_line.push_str(&format!("; error: {error}"));
}
let features = &report.features;
let permissions = &report.permissions;
let permission_line = if permissions.ready {
"ready".to_string()
} else {
format!(
"microphone {}, accessibility {}, model {}",
permissions.microphone,
permissions.accessibility,
if permissions.transcription_model {
"ready"
} else {
"missing"
}
)
};
let window = match (&report.window.pane, report.window.open) {
(Some(pane), true) => format!("open ({pane})"),
(None, true) => "open".into(),
_ => "closed".into(),
};
format!(
"Hex {} (pid {}{memory})\n\
Listener {}\n\
Capture {capture}{pending}\n\
Meeting {meeting}\n\
Model {model_line}\n\
Features meetings {}, release mic while idle {}, history {}\n\
Permissions {permission_line}\n\
Window {window}\n\
Update {}\n\
Labs {}\n",
report.app_version,
report.pid,
if report.listener.running {
format!("running: {}", report.listener.status)
} else {
format!("waiting: {}", report.listener.status)
},
on(features.meetings),
on(features.release_microphone_while_idle),
features.history_retention,
report.update,
match report.labs.as_slice() {
[] => "none".to_string(),
names => names.join(", "),
},
)
}