---
title: "Plan: live transcription, measured first"
---

> Documentation Index
> Fetch the complete documentation index at: https://jasonm4130-labs.github.io/ambient/llms.txt
> Use this file to discover all available pages before exploring further.

# Plan: live transcription, measured first

# Plan: live transcription, measured first

**Goal:** find out whether this machine can transcribe while it captures, as
numbers, before anyone designs live transcription.
**Architecture:** one benchmark binary replays a recording the way a live
transcriber would meet it: native-rate audio in 30 s blocks, each block
resampled, run through VAD, and its turns decoded, every stage timed. Those
timings feed a queue simulation that says how far behind a live transcriber
would fall with one track and with two. The existing `drainbench` supplies the
capture-side number. A measurements section turns both into a go/no-go with
the rule written down.
**Tech Stack:** Rust 1.85 floor, existing `vad`, `asr`, `features`, `resample`,
`session` modules, `src/bin/drainbench.rs`, ffmpeg, `/usr/bin/time -l` for
peak RSS. No new crates.

## Global Constraints

- The input is any mono wav; the default is the first entry of
  `~/.cache/ambient/fixtures/wer/manifest.json` when it exists, otherwise the
  binary needs `--wav <path>` and says so. The benchmark upsamples it to the
  capture's native rate (48 000 Hz) with `ffmpeg -ar 48000` into
  `~/.cache/ambient/bench/<name>.48k.wav` once, so the resampler is in the
  timed path.
- Numbers go in `docs/developing/measurements.md` with the date and machine, in
  the style of the sections already there.
- No capture is started at night: a Core Audio tap needs a LaunchServices
  launch and a TCC grant. The capture side is measured the way ADR-0015 does,
  with `drainbench`, which runs the real 200 ms drain loop without a tap.
- `scripts/check` does not look at docs. After any docs edit run
  `node .github/scripts/check-links.mjs` and quote its `… 0 broken link(s)` line.
- No `.github/workflows/`, `.claude/`, or `loop/` changes; no editing existing tests.

## Task 1: Cost per block, and the backlog it implies

**Files:**
- Create: `src/bin/asrbench.rs`
- Create: `src/bench.rs`; register `pub mod bench;` in `src/lib.rs` before writing the test
- Modify: `src/session.rs:520` (`fn model_paths` becomes `pub fn model_paths`, doc comment kept)
- Modify: `docs/developing/measurements.md` (new `## Live transcription: cost per block` section)

**Interfaces:**
- Consumes: `ambient::resample::{read_wav_any, to_16k}`, `ambient::vad::Vad::{load, turns}`,
  `ambient::asr::Recognizer::{load, transcribe_segments}`, `ambient::session::model_paths(None)`
  (now public; returns `(asr_dir, vad_path)` exactly as `record` resolves them).
- Produces in `src/bench.rs`:
  `pub struct BlockRow { pub index: usize, pub audio_s: f64, pub end_s: f64, pub prep_s: f64, pub decode_s: f64, pub turns: usize }`
  (`prep_s` is resample plus VAD for the block, `decode_s` the sum of its turns' decodes);
  `pub fn rtf(work_s: f64, audio_s: f64) -> Option<f64>` (`None` when `audio_s == 0.0`);
  `pub struct Lag { pub max_lag_s: f64, pub final_lag_s: f64 }`;
  `pub fn repeat(rows: &[BlockRow], times: usize) -> Vec<BlockRow>`: the rows concatenated `times`
  over, each copy's `end_s` offset by the previous copy's last `end_s`, indices renumbered;
  `pub fn simulate_queue(rows: &[BlockRow], tracks: usize) -> Lag`: each block arrives at `end_s`,
  `tracks` copies of every block arrive at the same instant, a single worker handles arrivals in
  arrival order, each taking `prep_s + decode_s`; lag of a block is `finish - end_s`; `final_lag_s` is
  the last block's lag.
- Produces the binary `cargo run --release --bin asrbench -- [--wav <path>] [--block-seconds <n>] [--json <path>]`:
  one row per block (`index, audio_s, end_s, prep_s, decode_s, turns, rtf`), a total row
  (`blocks, audio_s, prep_s, decode_s, rtf, max_block_work_s, model_load_s`), and four lag rows
  (`tracks=1` and `tracks=2`, each once for the workload and once for `repeat(rows, 2)`, each
  `max_lag_s, final_lag_s`). A backlog is bounded when the doubled workload's `max_lag_s` is within
  1 s of the single workload's; an accumulating backlog roughly doubles it. `--block-seconds` below 1
  is an argument error naming the flag. `--json` writes
  `{"blocks":[BlockRow…],"total":{…},"lag":{"1":Lag,"2":Lag},"lag_x2":{"1":Lag,"2":Lag}}`
  with `rtf` as JSON `null` where undefined.
  Blocks carry unfinished speech forward: after `vad.turns` on a block, any turn ending within
  `PAD_MS` of the block's end is not decoded yet; its samples (from that turn's start) are prepended
  to the next block, which is what a live pass reading a growing file would do, and what keeps a 40 s
  utterance splitting at an interior quiet frame as the whole-track path does instead of at the block
  edge. `Vad::turns` resets its recurrent state per call (`src/vad.rs:131`); the benchmark accepts that
  reset because a live pass built on the current `Vad` would pay it too, and says so in the
  measurements section.

- [ ] **Step 1:** write the failing tests in `src/bench.rs`: `rtf(1.5, 30.0) == Some(0.05)`;
  `rtf(1.0, 0.0) == None`; `simulate_queue` on blocks `(audio 10, end 10, prep 1, decode 1)`,
  `(10, 20, 1, 1)` with 1 track gives `max_lag_s == 2.0`; the same rows with 2 tracks give
  `max_lag_s == 4.0` (the second copy waits for the first); blocks `(10, 10, 2, 10), (10, 20, 2, 10)` with
  1 track give `max_lag_s == 14.0` and `final_lag_s == 14.0`; `repeat` of those two rows twice gives
  four rows with `end_s` 10, 20, 30, 40 and `simulate_queue` on them gives `max_lag_s == 18.0`
  (each block adds 2 s of backlog: lags 12, 14, 16, 18), while `repeat` of the first pair gives
  `max_lag_s == 2.0` (bounded); an empty slice gives zeros.
- [ ] **Step 2:** run `cargo test bench::`, expect FAIL "cannot find function".
- [ ] **Step 3:** implement `src/bench.rs`, make `model_paths` public, then the binary: build the 48 kHz
  copy if absent, `read_wav_any` it, load the VAD and recogniser (timing the load), then for each
  block of `block_seconds` (default 30) of native-rate samples, with the carried tail prepended: time
  `to_16k` plus `vad.turns(&block16k, block_seconds)` as `prep_s`, hold back a turn ending within
  `PAD_MS` of the block end, then time each `transcribe_segments` call on the rest and sum as
  `decode_s`. The last block decodes everything. Print the rows, total and the four lag rows, write
  the JSON when asked.
- [ ] **Step 4:** run `scripts/check`, expect `CHECK OK`. Then run
  `/usr/bin/time -l cargo run --release --bin asrbench` and record in the measurements section: the
  total row, both lag rows, `maximum resident set size` from `time -l`, and the block with the largest
  `prep_s + decode_s`. Run `node .github/scripts/check-links.mjs`, expect `0 broken link(s)`.
- [ ] **Step 5:** commit: `git add src/bin/asrbench.rs src/bench.rs src/lib.rs src/session.rs docs/developing/measurements.md && git commit -m "asrbench: cost per block and the backlog it implies"`.

## Task 2: The drain loop beside a decoder

**Files:**
- Modify: `src/bin/drainbench.rs` (the fixed three-second sleep at `src/bin/drainbench.rs:76` becomes
  a wait on a `std::sync::mpsc` channel: the ASR thread sends `Ok(())` once its models are loaded or
  `Err(<message>)` if loading failed, and the main thread uses `recv_timeout(120 s)`, exiting non-zero
  with the worker's error or `worker not ready after 120 s`; the ASR thread counts completed decodes
  in an `AtomicUsize`; the summary prints that count and the run exits non-zero with
  `no decode completed during the measured window` when it is zero)
- Modify: `docs/developing/measurements.md` (new `## Live transcription: drain overshoot beside ASR` section)

**Interfaces:**
- Consumes: `cargo run --release --bin drainbench -- <seconds> <wav>` (`src/bin/drainbench.rs`), which
  runs the capture's 200 ms drain loop on a default-QoS thread with ASR on a background-QoS thread and
  prints each period's overshoot; the fixture wav from Task 1.

- [ ] **Step 1:** read `src/bin/drainbench.rs`; make the readiness and decode-count changes above
  (the channel's sender moves into the worker; the `Arc<AtomicUsize>` is shared the same way its stop
  flag is).
- [ ] **Step 2:** run `scripts/check`, expect `CHECK OK`; run
  `cargo run --release --bin drainbench -- 60 <fixture wav>` three times and confirm each prints a
  non-zero decode count.
- [ ] **Step 3:** record a table: run, periods, decodes completed, mean overshoot, max overshoot, and
  the ring margin it leaves (the mic ring is 30 s, the call ring 60 s, per the binary's header comment).
  Quote the worst run's max overshoot in the section's first sentence. Run
  `node .github/scripts/check-links.mjs`, expect `0 broken link(s)`.
- [ ] **Step 4:** commit: `git add src/bin/drainbench.rs docs/developing/measurements.md && git commit -m "drainbench: wait for the decoder, count its passes"`.

## Task 3: The go/no-go and what a follow-up must respect

**Files:**
- Modify: `docs/developing/measurements.md` (a `### Verdict` paragraph under the cost-per-block section)
- Modify: `docs/developing/two-tracks.md` (one paragraph at the end pointing at the verdict)

**Interfaces:**
- Consumes: the total, lag and RSS rows from Task 1, the overshoot table from Task 2.

- [ ] **Step 1:** re-run `cargo run --release --bin asrbench -- --block-seconds 10` and
  `--block-seconds 20`; add both total rows and both `tracks=2` lag pairs beside the 30 s rows.
- [ ] **Step 2:** write the verdict, one paragraph, quoting the numbers, using this rule: live
  transcription is a **go** when, at 30 s blocks, the `tracks=2` `max_lag_s` is under 30 and the
  `tracks=2` doubled-workload `max_lag_s` is within 1 s of it (the backlog is bounded, not growing), the
  `tracks=2` implied `rtf` (`2 × (prep_s + decode_s) / audio_s`) is under 0.5, and the worst drain
  overshoot from Task 2 is under 1 s; otherwise **no-go**, naming the constraint that failed. Say
  in the same paragraph that the go covers one recogniser at a time: the queue's worker and the live
  pass never run together.
- [ ] **Step 3:** if go: add a paragraph naming what the follow-up plan must respect, as facts with
  file references: `transcribe_session` truncates `raw.jsonl` with `File::create` at
  `src/session.rs:841`, so a live writer needs a handoff (the final pass must fold or skip lines the
  live pass already wrote); `room.native.wav` is a `hound` WAV whose RIFF and data length fields stay
  stale until `finalize()` at `src/session.rs:698`, so a tail reader parses the header once, then
  reads samples past it by file growth and never trusts the length fields; the live pass must run on
  the background-QoS thread `queue::Queue` uses, since Task 2 measured that arrangement and no other;
  and the live pass and the queue's transcriber are one worker, never two: while a capture is live the
  queue does not start a queued session (back-to-back meetings otherwise put two recognisers on the
  CPU at once, a workload neither Task 1's single-worker simulation nor Task 2's single decoder
  measured, so a go here does not cover it; a follow-up that wants both at once re-runs both
  benchmarks with two workers first).
  If no-go: name the two levers to measure next, the fp16 model on Core ML and a shorter block, each
  with the figure that would flip the verdict. Do not implement either.
- [ ] **Step 4:** run `scripts/check`, expect `CHECK OK`; run `node .github/scripts/check-links.mjs`
  and quote its last line.
- [ ] **Step 5:** commit: `git add docs/developing/measurements.md docs/developing/two-tracks.md && git commit -m "measurements: live transcription verdict"`.

## Open Questions

Source: https://jasonm4130-labs.github.io/ambient/plans/2026-09-05-live-asr/index.mdx