--- title: "Plan: what was said" --- > Documentation Index > Fetch the complete documentation index at: https://jasonm4130-labs.github.io/ambient/llms.txt > Use this file to discover all available pages before exploring further. # Plan: what was said # Plan: what was said Make the transcript closer to what was said, measured. The first four tasks build the measurement: a fixture the loop can fetch, a scorer, a harness that runs Ambient's own recording path over the fixture, and a quality script the loop runs after `scripts/check`. Every task after that is one experiment with one number as its judge. An experiment that does not improve the number still lands: it records the measurement and reverts the code. Ground rules for every task in this plan: - The fixture is LibriSpeech test-clean (CC BY 4.0, openslr.org/12). It is fetched and cached under `~/.cache/ambient/`, never committed. - The number is word error rate as `scripts/quality` prints it, on the models `ambient record` uses. Report it in the pull request body. - A change to the pipeline that makes the number worse is reverted before the task ends; the finding goes into `docs/developing/measurements.md` instead. - Nothing here touches capture, the menu bar, or the settings UI. # Task 1: A word error rate scorer with tests Add `src/wer.rs` with two public functions and unit tests, no I/O: - `normalise(text: &str) -> Vec`: lowercase, split on whitespace, strip every character that is not a letter, digit or apostrophe from each word, drop empty words. `"Hello, world's END."` becomes `["hello", "world's", "end"]`. - `score(reference: &[String], hypothesis: &[String]) -> Wer` where `Wer` holds `substitutions`, `insertions`, `deletions`, `reference_words`, and a method `rate(&self) -> f64` returning `(s + i + d) / reference_words` (0.0 when the reference is empty and the hypothesis is too; 1.0 when only the reference is empty). Use word-level Levenshtein with the standard traceback so the three counts are exact, not just their sum. Tests: identical sequences score zero; one substitution, one insertion, one deletion each counted correctly; the empty cases; `normalise` on punctuation, mixed case, and a hyphenated word (a hyphen splits nothing: `"well-known"` stays one word with the hyphen removed, so document that choice in a comment). Register the module in `src/lib.rs`. Nothing else changes. # Task 2: Fetch and stitch the fixture Add `scripts/fetch-fixtures`, a bash script that builds the WER fixture and is safe to run every night: 1. Download `https://www.openslr.org/resources/12/test-clean.tar.gz` (about 330 MB) to `~/.cache/ambient/librispeech/test-clean.tar.gz` only if that file is absent, and extract it there only if the extracted directory is absent. Verify the download with `tar -tzf` before extracting; a truncated file is deleted and the script exits non-zero saying so. 2. Take the first three speaker directories of `test-clean` in sorted order, and for each the first chapter directory in sorted order, so the choice is deterministic and nothing is hard-coded. Sort the chapter's utterances by id, decode each FLAC to 16 kHz mono 16-bit wav with `ffmpeg` (present on this machine; `afconvert` is the fallback if it is not), and concatenate them in order with 700 ms of silence between utterances into `~/.cache/ambient/fixtures/wer/.wav`. 3. Concatenate the matching lines of the chapter's `.trans.txt` in the same order into `.txt`, one utterance per line. 4. Write `~/.cache/ambient/fixtures/wer/manifest.json`: an array of `{"wav": "", "reference": "", "speaker": "1089", "utterances": N, "seconds": S}`. Idempotent: a second run does no network and rewrites nothing that exists unless `--force` is passed. Print one line per file it produces and end with `FIXTURES OK files s`. Add the cache path to the `## Fixtures` section this task creates at the end of `docs/developing/measurements.md`, with the exact command. Prove it by running it: the report quotes the last line and the three durations. # Task 3: A harness that scores Ambient's own recording path Add `src/bin/wer.rs`. It must run the same path `ambient record` runs for a finished track, so the number means what a user gets: 1. Read the manifest (`--manifest `, default the path Task 2 writes). 2. For each entry: `features::read_wav`, then `Vad::load` on the silero model `record` uses, `vad.turns(&samples, 30)`, then `Recognizer::load` on the ASR model directory `record` uses (find how `src/session.rs` and `src/main.rs` resolve both paths and reuse that, do not hard-code a second copy), then `transcribe_segments`. Join the segment texts with single spaces. 3. Score with `wer::normalise` and `wer::score` against the reference file. 4. Print a table: speaker, seconds, reference words, S, I, D, WER, decode seconds, realtime factor. Then a total row where WER is computed over the summed counts, not averaged. With `--json ` also write those rows as JSON. Exit 0 on success and non-zero with a clear message if the models or the manifest are missing (say which file and which script fetches it). Add the binary to the table in `docs/developing/measurements.md` with its command. `cargo clippy --all-targets -D warnings` covers it, so it must be warning-free. Prove it by running it on the fixture and quoting the total row in the report. # Task 4: The quality script the loop runs Add `scripts/quality`, quiet like `scripts/check`: 1. Run `scripts/fetch-fixtures` (cached after the first night) and `cargo run --release --bin wer -- --json `, both with output captured and shown only on failure. 2. Compare each speaker's WER and the total against `quality/wer.json` (committed). If the file does not exist, write it from this run and print `QUALITY BASELINE ` as the last line. Otherwise, fail if any row's WER exceeds its baseline by more than 0.005 absolute (half a point), printing the offending rows, and end with `QUALITY OK ` otherwise. 3. `--update` rewrites `quality/wer.json` from the current run; that is how an improvement is recorded, and the diff to that file is the evidence in the pull request. Then add a paragraph to `docs/developing/landing.md` under "What a night looks like" saying the quality script runs after the verifier, what its last line means, and that pointing `CHECK_CMD` in `loop/config` at `scripts/check && scripts/quality` is a change a person commits (the hooks deny `loop/` commits from a task, so do not attempt it). Commit the first `quality/wer.json` from your own run. The report quotes the baseline line and its first line says `CHECK_CMD` still needs the daytime change. # Task 5: Both shipped models, one table `models/` holds `sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` and `sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-fp16`. Add `--model ` to the harness, run it against both, and record a table in `docs/developing/measurements.md` with WER, decode seconds and realtime factor per model, measured on this machine with the date. No pipeline change; the task ends with the table and the report says which model is currently the default and whether the table argues for changing it. Changing the default is not part of this task. # Task 6: Does the resampler cost words? The fixture is native 16 kHz, so `resample::to_16k` is never exercised. Add `--via ` to the harness: upsample each fixture wav to that rate with `ffmpeg` into a cache directory, read it, run `resample::to_16k`, and score as before. Run at 48000 (what Core Audio delivers) and 44100. Record the three-way table (native, 48 k, 44.1 k) in the measurements page. If the resampled WER is worse by more than half a point, the report names the resampler as the suspect and stops; fixing it is a task of its own. # Task 7: Padding at turn edges `vad.rs` pads each turn by `PAD_MS` (200 ms) and trims quiet edges. Words at cut points are the usual casualty. Run the harness at 100, 200, 300 and 400 ms (make the value a parameter of `turns` for the experiment, defaulting to the current constant) and record the table. Keep the value that gives the lowest total WER only if it beats the current one by more than half a point on every speaker; otherwise restore 200 and record that the default holds. Either way, update `quality/wer.json` only if the shipped value changed. # Task 8: Chunk length `turns(&samples, 30)` caps a turn at 30 seconds. Run the harness at 15, 20, 30 and 40 (again a parameter for the experiment) and record WER and peak RSS (from `/usr/bin/time -l`) for each. Same keep-or-restore rule as Task 7, with the extra condition that peak RSS must not rise above the 30-second figure already recorded in the measurements page. # Task 9: Insertions in the gaps The fixture has 700 ms of silence between utterances. From the harness's S/I/D breakdown, insertions are the words Parakeet invented. Add a `--gap ` option to `scripts/fetch-fixtures` so a second fixture set with 3 s gaps can be built, score both, and record the insertion count per fixture. If insertions rise with gap length, the report says so and names `last_confidence` as the lever; tuning it is a later task with its own number, because the confidence gate also drops real quiet speech. # Task 10: A second fixture that sounds like a call LibriSpeech is read audiobooks. Earnings-21 (github.com/revdotcom/speech-datasets) is real earnings-call audio: a conference bridge, several speakers, crosstalk, phone-codec sound, with human transcripts under CC BY-SA 4.0. The audio files carry no stated licence, so they are fetched to the cache and never committed, the same as LibriSpeech. 1. Extend `scripts/fetch-fixtures` with a `calls` set beside the existing one. Take the first three ids in sorted order from `earnings21/transcripts/nlp_references/` (list the directory with the GitHub API, or pin the three ids the first run found and say so in the script). Fetch each `earnings21/media/.mp3` from `https://raw.githubusercontent.com/revdotcom/speech-datasets/main/` (plain files, not LFS) and each `.nlp` from `nlp_references/`. 2. Decode the first five minutes of each mp3 to 16 kHz mono 16-bit wav with `ffmpeg -t 300`, into `~/.cache/ambient/fixtures/calls/.wav`. 3. The `.nlp` file is pipe-delimited with a header; the columns are `token|speaker|ts|endTs|punctuation|case|tags|wer_tags`. Build the reference from the `token` column of every row whose `ts` is under 300 seconds, joined with single spaces, into `.txt`. Ignore the punctuation and case columns; `wer::normalise` discards both anyway. 4. Write `~/.cache/ambient/fixtures/calls/manifest.json` in the same shape as the LibriSpeech one. Then make `scripts/quality` run the harness on both manifests and keep both totals in `quality/wer.json` under `clean` and `calls`; the regression rule applies to each set separately, and the last line becomes `QUALITY OK clean= calls=`. Add the calls row to the measurements page with the date. The report quotes both totals and says, in one sentence, how far apart they are; that gap is what the later experiments should close. Source: https://jasonm4130-labs.github.io/ambient/plans/2026-09-05-wer/index.mdx