Make the transcript closer to what was said, measured. The first four tasks build the measurement: a fixture the loop can fetch, a scorer, a harness that runs Ambient’s own recording path over the fixture, and a quality script the loop runs after scripts/check. Every task after that is one experiment with one number as its judge. An experiment that does not improve the number still lands: it records the measurement and reverts the code.

Ground rules for every task in this plan:

• The fixture is LibriSpeech test-clean (CC BY 4.0, openslr.org/12). It is fetched and cached under ~/.cache/ambient/, never committed.
• The number is word error rate as scripts/quality prints it, on the models ambient record uses. Report it in the pull request body.
• A change to the pipeline that makes the number worse is reverted before the task ends; the finding goes into docs/developing/measurements.md instead.
• Nothing here touches capture, the menu bar, or the settings UI.

Task 1: A word error rate scorer with tests

Add src/wer.rs with two public functions and unit tests, no I/O:

• normalise(text: &str) -> Vec<String>: lowercase, split on whitespace, strip every character that is not a letter, digit or apostrophe from each word, drop empty words. "Hello, world's END." becomes ["hello", "world's", "end"].
• score(reference: &[String], hypothesis: &[String]) -> Wer where Wer holds substitutions, insertions, deletions, reference_words, and a method rate(&self) -> f64 returning (s + i + d) / reference_words (0.0 when the reference is empty and the hypothesis is too; 1.0 when only the reference is empty). Use word-level Levenshtein with the standard traceback so the three counts are exact, not just their sum.

Tests: identical sequences score zero; one substitution, one insertion, one deletion each counted correctly; the empty cases; normalise on punctuation, mixed case, and a hyphenated word (a hyphen splits nothing: "well-known" stays one word with the hyphen removed, so document that choice in a comment). Register the module in src/lib.rs. Nothing else changes.

Task 2: Fetch and stitch the fixture

Add scripts/fetch-fixtures, a bash script that builds the WER fixture and is safe to run every night:

1. Download https://www.openslr.org/resources/12/test-clean.tar.gz (about 330 MB) to ~/.cache/ambient/librispeech/test-clean.tar.gz only if that file is absent, and extract it there only if the extracted directory is absent. Verify the download with tar -tzf before extracting; a truncated file is deleted and the script exits non-zero saying so.
2. Take the first three speaker directories of test-clean in sorted order, and for each the first chapter directory in sorted order, so the choice is deterministic and nothing is hard-coded. Sort the chapter’s utterances by id, decode each FLAC to 16 kHz mono 16-bit wav with ffmpeg (present on this machine; afconvert is the fallback if it is not), and concatenate them in order with 700 ms of silence between utterances into ~/.cache/ambient/fixtures/wer/<speaker>.wav.
3. Concatenate the matching lines of the chapter’s .trans.txt in the same order into <speaker>.txt, one utterance per line.
4. Write ~/.cache/ambient/fixtures/wer/manifest.json: an array of {"wav": "<abs path>", "reference": "<abs path>", "speaker": "1089", "utterances": N, "seconds": S}.

Idempotent: a second run does no network and rewrites nothing that exists unless --force is passed. Print one line per file it produces and end with FIXTURES OK <n> files <total seconds>s. Add the cache path to the ## Fixtures section this task creates at the end of docs/developing/measurements.md, with the exact command.

Prove it by running it: the report quotes the last line and the three durations.

Task 3: A harness that scores Ambient’s own recording path

Add src/bin/wer.rs. It must run the same path ambient record runs for a finished track, so the number means what a user gets:

1. Read the manifest (--manifest <path>, default the path Task 2 writes).
2. For each entry: features::read_wav, then Vad::load on the silero model record uses, vad.turns(&samples, 30), then Recognizer::load on the ASR model directory record uses (find how src/session.rs and src/main.rs resolve both paths and reuse that, do not hard-code a second copy), then transcribe_segments. Join the segment texts with single spaces.
3. Score with wer::normalise and wer::score against the reference file.
4. Print a table: speaker, seconds, reference words, S, I, D, WER, decode seconds, realtime factor. Then a total row where WER is computed over the summed counts, not averaged. With --json <path> also write those rows as JSON.

Exit 0 on success and non-zero with a clear message if the models or the manifest are missing (say which file and which script fetches it). Add the binary to the table in docs/developing/measurements.md with its command. cargo clippy --all-targets -D warnings covers it, so it must be warning-free. Prove it by running it on the fixture and quoting the total row in the report.

Task 4: The quality script the loop runs

Add scripts/quality, quiet like scripts/check:

1. Run scripts/fetch-fixtures (cached after the first night) and cargo run --release --bin wer -- --json <tmp>, both with output captured and shown only on failure.
2. Compare each speaker’s WER and the total against quality/wer.json (committed). If the file does not exist, write it from this run and print QUALITY BASELINE <total wer> as the last line. Otherwise, fail if any row’s WER exceeds its baseline by more than 0.005 absolute (half a point), printing the offending rows, and end with QUALITY OK <total wer> otherwise.
3. --update rewrites quality/wer.json from the current run; that is how an improvement is recorded, and the diff to that file is the evidence in the pull request.

Then add a paragraph to docs/developing/landing.md under “What a night looks like” saying the quality script runs after the verifier, what its last line means, and that pointing CHECK_CMD in loop/config at scripts/check && scripts/quality is a change a person commits (the hooks deny loop/ commits from a task, so do not attempt it). Commit the first quality/wer.json from your own run. The report quotes the baseline line and its first line says CHECK_CMD still needs the daytime change.

Task 5: Both shipped models, one table

models/ holds sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8 and sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-fp16. Add --model <dir> to the harness, run it against both, and record a table in docs/developing/measurements.md with WER, decode seconds and realtime factor per model, measured on this machine with the date. No pipeline change; the task ends with the table and the report says which model is currently the default and whether the table argues for changing it. Changing the default is not part of this task.

Task 6: Does the resampler cost words?

The fixture is native 16 kHz, so resample::to_16k is never exercised. Add --via <rate> to the harness: upsample each fixture wav to that rate with ffmpeg into a cache directory, read it, run resample::to_16k, and score as before. Run at 48000 (what Core Audio delivers) and 44100. Record the three-way table (native, 48 k, 44.1 k) in the measurements page. If the resampled WER is worse by more than half a point, the report names the resampler as the suspect and stops; fixing it is a task of its own.

Task 7: Padding at turn edges

vad.rs pads each turn by PAD_MS (200 ms) and trims quiet edges. Words at cut points are the usual casualty. Run the harness at 100, 200, 300 and 400 ms (make the value a parameter of turns for the experiment, defaulting to the current constant) and record the table. Keep the value that gives the lowest total WER only if it beats the current one by more than half a point on every speaker; otherwise restore 200 and record that the default holds. Either way, update quality/wer.json only if the shipped value changed.

Task 8: Chunk length

turns(&samples, 30) caps a turn at 30 seconds. Run the harness at 15, 20, 30 and 40 (again a parameter for the experiment) and record WER and peak RSS (from /usr/bin/time -l) for each. Same keep-or-restore rule as Task 7, with the extra condition that peak RSS must not rise above the 30-second figure already recorded in the measurements page.

Task 9: Insertions in the gaps

The fixture has 700 ms of silence between utterances. From the harness’s S/I/D breakdown, insertions are the words Parakeet invented. Add a --gap <ms> option to scripts/fetch-fixtures so a second fixture set with 3 s gaps can be built, score both, and record the insertion count per fixture. If insertions rise with gap length, the report says so and names last_confidence as the lever; tuning it is a later task with its own number, because the confidence gate also drops real quiet speech.

Task 10: A second fixture that sounds like a call

LibriSpeech is read audiobooks. Earnings-21 (github.com/revdotcom/speech-datasets) is real earnings-call audio: a conference bridge, several speakers, crosstalk, phone-codec sound, with human transcripts under CC BY-SA 4.0. The audio files carry no stated licence, so they are fetched to the cache and never committed, the same as LibriSpeech.

1. Extend scripts/fetch-fixtures with a calls set beside the existing one. Take the first three ids in sorted order from earnings21/transcripts/nlp_references/ (list the directory with the GitHub API, or pin the three ids the first run found and say so in the script). Fetch each earnings21/media/<id>.mp3 from https://raw.githubusercontent.com/revdotcom/speech-datasets/main/ (plain files, not LFS) and each <id>.nlp from nlp_references/.
2. Decode the first five minutes of each mp3 to 16 kHz mono 16-bit wav with ffmpeg -t 300, into ~/.cache/ambient/fixtures/calls/<id>.wav.
3. The .nlp file is pipe-delimited with a header; the columns are token|speaker|ts|endTs|punctuation|case|tags|wer_tags. Build the reference from the token column of every row whose ts is under 300 seconds, joined with single spaces, into <id>.txt. Ignore the punctuation and case columns; wer::normalise discards both anyway.
4. Write ~/.cache/ambient/fixtures/calls/manifest.json in the same shape as the LibriSpeech one.

Then make scripts/quality run the harness on both manifests and keep both totals in quality/wer.json under clean and calls; the regression rule applies to each set separately, and the last line becomes QUALITY OK clean=<wer> calls=<wer>. Add the calls row to the measurements page with the date. The report quotes both totals and says, in one sentence, how far apart they are; that gap is what the later experiments should close.