Skip to content
Ambient

Plan: who spoke

Updated View as Markdown

Goal: measure how often Ambient’s diarizer assigns speech to the wrong speaker, on a public meeting corpus, with the same Diarizer::diarize call ambient diarize makes; then tune the diarizer against that number. Architecture: a diarization error rate (DER) scorer in src/der.rs, an AMI fixture fetched into ~/.cache/ambient, a harness binary that runs Diarizer::diarize over the fixture and scores it, a der= figure on the quality script’s last line, then one experiment per task with a keep-or-restore rule. Tech Stack: Rust 1.85 floor, no new crates (anyhow, serde_json, hound only); bash + ffmpeg + curl + shasum for the fixture; ONNX models already in models/.

Global Constraints

  • The number is span-level DER: how the diarizer’s spans compare with the reference turns. The later step in session::diarize_session that gives each raw.jsonl record its majority speaker (src/session.rs:1241) is not scored here; a record straddling two turns is a transcript-level error and gets its own plan once raw.jsonl-level fixtures exist.
  • The fixture is AMI Mix-Headset audio (CC BY 4.0) with the pyannote only_words RTTM references pinned to commit 67c2d539286e89f68952d5dcf83912bd9f01dfae. Cached under ~/.cache/ambient/fixtures/der/, never committed.
  • The number is DER with a 0.25 s collar as cargo run --release --bin der prints it in its total row. Report it in the pull request body.
  • An experiment that makes the number worse reverts its code change before the task ends and records the table in docs/developing/measurements.md.
  • scripts/check does not look at docs. After any docs edit run node .github/scripts/check-links.mjs and quote its … 0 broken link(s) line.
  • Nothing here touches capture, the menu bar, the settings UI, or src/asr.rs.
  • No .github/workflows/, .claude/, or loop/ changes; no editing existing tests.

Task 1: A DER scorer with tests

Files:

  • Create: src/der.rs
  • Modify: src/lib.rs (add pub mod der; in alphabetical position, before writing any test)

Interfaces:

  • Produces: pub struct Turn { pub start_s: f64, pub end_s: f64, pub speaker: String }; pub fn parse_rttm(text: &str) -> anyhow::Result<Vec<Turn>>; pub struct Der { pub missed_s: f64, pub false_alarm_s: f64, pub confusion_s: f64, pub reference_s: f64 } with pub fn rate(&self) -> f64 = (missed + false_alarm + confusion) / reference; pub fn score(reference: &[Turn], hypothesis: &[Turn], collar_s: f64) -> anyhow::Result<Der>, an error only when the reference is empty (empty reference); any number of speakers on either side is scored, so an over-clustered candidate in a sweep gets a number, not an abort.

  • Step 1: add pub mod der; to src/lib.rs, create src/der.rs holding only the #[cfg(test)] mod tests below (every test passes collar_s 0.0 except the collar test), and run cargo test der:: once to see the red:

    • parse_rttm on SPEAKER ES2004a 1 12.34 5.00 <NA> <NA> FEE013 <NA> <NA> yields Turn{start_s:12.34,end_s:17.34,speaker:"FEE013"}; blank lines and lines starting with ;; are skipped; any other line not starting with SPEAKER, or with fewer than 8 fields, is an error naming the line number (an HTML error page saved as an RTTM must fail, not parse to nothing).
    • identical reference and hypothesis score rate() == 0.0.
    • reference A 0–10, hypothesis empty: missed_s == 10.0.
    • reference empty, hypothesis A 0–4: score is an error containing empty reference.
    • reference A 0–10, B 10–20, hypothesis X 0–10, Y 10–20: rate 0.0 (labels are mapped, not compared).
    • reference A 0–10, B 10–20, hypothesis X 0–20: confusion_s == 10.0 (within 0.05).
    • over-clustering: reference A 0–17, B 17–25, hypothesis X 0–9, Y 9–17, X 17–25: the optimal map is A→Y, B→X (16 s correct, confusion_s == 9.0 within 0.05); a greedy assignment takes A→X first (9 s) and leaves B nothing, scoring 16 s of confusion, so this test fails on greedy.
    • many clusters: reference A 0–10, B 10–20, hypothesis with twelve labels H0..H11, label Hi covering [i, i+1): score is Ok; one label maps to A and one to B (2 s correct), so confusion_s == 10.0, missed_s == 8.0 (12–20 has no hypothesis), false_alarm_s == 0.0.
    • collar: reference A 0–10, hypothesis A 0.2–10, collar 0.25: rate 0.0; collar 0.0: missed_s ≈ 0.2.

  • Step 2: expect FAIL with “cannot find function” for each item.

  • Step 3: implement. Discretise both sides into 10 ms frames over max(end) of both. A frame is excluded when it lies within collar_s of any reference turn boundary. Per frame collect the set of reference speakers and hypothesis speakers. Speaker mapping is exact for any count: build the overlap matrix (reference × hypothesis, in included frames), pad it to square with zeros, and solve the maximum-weight assignment with the Hungarian algorithm (O(n³), about sixty lines, written in src/der.rs with a comment naming it; no crate). Then per included frame, with n_ref the number of reference speakers, n_hyp the number of hypothesis speakers, and n_correct the reference speakers whose mapped hypothesis label is present: missed = max(n_ref − n_hyp, 0), false alarm = max(n_hyp − n_ref, 0), confusion = min(n_ref, n_hyp) − n_correct, each × 0.01 s (the NIST md-eval cardinality formula, so an unmapped label speaking over a reference speaker is confusion, not a miss plus a false alarm). reference_s = sum of n_ref × 0.01 s over included frames. Comment the frame size.

  • Step 4: run scripts/check, expect last line CHECK OK.

  • Step 5: commit: git add src/der.rs src/lib.rs && git commit -m "der: diarization error rate scorer".

Task 2: Fetch the AMI fixture

Files:

  • Create or modify: scripts/fetch-fixtures (if it exists, add a der set beside the existing sets and keep its existing behaviour and last line; if it does not exist, create it with only the der set, ending with FIXTURES OK <n> files <total seconds>s). Executable, set -euo pipefail.
  • Modify: docs/developing/measurements.md (a ## Fixtures section; create it at the end if absent, otherwise add a paragraph for the DER set with the exact command).

Interfaces:

  • Produces: ~/.cache/ambient/fixtures/der/manifest.json, a JSON array of {"wav": "<abs>", "rttm": "<abs>", "meeting": "ES2004a", "seconds": 300}.

  • Step 1: the meetings are pinned: ES2004a, IS1009a, TS3003a (the first meeting of three AMI test-set series; say so in a comment). For each meeting M, download with curl --fail --location --silent --show-error -o <tmp> then mv into place, only if absent: https://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus/<M>/audio/<M>.Mix-Headset.wav to ~/.cache/ambient/ami/<M>.Mix-Headset.wav, and https://raw.githubusercontent.com/pyannote/AMI-diarization-setup/67c2d539286e89f68952d5dcf83912bd9f01dfae/only_words/rttms/test/<M>.rttm to ~/.cache/ambient/ami/<M>.rttm. Verify each wav with ffprobe and each RTTM by checking it is non-empty and every non-blank line starts with SPEAKER <M> (delete the file, exit non-zero and name the URL otherwise).

  • Step 2: on the first run print shasum -a 256 of each downloaded file; then pin those six digests in the script as a sha256sum-style table and have the script verify cached files against it on every run, so an upstream change surfaces as a failure and not as a moved baseline.

  • Step 3: produce ~/.cache/ambient/fixtures/der/<M>.wav: the first 300 s, 16 kHz mono 16-bit, via ffmpeg -t 300 -ac 1 -ar 16000 -sample_fmt s16. Produce <M>.rttm: the reference lines whose start is under 300 s, with dur clipped so start + dur ≤ 300, written with awk in the same 10-field RTTM layout. Write manifest.json. Idempotent: a second run does no network and rewrites nothing unless --force. Print one line per file produced.

  • Step 4: run scripts/fetch-fixtures, then run it again. The report quotes the last line of both runs, the six digests now pinned, and ls -la ~/.cache/ambient/fixtures/der.

  • Step 5: run node .github/scripts/check-links.mjs, expect 0 broken link(s). Commit: git add scripts/fetch-fixtures docs/developing/measurements.md && git commit -m "fixtures: AMI diarization set".

Task 3: A harness that scores Ambient’s own diarize path

Files:

  • Create: src/bin/der.rs
  • Modify: src/der.rs (add spans_to_turns)
  • Modify: docs/developing/measurements.md (add the binary and its command to the table of tools, or a ## DER section if there is no such table)

Interfaces:

  • Consumes: ambient::der::{parse_rttm, score, Turn}; ambient::resample::{read_wav_any, to_16k}; ambient::diarize::{Diarizer, Span, DEFAULT_THRESHOLD}; ambient::session::models_root.

  • Produces: pub fn spans_to_turns(spans: &[Span], sr: usize) -> Vec<Turn> in src/der.rs; cargo run --release --bin der -- [--manifest <path>] [--threshold <f32>] [--collar <f64>] [--json <path>], a table with one row per meeting and a total row, JSON rows as {"meeting","seconds","reference_speakers","hypothesis_speakers","missed_s","false_alarm_s","confusion_s","der","run_s"}.

  • Step 1: write the failing test in src/der.rs’s test module: [Span{start:0,end:16000,speaker:0}, Span{start:32000,end:48000,speaker:1}] at 16 kHz becomes [Turn{0.0,1.0,"0"}, Turn{2.0,3.0,"1"}].

  • Step 2: run cargo test der::, expect FAIL “cannot find function spans_to_turns”.

  • Step 3: implement spans_to_turns, then the binary: parse args (default manifest ~/.cache/ambient/fixtures/der/manifest.json, threshold DEFAULT_THRESHOLD, collar 0.25). Resolve the models exactly as session::diarize_session does (src/session.rs:1188-1197): models_root()? joined with pyannote-segmentation-3.0/model.onnx and wespeaker_en_voxceleb_resnet34_LM.onnx, bailing with missing <path> — run ./fetch-models.sh. For each manifest entry: read_wav_any → to_16k → Diarizer::diarize(&samples, threshold) → spans_to_turns(.., 16_000) → score against parse_rttm of the rttm file; an empty reference is an error naming the file, never a zero. Print the table; the total row sums the four seconds fields and computes DER from the sums. Exit non-zero with the file name and the fetch script to run when the manifest is missing.

  • Step 4: run scripts/check, expect CHECK OK; then cargo run --release --bin der and quote the total row in the report; run node .github/scripts/check-links.mjs, expect 0 broken link(s).

  • Step 5: commit: git add src/bin/der.rs src/der.rs docs/developing/measurements.md && git commit -m "der: harness over the diarize path".

Task 4: The quality script reports DER

Files:

  • Create or modify: scripts/quality (if it exists, add a DER step and append der=<total> to its existing last line; if it does not exist, create it quiet like scripts/check, running scripts/fetch-fixtures then the DER harness, ending with QUALITY OK der=<total>).
  • Create: quality/der.json from your own run (the harness’s --json output).
  • Modify: docs/developing/landing.md (one sentence under “What a night looks like” saying the quality script’s last line carries der=, and that wiring scripts/quality into CHECK_CMD in loop/config is a change a person commits, because the hooks deny loop/ commits from a task; the script is the gate, the config line is the switch, and this plan delivers the gate).

Interfaces:

  • Consumes: cargo run --release --bin der -- --json <tmp>.

  • Produces: the regression rule: any meeting’s der or the total more than 0.01 above quality/der.json fails the script, printing the offending rows; --update rewrites quality/der.json.

  • Step 1: run scripts/quality before the change (or note it does not exist yet).

  • Step 2: implement the step with jq (present on this machine) comparing per-meeting and total der against the baseline file; when quality/der.json is absent, write it and print QUALITY BASELINE der=<total> as the last line instead.

  • Step 3: run scripts/quality twice: the first run writes the baseline, the second ends QUALITY OK … der=<total>. Quote both last lines.

  • Step 4: run scripts/check, expect CHECK OK; run node .github/scripts/check-links.mjs, expect 0 broken link(s).

  • Step 5: commit: git add scripts/quality quality/der.json docs/developing/landing.md && git commit -m "quality: DER baseline". The report’s first line says whether loop/config’s CHECK_CMD already runs scripts/quality; if not, it names that as the one-line daytime change for a human.

Task 5: The clustering threshold

Files:

  • Modify: src/diarize.rs:38 (DEFAULT_THRESHOLD), only if the rule below says so.
  • Modify: docs/developing/measurements.md (a ## DER by threshold table with the date).
  • Modify: quality/der.json via scripts/quality --update, only if the constant changed.

Interfaces:

  • Consumes: cargo run --release --bin der -- --threshold <t> --json <tmp>.

  • Step 1: run the harness at thresholds 0.30, 0.40, 0.50, 0.60, 0.70, 0.80 and record, per threshold, total DER and the hypothesis speaker count per meeting next to the reference count.

  • Step 2: keep the threshold with the lowest total DER only if it beats 0.50 by more than 0.01 on every meeting and never produces more than twice the reference speaker count; otherwise restore 0.50 and write that the default holds.

  • Step 3: if the constant changed, run scripts/quality --update and include the diff to quality/der.json.

  • Step 4: run scripts/check, expect CHECK OK; run node .github/scripts/check-links.mjs, expect 0 broken link(s).

  • Step 5: commit: git add src/diarize.rs docs/developing/measurements.md quality/der.json && git commit -m "diarize: threshold measured against AMI" (omit files that did not change).

Task 6: Minimum embedding length

Files:

  • Modify: src/diarize.rs:34 (MIN_EMBED, currently one second of samples), swapped between runs as a constant.
  • Modify: docs/developing/measurements.md (a ## DER by minimum embedding length table).

Interfaces:

  • Consumes: the DER harness.

  • Step 1: run the harness with MIN_EMBED at 0.5 s, 1.0 s (current), 1.5 s and 2.0 s of samples, recording total DER, missed_s, and the hypothesis speaker counts.

  • Step 2: same keep-or-restore rule: keep a value only if it beats the current one by more than 0.01 on every meeting; otherwise restore and record that the default holds. Explain in one sentence in the measurements table what moved (segments shorter than MIN_EMBED are not embedded, so a lower value trades confusion for missed speech or the reverse).

  • Step 3: if the constant changed, run scripts/quality --update.

  • Step 4: run scripts/check, expect CHECK OK; run node .github/scripts/check-links.mjs, expect 0 broken link(s).

  • Step 5: commit: git add src/diarize.rs docs/developing/measurements.md quality/der.json && git commit -m "diarize: minimum embedding length measured" (omit files that did not change).

Open Questions

Third-party licenses
Navigation

Type to search…

↑↓ navigate↵ selectEsc close