---
title: "Plan: who spoke"
---

> Documentation Index
> Fetch the complete documentation index at: https://jasonm4130-labs.github.io/ambient/llms.txt
> Use this file to discover all available pages before exploring further.

# Plan: who spoke

# Plan: who spoke

**Goal:** measure how often Ambient's diarizer assigns speech to the wrong
speaker, on a public meeting corpus, with the same `Diarizer::diarize` call
`ambient diarize` makes; then tune the diarizer against that number.
**Architecture:** a diarization error rate (DER) scorer in `src/der.rs`, an
AMI fixture fetched into `~/.cache/ambient`, a harness binary that runs
`Diarizer::diarize` over the fixture and scores it, a `der=` figure on the
quality script's last line, then one experiment per task with a keep-or-restore
rule.
**Tech Stack:** Rust 1.85 floor, no new crates (anyhow, serde_json, hound only);
bash + ffmpeg + curl + shasum for the fixture; ONNX models already in `models/`.

## Global Constraints

- The number is span-level DER: how the diarizer's spans compare with the
  reference turns. The later step in `session::diarize_session` that gives each
  `raw.jsonl` record its majority speaker (`src/session.rs:1241`) is not scored
  here; a record straddling two turns is a transcript-level error and gets its
  own plan once `raw.jsonl`-level fixtures exist.
- The fixture is AMI Mix-Headset audio (CC BY 4.0) with the pyannote
  `only_words` RTTM references pinned to commit
  `67c2d539286e89f68952d5dcf83912bd9f01dfae`. Cached under
  `~/.cache/ambient/fixtures/der/`, never committed.
- The number is DER with a 0.25 s collar as `cargo run --release --bin der`
  prints it in its total row. Report it in the pull request body.
- An experiment that makes the number worse reverts its code change before the
  task ends and records the table in `docs/developing/measurements.md`.
- `scripts/check` does not look at docs. After any docs edit run
  `node .github/scripts/check-links.mjs` and quote its `… 0 broken link(s)` line.
- Nothing here touches capture, the menu bar, the settings UI, or `src/asr.rs`.
- No `.github/workflows/`, `.claude/`, or `loop/` changes; no editing existing tests.

## Task 1: A DER scorer with tests

**Files:**
- Create: `src/der.rs`
- Modify: `src/lib.rs` (add `pub mod der;` in alphabetical position, before writing any test)

**Interfaces:**
- Produces: `pub struct Turn { pub start_s: f64, pub end_s: f64, pub speaker: String }`;
  `pub fn parse_rttm(text: &str) -> anyhow::Result<Vec<Turn>>`;
  `pub struct Der { pub missed_s: f64, pub false_alarm_s: f64, pub confusion_s: f64, pub reference_s: f64 }`
  with `pub fn rate(&self) -> f64` = `(missed + false_alarm + confusion) / reference`;
  `pub fn score(reference: &[Turn], hypothesis: &[Turn], collar_s: f64) -> anyhow::Result<Der>`,
  an error only when the reference is empty (`empty reference`); any number of speakers on either
  side is scored, so an over-clustered candidate in a sweep gets a number, not an abort.

- [ ] **Step 1:** add `pub mod der;` to `src/lib.rs`, create `src/der.rs` holding only the
  `#[cfg(test)] mod tests` below (every test passes `collar_s` 0.0 except the collar test), and run
  `cargo test der::` once to see the red:
  - `parse_rttm` on `SPEAKER ES2004a 1 12.34 5.00 <NA> <NA> FEE013 <NA> <NA>` yields
    `Turn{start_s:12.34,end_s:17.34,speaker:"FEE013"}`; blank lines and lines starting with `;;` are
    skipped; any other line not starting with `SPEAKER`, or with fewer than 8 fields, is an error naming
    the line number (an HTML error page saved as an RTTM must fail, not parse to nothing).
  - identical reference and hypothesis score `rate() == 0.0`.
  - reference `A 0–10`, hypothesis empty: `missed_s == 10.0`.
  - reference empty, hypothesis `A 0–4`: `score` is an error containing `empty reference`.
  - reference `A 0–10, B 10–20`, hypothesis `X 0–10, Y 10–20`: rate 0.0 (labels are mapped, not compared).
  - reference `A 0–10, B 10–20`, hypothesis `X 0–20`: `confusion_s == 10.0` (within 0.05).
  - over-clustering: reference `A 0–17, B 17–25`, hypothesis `X 0–9, Y 9–17, X 17–25`: the optimal
    map is A→Y, B→X (16 s correct, `confusion_s == 9.0` within 0.05); a greedy assignment takes A→X
    first (9 s) and leaves B nothing, scoring 16 s of confusion, so this test fails on greedy.
  - many clusters: reference `A 0–10, B 10–20`, hypothesis with twelve labels `H0..H11`, label `Hi`
    covering `[i, i+1)`: `score` is `Ok`; one label maps to A and one to B (2 s correct), so
    `confusion_s == 10.0`, `missed_s == 8.0` (12–20 has no hypothesis), `false_alarm_s == 0.0`.
  - collar: reference `A 0–10`, hypothesis `A 0.2–10`, collar 0.25: rate 0.0; collar 0.0: `missed_s ≈ 0.2`.
- [ ] **Step 2:** expect FAIL with "cannot find function" for each item.
- [ ] **Step 3:** implement. Discretise both sides into 10 ms frames over `max(end)` of both. A frame is
  excluded when it lies within `collar_s` of any reference turn boundary. Per frame collect the set of
  reference speakers and hypothesis speakers. Speaker mapping is exact for any count: build the
  overlap matrix (reference × hypothesis, in included frames), pad it to square with zeros, and solve
  the maximum-weight assignment with the Hungarian algorithm (O(n³), about sixty lines, written in
  `src/der.rs` with a comment naming it; no crate). Then per included frame, with `n_ref` the number of
  reference speakers, `n_hyp` the number of hypothesis speakers, and `n_correct` the reference
  speakers whose mapped hypothesis label is present: missed = `max(n_ref − n_hyp, 0)`, false alarm =
  `max(n_hyp − n_ref, 0)`, confusion = `min(n_ref, n_hyp) − n_correct`, each × 0.01 s (the NIST
  md-eval cardinality formula, so an unmapped label speaking over a reference speaker is confusion,
  not a miss plus a false alarm). `reference_s` = sum of `n_ref` × 0.01 s over included frames.
  Comment the frame size.
- [ ] **Step 4:** run `scripts/check`, expect last line `CHECK OK`.
- [ ] **Step 5:** commit: `git add src/der.rs src/lib.rs && git commit -m "der: diarization error rate scorer"`.

## Task 2: Fetch the AMI fixture

**Files:**
- Create or modify: `scripts/fetch-fixtures` (if it exists, add a `der` set beside the existing sets and
  keep its existing behaviour and last line; if it does not exist, create it with only the `der` set,
  ending with `FIXTURES OK <n> files <total seconds>s`). Executable, `set -euo pipefail`.
- Modify: `docs/developing/measurements.md` (a `## Fixtures` section; create it at the end if absent,
  otherwise add a paragraph for the DER set with the exact command).

**Interfaces:**
- Produces: `~/.cache/ambient/fixtures/der/manifest.json`, a JSON array of
  `{"wav": "<abs>", "rttm": "<abs>", "meeting": "ES2004a", "seconds": 300}`.

- [ ] **Step 1:** the meetings are pinned: `ES2004a`, `IS1009a`, `TS3003a` (the first meeting of three
  AMI test-set series; say so in a comment). For each meeting `M`, download with
  `curl --fail --location --silent --show-error -o <tmp>` then `mv` into place, only if absent:
  `https://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus/<M>/audio/<M>.Mix-Headset.wav` to
  `~/.cache/ambient/ami/<M>.Mix-Headset.wav`, and
  `https://raw.githubusercontent.com/pyannote/AMI-diarization-setup/67c2d539286e89f68952d5dcf83912bd9f01dfae/only_words/rttms/test/<M>.rttm`
  to `~/.cache/ambient/ami/<M>.rttm`. Verify each wav with `ffprobe` and each RTTM by checking it is
  non-empty and every non-blank line starts with `SPEAKER <M> ` (delete the file, exit non-zero and name
  the URL otherwise).
- [ ] **Step 2:** on the first run print `shasum -a 256` of each downloaded file; then pin those six
  digests in the script as a `sha256sum`-style table and have the script verify cached files against it
  on every run, so an upstream change surfaces as a failure and not as a moved baseline.
- [ ] **Step 3:** produce `~/.cache/ambient/fixtures/der/<M>.wav`: the first 300 s, 16 kHz mono 16-bit,
  via `ffmpeg -t 300 -ac 1 -ar 16000 -sample_fmt s16`. Produce `<M>.rttm`: the reference lines whose
  start is under 300 s, with `dur` clipped so `start + dur ≤ 300`, written with `awk` in the same
  10-field RTTM layout. Write `manifest.json`. Idempotent: a second run does no network and rewrites
  nothing unless `--force`. Print one line per file produced.
- [ ] **Step 4:** run `scripts/fetch-fixtures`, then run it again. The report quotes the last line of
  both runs, the six digests now pinned, and `ls -la ~/.cache/ambient/fixtures/der`.
- [ ] **Step 5:** run `node .github/scripts/check-links.mjs`, expect `0 broken link(s)`. Commit:
  `git add scripts/fetch-fixtures docs/developing/measurements.md && git commit -m "fixtures: AMI diarization set"`.

## Task 3: A harness that scores Ambient's own diarize path

**Files:**
- Create: `src/bin/der.rs`
- Modify: `src/der.rs` (add `spans_to_turns`)
- Modify: `docs/developing/measurements.md` (add the binary and its command to the table of tools, or a
  `## DER` section if there is no such table)

**Interfaces:**
- Consumes: `ambient::der::{parse_rttm, score, Turn}`; `ambient::resample::{read_wav_any, to_16k}`;
  `ambient::diarize::{Diarizer, Span, DEFAULT_THRESHOLD}`; `ambient::session::models_root`.
- Produces: `pub fn spans_to_turns(spans: &[Span], sr: usize) -> Vec<Turn>` in `src/der.rs`;
  `cargo run --release --bin der -- [--manifest <path>] [--threshold <f32>] [--collar <f64>] [--json <path>]`,
  a table with one row per meeting and a `total` row, JSON rows as
  `{"meeting","seconds","reference_speakers","hypothesis_speakers","missed_s","false_alarm_s","confusion_s","der","run_s"}`.

- [ ] **Step 1:** write the failing test in `src/der.rs`'s test module:
  `[Span{start:0,end:16000,speaker:0}, Span{start:32000,end:48000,speaker:1}]` at 16 kHz becomes
  `[Turn{0.0,1.0,"0"}, Turn{2.0,3.0,"1"}]`.
- [ ] **Step 2:** run `cargo test der::`, expect FAIL "cannot find function `spans_to_turns`".
- [ ] **Step 3:** implement `spans_to_turns`, then the binary: parse args (default manifest
  `~/.cache/ambient/fixtures/der/manifest.json`, threshold `DEFAULT_THRESHOLD`, collar 0.25). Resolve the
  models exactly as `session::diarize_session` does (`src/session.rs:1188-1197`): `models_root()?` joined
  with `pyannote-segmentation-3.0/model.onnx` and `wespeaker_en_voxceleb_resnet34_LM.onnx`, bailing with
  `missing <path> — run ./fetch-models.sh`. For each manifest entry: `read_wav_any` → `to_16k` →
  `Diarizer::diarize(&samples, threshold)` → `spans_to_turns(.., 16_000)` → `score` against
  `parse_rttm` of the rttm file; an empty reference is an error naming the file, never a zero. Print
  the table; the total row sums the four seconds fields and computes DER from the sums. Exit non-zero
  with the file name and the fetch script to run when the manifest is missing.
- [ ] **Step 4:** run `scripts/check`, expect `CHECK OK`; then
  `cargo run --release --bin der` and quote the total row in the report; run
  `node .github/scripts/check-links.mjs`, expect `0 broken link(s)`.
- [ ] **Step 5:** commit: `git add src/bin/der.rs src/der.rs docs/developing/measurements.md && git commit -m "der: harness over the diarize path"`.

## Task 4: The quality script reports DER

**Files:**
- Create or modify: `scripts/quality` (if it exists, add a DER step and append ` der=<total>` to its
  existing last line; if it does not exist, create it quiet like `scripts/check`, running
  `scripts/fetch-fixtures` then the DER harness, ending with `QUALITY OK der=<total>`).
- Create: `quality/der.json` from your own run (the harness's `--json` output).
- Modify: `docs/developing/landing.md` (one sentence under "What a night looks like" saying the quality
  script's last line carries `der=`, and that wiring `scripts/quality` into `CHECK_CMD` in `loop/config`
  is a change a person commits, because the hooks deny `loop/` commits from a task; the script is the
  gate, the config line is the switch, and this plan delivers the gate).

**Interfaces:**
- Consumes: `cargo run --release --bin der -- --json <tmp>`.
- Produces: the regression rule: any meeting's `der` or the total more than 0.01 above `quality/der.json`
  fails the script, printing the offending rows; `--update` rewrites `quality/der.json`.

- [ ] **Step 1:** run `scripts/quality` before the change (or note it does not exist yet).
- [ ] **Step 2:** implement the step with `jq` (present on this machine) comparing per-meeting and total
  `der` against the baseline file; when `quality/der.json` is absent, write it and print
  `QUALITY BASELINE der=<total>` as the last line instead.
- [ ] **Step 3:** run `scripts/quality` twice: the first run writes the baseline, the second ends
  `QUALITY OK … der=<total>`. Quote both last lines.
- [ ] **Step 4:** run `scripts/check`, expect `CHECK OK`; run `node .github/scripts/check-links.mjs`,
  expect `0 broken link(s)`.
- [ ] **Step 5:** commit: `git add scripts/quality quality/der.json docs/developing/landing.md && git commit -m "quality: DER baseline"`.
  The report's first line says whether `loop/config`'s `CHECK_CMD` already runs `scripts/quality`; if
  not, it names that as the one-line daytime change for a human.

## Task 5: The clustering threshold

**Files:**
- Modify: `src/diarize.rs:38` (`DEFAULT_THRESHOLD`), only if the rule below says so.
- Modify: `docs/developing/measurements.md` (a `## DER by threshold` table with the date).
- Modify: `quality/der.json` via `scripts/quality --update`, only if the constant changed.

**Interfaces:**
- Consumes: `cargo run --release --bin der -- --threshold <t> --json <tmp>`.

- [ ] **Step 1:** run the harness at thresholds 0.30, 0.40, 0.50, 0.60, 0.70, 0.80 and record, per
  threshold, total DER and the hypothesis speaker count per meeting next to the reference count.
- [ ] **Step 2:** keep the threshold with the lowest total DER only if it beats 0.50 by more than
  0.01 on every meeting and never produces more than twice the reference speaker count; otherwise
  restore 0.50 and write that the default holds.
- [ ] **Step 3:** if the constant changed, run `scripts/quality --update` and include the diff to
  `quality/der.json`.
- [ ] **Step 4:** run `scripts/check`, expect `CHECK OK`; run `node .github/scripts/check-links.mjs`,
  expect `0 broken link(s)`.
- [ ] **Step 5:** commit: `git add src/diarize.rs docs/developing/measurements.md quality/der.json && git commit -m "diarize: threshold measured against AMI"`
  (omit files that did not change).

## Task 6: Minimum embedding length

**Files:**
- Modify: `src/diarize.rs:34` (`MIN_EMBED`, currently one second of samples), swapped between runs as a
  constant.
- Modify: `docs/developing/measurements.md` (a `## DER by minimum embedding length` table).

**Interfaces:**
- Consumes: the DER harness.

- [ ] **Step 1:** run the harness with `MIN_EMBED` at 0.5 s, 1.0 s (current), 1.5 s and 2.0 s of samples,
  recording total DER, `missed_s`, and the hypothesis speaker counts.
- [ ] **Step 2:** same keep-or-restore rule: keep a value only if it beats the current one by more than
  0.01 on every meeting; otherwise restore and record that the default holds. Explain in one sentence in
  the measurements table what moved (segments shorter than `MIN_EMBED` are not embedded, so a lower
  value trades confusion for missed speech or the reverse).
- [ ] **Step 3:** if the constant changed, run `scripts/quality --update`.
- [ ] **Step 4:** run `scripts/check`, expect `CHECK OK`; run `node .github/scripts/check-links.mjs`,
  expect `0 broken link(s)`.
- [ ] **Step 5:** commit: `git add src/diarize.rs docs/developing/measurements.md quality/der.json && git commit -m "diarize: minimum embedding length measured"`
  (omit files that did not change).

## Open Questions

Source: https://jasonm4130-labs.github.io/ambient/plans/2026-09-05-der/index.mdx